From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id CIDIDh7hrWNfwAAAbAwnHQ (envelope-from ) for ; Thu, 29 Dec 2022 19:49:02 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id eLytDh7hrWNMrwAAauVa8A (envelope-from ) for ; Thu, 29 Dec 2022 19:49:02 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C23E03BED1 for ; Thu, 29 Dec 2022 19:49:00 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pAxQl-0000AP-JM; Thu, 29 Dec 2022 13:15:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pAxQk-0000A3-Om for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pAxQk-00042z-Ft for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pAxQk-0007RH-AF for guix-patches@gnu.org; Thu, 29 Dec 2022 13:15:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [PATCH v3 2/8] publish: Store ERIS encoded blocks to a local block store. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 29 Dec 2022 18:15:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.167233767628487 (code B ref 52555); Thu, 29 Dec 2022 18:15:02 +0000 Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:36 +0000 Received: from localhost ([127.0.0.1]:32804 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQJ-0007PO-MS for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:36 -0500 Received: from mout02.posteo.de ([185.67.36.66]:58729) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pAxQB-0007OH-UO for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:29 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 6ECD92401BC for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1672337662; bh=uHwlg5kIjSso0T2R2trMFAAlm0ymvpLunCNwcfsNJy8=; h=From:To:Cc:Subject:Date:From; b=f38MZYaBgd5YNNv18miJEWnShK3EJs5B0CbbFOj9w4nqv9rTNVhBpExcFgZvrT6B0 UFY+xOV6pYBdM3e2WxrZCTdGbGSzi19ov2hs+JTD6XaakFlykMD2os6btN1z91ugsL J1QxQ10LrBCdpMrxVOXX3DJQ+Yby1Y70bupb6fkHixnSn+EbRZcoXj43lUuoictbew VPI/mOVnTMFAIUl62tzromgjAhLdlMO4wCpU/VYhzz196YXi2awwrXU6ZA744NwuH4 7W8P34QUq0e5jkAE2SNdzAdC25hV6W7hrxATWjgvToPQv5wzWOa5IMzAUDOiCPTgIj 7/Li8N4ZNeeBg== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4Njc3d6jzwz6tmY; Thu, 29 Dec 2022 19:14:21 +0100 (CET) From: pukkamustard Date: Thu, 29 Dec 2022 18:13:21 +0000 Message-Id: <20221229181327.758-3-pukkamustard@posteo.net> In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net> References: <20221229181327.758-1-pukkamustard@posteo.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1672339741; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=CRMtvhf3dVF/3588/ohxOup9D1uWM2kCtLKwEfCgm0U=; b=UyU6MhWX28U4kFVigjdkOu6wEFmWqz8PGoN+oyof1eAD0db8lftHGqUW6TF+1MzEBYuWid W8bmpdcbNOnf5XKvj/Sjwgm5/F92jWyvdtsdgOTZBRuAYL0zIl6zCQ2zCjV5p4adfyIdzL XtJrXISpPQ0Fe7i0lTwOWLp3NZYMkCXo24EcvJXrx/SfInBlyPW0xXlGk965oo2L5zLHnu MOtzvQKMbM0rTLFgs5IpuS1U9y699xEY2qxfIWyfO0SfvLRMYlhnbHGJRnjGbDYp04T71Y JzFT1KGPsz08+Yb4GNhdzxaM12vQ2p+o9JfqKBlVpuae3ng/kGqvi+G/vhZOuw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=f38MZYaB; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) ARC-Seal: i=1; s=key1; d=yhetil.org; t=1672339741; a=rsa-sha256; cv=none; b=JMwWaW6qmiB35eV0U2OmY1kJN+gKwU2RBsjkMCLWyYQ6g5ff3cYk51rQIbp8Kp+O87rW5H AKJ6Mo3zQzI7c8wdFBYtZ/rjPQreXJjnfqnD8DYaxjofKvrJG3I5tsqYzPdAbj33pdgYvP p8u8bE+yc64mEYP+qPhddpduqMLY5N8RIL7PJWjUpvfC2gzjl1NPbde/upSvQPTBw8myap GUE3McWuYkYk5Aj3tPQudfqXmvwA3lZBewS7C/3AnhW6I0bgNEvpC7SYbMpmNO6OuOxOlW g2YN9r3uiUHQAmaoLJoHFPFaUaNFIV7nvnjTRcGoW5ws6ZSWCshP4pKFsvwjKg== X-Spam: Yes X-Migadu-Queue-Id: C23E03BED1 X-Migadu-Spam: Yes X-Spam-Score: 8.27 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=f38MZYaB; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org"; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none) X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: 8.27 X-TUID: s/9RdV5NflcJ * guix/eris.scm: New file. * guix/eris/fs-store.scm: New file. * Makefile.am (MODULES): Add new files. * guix/scripts/publish.scm (bake-narinfo+nar): Use guix-eris-block-reducer. --- Makefile.am | 2 ++ guix/eris.scm | 36 +++++++++++++++++++++ guix/eris/fs-store.scm | 67 ++++++++++++++++++++++++++++++++++++++++ guix/scripts/publish.scm | 14 +++++---- 4 files changed, 113 insertions(+), 6 deletions(-) create mode 100644 guix/eris.scm create mode 100644 guix/eris/fs-store.scm diff --git a/Makefile.am b/Makefile.am index b54288c0fc..c549fc8580 100644 --- a/Makefile.am +++ b/Makefile.am @@ -132,6 +132,8 @@ MODULES = \ guix/least-authority.scm \ guix/read-print.scm \ guix/ipfs.scm \ + guix/eris.scm \ + guix/eris/fs-store.scm \ guix/platform.scm \ guix/platforms/arm.scm \ guix/platforms/mips.scm \ diff --git a/guix/eris.scm b/guix/eris.scm new file mode 100644 index 0000000000..29d5e7b1db --- /dev/null +++ b/guix/eris.scm @@ -0,0 +1,36 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 pukkamustard +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix eris) + #:use-module (eris) + + #:use-module (guix config) + #:use-module (guix eris fs-store) + + #:export (guix-eris-block-reducer + + %eris-block-store-directory)) + +(define %eris-block-store-directory + (make-parameter + (or (getenv "GUIX_ERIS_BLOCK_STORE_DIRECTORY") + (string-append %state-directory "/eris")))) + +(define (guix-eris-block-reducer) + "Returns a block reducer that stores blocks of ERIS encoded content." + (eris-fs-store-reducer (%eris-block-store-directory))) diff --git a/guix/eris/fs-store.scm b/guix/eris/fs-store.scm new file mode 100644 index 0000000000..2ef7607988 --- /dev/null +++ b/guix/eris/fs-store.scm @@ -0,0 +1,67 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2022 pukkamustard +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix eris fs-store) + #:use-module (rnrs io ports) + #:use-module (guix build utils) ; for mkdir-p + #:use-module (eris utils base32) + + #:export (eris-fs-store-reducer + eris-fs-store-ref)) + +;;; Commentary: +;;; +;;; This module provides a file-system based store of ERIS encoded blocks. +;;; +;;; Code: + +(define (eris-fs-store-reducer store-directory) + (case-lambda + (() (mkdir-p store-directory)) + + ((result) result) + + ((_ ref-block) + (let* ((ref (car ref-block)) + (b32 (base32-encode ref)) + (pre (substring b32 0 2)) + (suf (substring b32 2)) + (pre-dir (string-append store-directory "/" pre)) + (path (string-append pre-dir "/" suf)) + (block (cdr ref-block))) + + (mkdir-p pre-dir) + + (unless (file-exists? path) + (call-with-output-file path + (lambda (port) (put-bytevector port block)) + #:binary #t)) + + #t)))) + +(define (eris-fs-store-ref store-directory) + (lambda (ref) + (let* ((b32 (base32-encode ref)) + (pre (substring b32 0 2)) + (suf (substring b32 2)) + (path (string-append store-directory "/" pre "/" suf))) + (if (file-exists? path) + (call-with-input-file path + (lambda (port) (get-bytevector-all port)) + #:binary #t) + #f)))) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 76b5a429f4..7f14e4d4d4 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -52,6 +52,7 @@ (define-module (guix scripts publish) #:use-module (guix base64) #:use-module (guix config) #:use-module (guix derivations) + #:use-module (guix eris) #:use-module (gcrypt hash) #:use-module (guix pki) #:use-module (gcrypt pk-crypto) @@ -647,12 +648,13 @@ (define (eris-encode-nar compressions) (and stat (call-with-input-file nar (lambda (port) - (let ((eris-urn _ - (eris-encode port - #:block-size 'large - #:block-reducer rcount - #:convergence-secret - %null-convergence-secret))) + (let ((eris-urn + _ (eris-encode port + #:block-size 'large + #:block-reducer + (guix-eris-block-reducer) + #:convergence-secret + %null-convergence-secret))) eris-urn))))))) (let ((compression (actual-compressions item compressions))) -- 2.38.1