From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id eNc1Ap9nu2EyVAEAgWs5BA (envelope-from ) for ; Thu, 16 Dec 2021 17:21:51 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id WINjOZ5nu2EAZwAAbx9fmQ (envelope-from ) for ; Thu, 16 Dec 2021 16:21:50 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8F08916968 for ; Thu, 16 Dec 2021 17:21:50 +0100 (CET) Received: from localhost ([::1]:43918 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mxtVt-0004xY-MM for larch@yhetil.org; Thu, 16 Dec 2021 11:21:49 -0500 Received: from eggs.gnu.org ([209.51.188.92]:56242) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mxtV9-0004TE-B6 for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:53197) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mxtV8-0003Oq-WE for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mxtV8-0003yC-SX for guix-patches@gnu.org; Thu, 16 Dec 2021 11:21:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#52555] [RFC PATCH 3/3] publish: Add IPFS support. Resent-From: pukkamustard Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 16 Dec 2021 16:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 52555 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 52555@debbugs.gnu.org Cc: pukkamustard , ~pukkamustard/eris@lists.sr.ht Received: via spool by 52555-submit@debbugs.gnu.org id=B52555.163967166115218 (code B ref 52555); Thu, 16 Dec 2021 16:21:02 +0000 Received: (at 52555) by debbugs.gnu.org; 16 Dec 2021 16:21:01 +0000 Received: from localhost ([127.0.0.1]:36508 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtV6-0003xO-Dp for submit@debbugs.gnu.org; Thu, 16 Dec 2021 11:21:00 -0500 Received: from mout01.posteo.de ([185.67.36.65]:34165) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mxtV2-0003wj-VK for 52555@debbugs.gnu.org; Thu, 16 Dec 2021 11:20:57 -0500 Received: from submission (posteo.de [89.146.220.130]) by mout01.posteo.de (Postfix) with ESMTPS id 66D91240026 for <52555@debbugs.gnu.org>; Thu, 16 Dec 2021 17:20:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1639671651; bh=8FUfzH+H19v7PInjtcmZ9loRZ+gv+GSpvf6oOoA0eR4=; h=From:To:Cc:Subject:Date:From; b=q3677HhT4ire8v1On3rXXk2hHSHVeTfXOJJMYNfGarL65V+/ZMjqP0fAYBPzROczE eas83QDMVCfP6Cru45FGGM4PktcKU+vVuCpRh3MmCqINvg7z46nVn8nJVxhE3r3HrW 2KZZ2vrzt8gZYeVgqymH+gIqvJnG31N8yTe2L8JhgEe4TBxqd2gM4hjE9Re42GZpxx TTVrU15muau8PE+/Z3KNb3VoOrRor3jJvJ0j8aJRHpycDpvEaoOkmAy9S9E9Ho1LDL BWQY3CgIy8F7qnsaDG7ODIY1al8FAVlboSijotsprngtlipcaQeOfTo4blayBSJMgu 1nMOmikrWOjTA== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4JFHR64dFHz9rxS; Thu, 16 Dec 2021 17:20:50 +0100 (CET) From: pukkamustard Date: Thu, 16 Dec 2021 16:20:36 +0000 Message-Id: <20211216162036.694-3-pukkamustard@posteo.net> In-Reply-To: <20211216162036.694-1-pukkamustard@posteo.net> References: <20211216161724.547-1-pukkamustard@posteo.net> <20211216162036.694-1-pukkamustard@posteo.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1639671710; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=6nSvEIlY6XA+QzqKdJrZgI4XMpXEtcVtDgihSYwhj7k=; b=pbqQC31vOfrtr0U0ecaqf3r9/M9yowUgJ+kMUFVfmdx7zQu3DaylySTkBuflnD4BItRwvx tk2cGs+CTWiQ6OwK7b0XyWV4t+teoOkGjsUu6w0QoloHCo1de9lmXHrnfDhdoVwzq6Cm+y DkpcW2YKiLl3OJKytyYyuDg4yo+oqK4TngWSsV5/hGilHbEtMSTGXPkReWdQ7EUsyWwLvh JSmL+kETvGW+repI9tnMWQPPOY2YwIEKIIFVaUtBgCMww/VWET3m83/VqHI2gXADZm+2Qt rwAcpsKb2Z9Y8c5Rvn4NjGMKvNTTiLLoe6nLEKgVf9Uy5YW3R+/M3tMmaUrXVQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1639671710; a=rsa-sha256; cv=none; b=HyM8WEoJtJKHPEzl+eBJl+CUab3r/24WUva3FAxLlOsMlj3OYawppcCtexRfS5+3x6CaaX 15BxL/Dusq33PcKUPRD80uhZF431Nd0Lv8U0j530y8lO4TTuiwIxOhckm4vfXMSK4JORZ0 83QUV2MHsy8fwfg4De3jXHV+XwZdK2qiQqx9CWN/tb0JgV4v+UthxOOaGjMWKtKCgBY4xD G64VkeGU4Yw36uRMNqmCheUg6Zz8IhaHHr2zRFHlyMq3DMYrg+2LXQ6qDe4B2Yz2M5jB13 Vp60dH1WNO2fDzOtl0Y9bmHGh0qm2jhwQVJryHEToicDyz+a4N8rVxGA8Ygr9A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=q3677HhT; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.69 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=posteo.net header.s=2017 header.b=q3677HhT; dmarc=fail reason="SPF not aligned (strict)" header.from=posteo.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 8F08916968 X-Spam-Score: -1.69 X-Migadu-Scanner: scn0.migadu.com X-TUID: Nl0vf8eVBHmY * guix/scripts/publish.scm: (show-help, %options): Add '--ipfs'. (render-narinfo/cached, bake-narinfo+nar, make-request-handler, run-publish-server): Add #:ipfs? and honor it. (guix-publish): Honor '--ipfs' and parameterize %ipfs-base-url. --- guix/scripts/publish.scm | 52 +++++++++++++++++++++++++++++++--------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 8e4b90789b..8e7fb47b9e 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -41,6 +41,8 @@ (define-module (guix scripts publish) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-37) + #:use-module (srfi srfi-71) + #:use-module (srfi srfi-171) #:use-module (web http) #:use-module (web request) #:use-module (web response) @@ -52,6 +54,7 @@ (define-module (guix scripts publish) #:use-module (guix base64) #:use-module (guix config) #:use-module (guix derivations) + #:use-module ((guix ipfs) #:prefix ipfs:) #:use-module (gcrypt hash) #:use-module (guix pki) #:use-module (gcrypt pk-crypto) @@ -59,6 +62,8 @@ (define-module (guix scripts publish) #:use-module (guix store) #:use-module ((guix serialization) #:select (write-file)) #:use-module (eris) + #:use-module (eris read-capability) + #:use-module (eris blocks ipfs) #:use-module (zlib) #:autoload (lzlib) (call-with-lzip-output-port make-lzip-output-port) @@ -83,6 +88,7 @@ (define-module (guix scripts publish) run-publish-server guix-publish)) + (define (show-help) (format #t (G_ "Usage: guix publish [OPTION]... Publish ~a over HTTP.\n") %store-directory) @@ -102,6 +108,8 @@ (define (show-help) (display (G_ " --cache-bypass-threshold=SIZE serve store items below SIZE even when not cached")) + (display (G_ " + --ipfs[=GATEWAY] publish items over IPFS via GATEWAY")) (display (G_ " --workers=N use N workers to bake items")) (display (G_ " @@ -220,6 +228,10 @@ (define %options (lambda (opt name arg result) (alist-cons 'cache-bypass-threshold (size->number arg) result))) + (option '("ipfs") #f #t + (lambda (opt name arg result) + (alist-cons 'ipfs (or arg (ipfs:%ipfs-base-url)) + result))) (option '("workers") #t #f (lambda (opt name arg result) (alist-cons 'workers (string->number* arg) @@ -526,7 +538,7 @@ (define (bypass-cache? store item) (define* (render-narinfo/cached store request hash #:key ttl (compressions (list %no-compression)) (nar-path "nar") negative-ttl - cache pool) + cache pool ipfs?) "Respond to the narinfo request for REQUEST. If the narinfo is available in CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo requested using POOL." @@ -571,7 +583,8 @@ (define (delete-entry narinfo) (bake-narinfo+nar cache item #:ttl ttl #:compressions compressions - #:nar-path nar-path))) + #:nar-path nar-path + #:ipfs? ipfs?))) (when ttl (single-baker 'cache-cleanup @@ -631,7 +644,7 @@ (define (write-compressed-file call-with-compressed-output-port) (define* (bake-narinfo+nar cache item #:key ttl (compressions (list %no-compression)) - (nar-path "/nar")) + (nar-path "/nar") ipfs?) "Write the narinfo and nar for ITEM to CACHE." (define (compressed-nar-size compression) (let* ((nar (nar-cache-file cache item #:compression compression)) @@ -641,9 +654,19 @@ (define (compressed-nar-size compression) (define (compressed-eris-urn compression) (let* ((nar (nar-cache-file cache item #:compression compression)) - (stat (stat nar #f))) + (stat (stat nar #f)) + (block-reducer (if ipfs? + (eris-blocks-ipfs-reducer + #:ipfs-base-url (ipfs:%ipfs-base-url)) + rcount))) (and stat - (cons compression (call-with-input-file nar eris-encode->urn))))) + (cons compression + (call-with-input-file nar + (lambda (port) + (let ((read-cap _ + (eris-encode port #:block-reducer + block-reducer))) + (read-capability->string read-cap)))))))) (let ((compression (actual-compressions item compressions))) @@ -1115,7 +1138,8 @@ (define* (make-request-handler store cache pool narinfo-ttl narinfo-negative-ttl (nar-path "nar") - (compressions (list %no-compression))) + (compressions (list %no-compression)) + ipfs?) (define compression-type? string->compression-type) @@ -1147,7 +1171,8 @@ (define (handle request body) #:ttl narinfo-ttl #:negative-ttl narinfo-negative-ttl #:nar-path nar-path - #:compressions compressions) + #:compressions compressions + #:ipfs? ipfs?) (render-narinfo store request hash #:ttl narinfo-ttl #:negative-ttl narinfo-negative-ttl @@ -1218,7 +1243,7 @@ (define* (run-publish-server socket store advertise? port (compressions (list %no-compression)) (nar-path "nar") narinfo-ttl narinfo-negative-ttl - cache pool) + cache pool ipfs?) (when advertise? (let ((name (service-name))) ;; XXX: Use a callback from Guile-Avahi here, as Avahi can pick a @@ -1234,7 +1259,8 @@ (define* (run-publish-server socket store #:nar-path nar-path #:narinfo-ttl narinfo-ttl #:narinfo-negative-ttl narinfo-negative-ttl - #:compressions compressions) + #:compressions compressions + #:ipfs? ipfs?) concurrent-http-server `(#:socket ,socket))) @@ -1296,6 +1322,8 @@ (define-command (guix-publish . args) (repl-port (assoc-ref opts 'repl)) (cache (assoc-ref opts 'cache)) (workers (assoc-ref opts 'workers)) + (ipfs (assoc-ref opts 'ipfs)) + (ipfs? (if ipfs #t #f)) ;; Read the key right away so that (1) we fail early on if we can't ;; access them, and (2) we can then drop privileges. @@ -1315,7 +1343,8 @@ (define-command (guix-publish . args) (%private-key private-key) (cache-bypass-threshold (or (assoc-ref opts 'cache-bypass-threshold) - (cache-bypass-threshold)))) + (cache-bypass-threshold))) + (ipfs:%ipfs-base-url ipfs)) (info (G_ "publishing ~a on ~a, port ~d~%") %store-directory (inet-ntop (sockaddr:fam address) (sockaddr:addr address)) @@ -1344,7 +1373,8 @@ (define-command (guix-publish . args) #:nar-path nar-path #:compressions compressions #:narinfo-negative-ttl negative-ttl - #:narinfo-ttl ttl)))))) + #:narinfo-ttl ttl + #:ipfs? ipfs?)))))) ;;; Local Variables: ;;; eval: (put 'single-baker 'scheme-indent-function 1) -- 2.34.0