From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id gFVRH8M5wV/nHAAA0tVLHw (envelope-from ) for ; Fri, 27 Nov 2020 17:39:15 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 4KgaG8M5wV9RDQAA1q6Kng (envelope-from ) for ; Fri, 27 Nov 2020 17:39:15 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 074F89401BC for ; Fri, 27 Nov 2020 17:39:14 +0000 (UTC) Received: from localhost ([::1]:52946 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kihiD-00054u-S2 for larch@yhetil.org; Fri, 27 Nov 2020 12:39:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53262) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kihi2-00054k-IF for guix-patches@gnu.org; Fri, 27 Nov 2020 12:39:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:34584) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kihi2-0005Aa-9o for guix-patches@gnu.org; Fri, 27 Nov 2020 12:39:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kihi2-0007o4-5r for guix-patches@gnu.org; Fri, 27 Nov 2020 12:39:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#44800] [PATCH v2 3/3] Use substitute servers on the local network. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 27 Nov 2020 17:39:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44800 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Mathieu Othacehe Cc: 44800@debbugs.gnu.org Received: via spool by 44800-submit@debbugs.gnu.org id=B44800.160649868729945 (code B ref 44800); Fri, 27 Nov 2020 17:39:02 +0000 Received: (at 44800) by debbugs.gnu.org; 27 Nov 2020 17:38:07 +0000 Received: from localhost ([127.0.0.1]:46129 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kihgu-0007mR-PE for submit@debbugs.gnu.org; Fri, 27 Nov 2020 12:38:06 -0500 Received: from eggs.gnu.org ([209.51.188.92]:34798) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kihgs-0007mD-Df for 44800@debbugs.gnu.org; Fri, 27 Nov 2020 12:37:51 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:55619) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kihgn-0004z6-6W for 44800@debbugs.gnu.org; Fri, 27 Nov 2020 12:37:45 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=47928 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kihgm-0005ag-C6; Fri, 27 Nov 2020 12:37:44 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20201124132145.217751-1-othacehe@gnu.org> <20201124132145.217751-4-othacehe@gnu.org> Date: Fri, 27 Nov 2020 18:37:42 +0100 In-Reply-To: <20201124132145.217751-4-othacehe@gnu.org> (Mathieu Othacehe's message of "Tue, 24 Nov 2020 14:21:45 +0100") Message-ID: <87lfemr995.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -1.0 (-) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -2.77 X-Scanner: ns3122888.ip-94-23-21.eu Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-TUID: wD8ckKs6SwZ4 Mathieu Othacehe skribis: > * guix/scripts/discover.scm: New file. > * Makefile.am (MODULES): Add it. > * nix/nix-daemon/guix-daemon.cc (options): Add "use-local-publish" option, > (parse-opt): parse it, > (main): start "guix discover" process when the option is set. > * nix/libstore/globals.hh (Settings): Add "useLocalPublish" public member. > * nix/libstore/globals.cc (Settings): Initialize it. > * guix/scripts/substitute.scm (%local-substitute-urls): New variable, > (substitute-urls): add it. > * gnu/services/base.scm (): Add "use-local-publish?" > field, > (guix-shepherd-service): honor it. > * doc/guix.texi (Invoking guix-daemon): Document "use-local-publish" opti= on, > (Base Services): ditto. [...] > +@item --use-local-publish[=3Dyes|no] > +Whether to use publish servers discovered a the local network, using > +Avahi, for substitutution. How about =E2=80=98--discover-substitute-servers=E2=80=99 or =E2=80=98--dis= cover-substitutes=E2=80=99 or even =E2=80=98--discover=E2=80=99? s/publish servers/substitute servers/ I think we need a note about the performance, security, and privacy implications of this here, namely: 0. It might be faster/less expensive than fetching from remote servers;=20 1. There are no security risks, only genuine substitutes will be used (add cross-ref); 2. An attacker advertising =E2=80=98guix publish=E2=80=99 on your LAN can= not serve you malicious binaries, but they can learn what software you=E2=80=99re installing. 3. Servers may serve substitute over HTTP, unencrypted, so anyone on the LAN can see what software you=E2=80=99re installing. IWBN to have an action of the Shepherd service to turn it on and off; you might want to do that depending on how much you trust the LAN you=E2=80= =99re on. (That can come later though.) > +++ b/gnu/services/base.scm > @@ -1529,6 +1529,8 @@ archive' public keys, with GUIX." > (default 0)) > (log-compression guix-configuration-log-compression > (default 'bzip2)) > + (use-local-publish? guix-configuration-use-local-publish? > + (default #f)) Same here. > +(define %publish-services > + ;; Set of discovered publish services. > + (make-hash-table)) > + > +(define (publish-file cache-directory) > + "Return the name of the file storing the discovered publish services i= nside > +CACHE-DIRECTORY." > + (let ((directory (string-append cache-directory "/discover"))) > + (string-append directory "/publish"))) > + > +(define %publish-file > + (make-parameter (publish-file %state-directory))) > + > +(define* (write-publish-file #:key (file (%publish-file))) > + "Dump the content of %PUBLISH-SERVICES hash table into FILE. Use a wr= ite > +lock on FILE to synchronize with any potential readers." Aren=E2=80=99t we partly duplicating what avahi-daemon=E2=80=99s already do= ing? avahi-daemon maintains a list of currently valid advertisements, which can be seen with: avahi-browse --cache _workstation._tcp However, that cache first needs to be initialized by running the same command without =E2=80=98--cache=E2=80=99. Hmm, maybe there=E2=80=99s no o= ther choice. I wonder how others deal with that. > +(define-command (guix-discover . args) > + (category plumbing) Should be =E2=80=9Cinternal=E2=80=9D IMO. > +++ b/nix/libstore/globals.cc > @@ -35,6 +35,7 @@ Settings::Settings() > maxSilentTime =3D 0; > buildTimeout =3D 0; > useBuildHook =3D true; > + useLocalPublish =3D false; > printBuildTrace =3D false; > multiplexedBuildOutput =3D false; > reservedSize =3D 8 * 1024 * 1024; > diff --git a/nix/libstore/globals.hh b/nix/libstore/globals.hh > index 27616a2283..43653aef48 100644 > --- a/nix/libstore/globals.hh > +++ b/nix/libstore/globals.hh > @@ -116,6 +116,10 @@ struct Settings { > users want to disable this from the command-line. */ > bool useBuildHook; >=20=20 > + /* Whether to use publish servers found on the local network for > + substitution. */ > + bool useLocalPublish; I think you don=E2=80=99t even need to field here since the variable is only used in guix-daemon.cc. > + case GUIX_OPT_USE_LOCAL_PUBLISH: > + settings.useLocalPublish =3D string_to_bool (arg); > + settings.set("use-local-publish", arg); > + break; Just set a variable local to this file and that=E2=80=99s enough. You still need the second line so that (guix scripts substitute) knows whether it should read the thing. > diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm > index ddb885d344..16e8fe6106 100755 > --- a/guix/scripts/substitute.scm > +++ b/guix/scripts/substitute.scm > @@ -27,6 +27,7 @@ > #:use-module (guix config) > #:use-module (guix records) > #:use-module ((guix serialization) #:select (restore-file)) > + #:use-module (guix scripts discover) > #:use-module (gcrypt hash) > #:use-module (guix base32) > #:use-module (guix base64) > @@ -1078,9 +1079,17 @@ found." > ;; daemon. > '("http://ci.guix.gnu.org")))) >=20=20 > +(define %local-substitute-urls > + ;; If the following option is passed to the daemon, use the substitute= s list > + ;; provided by "guix discover" process. > + (if (find-daemon-option "use-local-publish") > + (read-publish-urls) > + '())) > + > (define substitute-urls > ;; List of substitute URLs. > - (make-parameter %default-substitute-urls)) > + (make-parameter (append %local-substitute-urls > + %default-substitute-urls))) As discussed on IRC, we should probably need to set an upper limit. on the number of local substitute URLs. Imagine: you=E2=80=99re at GuixCon 2021, there are 500 participants all of = which are running =E2=80=98guix publish --advertise=E2=80=99; every Guix operatio= n leads to everyone=E2=80=99s Guix talking to every other person=E2=80=99s Guix, the w= hole thing gets slow as hell, 500 people staring at =E2=80=9Cupdating list of substitu= tes=E2=80=9D, 500 people eventually giving up and signing up for CONDACon. Also, we must make sure =E2=80=98guix substitute=E2=80=99 gracefully handle= s disconnects and servers still advertised but no longer around (timeouts etc.) We=E2=80=99ll need real world tests to see how it behaves I think. In the meantime, we can describe it as a technology preview=E2=84=A2 in the manual. WDYT? Ludo=E2=80=99.