From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49883) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dntxz-0004gs-23 for guix-patches@gnu.org; Fri, 01 Sep 2017 17:59:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dntxu-0005Qx-2X for guix-patches@gnu.org; Fri, 01 Sep 2017 17:59:07 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:60231) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dntxt-0005Qs-Vl for guix-patches@gnu.org; Fri, 01 Sep 2017 17:59:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dntxt-0001eF-NM for guix-patches@gnu.org; Fri, 01 Sep 2017 17:59:01 -0400 Subject: [bug#28324] [PATCH 0/2] Allow substitute downloads from unauthorized servers when they are undistinguishable Resent-Message-ID: Received: from eggs.gnu.org ([2001:4830:134:3::10]:49644) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dntxZ-0004RX-KC for guix-patches@gnu.org; Fri, 01 Sep 2017 17:58:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dntxY-0005CL-8S for guix-patches@gnu.org; Fri, 01 Sep 2017 17:58:41 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 1 Sep 2017 23:58:12 +0200 Message-Id: <20170901215812.24934-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: 28324@debbugs.gnu.org Hello Guix! ‘guix publish’ is really helpful, but it’s still inconvenient to share binaries among machines: one has to authorize the other machine’s key, which is tedious and is an all-or-nothing kind of decision. Ideally we’d like to support zero-configuration binary sharing on a local network, for example, and that means it should Just Work without having to fiddle with keys/ACLs and without having to give blanket permission to your colleague’s machine next-door. This patch allows substitutes to be downloaded from an unauthorized server (one whose public key is not in /etc/guix/acl), or from a server that does not even sign substitutes, *provided* that server advertises and serves the exact same content as one of the authorized servers. Concretely, here’s the metadata substitute servers provide: --8<---------------cut here---------------start------------->8--- $ wget -q -O - https://mirror.hydra.gnu.org/sfx1wh27i6gsrk21p87rdyikc64v7d51.narinfo StorePath: /gnu/store/sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11 URL: guix/nar/gzip/sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11 Compression: gzip NarHash: sha256:0ac82i3kn10lgb64d3mn0n062rj291bwjpgv7asn32ja4phri463 NarSize: 384240 References: 3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25 sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11 FileSize: 154865 System: x86_64-linux Deriver: bpxa7iypl2q2fxzf1dgq9k4daa3p3s39-zlib-1.2.11.drv Signature: 1;20121227-hydra.gnu.org;KHNpZ25… --8<---------------cut here---------------end--------------->8--- Let’s look at what another server provides for the same item: --8<---------------cut here---------------start------------->8--- $ wget -q -O - https://berlin.guixsd.org/sfx1wh27i6gsrk21p87rdyikc64v7d51.narinfo StorePath: /gnu/store/sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11 URL: nar/gzip/sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11 Compression: gzip NarHash: sha256:0ac82i3kn10lgb64d3mn0n062rj291bwjpgv7asn32ja4phri463 NarSize: 384240 References: 3x53yv4v144c9xp02rs64z7j597kkqax-gcc-5.4.0-lib n6nvxlk2j8ysffjh3jphn1k5silnakh6-glibc-2.25 sfx1wh27i6gsrk21p87rdyikc64v7d51-zlib-1.2.11 FileSize: 154511 System: x86_64-linux Deriver: 512mdjkpbyp2kggijxcxw0j0xw0scvn4-zlib-1.2.11.drv Signature: 1;berlin.guixsd.org;KHNpZ25… --8<---------------cut here---------------end--------------->8--- Apart from the signature, compressed size (“FileSize”), deriver, and URL, everything is the same. Thus, even if we only authorized hydra.gnu.org, it is safe to download from berlin.guixsd.org (in fact, it’d be enough to have the same “NarHash”) and more generally to trust the metadata it provides for this store item. With this patch, one can use: --substitute-urls="https://foo.example.org https://hydra.gnu.org" If foo.example.org serves narinfos equivalent to those of hydra (modulo signature, URL, compression, etc.), then Guix will download substitutes from foo.example.org. Thanks to reproducible builds, that means we’ll effectively be able to share binaries with pretty much any machine out there. Pretty neat no? >From there there are several things we can do: 1. Add optional service discovery support in ‘guix substitute’ and service advertisement in ‘guix publish’ via Guile-Avahi. Imagine going to a GHM or FOSDEM and getting binaries from your neighbor’s laptop. I think that’d be pretty cool. :-) 2. Change narinfos such that the signature is computed only over the relevant parts: StorePath, NarHash, and References (currently it’s computed over all the fields up to “Signature”.) Not strictly necessary, but would make sense. 3. Change ‘guix publish’ to use content-addressed URLs for nars, similar to what is done for /file URLs. 4. Maybe replace --substitute-urls with two options, say --metadata-urls and --nar-urls. You could, say, fetch narinfos only from hydra.gnu.org and fetch nars only from foo.example.org (using the content-addressed URLs to get nars from foo.example.org.) 5. Add support to download those content-addressed nars over Bittorrent, IPFS, and whatnot. We can keep downloading narinfos themselves over HTTPS. Sky is the limit. Until then, feedback welcome! I’d particularly like for people to read the patch and the associated unit tests; more eyeballs is a good thing. Ludo’. Ludovic Courtès (2): substitute: Make substitute URLs a SRFI-39 parameter. substitute: Download from unauthorized sources that provide the right content. doc/guix.texi | 28 ++++++- guix/scripts/substitute.scm | 152 ++++++++++++++++++++++------------ tests/substitute.scm | 193 ++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 303 insertions(+), 70 deletions(-) -- 2.14.1