From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#26201: hydra.gnu.org uses =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?publish=E2=80=99?= for nars and narinfos Date: Wed, 03 May 2017 11:25:38 +0200 Message-ID: <87k25ywaul.fsf@gnu.org> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> <877f1yjr64.fsf@netris.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:38296) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d5qXt-0001TC-J2 for bug-guix@gnu.org; Wed, 03 May 2017 05:26:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d5qXq-0005hH-ED for bug-guix@gnu.org; Wed, 03 May 2017 05:26:05 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:53710) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d5qXq-0005h9-AJ for bug-guix@gnu.org; Wed, 03 May 2017 05:26:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1d5qXp-0006Uj-Vr for bug-guix@gnu.org; Wed, 03 May 2017 05:26:01 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <877f1yjr64.fsf@netris.org> (Mark H. Weaver's message of "Wed, 03 May 2017 04:11:31 -0400") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Mark H Weaver Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org Hello, Mark H Weaver skribis: > Actually, IIUC, the build slaves are _already_ compressing everything, > and they always have. They compress the build outputs for transmission > back to the master machine. In the current framework, the master > machine immediately decompresses them upon receipt, and this compression > and decompression is considered an internal detail of the network > transport. > > Currently, the master machine stores all build outputs uncompressed in > /gnu/store, and then later recompresses them for transmission to users > and other build slaves. The needless decompression and recompression is > a tremendous amount of wasted work on our master machine. That it's all > stored uncompressed is also a significant waste of disk space, which > leads to significant additional costs during garbage collection. > > Essentially, my proposal is for the build slaves to be modified to > prepare the compressed NARs in a form suitable for delivery to end users > (and other build slaves) with minimal processing by our master node. > The master node would be significantly modified to receive, store, and > forward NARs explicitly, without ever decompressing them. As far as I > can tell, this would mean strictly less work to do and less data to > store for every machine and in every case. I agree that the redundant compression/decompression is terrible. Yet I=E2=80=99m not sure how to architect a solution where compression is perfo= rmed by build machines. The main issue is that offloading and publication are two independent mechanisms, as things are. Maybe each build machine for a build farm use-case we could have a =E2=80=9Csemi-offloading=E2=80=9D mechanism whereby the master spawns a rem= ote build without retrieving its result, something akin to: GUIX_DAEMON_SOCKET=3Dssh://build-machine.example.org \ guix build /gnu/store/=E2=80=A6-foo.drv In addition, the build machine would publish its result via =E2=80=98guix publish=E2=80=99, which the master could then simply mirror and cache with nginx. There=E2=80=99s the issue of signatures, but perhaps we could have a more sophisticated PKI and have the master delegate to build machines=E2=80=A6 Then there are other issues such as that of synchronizing the TTL of a narinfo and its corresponding nar, which --cache addresses. Tricky! > Ludovic has pointed out that we cannot do this because Hydra must add > its digital signature, and that this digital signature is stored within > the compressed NAR. Therefore, we cannot avoid having the master > machine decompress and recompress every NAR that is delivered to users. > > In my opinion, we should change the way we sign NARs. Signatures should > be external to the NARs, not internal. Not only would this allow us to > decentralize production of our NARs, but more importantly, it would > enable a community of independent builders to add their signatures to a > common pool of NARs. Having a common pool of NARs enables us to store > these NARs in a shared distribution network without duplication. We > cannot even have a common pool of NARs if they contain > build-farm-specific data such as signatures. Currently the signature is in the narinfos, not in nars proper=C2=B9. So we can already add signatures on an externally provided nar, for instance. There=E2=80=99s a silly limitation currently, which is that the signature is computed over all the fields of the narinfo. That=E2=80=99s silly because = it means that if you change, say, the compression format or the URL of the nar, then the signature becomes invalid. We should fix that at some point. Ludo=E2=80=99. =C2=B9 For =E2=80=98guix publish=E2=80=99. =E2=80=98guix archive --export= =E2=80=99 appends a signature to the nar set.