From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark H Weaver Subject: bug#26201: hydra.gnu.org uses =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?publish=E2=80=99?= for nars and narinfos Date: Wed, 03 May 2017 04:11:31 -0400 Message-ID: <877f1yjr64.fsf@netris.org> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:44741) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d5pOH-0000hW-UY for bug-guix@gnu.org; Wed, 03 May 2017 04:12:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d5pOE-0005oG-0A for bug-guix@gnu.org; Wed, 03 May 2017 04:12:05 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:53665) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d5pOD-0005oA-TD for bug-guix@gnu.org; Wed, 03 May 2017 04:12:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1d5pOD-0004ni-J2 for bug-guix@gnu.org; Wed, 03 May 2017 04:12:01 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> (Tobias Geerinckx-Rice's message of "Sun, 26 Mar 2017 19:35:25 +0200") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Tobias Geerinckx-Rice Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org Reviving an old thread... Tobias Geerinckx-Rice writes: >> IMO, the best solution is to *never* generate nars on Hydra in response >> to client requests, but rather to have the build slaves pack and >> compress the nars, copy them to Hydra, and then serve them as static >> files using nginx. > > A true mirror at last! Do we have the disc space for that? > > And could Hydra actually handle compressing *everything*, without an > infinitely growing back-log? I don't have access to any statistics, but > I'm guessing that a fair number of package+versions are never actually > requested, and hence never compressed. This would change that. Actually, IIUC, the build slaves are _already_ compressing everything, and they always have. They compress the build outputs for transmission back to the master machine. In the current framework, the master machine immediately decompresses them upon receipt, and this compression and decompression is considered an internal detail of the network transport. Currently, the master machine stores all build outputs uncompressed in /gnu/store, and then later recompresses them for transmission to users and other build slaves. The needless decompression and recompression is a tremendous amount of wasted work on our master machine. That it's all stored uncompressed is also a significant waste of disk space, which leads to significant additional costs during garbage collection. Essentially, my proposal is for the build slaves to be modified to prepare the compressed NARs in a form suitable for delivery to end users (and other build slaves) with minimal processing by our master node. The master node would be significantly modified to receive, store, and forward NARs explicitly, without ever decompressing them. As far as I can tell, this would mean strictly less work to do and less data to store for every machine and in every case. Ludovic has pointed out that we cannot do this because Hydra must add its digital signature, and that this digital signature is stored within the compressed NAR. Therefore, we cannot avoid having the master machine decompress and recompress every NAR that is delivered to users. In my opinion, we should change the way we sign NARs. Signatures should be external to the NARs, not internal. Not only would this allow us to decentralize production of our NARs, but more importantly, it would enable a community of independent builders to add their signatures to a common pool of NARs. Having a common pool of NARs enables us to store these NARs in a shared distribution network without duplication. We cannot even have a common pool of NARs if they contain build-farm-specific data such as signatures. Thoughts? Mark