From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#26201: hydra.gnu.org uses =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?publish=E2=80=99?= for nars and narinfos Date: Mon, 17 Apr 2017 23:36:06 +0200 Message-ID: <87inm2ogxl.fsf@gnu.org> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <87d1d710xc.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:50075) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d0EKW-0005Oy-V5 for bug-guix@gnu.org; Mon, 17 Apr 2017 17:37:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d0EKU-0001Ee-C7 for bug-guix@gnu.org; Mon, 17 Apr 2017 17:37:04 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:54868) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d0EKU-0001EW-8o for bug-guix@gnu.org; Mon, 17 Apr 2017 17:37:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1d0EKU-0003Wc-2R for bug-guix@gnu.org; Mon, 17 Apr 2017 17:37:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87d1d710xc.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 24 Mar 2017 10:25:35 +0100") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Mark H Weaver Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org Hello, ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > Other solutions I=E2=80=99ve thought about: > > 1. Produce narinfos and nars periodically rather than on-demand and > serve them as static files. > > pros: better HTTP latency and bandwidth > pros: allows us to add a Content-Length for nars > cons: doesn=E2=80=99t reduce load on hydra.gnu.org > cons: introduces arbitrary delays in delivering nars > cons: difficult/expensive to know what new store items are available > > 2. Produce a narinfo and corresponding nar the first time they are > requested. So, the first time we receive =E2=80=9CGET foo.narinfo= =E2=80=9D, return > 404 and spawn a thread to compute foo.narinfo and foo.nar. Return > 200 only when both are ready. > > The precomputed nar{,info}s would be kept in a cache and we could > make sure a narinfo and its nar have the same lifetime, which > addresses one of the problems we have. > > pros: better HTTP latency and bandwidth > pros: allows us to add a Content-Length for nars > pros: helps keep narinfo/nar lifetime in sync > cons: doesn=E2=80=99t reduce load on hydra.gnu.org > cons: exposes inconsistency between the store contents and the HTTP > response (you may get 404 even if the thing is actually in > store), but maybe that=E2=80=99s not a problem The =E2=80=98wip-publish-baking=E2=80=99 implements #2 as a new option to = =E2=80=98guix publish=E2=80=99. It gives some control on the upper bound on CPU usage si= nce we can specify how many worker threads are used. I=E2=80=99ll finish it soon so we can experiment with it. Thanks, Ludo=E2=80=99.