From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#26201: Heads-up: hydra.gnu.org uses =?UTF-8?Q?=E2=80=98guix?= publish =?UTF-8?Q?--cache=E2=80=99?= Date: Wed, 19 Apr 2017 16:24:46 +0200 Message-ID: <87vaq0o4pd.fsf_-_@gnu.org> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> <87d1d710xc.fsf@gnu.org> <87o9vts8xb.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:56075) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1d0qYX-0005e1-Jq for bug-guix@gnu.org; Wed, 19 Apr 2017 10:26:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1d0qYU-0002LN-Fv for bug-guix@gnu.org; Wed, 19 Apr 2017 10:26:05 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:58744) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1d0qYU-0002LI-CF for bug-guix@gnu.org; Wed, 19 Apr 2017 10:26:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1d0qYU-0001Uv-69 for bug-guix@gnu.org; Wed, 19 Apr 2017 10:26:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87o9vts8xb.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 18 Apr 2017 23:27:44 +0200") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Mark H Weaver Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > >> 2. Produce a narinfo and corresponding nar the first time they are >> requested. So, the first time we receive =E2=80=9CGET foo.narinfo= =E2=80=9D, return >> 404 and spawn a thread to compute foo.narinfo and foo.nar. Return >> 200 only when both are ready. >> >> The precomputed nar{,info}s would be kept in a cache and we could >> make sure a narinfo and its nar have the same lifetime, which >> addresses one of the problems we have. >> >> pros: better HTTP latency and bandwidth >> pros: allows us to add a Content-Length for nars >> pros: helps keep narinfo/nar lifetime in sync >> cons: doesn=E2=80=99t reduce load on hydra.gnu.org >> cons: exposes inconsistency between the store contents and the HTTP >> response (you may get 404 even if the thing is actually in >> store), but maybe that=E2=80=99s not a problem > > Implemented in commit 00753f7038234a0f5a79be3ec9ab949840a18743. > > I=E2=80=99ll set up a test instance shortly. I ended up deploying it on hydra.gnu.org directly. :-) Progressively the cached nar/narinfo at {,mirror.}hydra.gnu.org will be replaced with the new ones. Now, the /guix/nar URLs have a =E2=80=98Content-Length=E2=80=99 header you should see a progress bar when = downloading one of these: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix build vim The following file will be downloaded: /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566 @ substituter-started /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0= 566 /gnu/store/rnpz1svz4aw75kibb5qb02hhccy2m4y0-guix-0.12.0-7.aabe/libexec/= guix/substitute Downloading https://mirror.hydra.gnu.org/guix/nar/gzip/ax5cm9gr1741pcq17w7b= hgss5nvq5470-vim-8.0.0566 (23.4MiB installed)... vim-8.0.0566 7.8MiB 385KiB/s 00:21 [= ####################] 100.0% @ substituter-succeeded /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0= .0566 /gnu/store/ax5cm9gr1741pcq17w7bhgss5nvq5470-vim-8.0.0566 --8<---------------cut here---------------end--------------->8--- This new caching scheme should put an end to caching of truncated nars in nginx, which has been too frequent lately. It should also mostly avoid the problem where we have a narinfo for something but not the corresponding nar, which leads to user frustration (=E2=80=98guix=E2=80=99 reports that the thing will be downloaded and event= ually fails with 410 =E2=80=9CGone=E2=80=9D while trying to download it), because =E2= =80=98guix publish=E2=80=99 caches narinfo/nar pairs together. I say =E2=80=9Cmostly=E2=80=9D because = nginx caching in front of =E2=80=98guix publish=E2=80=99 makes things more complicated. The bandwidth issue reported at the beginning of this thread should be mostly fixed: serving a narinfo or nar URL is now just sendfile(2), which is the best we can do; 404s on narinfo should be immediate. Of course, when the machine is overloaded, we=E2=80=99ll still experience increased latency and lower bandwidth, but that should be less acute than with the previous setting. Please report any problems you may have! Ludo=E2=80=99.