From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark H Weaver Subject: bug#26201: hydra.gnu.org uses =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?publish=E2=80=99?= for nars and narinfos Date: Thu, 23 Mar 2017 14:36:20 -0400 Message-ID: <87inmzrgbf.fsf@netris.org> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43215) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cr7bd-0004OX-JS for bug-guix@gnu.org; Thu, 23 Mar 2017 14:37:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cr7ba-0008AU-B6 for bug-guix@gnu.org; Thu, 23 Mar 2017 14:37:05 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:43088) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1cr7ba-0008AQ-7j for bug-guix@gnu.org; Thu, 23 Mar 2017 14:37:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1cr7ba-0006Ww-2N for bug-guix@gnu.org; Thu, 23 Mar 2017 14:37:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87r31pyms2.fsf_-_@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\?\= \=\?utf-8\?Q\?\=22's\?\= message of "Wed, 22 Mar 2017 23:22:37 +0100") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Hi again! > > Until now hydra.gnu.org was using Hydra (the software) to serve not only > the Web interface but also all the .narinfo and /nar URLs (substitute > meta-data and substitutes). > > Starting from now, hydra.gnu.org directs all .narinfo and corresponding > nar requests to =E2=80=98guix publish=E2=80=99 instead of Hydra. > > =E2=80=98guix publish=E2=80=99 should be faster and less resource-hungry = than Hydra. It > uses in-process gzip for nar compression instead of bzip2 (I chose level > 7, which seems to provide compression ratios close to what bzip2 > provides with its default compression level, while being 3 times > faster). Unlike Hydra it never forks so for instance, 404 responses for > .narinfo URLs should be quicker. Hopefully, that will improve the > worst-case (cache miss) throughput. Excellent! Any improvement in 404 response time will be very helpful. I've noticed that spikes of narinfo requests resulting in 404 has been a major source of overloading on Hydra, because these requests cannot be cached for very long. The reason: if we cache those failures for N minutes, this effectively delays the appearance of new nars by N minutes (if it was requested before that). This forces us to choose a small N for negative cache entries, which means the cache is not much help here. One question: what will happen in the case of multiple concurrent requests for the same nar? Will multiple nar-pack-and-bzip2 processes be run on-demand? Recall that the nginx proxy will pass all of those requests through, and not create the cache entry until it has received a complete response. This has caused us severe problems with huge nars such as texinfo-texmf, to the point that we had to crudely block those nar requests. Unfortunately, it is not obvious how to block the associated narinfo requests due to the lack of job name in the URL, so this results in failures on the client side that must be manually worked around. Thanks, Mark