From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tobias Geerinckx-Rice Subject: bug#26201: hydra.gnu.org uses =?UTF-8?Q?=E2=80=98guix_?= =?UTF-8?Q?publish=E2=80=99?= for nars and narinfos Date: Sun, 26 Mar 2017 19:35:25 +0200 Message-ID: <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Tao7UKjCLBAtpINvT4ketSuXk9Iww88Mg" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:59295) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1csC5G-0004TF-7A for bug-guix@gnu.org; Sun, 26 Mar 2017 13:36:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1csC5C-00053a-8V for bug-guix@gnu.org; Sun, 26 Mar 2017 13:36:06 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:47922) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1csC5B-00053W-Vr for bug-guix@gnu.org; Sun, 26 Mar 2017 13:36:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1csC5B-0003M1-QK for bug-guix@gnu.org; Sun, 26 Mar 2017 13:36:01 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87y3vvozy5.fsf@netris.org> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: mhw@netris.org Cc: 26201@debbugs.gnu.org, guix-sysadmin@gnu.org This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Tao7UKjCLBAtpINvT4ketSuXk9Iww88Mg Content-Type: multipart/mixed; boundary="LWlJpkDdb2w3EIlUP7ojS6J4u0Lg6kf1c"; protected-headers="v1" From: Tobias Geerinckx-Rice To: mhw@netris.org Cc: ludo@gnu.org, 26201@debbugs.gnu.org, guix-sysadmin@gnu.org Message-ID: <1988d01c-1e67-bf47-2b43-cf3551d0651b@tobias.gr> Subject: =?UTF-8?Q?Re:_bug#26201:_hydra.gnu.org_uses_=e2=80=98guix_publish?= =?UTF-8?Q?=e2=80=99_for_nars_and_narinfos?= References: <20170320184449.5ac06051@khaalida> <144e9ba8-af93-fb18-d2b9-f198ae7c11e9@tobias.gr> <20170320195247.05f72fc9@khaalida> <8e7e07d1-563f-666f-2c32-2a772757c86f@tobias.gr> <8760j2wpfy.fsf@gnu.org> <9889a4b5-c300-cd03-1095-1115428067fb@tobias.gr> <87r31pyms2.fsf_-_@gnu.org> <87inmzrgbf.fsf@netris.org> <25b2472a-c705-53fe-f94f-04de9a2d484e@tobias.gr> <87y3vvozy5.fsf@netris.org> In-Reply-To: <87y3vvozy5.fsf@netris.org> --LWlJpkDdb2w3EIlUP7ojS6J4u0Lg6kf1c Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mark, On 24/03/17 09:12, Mark H Weaver wrote: > IIUC, with "proxy_cache_lock on", we have two choices of how other > client requests will be treated: >=20 > [badly, ed.] Eh. You're probably (and disappointingly) right. When configuring my little cache, I had a clear idea of how such a cache should work (basically, your last scenario below), then looked at the nginx documentation to find what I had in mind. =E2=80=98proxy_cache_lock= =E2=80=99 matched. I should have been more pessimistic and done more testing. Shame on me, &c. Too much other things on my mind. :-/ > Or at least that's what I'd expect based on my reading of the nginx doc= s > linked above. I haven't tried it. I can try to do some simple tests tomorrow. > IMO, the best solution is to *never* generate nars on Hydra in response= > to client requests, but rather to have the build slaves pack and > compress the nars, copy them to Hydra, and then serve them as static > files using nginx. A true mirror at last! Do we have the disc space for that? And could Hydra actually handle compressing *everything*, without an infinitely growing back-log? I don't have access to any statistics, but I'm guessing that a fair number of package+versions are never actually requested, and hence never compressed. This would change that. > A far inferior solution, but possibly acceptable and closer to the > current approach, would be to arrange for all concurrent responses for > the same nar to be sent incrementally from a single nar-packing process= =2E > More concretely, while packing and sending a nar response to the first > client, the data would also be written to a file. Subsequent requests > for the same nar would be serviced using the equivalent of: >=20 > tail --bytes=3D+0 --follow FILENAME >=20 > This way, no one would have to wait an hour to receive the first byte. ^ This is so obviously the right solution, that it would be disappointing if nginx really couldn't be made to do it. It already buffers proxy responses to a temporary file anyway... Kind regards, T G-R --LWlJpkDdb2w3EIlUP7ojS6J4u0Lg6kf1c-- --Tao7UKjCLBAtpINvT4ketSuXk9Iww88Mg Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEqBAEBCgAUBQJY1/veDRxtZUB0b2JpYXMuZ3IACgkQkczbm0hUG5ln2AgAmIsp fZMKZUsfwzzPNL+Cykyb0CoyfPODfBpUJiJNxiqnNrZDo6br7lLAO+oW5hHMDpwe B6+evORtaXpszT60NSQvRWLs5Re/tOkSU/cb+aoZ/7Fm7zgWezbbjWmzUobE4cbB gaPyMybvxuEXqbgMt/Cf7hjAjG3zY1RC3RteNbFym83st6+4NHw2QufjyFmXCzVY 93f+6j/cpOBk+na1LeVPJmVposs1qlhqxoWLfGVZrnDxFFXWb6W0VJhGDKk9i3So prEc5fUNegwmbMngExs/OD0IDuHedYJ5Eh/5ZseQdDLmNS2fDh6EXmYrf2KYkb3b a9xA/HOzmRFct45+tQ== =Y5jd -----END PGP SIGNATURE----- --Tao7UKjCLBAtpINvT4ketSuXk9Iww88Mg--