From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:49527) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hWiq9-0000ga-6o for guix-patches@gnu.org; Fri, 31 May 2019 10:49:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hWiq6-0006Bh-Kl for guix-patches@gnu.org; Fri, 31 May 2019 10:49:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:50614) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hWiq6-0006BY-Hm for guix-patches@gnu.org; Fri, 31 May 2019 10:49:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1hWiq6-0003O1-DW for guix-patches@gnu.org; Fri, 31 May 2019 10:49:02 -0400 Subject: [bug#36029] [PATCH 0/2] 'publish' and 'substitute' support several compression methods Resent-Message-ID: Received: from eggs.gnu.org ([209.51.188.92]:49454) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hWipl-0000gP-5l for guix-patches@gnu.org; Fri, 31 May 2019 10:48:42 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Fri, 31 May 2019 16:48:28 +0200 Message-Id: <20190531144828.9585-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: 36029@debbugs.gnu.org Cc: Pierre Neidhardt Hello Guix! This is a followup to . One idea we discussed there was to allow clients to pass an ‘X-Guix-Accepted-Encoding’ header in HTTP requests, and the server would return an lzip narinfo or a gzip narinfo depending on that. However, I thought that this was not very flexible, and that we were bound to mess up with caching. This patch implements a different solution: ‘guix publish’ can be passed multiple ‘-C’ options, in which case it compresses substitutes with all these compression methods. The corresponding narinfo looks like this: --8<---------------cut here---------------start------------->8--- StorePath: /gnu/store/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4 URL: nar/gzip/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4 Compression: gzip FileSize: 40308611 URL: nar/lzip/9czlz7ss3187l2vi1hvrlkwlgrggdg5p-inkscape-0.92.4 Compression: lzip FileSize: 19867767 NarHash: sha256:1jv4nkq68a7zwqhi9inrnh340a4jxcpb91wq7d25hgw0nk8isbbk NarSize: 136499024 References: … --8<---------------cut here---------------end--------------->8--- IOW, it’s like before, except that there are multiple URL/Compression/FileSize fields instead of just one of each. The trick is that old clients take the first occurrence of each of these fields and ignore subsequent occurrences. In the example above, they’d just take gzip and ignore the rest. The new ‘guix substitute’ (second patch) “sees” all these fields and is able to choose the most appropriate compression method (i.e., the best one among those it supports.) This adds a bit of complexity that is useless beyond the transitioning period from gzip to lzip, but I think that’s OK; plus there might be an lzip to super-lzip transition in the future, who knows. Thoughts? When we deploy that, we’ll obviously more use storage and more CPU on the build farm, but that seems unavoidable. OTOH, we’ll progressively end up sending less data over the wire (and paying less for the CDN!), given that lzip compresses better. Ludo’. Ludovic Courtès (2): publish: '--compression' can be repeated. substitute: Select the best compression methods. doc/guix.texi | 5 + guix/scripts/challenge.scm | 4 +- guix/scripts/publish.scm | 204 ++++++++++++++++++++++-------------- guix/scripts/substitute.scm | 141 ++++++++++++++++++------- guix/scripts/weather.scm | 5 +- tests/publish.scm | 89 ++++++++++++++-- tests/substitute.scm | 51 ++++++++- 7 files changed, 370 insertions(+), 129 deletions(-) -- 2.21.0