From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 6PbSNJGc6F8nbAAA0tVLHw (envelope-from ) for ; Sun, 27 Dec 2020 14:39:13 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id gMK+MJGc6F88fAAAB5/wlQ (envelope-from ) for ; Sun, 27 Dec 2020 14:39:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5BF129404C7 for ; Sun, 27 Dec 2020 14:39:13 +0000 (UTC) Received: from localhost ([::1]:34394 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ktXCS-0003hl-Aa for larch@yhetil.org; Sun, 27 Dec 2020 09:39:12 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55168) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ktXCJ-0003g4-BL for guix-patches@gnu.org; Sun, 27 Dec 2020 09:39:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:48960) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ktXCJ-0003On-42 for guix-patches@gnu.org; Sun, 27 Dec 2020 09:39:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1ktXCJ-0000Jl-27 for guix-patches@gnu.org; Sun, 27 Dec 2020 09:39:03 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45460] [PATCH 3/5] publish: Add support for zstd compression. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 27 Dec 2020 14:39:03 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45460 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 45460@debbugs.gnu.org Received: via spool by 45460-submit@debbugs.gnu.org id=B45460.16090799091142 (code B ref 45460); Sun, 27 Dec 2020 14:39:03 +0000 Received: (at 45460) by debbugs.gnu.org; 27 Dec 2020 14:38:29 +0000 Received: from localhost ([127.0.0.1]:60500 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ktXBk-0000IL-Dp for submit@debbugs.gnu.org; Sun, 27 Dec 2020 09:38:28 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33742) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ktXBd-0000HV-Vs for 45460@debbugs.gnu.org; Sun, 27 Dec 2020 09:38:22 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49702) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ktXBY-0003AQ-PB; Sun, 27 Dec 2020 09:38:16 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51504 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1ktXBY-0001Rq-9y; Sun, 27 Dec 2020 09:38:16 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Sun, 27 Dec 2020 15:38:07 +0100 Message-Id: <20201227143809.18554-3-ludo@gnu.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201227143809.18554-1-ludo@gnu.org> References: <20201227143809.18554-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: 2.18 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 5BF129404C7 X-Spam-Score: 2.18 X-Migadu-Scanner: scn1.migadu.com X-TUID: POgG8dWgdgdk * guix/scripts/publish.scm (compress-nar)[write-compressed-file]: New procedure. Use it for 'gzip' and 'lzip'. Add 'zstd. (nar-response-port, string->compression-type): Add case for 'zstd'. * tests/publish.scm (zstd-supported?): New procedure. ("/nar/zstd/*"): New test. * doc/guix.texi (Invoking guix publish): Document zstd compression. --- doc/guix.texi | 18 ++++++++++++------ guix/scripts/publish.scm | 31 ++++++++++++++++++------------- tests/publish.scm | 16 ++++++++++++++++ 3 files changed, 46 insertions(+), 19 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index b12cb11bdf..ed38f2e37b 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -12329,17 +12329,23 @@ server socket is open and the signing key has been read. @item --compression[=@var{method}[:@var{level}]] @itemx -C [@var{method}[:@var{level}]] Compress data using the given @var{method} and @var{level}. @var{method} is -one of @code{lzip} and @code{gzip}; when @var{method} is omitted, @code{gzip} -is used. +one of @code{lzip}, @code{zstd}, and @code{gzip}; when @var{method} is +omitted, @code{gzip} is used. When @var{level} is zero, disable compression. The range 1 to 9 corresponds to different compression levels: 1 is the fastest, and 9 is the best (CPU-intensive). The default is 3. -Usually, @code{lzip} compresses noticeably better than @code{gzip} for a small -increase in CPU usage; see -@uref{https://nongnu.org/lzip/lzip_benchmark.html,benchmarks on the lzip Web -page}. +Usually, @code{lzip} compresses noticeably better than @code{gzip} for a +small increase in CPU usage; see +@uref{https://nongnu.org/lzip/lzip_benchmark.html,benchmarks on the lzip +Web page}. However, @code{lzip} achieves low decompression throughput +(on the order of 50@tie{}MiB/s on modern hardware), which can be a +bottleneck for someone who downloads over a fast network connection. + +The compression ratio of @code{zstd} is between that of @code{lzip} and +that of @code{gzip}; its main advantage is a +@uref{https://facebook.github.io/zstd/,high decompression speed}. Unless @option{--cache} is used, compression occurs on the fly and the compressed streams are not diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm index 5a865c838d..fa85088ed0 100644 --- a/guix/scripts/publish.scm +++ b/guix/scripts/publish.scm @@ -56,6 +56,8 @@ #:use-module (zlib) #:autoload (lzlib) (call-with-lzip-output-port make-lzip-output-port) + #:autoload (zstd) (call-with-zstd-output-port + make-zstd-output-port) #:use-module (guix cache) #:use-module (guix ui) #:use-module (guix scripts) @@ -588,23 +590,22 @@ requested using POOL." (define nar (nar-cache-file cache item #:compression compression)) + (define (write-compressed-file call-with-compressed-output-port) + ;; Note: the file port gets closed along with the compressed port. + (call-with-compressed-output-port (open-output-file (string-append nar ".tmp")) + (lambda (port) + (write-file item port)) + #:level (compression-level compression)) + (rename-file (string-append nar ".tmp") nar)) + (mkdir-p (dirname nar)) (match (compression-type compression) ('gzip - ;; Note: the file port gets closed along with the gzip port. - (call-with-gzip-output-port (open-output-file (string-append nar ".tmp")) - (lambda (port) - (write-file item port)) - #:level (compression-level compression) - #:buffer-size %default-buffer-size) - (rename-file (string-append nar ".tmp") nar)) + (write-compressed-file call-with-gzip-output-port)) ('lzip - ;; Note: the file port gets closed along with the lzip port. - (call-with-lzip-output-port (open-output-file (string-append nar ".tmp")) - (lambda (port) - (write-file item port)) - #:level (compression-level compression)) - (rename-file (string-append nar ".tmp") nar)) + (write-compressed-file call-with-lzip-output-port)) + ('zstd + (write-compressed-file call-with-zstd-output-port)) ('none ;; Cache nars even when compression is disabled so that we can ;; guarantee the TTL (see .) @@ -871,6 +872,9 @@ example: \"/foo/bar\" yields '(\"foo\" \"bar\")." (($ 'lzip level) (make-lzip-output-port (response-port response) #:level level)) + (($ 'zstd level) + (make-zstd-output-port (response-port response) + #:level level)) (($ 'none) (response-port response)) (#f @@ -953,6 +957,7 @@ blocking." (match string ("gzip" 'gzip) ("lzip" 'lzip) + ("zstd" 'zstd) (_ #f))) (define (effective-compression requested-type compressions) diff --git a/tests/publish.scm b/tests/publish.scm index cafd0f13a2..52101876b5 100644 --- a/tests/publish.scm +++ b/tests/publish.scm @@ -38,6 +38,7 @@ #:use-module ((guix pki) #:select (%public-key-file %private-key-file)) #:use-module (zlib) #:use-module (lzlib) + #:autoload (zstd) (call-with-zstd-input-port) #:use-module (web uri) #:use-module (web client) #:use-module (web response) @@ -54,6 +55,9 @@ (define %store (open-connection-for-tests)) +(define (zstd-supported?) + (resolve-module '(zstd) #t #f #:ensure #f)) + (define %reference (add-text-to-store %store "ref" "foo")) (define %item (add-text-to-store %store "item" "bar" (list %reference))) @@ -237,6 +241,18 @@ References: ~%" (cut restore-file <> temp))) (call-with-input-file temp read-string)))) +(unless (zstd-supported?) (test-skip 1)) +(test-equal "/nar/zstd/*" + "bar" + (call-with-temporary-output-file + (lambda (temp port) + (let ((nar (http-get-port + (publish-uri + (string-append "/nar/zstd/" (basename %item)))))) + (call-with-zstd-input-port nar + (cut restore-file <> temp))) + (call-with-input-file temp read-string)))) + (test-equal "/*.narinfo with compression" `(("StorePath" . ,%item) ("URL" . ,(string-append "nar/gzip/" (basename %item))) -- 2.29.2