* [bug#45460] [PATCH 2/5] utils: Support zstd compression via Guile-zstd.
2020-12-27 14:38 ` [bug#45460] [PATCH 1/5] utils: Remove 'compressed-output-port' Ludovic Courtès
@ 2020-12-27 14:38 ` Ludovic Courtès
2020-12-27 14:38 ` [bug#45460] [PATCH 3/5] publish: Add support for zstd compression Ludovic Courtès
` (2 subsequent siblings)
3 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2020-12-27 14:38 UTC (permalink / raw)
To: 45460
* guix/utils.scm (lzip-port): Return a single value.
(zstd-port): New procedure.
(decompressed-port, compressed-output-port): Add 'zstd' case.
* tests/utils.scm (test-compression/decompression): Test 'zstd' when
the (zstd) module is available.
---
guix/utils.scm | 12 +++++++++++-
tests/utils.scm | 6 ++++--
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/guix/utils.scm b/guix/utils.scm
index e3c78959ed..09eee0ecfb 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -209,7 +209,13 @@ buffered data is lost."
"Return the lzip port produced by calling PROC (a symbol) on PORT and ARGS.
Raise an error if lzlib support is missing."
(let ((make-port (module-ref (resolve-interface '(lzlib)) proc)))
- (values (make-port port) '())))
+ (make-port port)))
+
+(define (zstd-port proc port . args)
+ "Return the zstd port produced by calling PROC (a symbol) on PORT and ARGS.
+Raise an error if zstd support is missing."
+ (let ((make-port (module-ref (resolve-interface '(zstd)) proc)))
+ (make-port port)))
(define (decompressed-port compression input)
"Return an input port where INPUT is decompressed according to COMPRESSION,
@@ -221,6 +227,8 @@ a symbol such as 'xz."
('gzip (filtered-port `(,%gzip "-dc") input))
('lzip (values (lzip-port 'make-lzip-input-port input)
'()))
+ ('zstd (values (zstd-port 'make-zstd-input-port input)
+ '()))
(_ (error "unsupported compression scheme" compression))))
(define (call-with-decompressed-port compression port proc)
@@ -280,6 +288,8 @@ program--e.g., '(\"--fast\")."
('gzip (filtered-output-port `(,%gzip "-c" ,@options) output))
('lzip (values (lzip-port 'make-lzip-output-port output)
'()))
+ ('zstd (values (zstd-port 'make-zstd-output-port output)
+ '()))
(_ (error "unsupported compression scheme" compression))))
(define* (call-with-compressed-output-port compression port proc
diff --git a/tests/utils.scm b/tests/utils.scm
index c278b2a277..9bce446d98 100644
--- a/tests/utils.scm
+++ b/tests/utils.scm
@@ -228,8 +228,10 @@ skip these tests."
get-bytevector-all)))))
(for-each test-compression/decompression
- '(gzip xz lzip)
- (list (const #t) (const #t) (const #t)))
+ `(gzip xz lzip zstd)
+ (list (const #t) (const #t) (const #t)
+ (lambda ()
+ (resolve-module '(zstd) #t #f #:ensure #f))))
;; This is actually in (guix store).
(test-equal "store-path-package-name"
--
2.29.2
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [bug#45460] [PATCH 3/5] publish: Add support for zstd compression.
2020-12-27 14:38 ` [bug#45460] [PATCH 1/5] utils: Remove 'compressed-output-port' Ludovic Courtès
2020-12-27 14:38 ` [bug#45460] [PATCH 2/5] utils: Support zstd compression via Guile-zstd Ludovic Courtès
@ 2020-12-27 14:38 ` Ludovic Courtès
2020-12-28 9:17 ` Jonathan Brielmaier
2020-12-27 14:38 ` [bug#45460] [PATCH 4/5] substitute: Add zstd support Ludovic Courtès
2020-12-27 14:38 ` [bug#45460] [PATCH 5/5] doc: Mention optional dependency on Guile-zstd Ludovic Courtès
3 siblings, 1 reply; 11+ messages in thread
From: Ludovic Courtès @ 2020-12-27 14:38 UTC (permalink / raw)
To: 45460
* guix/scripts/publish.scm (compress-nar)[write-compressed-file]: New
procedure.
Use it for 'gzip' and 'lzip'. Add 'zstd.
(nar-response-port, string->compression-type): Add case for 'zstd'.
* tests/publish.scm (zstd-supported?): New procedure.
("/nar/zstd/*"): New test.
* doc/guix.texi (Invoking guix publish): Document zstd compression.
---
doc/guix.texi | 18 ++++++++++++------
guix/scripts/publish.scm | 31 ++++++++++++++++++-------------
tests/publish.scm | 16 ++++++++++++++++
3 files changed, 46 insertions(+), 19 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index b12cb11bdf..ed38f2e37b 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12329,17 +12329,23 @@ server socket is open and the signing key has been read.
@item --compression[=@var{method}[:@var{level}]]
@itemx -C [@var{method}[:@var{level}]]
Compress data using the given @var{method} and @var{level}. @var{method} is
-one of @code{lzip} and @code{gzip}; when @var{method} is omitted, @code{gzip}
-is used.
+one of @code{lzip}, @code{zstd}, and @code{gzip}; when @var{method} is
+omitted, @code{gzip} is used.
When @var{level} is zero, disable compression. The range 1 to 9 corresponds
to different compression levels: 1 is the fastest, and 9 is the best
(CPU-intensive). The default is 3.
-Usually, @code{lzip} compresses noticeably better than @code{gzip} for a small
-increase in CPU usage; see
-@uref{https://nongnu.org/lzip/lzip_benchmark.html,benchmarks on the lzip Web
-page}.
+Usually, @code{lzip} compresses noticeably better than @code{gzip} for a
+small increase in CPU usage; see
+@uref{https://nongnu.org/lzip/lzip_benchmark.html,benchmarks on the lzip
+Web page}. However, @code{lzip} achieves low decompression throughput
+(on the order of 50@tie{}MiB/s on modern hardware), which can be a
+bottleneck for someone who downloads over a fast network connection.
+
+The compression ratio of @code{zstd} is between that of @code{lzip} and
+that of @code{gzip}; its main advantage is a
+@uref{https://facebook.github.io/zstd/,high decompression speed}.
Unless @option{--cache} is used, compression occurs on the fly and
the compressed streams are not
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 5a865c838d..fa85088ed0 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -56,6 +56,8 @@
#:use-module (zlib)
#:autoload (lzlib) (call-with-lzip-output-port
make-lzip-output-port)
+ #:autoload (zstd) (call-with-zstd-output-port
+ make-zstd-output-port)
#:use-module (guix cache)
#:use-module (guix ui)
#:use-module (guix scripts)
@@ -588,23 +590,22 @@ requested using POOL."
(define nar
(nar-cache-file cache item #:compression compression))
+ (define (write-compressed-file call-with-compressed-output-port)
+ ;; Note: the file port gets closed along with the compressed port.
+ (call-with-compressed-output-port (open-output-file (string-append nar ".tmp"))
+ (lambda (port)
+ (write-file item port))
+ #:level (compression-level compression))
+ (rename-file (string-append nar ".tmp") nar))
+
(mkdir-p (dirname nar))
(match (compression-type compression)
('gzip
- ;; Note: the file port gets closed along with the gzip port.
- (call-with-gzip-output-port (open-output-file (string-append nar ".tmp"))
- (lambda (port)
- (write-file item port))
- #:level (compression-level compression)
- #:buffer-size %default-buffer-size)
- (rename-file (string-append nar ".tmp") nar))
+ (write-compressed-file call-with-gzip-output-port))
('lzip
- ;; Note: the file port gets closed along with the lzip port.
- (call-with-lzip-output-port (open-output-file (string-append nar ".tmp"))
- (lambda (port)
- (write-file item port))
- #:level (compression-level compression))
- (rename-file (string-append nar ".tmp") nar))
+ (write-compressed-file call-with-lzip-output-port))
+ ('zstd
+ (write-compressed-file call-with-zstd-output-port))
('none
;; Cache nars even when compression is disabled so that we can
;; guarantee the TTL (see <https://bugs.gnu.org/28664>.)
@@ -871,6 +872,9 @@ example: \"/foo/bar\" yields '(\"foo\" \"bar\")."
(($ <compression> 'lzip level)
(make-lzip-output-port (response-port response)
#:level level))
+ (($ <compression> 'zstd level)
+ (make-zstd-output-port (response-port response)
+ #:level level))
(($ <compression> 'none)
(response-port response))
(#f
@@ -953,6 +957,7 @@ blocking."
(match string
("gzip" 'gzip)
("lzip" 'lzip)
+ ("zstd" 'zstd)
(_ #f)))
(define (effective-compression requested-type compressions)
diff --git a/tests/publish.scm b/tests/publish.scm
index cafd0f13a2..52101876b5 100644
--- a/tests/publish.scm
+++ b/tests/publish.scm
@@ -38,6 +38,7 @@
#:use-module ((guix pki) #:select (%public-key-file %private-key-file))
#:use-module (zlib)
#:use-module (lzlib)
+ #:autoload (zstd) (call-with-zstd-input-port)
#:use-module (web uri)
#:use-module (web client)
#:use-module (web response)
@@ -54,6 +55,9 @@
(define %store
(open-connection-for-tests))
+(define (zstd-supported?)
+ (resolve-module '(zstd) #t #f #:ensure #f))
+
(define %reference (add-text-to-store %store "ref" "foo"))
(define %item (add-text-to-store %store "item" "bar" (list %reference)))
@@ -237,6 +241,18 @@ References: ~%"
(cut restore-file <> temp)))
(call-with-input-file temp read-string))))
+(unless (zstd-supported?) (test-skip 1))
+(test-equal "/nar/zstd/*"
+ "bar"
+ (call-with-temporary-output-file
+ (lambda (temp port)
+ (let ((nar (http-get-port
+ (publish-uri
+ (string-append "/nar/zstd/" (basename %item))))))
+ (call-with-zstd-input-port nar
+ (cut restore-file <> temp)))
+ (call-with-input-file temp read-string))))
+
(test-equal "/*.narinfo with compression"
`(("StorePath" . ,%item)
("URL" . ,(string-append "nar/gzip/" (basename %item)))
--
2.29.2
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [bug#45460] [PATCH 3/5] publish: Add support for zstd compression.
2020-12-27 14:38 ` [bug#45460] [PATCH 3/5] publish: Add support for zstd compression Ludovic Courtès
@ 2020-12-28 9:17 ` Jonathan Brielmaier
2021-01-03 15:18 ` Ludovic Courtès
0 siblings, 1 reply; 11+ messages in thread
From: Jonathan Brielmaier @ 2020-12-28 9:17 UTC (permalink / raw)
To: Ludovic Courtès, 45460
Nice thing, only one nitpick
On 27.12.20 15:38, Ludovic Courtès wrote:
> * doc/guix.texi (Invoking guix publish): Document zstd compression.
> ---
> doc/guix.texi | 18 ++++++++++++------
> guix/scripts/publish.scm | 31 ++++++++++++++++++-------------
> tests/publish.scm | 16 ++++++++++++++++
> 3 files changed, 46 insertions(+), 19 deletions(-)
>
> diff --git a/doc/guix.texi b/doc/guix.texi
> index b12cb11bdf..ed38f2e37b 100644
> --- a/doc/guix.texi
> +++ b/doc/guix.texi
> @@ -12329,17 +12329,23 @@ server socket is open and the signing key has been read.
> @item --compression[=@var{method}[:@var{level}]]
> @itemx -C [@var{method}[:@var{level}]]
> Compress data using the given @var{method} and @var{level}. @var{method} is
> -one of @code{lzip} and @code{gzip}; when @var{method} is omitted, @code{gzip}
> -is used.
> +one of @code{lzip}, @code{zstd}, and @code{gzip}; when @var{method} is
> +omitted, @code{gzip} is used.
>
> When @var{level} is zero, disable compression. The range 1 to 9 corresponds
> to different compression levels: 1 is the fastest, and 9 is the best
> (CPU-intensive). The default is 3.
>
> -Usually, @code{lzip} compresses noticeably better than @code{gzip} for a small
> -increase in CPU usage; see
> -@uref{https://nongnu.org/lzip/lzip_benchmark.html,benchmarks on the lzip Web
> -page}.
> +Usually, @code{lzip} compresses noticeably better than @code{gzip} for a
> +small increase in CPU usage; see
> +@uref{https://nongnu.org/lzip/lzip_benchmark.html,benchmarks on the lzip
> +Web page}. However, @code{lzip} achieves low decompression throughput
> +(on the order of 50@tie{}MiB/s on modern hardware), which can be a
> +bottleneck for someone who downloads over a fast network connection.
> +
> +The compression ratio of @code{zstd} is between that of @code{lzip} and
> +that of @code{gzip}; its main advantage is a
> +@uref{https://facebook.github.io/zstd/,high decompression speed}.
>
> Unless @option{--cache} is used, compression occurs on the fly and
> the compressed streams are not
It should be also documented at the `guix-publish-configuration` section
in the manual. As I guess that a lot of publish server admins look there...
^ permalink raw reply [flat|nested] 11+ messages in thread
* [bug#45460] [PATCH 3/5] publish: Add support for zstd compression.
2020-12-28 9:17 ` Jonathan Brielmaier
@ 2021-01-03 15:18 ` Ludovic Courtès
0 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2021-01-03 15:18 UTC (permalink / raw)
To: Jonathan Brielmaier; +Cc: 45460
Hallo!
Jonathan Brielmaier <jonathan.brielmaier@web.de> skribis:
> Nice thing, only one nitpick
[...]
>> +The compression ratio of @code{zstd} is between that of @code{lzip} and
>> +that of @code{gzip}; its main advantage is a
>> +@uref{https://facebook.github.io/zstd/,high decompression speed}.
>>
>> Unless @option{--cache} is used, compression occurs on the fly and
>> the compressed streams are not
>
> It should be also documented at the `guix-publish-configuration` section
> in the manual. As I guess that a lot of publish server admins look there...
Good point, I’ll add a cross-reference there.
Ludo’.
^ permalink raw reply [flat|nested] 11+ messages in thread
* [bug#45460] [PATCH 4/5] substitute: Add zstd support.
2020-12-27 14:38 ` [bug#45460] [PATCH 1/5] utils: Remove 'compressed-output-port' Ludovic Courtès
2020-12-27 14:38 ` [bug#45460] [PATCH 2/5] utils: Support zstd compression via Guile-zstd Ludovic Courtès
2020-12-27 14:38 ` [bug#45460] [PATCH 3/5] publish: Add support for zstd compression Ludovic Courtès
@ 2020-12-27 14:38 ` Ludovic Courtès
2020-12-27 14:38 ` [bug#45460] [PATCH 5/5] doc: Mention optional dependency on Guile-zstd Ludovic Courtès
3 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2020-12-27 14:38 UTC (permalink / raw)
To: 45460
* guix/scripts/substitute.scm (%compression-methods): Add zstd.
(compresses-better?): "lzip" always wins.
---
guix/scripts/substitute.scm | 3 +++
1 file changed, 3 insertions(+)
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 8084c89ae5..32ebcf1bb9 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -944,6 +944,8 @@ authorized substitutes."
;; supported. See 'decompressed-port' in (guix utils).
`(("gzip" . ,(const #t))
("lzip" . ,(const #t))
+ ("zstd" . ,(lambda ()
+ (resolve-module '(zstd) #t #f #:ensure #f)))
("xz" . ,(const #t))
("bzip2" . ,(const #t))
("none" . ,(const #t))))
@@ -961,6 +963,7 @@ this is a rough approximation."
(match compression1
("none" #f)
("gzip" (string=? compression2 "none"))
+ ("lzip" #t)
(_ (or (string=? compression2 "none")
(string=? compression2 "gzip")))))
--
2.29.2
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [bug#45460] [PATCH 5/5] doc: Mention optional dependency on Guile-zstd.
2020-12-27 14:38 ` [bug#45460] [PATCH 1/5] utils: Remove 'compressed-output-port' Ludovic Courtès
` (2 preceding siblings ...)
2020-12-27 14:38 ` [bug#45460] [PATCH 4/5] substitute: Add zstd support Ludovic Courtès
@ 2020-12-27 14:38 ` Ludovic Courtès
3 siblings, 0 replies; 11+ messages in thread
From: Ludovic Courtès @ 2020-12-27 14:38 UTC (permalink / raw)
To: 45460
* doc/guix.texi (Requirements): Add Guile-zstd.
---
doc/guix.texi | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/doc/guix.texi b/doc/guix.texi
index ed38f2e37b..dafa1ffc56 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -850,6 +850,11 @@ Support for build offloading (@pxref{Daemon Offload Setup}) and
@uref{https://github.com/artyom-poptsov/guile-ssh, Guile-SSH},
version 0.13.0 or later.
+@item
+@uref{https://notabug.org/guile-zstd/guile-zstd, Guile-zstd}, for zstd
+compression and decompression in @command{guix publish} and for
+substitutes (@pxref{Invoking guix publish}).
+
@item
@uref{https://ngyro.com/software/guile-semver.html, Guile-Semver} for
the @code{crate} importer (@pxref{Invoking guix import}).
--
2.29.2
^ permalink raw reply related [flat|nested] 11+ messages in thread