[PATCH 0/1] Add lz4 (C implementation)

[PATCH 0/1] Add lz4 (C implementation)

[Leo Famulari]

This patch adds the C reference implementation of the lz4 compression
algorithm.

I'm looking for advice regarding the tests. The tests take >30 minutes on my on
my quad-core i5 with 8 gigabytes of RAM and they require Valgrind as a
native-input. This seems excessive to me but if users install binary
substitutes, they won't run the tests or need to install Valgrind, right? And 
if distributions won't test software, then who will?

It might be possible to cherry-pick which tests are used. `make test` is a
recursive call to a Makefile in the programs/ subdirectory of the tarball. That
Makefile builds the lists of test to run. For this reason, setting
'test-target' in the package definition won't work because it works on the 
top-level Makefile. If someone knows how to specify the test-targets on the 
second Makefile, we could ask upstream for advice on what tests to exclude.

Also, I have sent a message to the lz4 author asking if tests are expected to
fail when run in parallel. I expect the answer is yes, but I'd like to be sure.

Leo Famulari (1):
  gnu: Add lz4.

 gnu/packages/compression.scm | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

-- 
2.4.3

[PATCH 1/1] gnu: Add lz4.

[Leo Famulari]

* gnu/packages/compression.scm (lz4): New variable.
---
 gnu/packages/compression.scm | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index 287ae25..e96518c 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -5,6 +5,7 @@
 ;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
 ;;; Copyright © 2015 Eric Bavier <bavier@member.fsf.org>
 ;;; Copyright © 2015 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2015 Leo Famulari <leo@famulari.name>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -30,6 +31,7 @@
   #:use-module (guix build-system perl)
   #:use-module (gnu packages base)
   #:use-module (gnu packages perl)
+  #:use-module (gnu packages valgrind)
   #:use-module ((srfi srfi-1) #:select (last)))
 
 (define-public zlib
@@ -513,3 +515,34 @@ compression library.")
     (description "IO-Compress provides a Perl interface to allow reading and
 writing of compressed data created with the zlib and bzip2 libraries.")
     (license (package-license perl))))
+
+(define-public lz4
+  (package
+    (name "lz4")
+    (version "r131")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append "https://github.com/Cyan4973/lz4/archive/"
+                            version ".tar.gz"))
+       (sha256
+        (base32 "1vfg305zvj50hwscad24wan9jar6nqj14gdk2hqyr7bb9mhh0kcx"))))
+    (build-system gnu-build-system)
+    (native-inputs `(("valgrind" ,valgrind)))
+    (arguments
+     `(#:test-target "test"
+       #:parallel-tests? #f ; tests fail if run in parallel
+       #:make-flags (list "CC=gcc"
+                          (string-append "PREFIX=" (assoc-ref %outputs "out")))
+       #:phases (modify-phases %standard-phases
+                  (delete 'configure))))
+    (home-page "https://github.com/Cyan4973/lz4")
+    (synopsis "Compression algorithm focused on speed")
+    (description "LZ4 is a lossless compression algorithm, providing
+compression speed at 400 MB/s per core (0.16 Bytes/cycle).  It also features an
+extremely fast decoder, with speed in multiple GB/s per core (0.71 Bytes/cycle).
+A high compression derivative, called LZ4_HC, is also provided.  It trades CPU
+time for compression ratio.")
+    ;; The libraries (lz4, lz4hc, and xxhash are BSD licenced. The command
+    ;; line interface programs (lz4, fullbench, fuzzer, datagen) are GPL2+.
+    (license (list license:bsd-2 license:gpl2+))))
-- 
2.4.3

Re: [PATCH 0/1] Add lz4 (C implementation)

[Taylan Ulrich Bayırlı/Kammer]

Leo Famulari <leo@famulari.name> writes:

> This patch adds the C reference implementation of the lz4 compression
> algorithm.
>
> I'm looking for advice regarding the tests. The tests take >30 minutes
> on my on my quad-core i5 with 8 gigabytes of RAM and they require
> Valgrind as a native-input. This seems excessive to me but if users
> install binary substitutes, they won't run the tests or need to
> install Valgrind, right? And if distributions won't test software,
> then who will?

Indeed users won't have to go through that unless they build from
source, and Valgrind won't become a run-time dependency so long as the
produced package does not contain any references to any files from the
Valgrind package.

Guix contains a lot of software with build and test phases that take a
very long time.  When I was working on Qt it would take 5 GB of disk
space and 6 hours to build every time.  Though that was a pathological
case and this program is tiny in comparison, I'd still say ~30 minutes
is harmless.  It's annoying, but that's the price of disciplined work.

Taylan

Re: [PATCH 0/1] Add lz4 (C implementation)

[Leo Famulari]

So, it is confirmed that parallel testing is not supposed to work:
https://groups.google.com/forum/#!topic/lz4c/Egwtrrbb66M
"If you are talking about doing multiple `make test` in parallel,
then indeed, some tests involved writing, reading, and removing test
files from local directory,
so should multiple such operations happen randomly in parallel, there is
a high probability that it will not end well."

On Wed, Sep 9, 2015, at 17:26, Taylan Ulrich Bayırlı/Kammer wrote:
> Guix contains a lot of software with build and test phases that take a
> very long time.  When I was working on Qt it would take 5 GB of disk
> space and 6 hours to build every time.  Though that was a pathological
> case and this program is tiny in comparison, I'd still say ~30 minutes
> is harmless.  It's annoying, but that's the price of disciplined work.

In that case, I say we leave the tests in.

Re: [PATCH 0/1] Add lz4 (C implementation)

[Ludovic Courtès]

Leo Famulari <leo@famulari.name> skribis:

> I'm looking for advice regarding the tests. The tests take >30 minutes on my on
> my quad-core i5 with 8 gigabytes of RAM and they require Valgrind as a
> native-input. This seems excessive to me but if users install binary
> substitutes, they won't run the tests or need to install Valgrind, right? And 
> if distributions won't test software, then who will?

Indeed, I think it’s fine.  As Taylan noted, there are already several
cases of long test suites.

> +    (name "lz4")
> +    (version "r131")

I changed that to “131”, which conforms to the expectations of what a
version number is.  For instance:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (package-name->name+version "lz4-r131")
$3 = "lz4-r131"
$4 = #f
scheme@(guile-user)> (package-name->name+version "lz4-131")
$5 = "lz4"
$6 = "131"
--8<---------------cut here---------------end--------------->8---


> +     (origin
> +       (method url-fetch)
> +       (uri (string-append "https://github.com/Cyan4973/lz4/archive/"
> +                            version ".tar.gz"))
> +       (sha256
> +        (base32 "1vfg305zvj50hwscad24wan9jar6nqj14gdk2hqyr7bb9mhh0kcx"))))

Added a ‘file-name’ field here so we get “lz4-131.tar.gz” instead of
“r131.tar.gz” (this is purely cosmetic; no functional change.)

Thanks!

Ludo’.