[bug#29995] [PATCH 0/5] Publish build logs, compress as gzip

[bug#29995] [PATCH 0/5] Publish build logs, compress as gzip

[Ludovic Courtès]

Hello Guix!

This patch series implements build log publication in ‘guix publish’.
Build logs are published as-is—i.e., compressed if they are already
compressed, uncompressed otherwise.

Until now we were unable to view build logs from berlin.guixsd.org, so
this bridges a gap between Hydra and publish/Cuirass.

Additionally, these patches add support for gzip compression of build
logs, which is probably more appropriate for ‘guix publish’ (it can
return a text/plain with gzip Content-Encoding, which plays nicely with
browsers and all.)  The last patches make libbz2 an optional dependency.

Comments welcome!

Ludo’.

Ludovic Courtès (5):
  publish: Use 'x-raw-file' internal response header.
  publish: Publish build logs.
  daemon: Add gzip log compression.
  daemon: Make libbz2 an optional dependency.
  doc: Mark zlib as mandatory, libbz2 as optional.

 README                        |   3 +-
 config-daemon.ac              |  18 +++++--
 doc/guix.texi                 |  29 ++++++++---
 guix/scripts/publish.scm      | 117 +++++++++++++++++++++++++++---------------
 guix/store.scm                |   6 ++-
 nix/libstore/build.cc         |  60 ++++++++++++++++++++--
 nix/libstore/globals.cc       |   8 ++-
 nix/libstore/globals.hh       |  11 +++-
 nix/local.mk                  |  12 +++--
 nix/nix-daemon/guix-daemon.cc |  27 ++++++++--
 tests/guix-daemon.sh          |  38 +++++++++++++-
 tests/publish.scm             |  28 ++++++++++
 12 files changed, 285 insertions(+), 72 deletions(-)

-- 
2.15.1

[bug#29995] [PATCH 1/5] publish: Use 'x-raw-file' internal response header.

[Ludovic Courtès]

This adjusts the workaround for <http://bugs.gnu.org/21093> so that it's
not limited to a single content-type.

* guix/scripts/publish.scm (render-nar/cached): Add the 'x-raw-file'
header on the response.
(render-content-addressed-file): Likewise.
(with-content-length): Remove the 'x-raw-file' header.
(http-write): Instead of dispatching on 'application/octet-stream',
check whether 'x-raw-file' is set to determine whether to spawn a
thread.
---
 guix/scripts/publish.scm | 88 +++++++++++++++++++++++++-----------------------
 1 file changed, 46 insertions(+), 42 deletions(-)

diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index dd54f0399..7e6d3fb23 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -1,6 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
-;;; Copyright © 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -544,11 +544,12 @@ return it; otherwise, return 404."
                                 #:compression compression)))
     (if (file-exists? cached)
         (values `((content-type . (application/octet-stream
-                                   (charset . "ISO-8859-1"))))
-                ;; XXX: We're not returning the actual contents, deferring
-                ;; instead to 'http-write'.  This is a hack to work around
-                ;; <http://bugs.gnu.org/21093>.
-                cached)
+                                   (charset . "ISO-8859-1")))
+                  ;; XXX: We're not returning the actual contents, deferring
+                  ;; instead to 'http-write'.  This is a hack to work around
+                  ;; <http://bugs.gnu.org/21093>.
+                  (x-raw-file . ,cached))
+                #f)
         (not-found request))))
 
 (define (render-content-addressed-file store request
@@ -562,11 +563,12 @@ has the given HASH of type ALGO."
                                      #:recursive? #f)))
         (if (valid-path? store item)
             (values `((content-type . (application/octet-stream
-                                       (charset . "ISO-8859-1"))))
-                    ;; XXX: We're not returning the actual contents, deferring
-                    ;; instead to 'http-write'.  This is a hack to work around
-                    ;; <http://bugs.gnu.org/21093>.
-                    item)
+                                       (charset . "ISO-8859-1")))
+                      ;; XXX: We're not returning the actual contents,
+                      ;; deferring instead to 'http-write'.  This is a hack to
+                      ;; work around <http://bugs.gnu.org/21093>.
+                      (x-raw-file . ,item))
+                    #f)
             (not-found request)))
       (not-found request)))
 
@@ -622,9 +624,9 @@ example: \"/foo/bar\" yields '(\"foo\" \"bar\")."
   "Return RESPONSE with a 'content-length' header set to LENGTH."
   (set-field response (response-headers)
              (alist-cons 'content-length length
-                         (alist-delete 'content-length
-                                       (response-headers response)
-                                       eq?))))
+                         (fold alist-delete
+                               (response-headers response)
+                               '(content-length x-raw-file)))))
 
 (define-syntax-rule (swallow-EPIPE exp ...)
   "Swallow EPIPE errors raised by EXP..."
@@ -685,35 +687,37 @@ blocking."
           (swallow-zlib-error
            (close-port port))
           (values)))))
-    (('application/octet-stream . _)
-     ;; Send a raw file in a separate thread.
-     (call-with-new-thread
-      (lambda ()
-        (set-thread-name "publish file")
-        (catch 'system-error
-          (lambda ()
-            (call-with-input-file (utf8->string body)
-              (lambda (input)
-                (let* ((size     (stat:size (stat input)))
-                       (response (write-response (with-content-length response
-                                                                      size)
-                                                 client))
-                       (output   (response-port response)))
-                  (if (file-port? output)
-                      (sendfile output input size)
-                      (dump-port input output))
-                  (close-port output)
-                  (values)))))
-          (lambda args
-            ;; If the file was GC'd behind our back, that's fine.  Likewise if
-            ;; the client closes the connection.
-            (unless (memv (system-error-errno args)
-                          (list ENOENT EPIPE ECONNRESET))
-              (apply throw args))
-            (values))))))
     (_
-     ;; Handle other responses sequentially.
-     (%http-write server client response body))))
+     (match (assoc-ref (response-headers response) 'x-raw-file)
+       ((? string? file)
+        ;; Send a raw file in a separate thread.
+        (call-with-new-thread
+         (lambda ()
+           (set-thread-name "publish file")
+           (catch 'system-error
+             (lambda ()
+               (call-with-input-file file
+                 (lambda (input)
+                   (let* ((size     (stat:size (stat input)))
+                          (response (write-response (with-content-length response
+                                                                         size)
+                                                    client))
+                          (output   (response-port response)))
+                     (if (file-port? output)
+                         (sendfile output input size)
+                         (dump-port input output))
+                     (close-port output)
+                     (values)))))
+             (lambda args
+               ;; If the file was GC'd behind our back, that's fine.  Likewise if
+               ;; the client closes the connection.
+               (unless (memv (system-error-errno args)
+                             (list ENOENT EPIPE ECONNRESET))
+                 (apply throw args))
+               (values))))))
+       (#f
+        ;; Handle other responses sequentially.
+        (%http-write server client response body))))))
 
 (define-server-impl concurrent-http-server
   ;; A variant of Guile's built-in HTTP server that offloads possibly long
-- 
2.15.1

[bug#29995] [PATCH 2/5] publish: Publish build logs.

[Ludovic Courtès]

* guix/scripts/publish.scm (render-log-file): New procedure.
(make-request-handler): Add "log" case.
* tests/publish.scm ("/log/NAME")
("/log/NAME not found"): New tests.
* doc/guix.texi (Invoking guix publish): Document /log URLs.
---
 doc/guix.texi            | 14 ++++++++++++++
 guix/scripts/publish.scm | 29 +++++++++++++++++++++++++++++
 tests/publish.scm        | 28 ++++++++++++++++++++++++++++
 3 files changed, 71 insertions(+)

diff --git a/doc/guix.texi b/doc/guix.texi
index 1d1f30f00..f6ed3ef87 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -5674,6 +5674,7 @@ collection as soon as the build completes.  @xref{Invoking guix gc}, for
 more on GC roots.
 
 @item --log-file
+@cindex build logs, access
 Return the build log file names or URLs for the given
 @var{package-or-derivation}, or raise an error if build logs are
 missing.
@@ -7236,6 +7237,19 @@ http://example.org/file/hello-2.10.tar.gz/sha256/0ssi1@dots{}ndq1i
 Obviously, these URLs only work for files that are in the store; in
 other cases, they return 404 (``Not Found'').
 
+@cindex build logs, publication
+Build logs are available from @code{/log} URLs like:
+
+@example
+http://example.org/log/gwspk@dots{}-guile-2.2.3
+@end example
+
+@noindent
+When @command{guix-daemon} is configured to save compressed build logs,
+as is the case by default (@pxref{Invoking guix-daemon}), @code{/log}
+URLs return the compressed log as-is, with an appropriate
+@code{Content-Type} and/or @code{Content-Encoding} header.
+
 The following options are available:
 
 @table @code
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 7e6d3fb23..1bc0ff3b6 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -572,6 +572,31 @@ has the given HASH of type ALGO."
             (not-found request)))
       (not-found request)))
 
+(define (render-log-file store request name)
+  "Render the log file for NAME, the base name of a store item.  Don't attempt
+to compress or decompress the log file; just return it as-is."
+  (define (response-headers file)
+    ;; XXX: We're not returning the actual contents, deferring instead to
+    ;; 'http-write'.  This is a hack to work around
+    ;; <http://bugs.gnu.org/21093>.
+    (cond ((string-suffix? ".gz" file)
+           `((content-type . (text/plain (charset . "UTF-8")))
+             (content-encoding . gzip)
+             (x-raw-file . ,file)))
+          ((string-suffix? ".bz2" file)
+           `((content-type . (application/x-bzip2
+                              (charset . "ISO-8859-1")))
+             (x-raw-file . ,file)))
+          (else                                   ;uncompressed
+           `((content-type . (text/plain (charset . "UTF-8")))
+             (x-raw-file . ,file)))))
+
+  (let ((log (log-file store
+                       (string-append (%store-prefix) "/" name))))
+    (if log
+        (values (response-headers log) log)
+        (not-found request))))
+
 (define (render-home-page request)
   "Render the home page."
   (values `((content-type . (text/html (charset . "UTF-8"))))
@@ -772,6 +797,10 @@ blocking."
                (render-content-addressed-file store request
                                               name 'sha256 hash))))
 
+          ;; /log/OUTPUT
+          (("log" name)
+           (render-log-file store request name))
+
           ;; Use different URLs depending on the compression type.  This
           ;; guarantees that /nar URLs remain valid even when 'guix publish'
           ;; is restarted with different compression parameters.
diff --git a/tests/publish.scm b/tests/publish.scm
index 352caf532..bd1a75cf0 100644
--- a/tests/publish.scm
+++ b/tests/publish.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet@gnu.org>
+;;; Copyright © 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -439,4 +440,31 @@ FileSize: ~a~%"
                          (assoc-ref narinfo "FileSize"))
                         (response-code compressed))))))))))
 
+(test-equal "/log/NAME"
+  `(200 #t application/x-bzip2)
+  (let ((drv (run-with-store %store
+               (gexp->derivation "with-log"
+                                 #~(call-with-output-file #$output
+                                     (lambda (port)
+                                       (display "Hello, build log!"
+                                                (current-error-port))
+                                       (display "" port)))))))
+    (build-derivations %store (list drv))
+    (let* ((response (http-get
+                      (publish-uri (string-append "/log/"
+                                                  (basename (derivation->output-path drv))))
+                      #:decode-body? #f))
+           (base     (basename (derivation-file-name drv)))
+           (log      (string-append (dirname %state-directory)
+                                    "/log/guix/drvs/" (string-take base 2)
+                                    "/" (string-drop base 2) ".bz2")))
+      (list (response-code response)
+            (= (response-content-length response) (stat:size (stat log)))
+            (first (response-content-type response))))))
+
+(test-equal "/log/NAME not found"
+  404
+  (let ((uri (publish-uri "/log/does-not-exist")))
+    (response-code (http-get uri))))
+
 (test-end "publish")
-- 
2.15.1

[bug#29995] [PATCH 3/5] daemon: Add gzip log compression.

[Ludovic Courtès]

* nix/nix-daemon/guix-daemon.cc (GUIX_OPT_LOG_COMPRESSION): New macro.
(options): Mark "disable-log-compression" as hidden and add
"log-compression".
(parse_opt): Handle GUIX_OPT_LOG_COMPRESSION.
* nix/libstore/build.cc (DerivationGoal): Add 'gzLogFile'.
(openLogFile): Initialize it when 'logCompression' is COMPRESSION_GZIP.
(closeLogFile, handleChildOutput): Honor 'gzLogFile'.
* nix/libstore/globals.hh (Settings)[compressLog]: Remove.
[logCompression]: New field.
(CompressionType): New enum.
* nix/libstore/globals.cc (Settings::Settings): Initialize it.
(update): Remove '_get' call for 'compressLog'.
* nix/local.mk (guix_daemon_LDADD, guix_register_LDADD): Add -lz.
* guix/store.scm (log-file): Handle '.gz' log files.
* tests/guix-daemon.sh: Add test with '--log-compression=gzip'.
* doc/guix.texi (Invoking guix-daemon): Adjust accordingly.
* config-daemon.ac: Check for libz and zlib.h.
---
 config-daemon.ac              |  6 ++++++
 doc/guix.texi                 |  9 +++++----
 guix/store.scm                |  6 ++++--
 nix/libstore/build.cc         | 45 +++++++++++++++++++++++++++++++++++++++----
 nix/libstore/globals.cc       |  4 ++--
 nix/libstore/globals.hh       |  8 +++++++-
 nix/local.mk                  |  6 +++---
 nix/nix-daemon/guix-daemon.cc | 25 +++++++++++++++++++++---
 tests/guix-daemon.sh          | 38 +++++++++++++++++++++++++++++++++++-
 9 files changed, 127 insertions(+), 20 deletions(-)

diff --git a/config-daemon.ac b/config-daemon.ac
index 42b59819d..59f6f2713 100644
--- a/config-daemon.ac
+++ b/config-daemon.ac
@@ -18,6 +18,12 @@ if test "x$guix_build_daemon" = "xyes"; then
   dnl Use 64-bit file system calls so that we can support files > 2 GiB.
   AC_SYS_LARGEFILE
 
+  dnl Look for zlib, a required dependency.
+  AC_CHECK_LIB([z], [gzdopen], [true],
+    [AC_MSG_ERROR([Guix requires zlib.  See http://www.zlib.net/.])])
+  AC_CHECK_HEADERS([zlib.h], [true],
+    [AC_MSG_ERROR([Guix requires zlib.  See http://www.zlib.net/.])])
+
   dnl Look for libbz2, a required dependency.
   AC_CHECK_LIB([bz2], [BZ2_bzWriteOpen], [true],
     [AC_MSG_ERROR([Guix requires libbz2, which is part of bzip2.  See http://www.bzip.org/.])])
diff --git a/doc/guix.texi b/doc/guix.texi
index f6ed3ef87..bd9bcb73a 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -13,7 +13,7 @@
 @set OPENPGP-SIGNING-KEY-ID 3CE464558A84FDC69DB40CFB090B11993D9AEBB5
 
 @copying
-Copyright @copyright{} 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès@*
+Copyright @copyright{} 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès@*
 Copyright @copyright{} 2013, 2014, 2016 Andreas Enge@*
 Copyright @copyright{} 2013 Nikita Karetnikov@*
 Copyright @copyright{} 2014, 2015, 2016 Alex Kost@*
@@ -1234,12 +1234,13 @@ processes to gain access to undeclared dependencies.  It is necessary,
 though, when @command{guix-daemon} is running under an unprivileged user
 account.
 
-@item --disable-log-compression
-Disable compression of the build logs.
+@item --log-compression=@var{type}
+Compress build logs according to @var{type}, one of @code{gzip},
+@code{bzip2}, or @code{none}.
 
 Unless @code{--lose-logs} is used, all the build logs are kept in the
 @var{localstatedir}.  To save space, the daemon automatically compresses
-them with bzip2 by default.  This option disables that.
+them with bzip2 by default.
 
 @item --disable-deduplication
 @cindex deduplication
diff --git a/guix/store.scm b/guix/store.scm
index e6e45ba89..89db46b8e 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -1567,8 +1567,10 @@ must be an absolute store file name, or a derivation file name."
                                         "/log/guix/drvs/"
                                         (string-take base 2) "/"
                                         (string-drop base 2)))
+                (log.gz  (string-append log ".gz"))
                 (log.bz2 (string-append log ".bz2")))
-           (cond ((file-exists? log.bz2) log.bz2)
+           (cond ((file-exists? log.gz) log.gz)
+                 ((file-exists? log.bz2) log.bz2)
                  ((file-exists? log) log)
                  (else #f))))
         (else
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index d68e8b2bc..5bf3e3aac 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -31,6 +31,7 @@
 #include <pwd.h>
 #include <grp.h>
 
+#include <zlib.h>
 #include <bzlib.h>
 
 /* Includes required for chroot support. */
@@ -744,6 +745,7 @@ private:
 
     /* File descriptor for the log file. */
     FILE * fLogFile;
+    gzFile   gzLogFile;
     BZFILE * bzLogFile;
     AutoCloseFD fdLogFile;
 
@@ -892,6 +894,7 @@ DerivationGoal::DerivationGoal(const Path & drvPath, const StringSet & wantedOut
     , needRestart(false)
     , retrySubstitution(false)
     , fLogFile(0)
+    , gzLogFile(0)
     , bzLogFile(0)
     , useChroot(false)
     , buildMode(buildMode)
@@ -2599,8 +2602,25 @@ Path DerivationGoal::openLogFile()
     Path dir = (format("%1%/%2%/%3%/") % settings.nixLogDir % drvsLogDir % string(baseName, 0, 2)).str();
     createDirs(dir);
 
-    if (settings.compressLog) {
+    switch (settings.logCompression)
+      {
+      case COMPRESSION_GZIP: {
+        Path logFileName = (format("%1%/%2%.gz") % dir % string(baseName, 2)).str();
+        AutoCloseFD fd = open(logFileName.c_str(), O_CREAT | O_WRONLY | O_TRUNC, 0666);
+        if (fd == -1) throw SysError(format("creating log file `%1%'") % logFileName);
+        closeOnExec(fd);
 
+	/* Note: FD will be closed by 'gzclose'.  */
+        if (!(gzLogFile = gzdopen(fd.borrow(), "w")))
+            throw Error(format("cannot open compressed log file `%1%'") % logFileName);
+
+        gzbuffer(gzLogFile, 32768);
+        gzsetparams(gzLogFile, Z_BEST_COMPRESSION, Z_DEFAULT_STRATEGY);
+
+        return logFileName;
+      }
+
+      case COMPRESSION_BZIP2: {
         Path logFileName = (format("%1%/%2%.bz2") % dir % string(baseName, 2)).str();
         AutoCloseFD fd = open(logFileName.c_str(), O_CREAT | O_WRONLY | O_TRUNC, 0666);
         if (fd == -1) throw SysError(format("creating log file `%1%'") % logFileName);
@@ -2614,20 +2634,30 @@ Path DerivationGoal::openLogFile()
             throw Error(format("cannot open compressed log file `%1%'") % logFileName);
 
         return logFileName;
+      }
 
-    } else {
+      case COMPRESSION_NONE: {
         Path logFileName = (format("%1%/%2%") % dir % string(baseName, 2)).str();
         fdLogFile = open(logFileName.c_str(), O_CREAT | O_WRONLY | O_TRUNC, 0666);
         if (fdLogFile == -1) throw SysError(format("creating log file `%1%'") % logFileName);
         closeOnExec(fdLogFile);
         return logFileName;
+      }
     }
+
+    abort();
 }
 
 
 void DerivationGoal::closeLogFile()
 {
-    if (bzLogFile) {
+    if (gzLogFile) {
+	int err;
+	err = gzclose(gzLogFile);
+	gzLogFile = NULL;
+	if (err != Z_OK) throw Error(format("cannot close compressed log file (gzip error = %1%)") % err);
+    }
+    else if (bzLogFile) {
         int err;
         BZ2_bzWriteClose(&err, bzLogFile, 0, 0, 0);
         bzLogFile = 0;
@@ -2695,7 +2725,14 @@ void DerivationGoal::handleChildOutput(int fd, const string & data)
         }
         if (verbosity >= settings.buildVerbosity)
             writeToStderr(data);
-        if (bzLogFile) {
+
+	if (gzLogFile) {
+	    if (data.size() > 0) {
+		int count, err;
+		count = gzwrite(gzLogFile, data.data(), data.size());
+		if (count == 0) throw Error(format("cannot write to compressed log file (gzip error = %1%)") % gzerror(gzLogFile, &err));
+	    }
+	} else if (bzLogFile) {
             int err;
             BZ2_bzWrite(&err, bzLogFile, (unsigned char *) data.data(), data.size());
             if (err != BZ_OK) throw Error(format("cannot write to compressed log file (BZip2 error = %1%)") % err);
diff --git a/nix/libstore/globals.cc b/nix/libstore/globals.cc
index 65dad24d9..82d528dc9 100644
--- a/nix/libstore/globals.cc
+++ b/nix/libstore/globals.cc
@@ -45,7 +45,7 @@ Settings::Settings()
     useSshSubstituter = false;
     impersonateLinux26 = false;
     keepLog = true;
-    compressLog = true;
+    logCompression = COMPRESSION_BZIP2;
     maxLogSize = 0;
     cacheFailure = false;
     pollInterval = 5;
@@ -162,7 +162,7 @@ void Settings::update()
     _get(useChroot, "build-use-chroot");
     _get(impersonateLinux26, "build-impersonate-linux-26");
     _get(keepLog, "build-keep-log");
-    _get(compressLog, "build-compress-log");
+    // _get(logCompression, "build-log-compression");
     _get(maxLogSize, "build-max-log-size");
     _get(cacheFailure, "build-cache-failure");
     _get(pollInterval, "build-poll-interval");
diff --git a/nix/libstore/globals.hh b/nix/libstore/globals.hh
index 7beb1a55c..81cf2f52d 100644
--- a/nix/libstore/globals.hh
+++ b/nix/libstore/globals.hh
@@ -8,6 +8,12 @@
 
 namespace nix {
 
+enum CompressionType
+{
+    COMPRESSION_NONE = 0,
+    COMPRESSION_GZIP = 1,
+    COMPRESSION_BZIP2 = 2
+};
 
 struct Settings {
 
@@ -169,7 +175,7 @@ struct Settings {
     bool keepLog;
 
     /* Whether to compress logs. */
-    bool compressLog;
+    enum CompressionType logCompression;
 
     /* Maximum number of bytes a builder can write to stdout/stderr
        before being killed (0 means no limit). */
diff --git a/nix/local.mk b/nix/local.mk
index 9e0c457be..d802da617 100644
--- a/nix/local.mk
+++ b/nix/local.mk
@@ -1,5 +1,5 @@
 # GNU Guix --- Functional package management for GNU
-# Copyright © 2012, 2013, 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org>
+# Copyright © 2012, 2013, 2014, 2015, 2016, 2018 Ludovic Courtès <ludo@gnu.org>
 # Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
 #
 # This file is part of GNU Guix.
@@ -132,7 +132,7 @@ guix_daemon_CPPFLAGS =				\
   -I$(top_srcdir)/%D%/libstore
 
 guix_daemon_LDADD =				\
-  libstore.a libutil.a libformat.a -lbz2	\
+  libstore.a libutil.a libformat.a -lz -lbz2	\
   $(SQLITE3_LIBS) $(LIBGCRYPT_LIBS)
 
 guix_daemon_headers =				\
@@ -149,7 +149,7 @@ guix_register_CPPFLAGS =			\
 
 # XXX: Should we start using shared libs?
 guix_register_LDADD =				\
-  libstore.a libutil.a libformat.a -lbz2	\
+  libstore.a libutil.a libformat.a -lz -lbz2	\
   $(SQLITE3_LIBS) $(LIBGCRYPT_LIBS)
 
 
diff --git a/nix/nix-daemon/guix-daemon.cc b/nix/nix-daemon/guix-daemon.cc
index 796335820..a1ef90dfd 100644
--- a/nix/nix-daemon/guix-daemon.cc
+++ b/nix/nix-daemon/guix-daemon.cc
@@ -1,5 +1,5 @@
 /* GNU Guix --- Functional package management for GNU
-   Copyright (C) 2012, 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+   Copyright (C) 2012, 2013, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
    Copyright (C) 2006, 2010, 2012, 2014 Eelco Dolstra <e.dolstra@tudelft.nl>
 
    This file is part of GNU Guix.
@@ -88,6 +88,7 @@ builds derivations on behalf of its clients.");
 #define GUIX_OPT_BUILD_ROUNDS 17
 #define GUIX_OPT_TIMEOUT 18
 #define GUIX_OPT_MAX_SILENT_TIME 19
+#define GUIX_OPT_LOG_COMPRESSION 20
 
 static const struct argp_option options[] =
   {
@@ -120,8 +121,11 @@ static const struct argp_option options[] =
       n_("build each derivation N times in a row") },
     { "lose-logs", GUIX_OPT_LOSE_LOGS, 0, 0,
       n_("do not keep build logs") },
-    { "disable-log-compression", GUIX_OPT_DISABLE_LOG_COMPRESSION, 0, 0,
+    { "disable-log-compression", GUIX_OPT_DISABLE_LOG_COMPRESSION, 0,
+      OPTION_HIDDEN,				  // deprecated
       n_("disable compression of the build logs") },
+    { "log-compression", GUIX_OPT_LOG_COMPRESSION, "TYPE", 0,
+      n_("use the specified compression type for build logs") },
 
     /* '--disable-deduplication' was known as '--disable-store-optimization'
        up to Guix 0.7 included, so keep the alias around.  */
@@ -197,8 +201,21 @@ parse_opt (int key, char *arg, struct argp_state *state)
 	settings.set("build-extra-chroot-dirs", chroot_dirs);
 	break;
       }
+    case GUIX_OPT_LOG_COMPRESSION:
+      if (strcmp (arg, "none") == 0)
+	settings.logCompression = COMPRESSION_NONE;
+      else if (strcmp (arg, "gzip") == 0)
+	settings.logCompression = COMPRESSION_GZIP;
+      else if (strcmp (arg, "bzip2") == 0)
+	settings.logCompression = COMPRESSION_BZIP2;
+      else
+	{
+	  fprintf (stderr, _("error: %s: unknown compression type\n"), arg);
+	  exit (EXIT_FAILURE);
+	}
+      break;
     case GUIX_OPT_DISABLE_LOG_COMPRESSION:
-      settings.compressLog = false;
+      settings.logCompression = COMPRESSION_NONE;
       break;
     case GUIX_OPT_BUILD_USERS_GROUP:
       settings.buildUsersGroup = arg;
@@ -487,6 +504,8 @@ main (int argc, char *argv[])
 
       /* Effect all the changes made via 'settings.set'.  */
       settings.update ();
+      printMsg(lvlDebug,
+	       format ("build log compression: %1%") % settings.logCompression);
 
       if (settings.useSubstitutes)
 	{
diff --git a/tests/guix-daemon.sh b/tests/guix-daemon.sh
index 7212e3eb6..6f91eb58b 100644
--- a/tests/guix-daemon.sh
+++ b/tests/guix-daemon.sh
@@ -1,5 +1,5 @@
 # GNU Guix --- Functional package management for GNU
-# Copyright © 2012, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
+# Copyright © 2012, 2014, 2015, 2016, 2017, 2018 Ludovic Courtès <ludo@gnu.org>
 #
 # This file is part of GNU Guix.
 #
@@ -193,3 +193,39 @@ do
     GUIX_DAEMON_SOCKET="$socket" guile -c "$client_code"
     kill "$daemon_pid"
 done
+
+# Log compression.
+
+guix-daemon --listen="$socket" --disable-chroot --debug --log-compression=gzip &
+daemon_pid=$!
+
+stamp="compressed-build-log-test-$$-`date +%H%M%S`"
+client_code="
+  (use-modules (guix) (gnu packages bootstrap))
+
+  (with-store store
+    (run-with-store store
+      (mlet %store-monad ((drv (lower-object
+				(computed-file \"compressed-log-test\"
+					       #~(begin
+						   (display \"$stamp\")
+                                                   (newline)
+						   (mkdir #\$output))
+					       #:guile %bootstrap-guile))))
+	(display (derivation-file-name drv))
+	(newline)
+	(return #t))))
+"
+
+GUIX_DAEMON_SOCKET="$socket"
+export GUIX_DAEMON_SOCKET
+
+drv=`guile -c "$client_code"`
+guix build "$drv"
+
+log=`guix build "$drv" --log-file`
+test -f "$log"
+case "$log" in
+    *.gz) test "`gunzip -c < "$log"`" = "$stamp" ;;
+    *)    false ;;
+esac
-- 
2.15.1

[bug#29995] [PATCH 4/5] daemon: Make libbz2 an optional dependency.

[Ludovic Courtès]

* config-daemon.ac: Don't bail out when libbz2 is missing.  Define
'HAVE_LIBBZ2' Automake conditional.
* nix/libstore/build.cc: Wrap relevant bits in '#if HAVE_BZLIB_H'.
* nix/libstore/globals.cc (Settings::Settings): 'logCompression'
defaults to COMPRESSION_GZIP when HAVE_BZLIB_H is false.
* nix/libstore/globals.hh (CompressionType): Make 'COMPRESSION_BZIP2'
conditional on HAVE_BZLIB_H.
* nix/local.mk (guix_register_LDADD, guix_daemon_LDADD): Add -lbz2 only
when HAVE_LIBBZ2.
* nix/nix-daemon/guix-daemon.cc (parse_opt): Ignore "bzip2" when not
HAVE_BZLIB_H.
---
 config-daemon.ac              | 12 +++++++-----
 nix/libstore/build.cc         | 15 ++++++++++++++-
 nix/libstore/globals.cc       |  4 ++++
 nix/libstore/globals.hh       |  7 +++++--
 nix/local.mk                  | 10 ++++++++--
 nix/nix-daemon/guix-daemon.cc |  2 ++
 6 files changed, 40 insertions(+), 10 deletions(-)

diff --git a/config-daemon.ac b/config-daemon.ac
index 59f6f2713..80d84cbdb 100644
--- a/config-daemon.ac
+++ b/config-daemon.ac
@@ -24,11 +24,12 @@ if test "x$guix_build_daemon" = "xyes"; then
   AC_CHECK_HEADERS([zlib.h], [true],
     [AC_MSG_ERROR([Guix requires zlib.  See http://www.zlib.net/.])])
 
-  dnl Look for libbz2, a required dependency.
-  AC_CHECK_LIB([bz2], [BZ2_bzWriteOpen], [true],
-    [AC_MSG_ERROR([Guix requires libbz2, which is part of bzip2.  See http://www.bzip.org/.])])
-  AC_CHECK_HEADERS([bzlib.h], [true],
-    [AC_MSG_ERROR([Guix requires libbz2, which is part of bzip2.  See http://www.bzip.org/.])])
+  dnl Look for libbz2, an optional dependency.
+  AC_CHECK_LIB([bz2], [BZ2_bzWriteOpen], [HAVE_LIBBZ2=yes], [HAVE_LIBBZ2=no])
+  if test "x$HAVE_LIBBZ2" = xyes; then
+    AC_CHECK_HEADERS([bzlib.h])
+    HAVE_LIBBZ2="$ac_cv_header_bzlib_h"
+  fi
 
   dnl Look for SQLite, a required dependency.
   PKG_CHECK_MODULES([SQLITE3], [sqlite3 >= 3.6.19])
@@ -169,6 +170,7 @@ if test "x$guix_build_daemon" = "xyes"; then
     [chmod +x nix/scripts/offload])
 fi
 
+AM_CONDITIONAL([HAVE_LIBBZ2], [test "x$HAVE_LIBBZ2" = "xyes"])
 AM_CONDITIONAL([BUILD_DAEMON], [test "x$guix_build_daemon" = "xyes"])
 AM_CONDITIONAL([BUILD_DAEMON_OFFLOAD],			\
   [test "x$guix_build_daemon" = "xyes"			\
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index 5bf3e3aac..275d6a5f7 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -32,7 +32,10 @@
 #include <grp.h>
 
 #include <zlib.h>
-#include <bzlib.h>
+
+#if HAVE_BZLIB_H
+# include <bzlib.h>
+#endif
 
 /* Includes required for chroot support. */
 #if HAVE_SYS_PARAM_H
@@ -746,7 +749,9 @@ private:
     /* File descriptor for the log file. */
     FILE * fLogFile;
     gzFile   gzLogFile;
+#if HAVE_BZLIB_H
     BZFILE * bzLogFile;
+#endif
     AutoCloseFD fdLogFile;
 
     /* Number of bytes received from the builder's stdout/stderr. */
@@ -895,7 +900,9 @@ DerivationGoal::DerivationGoal(const Path & drvPath, const StringSet & wantedOut
     , retrySubstitution(false)
     , fLogFile(0)
     , gzLogFile(0)
+#if HAVE_BZLIB_H
     , bzLogFile(0)
+#endif
     , useChroot(false)
     , buildMode(buildMode)
 {
@@ -2620,6 +2627,7 @@ Path DerivationGoal::openLogFile()
         return logFileName;
       }
 
+#if HAVE_BZLIB_H
       case COMPRESSION_BZIP2: {
         Path logFileName = (format("%1%/%2%.bz2") % dir % string(baseName, 2)).str();
         AutoCloseFD fd = open(logFileName.c_str(), O_CREAT | O_WRONLY | O_TRUNC, 0666);
@@ -2635,6 +2643,7 @@ Path DerivationGoal::openLogFile()
 
         return logFileName;
       }
+#endif
 
       case COMPRESSION_NONE: {
         Path logFileName = (format("%1%/%2%") % dir % string(baseName, 2)).str();
@@ -2657,12 +2666,14 @@ void DerivationGoal::closeLogFile()
 	gzLogFile = NULL;
 	if (err != Z_OK) throw Error(format("cannot close compressed log file (gzip error = %1%)") % err);
     }
+#if HAVE_BZLIB_H
     else if (bzLogFile) {
         int err;
         BZ2_bzWriteClose(&err, bzLogFile, 0, 0, 0);
         bzLogFile = 0;
         if (err != BZ_OK) throw Error(format("cannot close compressed log file (BZip2 error = %1%)") % err);
     }
+#endif
 
     if (fLogFile) {
         fclose(fLogFile);
@@ -2732,10 +2743,12 @@ void DerivationGoal::handleChildOutput(int fd, const string & data)
 		count = gzwrite(gzLogFile, data.data(), data.size());
 		if (count == 0) throw Error(format("cannot write to compressed log file (gzip error = %1%)") % gzerror(gzLogFile, &err));
 	    }
+#if HAVE_BZLIB_H
 	} else if (bzLogFile) {
             int err;
             BZ2_bzWrite(&err, bzLogFile, (unsigned char *) data.data(), data.size());
             if (err != BZ_OK) throw Error(format("cannot write to compressed log file (BZip2 error = %1%)") % err);
+#endif
         } else if (fdLogFile != -1)
             writeFull(fdLogFile, data);
     }
diff --git a/nix/libstore/globals.cc b/nix/libstore/globals.cc
index 82d528dc9..4ab6c3a0f 100644
--- a/nix/libstore/globals.cc
+++ b/nix/libstore/globals.cc
@@ -45,7 +45,11 @@ Settings::Settings()
     useSshSubstituter = false;
     impersonateLinux26 = false;
     keepLog = true;
+#if HAVE_BZLIB_H
     logCompression = COMPRESSION_BZIP2;
+#else
+    logCompression = COMPRESSION_GZIP;
+#endif
     maxLogSize = 0;
     cacheFailure = false;
     pollInterval = 5;
diff --git a/nix/libstore/globals.hh b/nix/libstore/globals.hh
index 81cf2f52d..243993695 100644
--- a/nix/libstore/globals.hh
+++ b/nix/libstore/globals.hh
@@ -1,5 +1,6 @@
 #pragma once
 
+#include "config.h"
 #include "types.hh"
 
 #include <map>
@@ -11,8 +12,10 @@ namespace nix {
 enum CompressionType
 {
     COMPRESSION_NONE = 0,
-    COMPRESSION_GZIP = 1,
-    COMPRESSION_BZIP2 = 2
+    COMPRESSION_GZIP = 1
+#if HAVE_BZLIB_H
+    , COMPRESSION_BZIP2 = 2
+#endif
 };
 
 struct Settings {
diff --git a/nix/local.mk b/nix/local.mk
index d802da617..4452301c6 100644
--- a/nix/local.mk
+++ b/nix/local.mk
@@ -132,7 +132,7 @@ guix_daemon_CPPFLAGS =				\
   -I$(top_srcdir)/%D%/libstore
 
 guix_daemon_LDADD =				\
-  libstore.a libutil.a libformat.a -lz -lbz2	\
+  libstore.a libutil.a libformat.a -lz		\
   $(SQLITE3_LIBS) $(LIBGCRYPT_LIBS)
 
 guix_daemon_headers =				\
@@ -149,9 +149,15 @@ guix_register_CPPFLAGS =			\
 
 # XXX: Should we start using shared libs?
 guix_register_LDADD =				\
-  libstore.a libutil.a libformat.a -lz -lbz2	\
+  libstore.a libutil.a libformat.a -lz		\
   $(SQLITE3_LIBS) $(LIBGCRYPT_LIBS)
 
+if HAVE_LIBBZ2
+
+guix_daemon_LDADD += -lbz2
+guix_register_LDADD += -lbz2
+
+endif HAVE_LIBBZ2
 
 noinst_HEADERS =						\
   $(libformat_headers) $(libutil_headers) $(libstore_headers)	\
diff --git a/nix/nix-daemon/guix-daemon.cc b/nix/nix-daemon/guix-daemon.cc
index a1ef90dfd..b71b100f6 100644
--- a/nix/nix-daemon/guix-daemon.cc
+++ b/nix/nix-daemon/guix-daemon.cc
@@ -206,8 +206,10 @@ parse_opt (int key, char *arg, struct argp_state *state)
 	settings.logCompression = COMPRESSION_NONE;
       else if (strcmp (arg, "gzip") == 0)
 	settings.logCompression = COMPRESSION_GZIP;
+#if HAVE_BZLIB_H
       else if (strcmp (arg, "bzip2") == 0)
 	settings.logCompression = COMPRESSION_BZIP2;
+#endif
       else
 	{
 	  fprintf (stderr, _("error: %s: unknown compression type\n"), arg);
-- 
2.15.1

[bug#29995] [PATCH 5/5] doc: Mark zlib as mandatory, libbz2 as optional.

[Ludovic Courtès]

* doc/guix.texi (Requirements): Move zlib to mandatory and libbz2 to
optional.
* README: Ditto.
---
 README        | 3 ++-
 doc/guix.texi | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/README b/README
index 4192eb412..d96a43bbe 100644
--- a/README
+++ b/README
@@ -25,13 +25,14 @@ GNU Guix currently depends on the following packages:
   - [[https://www.gnu.org/software/make/][GNU Make]]
   - [[https://www.gnutls.org][GnuTLS]] compiled with guile support enabled.
   - [[https://gitlab.com/guile-git/guile-git][Guile-Git]]
+  - [[http://www.zlib.net/][zlib]]
   - optionally [[https://savannah.nongnu.org/projects/guile-json/][Guile-JSON]], for the 'guix import pypi' command
 
 Unless `--disable-daemon' was passed, the following packages are needed:
 
   - [[https://sqlite.org/][SQLite 3]]
-  - [[http://www.bzip.org][libbz2]]
   - [[https://gcc.gnu.org][GCC's g++]]
+  - optionally [[http://www.bzip.org][libbz2]]
 
 When `--disable-daemon' was passed, you instead need the following:
 
diff --git a/doc/guix.texi b/doc/guix.texi
index bd9bcb73a..3f4dead82 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -604,6 +604,7 @@ Guile,, gnutls-guile, GnuTLS-Guile});
 @c FIXME: Specify a version number once a release has been made.
 @uref{https://gitlab.com/guile-git/guile-git, Guile-Git}, from August
 2017 or later;
+@item @url{http://zlib.net, zlib};
 @item @url{http://www.gnu.org/software/make/, GNU Make}.
 @end itemize
 
@@ -625,8 +626,8 @@ Support for build offloading (@pxref{Daemon Offload Setup}) and
 version 0.10.2 or later.
 
 @item
-When @url{http://zlib.net, zlib} is available, @command{guix publish}
-can compress build byproducts (@pxref{Invoking guix publish}).
+When @url{http://www.bzip.org, libbz2} is available,
+@command{guix-daemon} can use it to compress build logs.
 @end itemize
 
 Unless @code{--disable-daemon} was passed to @command{configure}, the
@@ -634,7 +635,6 @@ following packages are also needed:
 
 @itemize
 @item @url{http://sqlite.org, SQLite 3};
-@item @url{http://www.bzip.org, libbz2};
 @item @url{http://gcc.gnu.org, GCC's g++}, with support for the
 C++11 standard.
 @end itemize
-- 
2.15.1

[bug#29995] [PATCH 0/5] Publish build logs, compress as gzip

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> Additionally, these patches add support for gzip compression of build
> logs, which is probably more appropriate for ‘guix publish’ (it can
> return a text/plain with gzip Content-Encoding, which plays nicely with
> browsers and all.)  The last patches make libbz2 an optional dependency.

For reference, here’s the compressed build log of GCC 7.2.0 with gzip
and bzip2, -9 and default compression, and no compression:

  -rw-r--r-- 1 ludo users 13890768 Jan  5 15:45 /tmp/log
  -rw-r--r-- 1 ludo users   546709 Jan  5 15:45 /tmp/log.9.bz2
  -rw-r--r-- 1 ludo users   693638 Jan  5 15:45 /tmp/log.9.gz
  -rw-r--r-- 1 ludo users   546709 Jan  5 15:45 /tmp/log.bz2
  -rw-r--r-- 1 ludo users   737772 Jan  5 15:45 /tmp/log.gz

Ludo’.

[bug#29995] [PATCH 0/5] Publish build logs, compress as gzip

[Tobias Geerinckx-Rice]

Ludovic,

Nice! As someone with a (hacky) substitute server that doesn't run
Hydra, this should be very welcome.

Ludovic Courtès wrote on 05/01/18 at 18:04:
> For reference, here’s the compressed build log of GCC 7.2.0 with gzip
> and bzip2, -9 and default compression, and no compression:

I've wondered before: is bzip2 considered part of the API at this point?
It seems so suboptimal in this age of LZMA.

(I'm explicitly not suggesting adding yet another compression format
without deprecating another.)

Kind regards,

T G-R

[bug#29995] [PATCH 0/5] Publish build logs, compress as gzip

[Ludovic Courtès]

Heya Tobias,

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

> Ludovic Courtès wrote on 05/01/18 at 18:04:
>> For reference, here’s the compressed build log of GCC 7.2.0 with gzip
>> and bzip2, -9 and default compression, and no compression:
>
> I've wondered before: is bzip2 considered part of the API at this point?

I think guix-daemon doesn’t have to stick to bzip2, and these patches
show how we can add support for more compression algorithms for build
logs.

Or did you mean another API?

> It seems so suboptimal in this age of LZMA.

For build logs I think it doesn’t matter much as long as it’s
compressed.  gzip is far from ideal but it does a good-enough job IMO.

Ludo’.

bug#29995: [PATCH 0/5] Publish build logs, compress as gzip

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> Ludovic Courtès (5):
>   publish: Use 'x-raw-file' internal response header.
>   publish: Publish build logs.
>   daemon: Add gzip log compression.
>   daemon: Make libbz2 an optional dependency.
>   doc: Mark zlib as mandatory, libbz2 as optional.

I went ahead and pushed these.

I reordered commits and added --log-compression=gzip as a recommendation
in “Invoking guix publish”.

Ludo’.

[bug#29995] [PATCH 0/5] Publish build logs, compress as gzip

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> This patch series implements build log publication in ‘guix publish’.
> Build logs are published as-is—i.e., compressed if they are already
> compressed, uncompressed otherwise.

As a followup, I modified Cuirass so that /build/N/log/raw redirects to
/log/OUTPUT:

  https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=9588e4d4a7c7eed0b3d3729d68f3c8c687c1434e

Ludo’.