[PATCH 0/6] gnu: Add ceph.

[PATCH 0/6] gnu: Add ceph.

[Marius Bakke]

Hello Guix,

This series adds the Ceph distributed filesystem to Guix.

Ceph is not fully working yet, the executables need to be wrapped with
PYTHONPATH at the very least. But the library seems to work.

Feedback wanted, will send updated Ceph patch when I've tested it.

Note: "crypto++" ended up unused in favor of "nss" since that is what
upstream uses in their release builds.

Marius Bakke (6):
  gnu: Add leveldb.
  gnu: Add crypto++.
  gnu: Add python-cram.
  gnu: Add rocksdb.
  gnu: Add ceph.
  gnu: fio: Enable rbd support.

 gnu/local.mk                             |   1 +
 gnu/packages/benchmark.scm               |   4 +-
 gnu/packages/crypto.scm                  |  90 +++++++++++++++-
 gnu/packages/databases.scm               | 123 ++++++++++++++++++++++
 gnu/packages/distributed-filesystems.scm | 172 +++++++++++++++++++++++++++++++
 gnu/packages/python.scm                  |  64 +++++++++++-
 6 files changed, 451 insertions(+), 3 deletions(-)
 create mode 100644 gnu/packages/distributed-filesystems.scm

-- 
2.11.0

[PATCH 1/6] gnu: Add leveldb.

[Marius Bakke]

* gnu/packages/databases.scm (leveldb): New variable.
---
 gnu/packages/databases.scm | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index e05a337e4..4bbe55bab 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -13,6 +13,7 @@
 ;;; Copyright © 2016 David Craven <david@craven.ch>
 ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;; Copyright © 2016 Andy Patterson <ajpatter@uwaterloo.ca>
+;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -207,6 +208,42 @@ SQL, Key/Value, XML/XQuery or Java Object storage for their data model.")
                (base32
                 "0a1n5hbl7027fbz5lm0vp0zzfp1hmxnz14wx3zl9563h83br5ag0"))))))
 
+(define-public leveldb
+  (package
+    (name "leveldb")
+    (version "1.19")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://github.com/google/leveldb"
+                                  "/archive/v" version ".tar.gz"))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "00jjgs9xlwycfkg0xd7n1rj6v9zrx7xc7hann6zalrjyhap18ykx"))))
+    (build-system gnu-build-system)
+    (arguments
+     '(#:make-flags (list "CC=gcc")
+       #:phases
+       (modify-phases %standard-phases
+         (delete 'configure)
+         (replace 'install
+           ;; There is no install target, so we do it here.
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (lib (string-append out "/lib"))
+                    (include (string-append out "/include")))
+               (for-each (lambda (file)
+                           (install-file file lib))
+                         (find-files "out-shared" "^libleveldb.so.*"))
+               (copy-recursively "include" include)
+               #t))))))
+    (home-page "http://leveldb.org/")
+    (synopsis "Fast key-value storage library")
+    (description
+     "LevelDB is a fast key-value storage library that provides an ordered
+mapping from string keys to string values.")
+    (license bsd-3)))
+
 (define-public mysql
   (package
     (name "mysql")
-- 
2.11.0

[PATCH 2/6] gnu: Add crypto++.

[Marius Bakke]

* gnu/packages/crypto.scm (crypto++): New variable.
---
 gnu/packages/crypto.scm | 90 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 89 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/crypto.scm b/gnu/packages/crypto.scm
index e4a8a4bd5..2bf64f1f6 100644
--- a/gnu/packages/crypto.scm
+++ b/gnu/packages/crypto.scm
@@ -6,6 +6,7 @@
 ;;; Copyright © 2016 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2016 ng0 <ng0@we.make.ritual.n0.is>
 ;;; Copyright © 2016 Eric Bavier <bavier@member.fsf.org>
+;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -46,12 +47,99 @@
   #:use-module (gnu packages tcl)
   #:use-module (gnu packages tls)
   #:use-module (gnu packages xml)
+  #:use-module (gnu packages zip)
   #:use-module ((guix licenses) #:prefix license:)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix build-system cmake)
-  #:use-module (guix build-system gnu))
+  #:use-module (guix build-system gnu)
+  #:use-module (guix utils)
+  #:use-module (srfi srfi-26)
+  #:use-module (ice-9 match))
+
+(define-public crypto++
+  (package
+    (name "crypto++")
+    (version "5.6.5")
+    (source (origin
+              (method url-fetch)
+              (uri (let ((numeric-version
+                          (match (string-split version #\.)
+                            ((first-digit other-digits ...)
+                             (string-append first-digit
+                                            (string-concatenate
+                                             other-digits))))))
+                     (string-append "https://cryptopp.com/cryptopp"
+                                    numeric-version ".zip")))
+              (sha256
+               (base32
+                "0d1cqdz369ivi082k59025wvxzywvkizw7i0pf5h0a1izs3g8pm7"))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:make-flags
+       (list (string-append "PREFIX=" (assoc-ref %outputs "out"))
+             (string-append "BINDIR=" (assoc-ref %outputs "bin") "/bin")
+             (string-append "DATADIR=" (assoc-ref %outputs "doc") "/share")
+             "DISABLE_CXXFLAGS_OPTIMIZATIONS=1"
+             ;; Override "/sbin/ldconfig" with simply "echo" since
+             ;; we don't need ldconfig(8).
+             "LDCONF=echo")
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'enter-source
+           ;; ??? Why are we in the TestData folder.
+           (lambda _ (chdir "..") #t))
+         (add-after 'enter-source 'disable-optimizations
+           (lambda _
+             ;; XXX: The disable optimizations flag above is not recognized in
+             ;; the current version. See https://github.com/weidai11/cryptopp/pull/354 .
+             ;; For now, just remove it the dirty way.
+             (substitute* "GNUmakefile"
+               (("-march=native") ""))
+             #t))
+         (delete 'configure)
+         (add-after 'build 'build-shared
+           (lambda _
+             ;; By default, only the static library is built.
+             (zero? (system* "make" "shared"))))
+         (add-after 'install 'move-static-library
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (lib (string-append out "/lib"))
+                    (static (assoc-ref outputs "static"))
+                    (slib (string-append static "/lib")))
+               (mkdir-p slib)
+               (for-each (lambda (file)
+                           (install-file file slib)
+                           (delete-file file))
+                         (find-files lib "\\.l?a$"))
+               #t)))
+         (add-after 'move-static-library 'add-so-version-symlink
+           ;; The library is named MAJOR.MINOR.PATCHLEVEL. Some programs
+           ;; expect a MAJOR.MINOR symlink.
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let ((out (assoc-ref outputs "out"))
+                   (major+minor ,(version-major+minor version)))
+               (with-directory-excursion (string-append out "/lib")
+                 (symlink (string-append "libcryptopp.so." ,version)
+                          (string-append "libcryptopp.so." major+minor))
+                 #t)))))))
+    (outputs '("out"              ; 6.4M shared library and headers
+               "bin"              ; 6.3M cryptest.exe
+               "doc"              ; 6.4M documentation and examples
+               "static"))         ; 15M static library
+    (native-inputs
+     `(("unzip" ,unzip)))
+    (synopsis "C++ class library of cryptographic schemes")
+    (description
+     "Crypto++ is a large collection of cryptograhic algorithms and related
+utilities for C++.")
+    (home-page "https://cryptopp.com")
+    ;; The compilation is licensed under Boost 1.0, while most individual
+    ;; files are in the public domain.
+    (license (list license:boost1.0
+                   license:public-domain))))
 
 (define-public libsodium
   (package
-- 
2.11.0

[PATCH 3/6] gnu: Add python-cram.

[Marius Bakke]

* gnu/packages/python.scm (python-cram, python2-cram): New variables.
---
 gnu/packages/python.scm | 64 ++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index ddf276de0..e61d7d1fb 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -26,7 +26,7 @@
 ;;; Copyright © 2016 ng0 <ng0@we.make.ritual.n0.is>
 ;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org>
 ;;; Copyright © 2016 David Craven <david@craven.ch>
-;;; Copyright © 2016 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2016, 2017 Marius Bakke <mbakke@fastmail.com>
 ;;; Copyright © 2016 Stefan Reichoer <stefan@xsteve.at>
 ;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org>
 ;;; Copyright © 2016 Alex Vong <alexvong1995@gmail.com>
@@ -6057,6 +6057,68 @@ pseudo terminal (pty), and interact with both the process and its pty.")
 (define-public python2-ptyprocess
   (package-with-python2 python-ptyprocess))
 
+(define-public python-cram
+  (package
+    (name "python-cram")
+    (version "0.7")
+    (home-page "https://bitheap.org/cram/")
+    (source (origin
+              (method url-fetch)
+              (uri (list (string-append home-page "cram-"
+                                        version ".tar.gz")
+                         (pypi-uri "cram" version)))
+              (sha256
+               (base32
+                "0bvz6fwdi55rkrz3f50zsy35gvvwhlppki2yml5bj5ffy9d499vx"))))
+    (arguments
+     '(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch-source
+           (lambda _
+             (substitute* (find-files "cram" ".*\\.py")
+               ;; Replace default shell path. This is necessary for tests,
+               ;; and convenient for programs using the main "cram" binary
+               ;; in environments lacking /bin/sh (e.g. if this program is
+               ;; used in other builds).
+               (("/bin/sh") (which "sh")))
+             (substitute* (find-files "tests" ".*\\.t")
+               (("md5") "md5sum")
+               (("/bin/bash") (which "bash"))
+               (("/bin/sh") (which "sh")))
+             (substitute* "cram/_test.py"
+               ;; This hack works around a bug triggered by substituting
+               ;; the /bin/sh paths. "tests/usage.t" compares the output of
+               ;; "cram -h", which breaks the output at 80 characters. This
+               ;; causes the line showing the default shell to break into two
+               ;; lines, but the test expects a single line...
+               (("env\\['COLUMNS'\\] = '80'")
+                "env['COLUMNS'] = '160'"))
+             #t))
+         (delete 'check)
+         (add-after 'install 'check
+           ;; The test phase uses the built binary and library.
+           ;; It's easier to run it after install since the build
+           ;; directory contains version-specific PATH.
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (add-installed-pythonpath inputs outputs)
+             (setenv "PATH" (string-append (getenv "PATH") ":"
+                                           (assoc-ref outputs "out") "/bin"))
+             (zero? (system* "make" "test")))))))
+    (build-system python-build-system)
+    (native-inputs
+     `(("python-coverage" ,python-coverage)
+       ("which" ,which)))
+    (synopsis "Simple testing framework for command line applications")
+    (description
+     "Cram is a functional testing framework for command line applications.
+Cram tests look like snippets of interactive shell sessions.  Cram runs each
+command and compares the command output in the test with the command’s actual
+output.")
+    (license license:gpl2+)))
+
+(define-public python2-cram
+  (package-with-python2 python-cram))
+
 (define-public python-terminado
   (package
     (name "python-terminado")
-- 
2.11.0

[PATCH 4/6] gnu: Add rocksdb.

[Marius Bakke]

* gnu/packages/databases.scm (rocksdb): New variable.
---
 gnu/packages/databases.scm | 86 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 4bbe55bab..610749d98 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -54,8 +54,10 @@
   #:use-module (gnu packages cyrus-sasl)
   #:use-module (gnu packages gnupg)
   #:use-module (gnu packages python)
+  #:use-module (gnu packages parallel)
   #:use-module (gnu packages pcre)
   #:use-module (gnu packages pkg-config)
+  #:use-module (gnu packages popt)
   #:use-module (gnu packages rdf)
   #:use-module (gnu packages xml)
   #:use-module (gnu packages bison)
@@ -480,6 +482,90 @@ types are supported, as is encryption.")
     (license gpl3+)
     (home-page "http://www.gnu.org/software/recutils/")))
 
+(define-public rocksdb
+  (package
+    (name "rocksdb")
+    (version "5.0.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://github.com/facebook/rocksdb"
+                                  "/archive/v" version ".tar.gz"))
+              (file-name (string-append name "-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0kija4q6nbkjaj1x94q6qb73abgc5i49rakppxj3a368pg9nwz54"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; TODO: unbundle gtest.
+                  (delete-file "build_tools/gnu_parallel")
+                  #t))))
+    (build-system gnu-build-system)
+    (arguments
+     '(#:make-flags (list "CC=gcc"
+                          ;; Make the resulting library position-independent so the
+                          ;; static version can be included in shared objects.
+                          "EXTRA_CXXFLAGS=-fPIC"
+                          (string-append "INSTALL_PATH="
+                                         (assoc-ref %outputs "out")))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch-source
+           (lambda _
+             (substitute* "Makefile"
+               (("build_tools/gnu_parallel") "parallel")
+               (("#!/bin/sh") (string-append "#!" (which "sh"))))
+             #t))
+         (delete 'configure)
+         (add-before 'check 'disable-failing-tests
+           (lambda _
+             (substitute* "Makefile"
+               ;; These tests reliably fail due to "Too many open files".
+               (("^[[:blank:]]+env_test[[:blank:]]+\\\\") "\\")
+               (("^[[:blank:]]+persistent_cache_test[[:blank:]]+\\\\") "\\"))
+             #t))
+         (add-after 'check 'build-release-libraries
+           ;; The 'check' target depends on the default target which
+           ;; is compiled with debug symbols. The 'install' target depends
+           ;; on custom release targets so we build them here for clarity.
+           ;; TODO: Add debug output.
+           (lambda* (#:key (make-flags '()) #:allow-other-keys)
+             ;; Prevent the build from adding machine-specific optimizations.
+             ;; This does not work if passed as a make flag...
+             (setenv "PORTABLE" "1")
+             (and (zero? (apply system* "make" "static_lib" make-flags))
+                  (zero? (apply system* "make" "shared_lib" make-flags)))))
+         (add-after 'install 'move-static-library
+           (lambda* (#:key outputs #:allow-other-keys)
+             (let* ((out (assoc-ref outputs "out"))
+                    (lib (string-append out "/lib"))
+                    (static (assoc-ref outputs "static"))
+                    (slib (string-append static "/lib")))
+               (mkdir-p slib)
+               (for-each (lambda (file)
+                           (install-file file slib)
+                           (delete-file file))
+                         (find-files lib "\\.l?a$"))
+               #t))))))
+    (outputs
+     '("out" "static"))
+    (native-inputs
+     `(("parallel" ,parallel)
+       ("perl" ,perl)
+       ("procps" ,procps)
+       ("python" ,python-2)))
+    (inputs
+     `(("bzip2" ,bzip2)
+       ("gflags" ,gflags)
+       ("snappy" ,snappy)
+       ("zlib" ,zlib)))
+    (home-page "http://rocksdb.org/")
+    (synopsis "Persistent key-value store for fast storage")
+    (description
+     "RocksDB is an embeddable, persistent key-value storage library that is
+designed for flash and RAM storage.")
+    (license bsd-3)))
+
 (define-public sparql-query
   (package
     (name "sparql-query")
-- 
2.11.0

[PATCH 5/6] gnu: Add ceph.

[Marius Bakke]

* gnu/packages/distributed-filesystems.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
---
 gnu/local.mk                             |   1 +
 gnu/packages/distributed-filesystems.scm | 172 +++++++++++++++++++++++++++++++
 2 files changed, 173 insertions(+)
 create mode 100644 gnu/packages/distributed-filesystems.scm

diff --git a/gnu/local.mk b/gnu/local.mk
index 81d774eb6..417e7a0a5 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -102,6 +102,7 @@ GNU_SYSTEM_MODULES =				\
   %D%/packages/dillo.scm			\
   %D%/packages/disk.scm				\
   %D%/packages/display-managers.scm		\
+  %D%/packages/distributed-filesystems.scm	\
   %D%/packages/django.scm			\
   %D%/packages/djvu.scm				\
   %D%/packages/dns.scm				\
diff --git a/gnu/packages/distributed-filesystems.scm b/gnu/packages/distributed-filesystems.scm
new file mode 100644
index 000000000..f215a847e
--- /dev/null
+++ b/gnu/packages/distributed-filesystems.scm
@@ -0,0 +1,172 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages distributed-filesystems)
+  #:use-module ((guix licenses) #:prefix license:)
+  #:use-module (guix packages)
+  #:use-module (guix download)
+  #:use-module (guix build-system gnu)
+  #:use-module (gnu packages bdw-gc)
+  #:use-module (gnu packages boost)
+  #:use-module (gnu packages compression)
+  #:use-module (gnu packages crypto)
+  #:use-module (gnu packages curl)
+  #:use-module (gnu packages databases)
+  #:use-module (gnu packages gnuzilla)
+  #:use-module (gnu packages jemalloc)
+  #:use-module (gnu packages linux)
+  #:use-module (gnu packages openldap)
+  #:use-module (gnu packages pkg-config)
+  #:use-module (gnu packages python)
+  #:use-module (gnu packages tls)
+  #:use-module (gnu packages web)
+  #:use-module (gnu packages xml))
+
+(define-public ceph
+  (package
+    (name "ceph")
+    (version "10.2.5")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://download.ceph.com/tarballs/ceph-"
+                                  version ".tar.gz"))
+              (sha256
+               (base32
+                "1x6m69il34x4rjhybk5cpw4yiad4a193l9vgy57vidwfy5ql5pc2"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Delete bundled software.
+                  (delete-file-recursively "src/test/downloads") ; python-cram
+                  (delete-file-recursively "src/rocksdb")
+                  ;; TODO: unbundle gtest, civetweb, DPDK, SPDK, xxHash.
+                  #t))))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:configure-flags
+       (list (string-append "--exec_prefix=" (assoc-ref %outputs "out"))
+             (string-append "--libdir=" (assoc-ref %outputs "lib") "/lib")
+             (string-append "--includedir=" (assoc-ref %outputs "lib") "/include")
+             "--localstatedir=/var"
+             "--sysconfdir=/etc"
+             "--enable-static=no" ; TODO: separate output
+             "--with-man-pages"
+             (string-append "--with-systemd-unit-dir="
+                            (assoc-ref %outputs "out") "/etc/systemd/system")
+             "--without-libxfs" ; TODO: enable when xfsprogs is added.
+             ;; Use jemalloc instead of tcmalloc.
+             "--with-jemalloc")
+       #:make-flags
+       ;; Pass sysconfdir here too so that the sample configuration files
+       ;; and directories are installed to the output instead of root level.
+       (list (string-append "sysconfdir=" (assoc-ref %outputs "out") "/etc")
+             (string-append "LDFLAGS=-Wl,-rpath="
+                            (assoc-ref %outputs "lib") "/lib")
+             ;; The python libraries depend on the ceph libraries,
+             ;; so make sure they are in RUNPATH.
+             (string-append "PYTHON_LDFLAGS=-Wl,-rpath="
+                            (assoc-ref %outputs "lib") "/lib"))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'patch-source
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (substitute* "src/Makefile.in"
+               ;; By default, the PYTHONPATH is set to "src/pybind".
+               ;; Make sure our libraries are found too.
+               (("export PYTHONPATH=") (string-append "export PYTHONPATH="
+                                                      (getenv "PYTHONPATH")
+                                                      ":"))
+
+               ;; Replace 'rocksdb/librocksdb.a' with the rocksdb store path.
+               (("rocksdb/librocksdb\\.a") (string-append
+                                            (assoc-ref inputs "rocksdb:static")
+                                            "/lib/librocksdb.a"))
+               ;; The above string is used both as a make target and a compile
+               ;; flag. No-op the make target.
+               (("^.*cd rocksdb &&.*$") "\ttrue\n"))
+
+             (substitute* "src/test/run-cli-tests"
+               ;; Use our python-cram instead of the (un)bundled one.
+               (("CRAM_BIN=.*$")
+                (string-append "CRAM_BIN=" (which "cram") "\n")))
+
+             (substitute* "src/Makefile.in"
+               ;; Install python packages to default output.
+               (("\\$\\$root \\$\\$options") (string-append
+                                              "--root=/ --prefix="
+                                              (assoc-ref outputs "out"))))
+             #t)))
+       ;; XXX: Tests need more work. The bundled cram tarball (!) needs
+       ;; patching to work on Guix, and the system version does not support
+       ;; '--error-dir'. TODO: How to run other tests/targets?
+       #:tests? #f
+       ;; Tests uses lots of file descriptors and warns about using
+       ;; -j without raising limits on large systems.
+       #:parallel-tests? #f))
+    (outputs
+     '("out"
+       "lib"))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)
+       ("rocksdb" ,rocksdb) ; for headers
+       ("python2-cram" ,python2-cram)
+       ("python2-cython" ,python2-cython)
+       ("python2-nose" ,python2-nose)
+       ("python2-pip" ,python2-pip)
+       ("python2-sphinx" ,python2-sphinx)
+       ("python2-tox" ,python2-tox)
+       ("python2-virtualenv" ,python2-virtualenv)))
+    (inputs
+     `(("boost" ,boost)
+       ("curl" ,curl)
+       ("expat" ,expat)
+       ("fcgi" ,fcgi)
+       ("fuse" ,fuse)
+       ("jemalloc" ,jemalloc)
+       ("keyutils" ,keyutils)
+       ("leveldb" ,leveldb)
+       ("libaio" ,libaio)
+       ("libatomic-ops" ,libatomic-ops)
+       ("lz4" ,lz4)
+       ("openldap" ,openldap)
+       ("openssl" ,openssl)
+       ("nss" ,nss)
+       ("python" ,python-2)
+       ("rocksdb:static" ,rocksdb "static")
+       ("snappy" ,snappy)
+       ("udev" ,eudev)
+       ("util-linux" ,util-linux)
+       ("zlib" ,zlib)))
+    (home-page "https://ceph.com")
+    (synopsis "Distributed object store and file system")
+    (description
+     "Ceph is a distributed storage system designed for reliability and
+performance.  It provides network-based block devices (RBD), a POSIX
+compliant filesystem (CephFS), and offers compatibility with various
+storage protocols (S3, NFS, and others) through the RADOS gateway.")
+    ;; Ceph is licensed under LGPL2.1, but includes a number of components
+    ;; covered by other licenses. Consult COPYING for full information.
+    (license (list license:lgpl2.1
+                   license:cc-by-sa3.0    ; documentation
+                   license:bsd-2          ; xxHash
+                   license:bsd-3
+                   license:gpl3
+                   license:gpl2
+                   license:public-domain
+                   license:boost1.0
+                   license:expat))))
-- 
2.11.0

[PATCH 6/6] gnu: fio: Enable rbd support.

[Marius Bakke]

* gnu/packages/benchmark.scm (fio)[inputs]: Add ceph:lib.
---
 gnu/packages/benchmark.scm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/benchmark.scm b/gnu/packages/benchmark.scm
index 465c81b43..6b3c28df3 100644
--- a/gnu/packages/benchmark.scm
+++ b/gnu/packages/benchmark.scm
@@ -22,6 +22,7 @@
   #:use-module (guix download)
   #:use-module (guix build-system gnu)
   #:use-module (gnu packages compression)
+  #:use-module (gnu packages distributed-filesystems)
   #:use-module (gnu packages linux))
 
 (define-public fio
@@ -49,7 +50,8 @@
                (zero? (system* "./configure"
                                (string-append "--prefix=" out)))))))))
     (inputs
-     `(("libaio" ,libaio)
+     `(("ceph" ,ceph "lib")
+       ("libaio" ,libaio)
        ("zlib" ,zlib)))
     (home-page "https://github.com/axboe/fio")
     (synopsis "Flexible I/O tester")
-- 
2.11.0

Re: [PATCH 1/6] gnu: Add leveldb.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/databases.scm (leveldb): New variable.

LGTM!

Re: [PATCH 3/6] gnu: Add python-cram.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/python.scm (python-cram, python2-cram): New variables.

LGTM!

> +             (substitute* "cram/_test.py"
> +               ;; This hack works around a bug triggered by substituting
> +               ;; the /bin/sh paths. "tests/usage.t" compares the output of
> +               ;; "cram -h", which breaks the output at 80 characters. This
> +               ;; causes the line showing the default shell to break into two
> +               ;; lines, but the test expects a single line...
> +               (("env\\['COLUMNS'\\] = '80'")
> +                "env['COLUMNS'] = '160'"))

Woow, good catch.

Ludo’.

Re: [PATCH 2/6] gnu: Add crypto++.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/crypto.scm (crypto++): New variable.

LGTM!  That was not an easy one either…

> +    (outputs '("out"              ; 6.4M shared library and headers
> +               "bin"              ; 6.3M cryptest.exe
> +               "doc"              ; 6.4M documentation and examples
> +               "static"))         ; 15M static library

If static libs are not used, we could even drop the “static” output.

Thanks,
Ludo’.

Re: [PATCH 4/6] gnu: Add rocksdb.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/databases.scm (rocksdb): New variable.

[...]

> +           (lambda* (#:key (make-flags '()) #:allow-other-keys)
> +             ;; Prevent the build from adding machine-specific optimizations.
> +             ;; This does not work if passed as a make flag...
> +             (setenv "PORTABLE" "1")
> +             (and (zero? (apply system* "make" "static_lib" make-flags))
> +                  (zero? (apply system* "make" "shared_lib" make-flags)))))

We could avoid building the static libs if nothing requires it; no
strong opinion though.

> +    (home-page "http://rocksdb.org/")
> +    (synopsis "Persistent key-value store for fast storage")
> +    (description
> +     "RocksDB is an embeddable, persistent key-value storage library that is
> +designed for flash and RAM storage.")

Could you expound a little bit?

LGTM!

Ludo’.

Re: [PATCH 2/6] gnu: Add crypto++.

[Leo Famulari]

On Sun, Jan 15, 2017 at 09:33:34PM +0100, Marius Bakke wrote:
> * gnu/packages/crypto.scm (crypto++): New variable.

We should make sure to get this change as well:
https://github.com/weidai11/cryptopp/pull/347

Re: [PATCH 5/6] gnu: Add ceph.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/distributed-filesystems.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.

[...]

> +             (substitute* "src/Makefile.in"
> +               ;; By default, the PYTHONPATH is set to "src/pybind".
> +               ;; Make sure our libraries are found too.
> +               (("export PYTHONPATH=") (string-append "export PYTHONPATH="
> +                                                      (getenv "PYTHONPATH")
> +                                                      ":"))

I’d move (string-append …) on the next line (for the following
substitutions as well.)

> +    ;; Ceph is licensed under LGPL2.1, but includes a number of components
> +    ;; covered by other licenses. Consult COPYING for full information.
> +    (license (list license:lgpl2.1
> +                   license:cc-by-sa3.0    ; documentation
> +                   license:bsd-2          ; xxHash
> +                   license:bsd-3
> +                   license:gpl3
> +                   license:gpl2
> +                   license:public-domain
> +                   license:boost1.0
> +                   license:expat))))

No “or any later version” clause for the GNU licenses here?

Then apart from the wrapping that you mentioned in the cover letter and
other issues, that looks good.  Great work!

Ludo’.

Re: [PATCH 6/6] gnu: fio: Enable rbd support.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/benchmark.scm (fio)[inputs]: Add ceph:lib.

LGTM!

Quite a lot of code for an “I/O tester”.  ;-)

Ludo’.

Re: [PATCH 2/6] gnu: Add crypto++.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

Leo Famulari <leo@famulari.name> writes:

> On Sun, Jan 15, 2017 at 09:33:34PM +0100, Marius Bakke wrote:
>> * gnu/packages/crypto.scm (crypto++): New variable.
>
> We should make sure to get this change as well:
> https://github.com/weidai11/cryptopp/pull/347

Wow, good catch, will add it!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [PATCH 5/6] gnu: Add ceph.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1941 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

>> +             (substitute* "src/Makefile.in"
>> +               ;; By default, the PYTHONPATH is set to "src/pybind".
>> +               ;; Make sure our libraries are found too.
>> +               (("export PYTHONPATH=") (string-append "export PYTHONPATH="
>> +                                                      (getenv "PYTHONPATH")
>> +                                                      ":"))
>
> I’d move (string-append …) on the next line (for the following
> substitutions as well.)

I tried that, but actually found this easier to read. :-)

Will double-check.

>> +    ;; Ceph is licensed under LGPL2.1, but includes a number of components
>> +    ;; covered by other licenses. Consult COPYING for full information.
>> +    (license (list license:lgpl2.1
>> +                   license:cc-by-sa3.0    ; documentation
>> +                   license:bsd-2          ; xxHash
>> +                   license:bsd-3
>> +                   license:gpl3
>> +                   license:gpl2
>> +                   license:public-domain
>> +                   license:boost1.0
>> +                   license:expat))))
>
> No “or any later version” clause for the GNU licenses here?

Oops, yes, there are some gpl2+ files at least. But Ceph itself is
LGPL2.1 only AFAICT; the COPYING file has a debian-style copyright
format and adds "or later" on some components but not on '*'. No
copyright headers in the source files I've checked.

Unfortunately COPYING does not cover the bundled components, so I still
have some digging to do.

> Then apart from the wrapping that you mentioned in the cover letter and
> other issues, that looks good.  Great work!

Thanks! I'm sure there is a lot of work remaining to make *everything*
work (e.g. 'ceph-deploy'), but will make sure it's possible to provision
a cluster the "old school" way at least :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [PATCH 4/6] gnu: Add rocksdb.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2193 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> * gnu/packages/databases.scm (rocksdb): New variable.
>
> [...]
>
>> +           (lambda* (#:key (make-flags '()) #:allow-other-keys)
>> +             ;; Prevent the build from adding machine-specific optimizations.
>> +             ;; This does not work if passed as a make flag...
>> +             (setenv "PORTABLE" "1")
>> +             (and (zero? (apply system* "make" "static_lib" make-flags))
>> +                  (zero? (apply system* "make" "shared_lib" make-flags)))))
>
> We could avoid building the static libs if nothing requires it; no
> strong opinion though.

I do tend to simply drop the static libs (see e.g. "dlib" or
"capnproto"), but in this case Ceph requires the static library. I will
make another attempt at making it use "-lrocksdb" instead of the ".a"
before pushing this package however, now that I know the build better.

In the crypto++ case, static is the default make target, so I suspect
it's "normal" to use. But as I recently realized, grafting most likely
won't work with static libraries, so I will remove it. Being a crypto
library and all. We should discourage it indeed.

>> +    (home-page "http://rocksdb.org/")
>> +    (synopsis "Persistent key-value store for fast storage")
>> +    (description
>> +     "RocksDB is an embeddable, persistent key-value storage library that is
>> +designed for flash and RAM storage.")
>
> Could you expound a little bit?

How about:

(synopsis "Persistent key-value store for flash and RAM storage")
(description
  "RocksDB is a library that forms the core building block for a fast
  key-value server, especially suited for storing data on flash drives.
  It has a Log-Structured-Merge-Database (LSM) design with flexible
  tradeoffs between Write-Amplification-Factor (WAF),
  Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF).
  It has multi-threaded compactions, making it specially suitable for
  storing multiple terabytes of data in a single database.
  It is based on @code{LevelDB}.")

(taken from the README)

Thanks a lot for the reviews!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [PATCH 5/6] gnu: Add ceph.

[Ludovic Courtès]

Hi!

Marius Bakke <mbakke@fastmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>>> +    ;; Ceph is licensed under LGPL2.1, but includes a number of components
>>> +    ;; covered by other licenses. Consult COPYING for full information.
>>> +    (license (list license:lgpl2.1
>>> +                   license:cc-by-sa3.0    ; documentation
>>> +                   license:bsd-2          ; xxHash
>>> +                   license:bsd-3
>>> +                   license:gpl3
>>> +                   license:gpl2
>>> +                   license:public-domain
>>> +                   license:boost1.0
>>> +                   license:expat))))
>>
>> No “or any later version” clause for the GNU licenses here?
>
> Oops, yes, there are some gpl2+ files at least. But Ceph itself is
> LGPL2.1 only AFAICT; the COPYING file has a debian-style copyright
> format and adds "or later" on some components but not on '*'. No
> copyright headers in the source files I've checked.
>
> Unfortunately COPYING does not cover the bundled components, so I still
> have some digging to do.

OK.

>> other issues, that looks good.  Great work!
>
> Thanks! I'm sure there is a lot of work remaining to make *everything*
> work (e.g. 'ceph-deploy'), but will make sure it's possible to provision
> a cluster the "old school" way at least :-)

Cool.  :-)

I guess it can be fixed incrementally as you stumble upon problems.

Cheers,
Ludo’.

Re: [PATCH 4/6] gnu: Add rocksdb.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Marius Bakke <mbakke@fastmail.com> skribis:
>>
>>> * gnu/packages/databases.scm (rocksdb): New variable.
>>
>> [...]
>>
>>> +           (lambda* (#:key (make-flags '()) #:allow-other-keys)
>>> +             ;; Prevent the build from adding machine-specific optimizations.
>>> +             ;; This does not work if passed as a make flag...
>>> +             (setenv "PORTABLE" "1")
>>> +             (and (zero? (apply system* "make" "static_lib" make-flags))
>>> +                  (zero? (apply system* "make" "shared_lib" make-flags)))))
>>
>> We could avoid building the static libs if nothing requires it; no
>> strong opinion though.
>
> I do tend to simply drop the static libs (see e.g. "dlib" or
> "capnproto"), but in this case Ceph requires the static library. I will
> make another attempt at making it use "-lrocksdb" instead of the ".a"
> before pushing this package however, now that I know the build better.

OK.

> In the crypto++ case, static is the default make target, so I suspect
> it's "normal" to use. But as I recently realized, grafting most likely
> won't work with static libraries, so I will remove it. Being a crypto
> library and all. We should discourage it indeed.

Indeed.  So it’s best to avoid static libs whenever possible.

>>> +    (home-page "http://rocksdb.org/")
>>> +    (synopsis "Persistent key-value store for fast storage")
>>> +    (description
>>> +     "RocksDB is an embeddable, persistent key-value storage library that is
>>> +designed for flash and RAM storage.")
>>
>> Could you expound a little bit?
>
> How about:
>
> (synopsis "Persistent key-value store for flash and RAM storage")
> (description
>   "RocksDB is a library that forms the core building block for a fast
>   key-value server, especially suited for storing data on flash drives.
>   It has a Log-Structured-Merge-Database (LSM) design with flexible
>   tradeoffs between Write-Amplification-Factor (WAF),
>   Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF).
>   It has multi-threaded compactions, making it specially suitable for
>   storing multiple terabytes of data in a single database.
>   It is based on @code{LevelDB}.")

Perfect, thanks!

Ludo’.