* [PATCH 1/6] gnu: Add leveldb.
2017-01-15 20:33 [PATCH 0/6] gnu: Add ceph Marius Bakke
@ 2017-01-15 20:33 ` Marius Bakke
2017-01-17 22:34 ` Ludovic Courtès
2017-01-15 20:33 ` [PATCH 2/6] gnu: Add crypto++ Marius Bakke
` (4 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Marius Bakke @ 2017-01-15 20:33 UTC (permalink / raw)
To: guix-devel; +Cc: Marius Bakke
* gnu/packages/databases.scm (leveldb): New variable.
---
gnu/packages/databases.scm | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index e05a337e4..4bbe55bab 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -13,6 +13,7 @@
;;; Copyright © 2016 David Craven <david@craven.ch>
;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org>
;;; Copyright © 2016 Andy Patterson <ajpatter@uwaterloo.ca>
+;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -207,6 +208,42 @@ SQL, Key/Value, XML/XQuery or Java Object storage for their data model.")
(base32
"0a1n5hbl7027fbz5lm0vp0zzfp1hmxnz14wx3zl9563h83br5ag0"))))))
+(define-public leveldb
+ (package
+ (name "leveldb")
+ (version "1.19")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append "https://github.com/google/leveldb"
+ "/archive/v" version ".tar.gz"))
+ (file-name (string-append name "-" version ".tar.gz"))
+ (sha256
+ (base32
+ "00jjgs9xlwycfkg0xd7n1rj6v9zrx7xc7hann6zalrjyhap18ykx"))))
+ (build-system gnu-build-system)
+ (arguments
+ '(#:make-flags (list "CC=gcc")
+ #:phases
+ (modify-phases %standard-phases
+ (delete 'configure)
+ (replace 'install
+ ;; There is no install target, so we do it here.
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (lib (string-append out "/lib"))
+ (include (string-append out "/include")))
+ (for-each (lambda (file)
+ (install-file file lib))
+ (find-files "out-shared" "^libleveldb.so.*"))
+ (copy-recursively "include" include)
+ #t))))))
+ (home-page "http://leveldb.org/")
+ (synopsis "Fast key-value storage library")
+ (description
+ "LevelDB is a fast key-value storage library that provides an ordered
+mapping from string keys to string values.")
+ (license bsd-3)))
+
(define-public mysql
(package
(name "mysql")
--
2.11.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 2/6] gnu: Add crypto++.
2017-01-15 20:33 [PATCH 0/6] gnu: Add ceph Marius Bakke
2017-01-15 20:33 ` [PATCH 1/6] gnu: Add leveldb Marius Bakke
@ 2017-01-15 20:33 ` Marius Bakke
2017-01-17 22:38 ` Ludovic Courtès
2017-01-17 22:46 ` Leo Famulari
2017-01-15 20:33 ` [PATCH 3/6] gnu: Add python-cram Marius Bakke
` (3 subsequent siblings)
5 siblings, 2 replies; 19+ messages in thread
From: Marius Bakke @ 2017-01-15 20:33 UTC (permalink / raw)
To: guix-devel; +Cc: Marius Bakke
* gnu/packages/crypto.scm (crypto++): New variable.
---
gnu/packages/crypto.scm | 90 ++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 89 insertions(+), 1 deletion(-)
diff --git a/gnu/packages/crypto.scm b/gnu/packages/crypto.scm
index e4a8a4bd5..2bf64f1f6 100644
--- a/gnu/packages/crypto.scm
+++ b/gnu/packages/crypto.scm
@@ -6,6 +6,7 @@
;;; Copyright © 2016 Tobias Geerinckx-Rice <me@tobias.gr>
;;; Copyright © 2016 ng0 <ng0@we.make.ritual.n0.is>
;;; Copyright © 2016 Eric Bavier <bavier@member.fsf.org>
+;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -46,12 +47,99 @@
#:use-module (gnu packages tcl)
#:use-module (gnu packages tls)
#:use-module (gnu packages xml)
+ #:use-module (gnu packages zip)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
#:use-module (guix download)
#:use-module (guix git-download)
#:use-module (guix build-system cmake)
- #:use-module (guix build-system gnu))
+ #:use-module (guix build-system gnu)
+ #:use-module (guix utils)
+ #:use-module (srfi srfi-26)
+ #:use-module (ice-9 match))
+
+(define-public crypto++
+ (package
+ (name "crypto++")
+ (version "5.6.5")
+ (source (origin
+ (method url-fetch)
+ (uri (let ((numeric-version
+ (match (string-split version #\.)
+ ((first-digit other-digits ...)
+ (string-append first-digit
+ (string-concatenate
+ other-digits))))))
+ (string-append "https://cryptopp.com/cryptopp"
+ numeric-version ".zip")))
+ (sha256
+ (base32
+ "0d1cqdz369ivi082k59025wvxzywvkizw7i0pf5h0a1izs3g8pm7"))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:make-flags
+ (list (string-append "PREFIX=" (assoc-ref %outputs "out"))
+ (string-append "BINDIR=" (assoc-ref %outputs "bin") "/bin")
+ (string-append "DATADIR=" (assoc-ref %outputs "doc") "/share")
+ "DISABLE_CXXFLAGS_OPTIMIZATIONS=1"
+ ;; Override "/sbin/ldconfig" with simply "echo" since
+ ;; we don't need ldconfig(8).
+ "LDCONF=echo")
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'enter-source
+ ;; ??? Why are we in the TestData folder.
+ (lambda _ (chdir "..") #t))
+ (add-after 'enter-source 'disable-optimizations
+ (lambda _
+ ;; XXX: The disable optimizations flag above is not recognized in
+ ;; the current version. See https://github.com/weidai11/cryptopp/pull/354 .
+ ;; For now, just remove it the dirty way.
+ (substitute* "GNUmakefile"
+ (("-march=native") ""))
+ #t))
+ (delete 'configure)
+ (add-after 'build 'build-shared
+ (lambda _
+ ;; By default, only the static library is built.
+ (zero? (system* "make" "shared"))))
+ (add-after 'install 'move-static-library
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (lib (string-append out "/lib"))
+ (static (assoc-ref outputs "static"))
+ (slib (string-append static "/lib")))
+ (mkdir-p slib)
+ (for-each (lambda (file)
+ (install-file file slib)
+ (delete-file file))
+ (find-files lib "\\.l?a$"))
+ #t)))
+ (add-after 'move-static-library 'add-so-version-symlink
+ ;; The library is named MAJOR.MINOR.PATCHLEVEL. Some programs
+ ;; expect a MAJOR.MINOR symlink.
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let ((out (assoc-ref outputs "out"))
+ (major+minor ,(version-major+minor version)))
+ (with-directory-excursion (string-append out "/lib")
+ (symlink (string-append "libcryptopp.so." ,version)
+ (string-append "libcryptopp.so." major+minor))
+ #t)))))))
+ (outputs '("out" ; 6.4M shared library and headers
+ "bin" ; 6.3M cryptest.exe
+ "doc" ; 6.4M documentation and examples
+ "static")) ; 15M static library
+ (native-inputs
+ `(("unzip" ,unzip)))
+ (synopsis "C++ class library of cryptographic schemes")
+ (description
+ "Crypto++ is a large collection of cryptograhic algorithms and related
+utilities for C++.")
+ (home-page "https://cryptopp.com")
+ ;; The compilation is licensed under Boost 1.0, while most individual
+ ;; files are in the public domain.
+ (license (list license:boost1.0
+ license:public-domain))))
(define-public libsodium
(package
--
2.11.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 3/6] gnu: Add python-cram.
2017-01-15 20:33 [PATCH 0/6] gnu: Add ceph Marius Bakke
2017-01-15 20:33 ` [PATCH 1/6] gnu: Add leveldb Marius Bakke
2017-01-15 20:33 ` [PATCH 2/6] gnu: Add crypto++ Marius Bakke
@ 2017-01-15 20:33 ` Marius Bakke
2017-01-17 22:36 ` Ludovic Courtès
2017-01-15 20:33 ` [PATCH 4/6] gnu: Add rocksdb Marius Bakke
` (2 subsequent siblings)
5 siblings, 1 reply; 19+ messages in thread
From: Marius Bakke @ 2017-01-15 20:33 UTC (permalink / raw)
To: guix-devel; +Cc: Marius Bakke
* gnu/packages/python.scm (python-cram, python2-cram): New variables.
---
gnu/packages/python.scm | 64 ++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 63 insertions(+), 1 deletion(-)
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index ddf276de0..e61d7d1fb 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -26,7 +26,7 @@
;;; Copyright © 2016 ng0 <ng0@we.make.ritual.n0.is>
;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org>
;;; Copyright © 2016 David Craven <david@craven.ch>
-;;; Copyright © 2016 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2016, 2017 Marius Bakke <mbakke@fastmail.com>
;;; Copyright © 2016 Stefan Reichoer <stefan@xsteve.at>
;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org>
;;; Copyright © 2016 Alex Vong <alexvong1995@gmail.com>
@@ -6057,6 +6057,68 @@ pseudo terminal (pty), and interact with both the process and its pty.")
(define-public python2-ptyprocess
(package-with-python2 python-ptyprocess))
+(define-public python-cram
+ (package
+ (name "python-cram")
+ (version "0.7")
+ (home-page "https://bitheap.org/cram/")
+ (source (origin
+ (method url-fetch)
+ (uri (list (string-append home-page "cram-"
+ version ".tar.gz")
+ (pypi-uri "cram" version)))
+ (sha256
+ (base32
+ "0bvz6fwdi55rkrz3f50zsy35gvvwhlppki2yml5bj5ffy9d499vx"))))
+ (arguments
+ '(#:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'patch-source
+ (lambda _
+ (substitute* (find-files "cram" ".*\\.py")
+ ;; Replace default shell path. This is necessary for tests,
+ ;; and convenient for programs using the main "cram" binary
+ ;; in environments lacking /bin/sh (e.g. if this program is
+ ;; used in other builds).
+ (("/bin/sh") (which "sh")))
+ (substitute* (find-files "tests" ".*\\.t")
+ (("md5") "md5sum")
+ (("/bin/bash") (which "bash"))
+ (("/bin/sh") (which "sh")))
+ (substitute* "cram/_test.py"
+ ;; This hack works around a bug triggered by substituting
+ ;; the /bin/sh paths. "tests/usage.t" compares the output of
+ ;; "cram -h", which breaks the output at 80 characters. This
+ ;; causes the line showing the default shell to break into two
+ ;; lines, but the test expects a single line...
+ (("env\\['COLUMNS'\\] = '80'")
+ "env['COLUMNS'] = '160'"))
+ #t))
+ (delete 'check)
+ (add-after 'install 'check
+ ;; The test phase uses the built binary and library.
+ ;; It's easier to run it after install since the build
+ ;; directory contains version-specific PATH.
+ (lambda* (#:key inputs outputs #:allow-other-keys)
+ (add-installed-pythonpath inputs outputs)
+ (setenv "PATH" (string-append (getenv "PATH") ":"
+ (assoc-ref outputs "out") "/bin"))
+ (zero? (system* "make" "test")))))))
+ (build-system python-build-system)
+ (native-inputs
+ `(("python-coverage" ,python-coverage)
+ ("which" ,which)))
+ (synopsis "Simple testing framework for command line applications")
+ (description
+ "Cram is a functional testing framework for command line applications.
+Cram tests look like snippets of interactive shell sessions. Cram runs each
+command and compares the command output in the test with the command’s actual
+output.")
+ (license license:gpl2+)))
+
+(define-public python2-cram
+ (package-with-python2 python-cram))
+
(define-public python-terminado
(package
(name "python-terminado")
--
2.11.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 3/6] gnu: Add python-cram.
2017-01-15 20:33 ` [PATCH 3/6] gnu: Add python-cram Marius Bakke
@ 2017-01-17 22:36 ` Ludovic Courtès
0 siblings, 0 replies; 19+ messages in thread
From: Ludovic Courtès @ 2017-01-17 22:36 UTC (permalink / raw)
To: Marius Bakke; +Cc: guix-devel
Marius Bakke <mbakke@fastmail.com> skribis:
> * gnu/packages/python.scm (python-cram, python2-cram): New variables.
LGTM!
> + (substitute* "cram/_test.py"
> + ;; This hack works around a bug triggered by substituting
> + ;; the /bin/sh paths. "tests/usage.t" compares the output of
> + ;; "cram -h", which breaks the output at 80 characters. This
> + ;; causes the line showing the default shell to break into two
> + ;; lines, but the test expects a single line...
> + (("env\\['COLUMNS'\\] = '80'")
> + "env['COLUMNS'] = '160'"))
Woow, good catch.
Ludo’.
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 4/6] gnu: Add rocksdb.
2017-01-15 20:33 [PATCH 0/6] gnu: Add ceph Marius Bakke
` (2 preceding siblings ...)
2017-01-15 20:33 ` [PATCH 3/6] gnu: Add python-cram Marius Bakke
@ 2017-01-15 20:33 ` Marius Bakke
2017-01-17 22:42 ` Ludovic Courtès
2017-01-15 20:33 ` [PATCH 5/6] gnu: Add ceph Marius Bakke
2017-01-15 20:33 ` [PATCH 6/6] gnu: fio: Enable rbd support Marius Bakke
5 siblings, 1 reply; 19+ messages in thread
From: Marius Bakke @ 2017-01-15 20:33 UTC (permalink / raw)
To: guix-devel; +Cc: Marius Bakke
* gnu/packages/databases.scm (rocksdb): New variable.
---
gnu/packages/databases.scm | 86 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 86 insertions(+)
diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 4bbe55bab..610749d98 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -54,8 +54,10 @@
#:use-module (gnu packages cyrus-sasl)
#:use-module (gnu packages gnupg)
#:use-module (gnu packages python)
+ #:use-module (gnu packages parallel)
#:use-module (gnu packages pcre)
#:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages popt)
#:use-module (gnu packages rdf)
#:use-module (gnu packages xml)
#:use-module (gnu packages bison)
@@ -480,6 +482,90 @@ types are supported, as is encryption.")
(license gpl3+)
(home-page "http://www.gnu.org/software/recutils/")))
+(define-public rocksdb
+ (package
+ (name "rocksdb")
+ (version "5.0.1")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append "https://github.com/facebook/rocksdb"
+ "/archive/v" version ".tar.gz"))
+ (file-name (string-append name "-" version ".tar.gz"))
+ (sha256
+ (base32
+ "0kija4q6nbkjaj1x94q6qb73abgc5i49rakppxj3a368pg9nwz54"))
+ (modules '((guix build utils)))
+ (snippet
+ '(begin
+ ;; TODO: unbundle gtest.
+ (delete-file "build_tools/gnu_parallel")
+ #t))))
+ (build-system gnu-build-system)
+ (arguments
+ '(#:make-flags (list "CC=gcc"
+ ;; Make the resulting library position-independent so the
+ ;; static version can be included in shared objects.
+ "EXTRA_CXXFLAGS=-fPIC"
+ (string-append "INSTALL_PATH="
+ (assoc-ref %outputs "out")))
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'patch-source
+ (lambda _
+ (substitute* "Makefile"
+ (("build_tools/gnu_parallel") "parallel")
+ (("#!/bin/sh") (string-append "#!" (which "sh"))))
+ #t))
+ (delete 'configure)
+ (add-before 'check 'disable-failing-tests
+ (lambda _
+ (substitute* "Makefile"
+ ;; These tests reliably fail due to "Too many open files".
+ (("^[[:blank:]]+env_test[[:blank:]]+\\\\") "\\")
+ (("^[[:blank:]]+persistent_cache_test[[:blank:]]+\\\\") "\\"))
+ #t))
+ (add-after 'check 'build-release-libraries
+ ;; The 'check' target depends on the default target which
+ ;; is compiled with debug symbols. The 'install' target depends
+ ;; on custom release targets so we build them here for clarity.
+ ;; TODO: Add debug output.
+ (lambda* (#:key (make-flags '()) #:allow-other-keys)
+ ;; Prevent the build from adding machine-specific optimizations.
+ ;; This does not work if passed as a make flag...
+ (setenv "PORTABLE" "1")
+ (and (zero? (apply system* "make" "static_lib" make-flags))
+ (zero? (apply system* "make" "shared_lib" make-flags)))))
+ (add-after 'install 'move-static-library
+ (lambda* (#:key outputs #:allow-other-keys)
+ (let* ((out (assoc-ref outputs "out"))
+ (lib (string-append out "/lib"))
+ (static (assoc-ref outputs "static"))
+ (slib (string-append static "/lib")))
+ (mkdir-p slib)
+ (for-each (lambda (file)
+ (install-file file slib)
+ (delete-file file))
+ (find-files lib "\\.l?a$"))
+ #t))))))
+ (outputs
+ '("out" "static"))
+ (native-inputs
+ `(("parallel" ,parallel)
+ ("perl" ,perl)
+ ("procps" ,procps)
+ ("python" ,python-2)))
+ (inputs
+ `(("bzip2" ,bzip2)
+ ("gflags" ,gflags)
+ ("snappy" ,snappy)
+ ("zlib" ,zlib)))
+ (home-page "http://rocksdb.org/")
+ (synopsis "Persistent key-value store for fast storage")
+ (description
+ "RocksDB is an embeddable, persistent key-value storage library that is
+designed for flash and RAM storage.")
+ (license bsd-3)))
+
(define-public sparql-query
(package
(name "sparql-query")
--
2.11.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 4/6] gnu: Add rocksdb.
2017-01-15 20:33 ` [PATCH 4/6] gnu: Add rocksdb Marius Bakke
@ 2017-01-17 22:42 ` Ludovic Courtès
2017-01-18 0:10 ` Marius Bakke
0 siblings, 1 reply; 19+ messages in thread
From: Ludovic Courtès @ 2017-01-17 22:42 UTC (permalink / raw)
To: Marius Bakke; +Cc: guix-devel
Marius Bakke <mbakke@fastmail.com> skribis:
> * gnu/packages/databases.scm (rocksdb): New variable.
[...]
> + (lambda* (#:key (make-flags '()) #:allow-other-keys)
> + ;; Prevent the build from adding machine-specific optimizations.
> + ;; This does not work if passed as a make flag...
> + (setenv "PORTABLE" "1")
> + (and (zero? (apply system* "make" "static_lib" make-flags))
> + (zero? (apply system* "make" "shared_lib" make-flags)))))
We could avoid building the static libs if nothing requires it; no
strong opinion though.
> + (home-page "http://rocksdb.org/")
> + (synopsis "Persistent key-value store for fast storage")
> + (description
> + "RocksDB is an embeddable, persistent key-value storage library that is
> +designed for flash and RAM storage.")
Could you expound a little bit?
LGTM!
Ludo’.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 4/6] gnu: Add rocksdb.
2017-01-17 22:42 ` Ludovic Courtès
@ 2017-01-18 0:10 ` Marius Bakke
2017-01-18 21:29 ` Ludovic Courtès
0 siblings, 1 reply; 19+ messages in thread
From: Marius Bakke @ 2017-01-18 0:10 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 2193 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> * gnu/packages/databases.scm (rocksdb): New variable.
>
> [...]
>
>> + (lambda* (#:key (make-flags '()) #:allow-other-keys)
>> + ;; Prevent the build from adding machine-specific optimizations.
>> + ;; This does not work if passed as a make flag...
>> + (setenv "PORTABLE" "1")
>> + (and (zero? (apply system* "make" "static_lib" make-flags))
>> + (zero? (apply system* "make" "shared_lib" make-flags)))))
>
> We could avoid building the static libs if nothing requires it; no
> strong opinion though.
I do tend to simply drop the static libs (see e.g. "dlib" or
"capnproto"), but in this case Ceph requires the static library. I will
make another attempt at making it use "-lrocksdb" instead of the ".a"
before pushing this package however, now that I know the build better.
In the crypto++ case, static is the default make target, so I suspect
it's "normal" to use. But as I recently realized, grafting most likely
won't work with static libraries, so I will remove it. Being a crypto
library and all. We should discourage it indeed.
>> + (home-page "http://rocksdb.org/")
>> + (synopsis "Persistent key-value store for fast storage")
>> + (description
>> + "RocksDB is an embeddable, persistent key-value storage library that is
>> +designed for flash and RAM storage.")
>
> Could you expound a little bit?
How about:
(synopsis "Persistent key-value store for flash and RAM storage")
(description
"RocksDB is a library that forms the core building block for a fast
key-value server, especially suited for storing data on flash drives.
It has a Log-Structured-Merge-Database (LSM) design with flexible
tradeoffs between Write-Amplification-Factor (WAF),
Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF).
It has multi-threaded compactions, making it specially suitable for
storing multiple terabytes of data in a single database.
It is based on @code{LevelDB}.")
(taken from the README)
Thanks a lot for the reviews!
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 4/6] gnu: Add rocksdb.
2017-01-18 0:10 ` Marius Bakke
@ 2017-01-18 21:29 ` Ludovic Courtès
0 siblings, 0 replies; 19+ messages in thread
From: Ludovic Courtès @ 2017-01-18 21:29 UTC (permalink / raw)
To: Marius Bakke; +Cc: guix-devel
Marius Bakke <mbakke@fastmail.com> skribis:
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Marius Bakke <mbakke@fastmail.com> skribis:
>>
>>> * gnu/packages/databases.scm (rocksdb): New variable.
>>
>> [...]
>>
>>> + (lambda* (#:key (make-flags '()) #:allow-other-keys)
>>> + ;; Prevent the build from adding machine-specific optimizations.
>>> + ;; This does not work if passed as a make flag...
>>> + (setenv "PORTABLE" "1")
>>> + (and (zero? (apply system* "make" "static_lib" make-flags))
>>> + (zero? (apply system* "make" "shared_lib" make-flags)))))
>>
>> We could avoid building the static libs if nothing requires it; no
>> strong opinion though.
>
> I do tend to simply drop the static libs (see e.g. "dlib" or
> "capnproto"), but in this case Ceph requires the static library. I will
> make another attempt at making it use "-lrocksdb" instead of the ".a"
> before pushing this package however, now that I know the build better.
OK.
> In the crypto++ case, static is the default make target, so I suspect
> it's "normal" to use. But as I recently realized, grafting most likely
> won't work with static libraries, so I will remove it. Being a crypto
> library and all. We should discourage it indeed.
Indeed. So it’s best to avoid static libs whenever possible.
>>> + (home-page "http://rocksdb.org/")
>>> + (synopsis "Persistent key-value store for fast storage")
>>> + (description
>>> + "RocksDB is an embeddable, persistent key-value storage library that is
>>> +designed for flash and RAM storage.")
>>
>> Could you expound a little bit?
>
> How about:
>
> (synopsis "Persistent key-value store for flash and RAM storage")
> (description
> "RocksDB is a library that forms the core building block for a fast
> key-value server, especially suited for storing data on flash drives.
> It has a Log-Structured-Merge-Database (LSM) design with flexible
> tradeoffs between Write-Amplification-Factor (WAF),
> Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF).
> It has multi-threaded compactions, making it specially suitable for
> storing multiple terabytes of data in a single database.
> It is based on @code{LevelDB}.")
Perfect, thanks!
Ludo’.
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 5/6] gnu: Add ceph.
2017-01-15 20:33 [PATCH 0/6] gnu: Add ceph Marius Bakke
` (3 preceding siblings ...)
2017-01-15 20:33 ` [PATCH 4/6] gnu: Add rocksdb Marius Bakke
@ 2017-01-15 20:33 ` Marius Bakke
2017-01-17 22:47 ` Ludovic Courtès
2017-01-15 20:33 ` [PATCH 6/6] gnu: fio: Enable rbd support Marius Bakke
5 siblings, 1 reply; 19+ messages in thread
From: Marius Bakke @ 2017-01-15 20:33 UTC (permalink / raw)
To: guix-devel; +Cc: Marius Bakke
* gnu/packages/distributed-filesystems.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
---
gnu/local.mk | 1 +
gnu/packages/distributed-filesystems.scm | 172 +++++++++++++++++++++++++++++++
2 files changed, 173 insertions(+)
create mode 100644 gnu/packages/distributed-filesystems.scm
diff --git a/gnu/local.mk b/gnu/local.mk
index 81d774eb6..417e7a0a5 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -102,6 +102,7 @@ GNU_SYSTEM_MODULES = \
%D%/packages/dillo.scm \
%D%/packages/disk.scm \
%D%/packages/display-managers.scm \
+ %D%/packages/distributed-filesystems.scm \
%D%/packages/django.scm \
%D%/packages/djvu.scm \
%D%/packages/dns.scm \
diff --git a/gnu/packages/distributed-filesystems.scm b/gnu/packages/distributed-filesystems.scm
new file mode 100644
index 000000000..f215a847e
--- /dev/null
+++ b/gnu/packages/distributed-filesystems.scm
@@ -0,0 +1,172 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2017 Marius Bakke <mbakke@fastmail.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu packages distributed-filesystems)
+ #:use-module ((guix licenses) #:prefix license:)
+ #:use-module (guix packages)
+ #:use-module (guix download)
+ #:use-module (guix build-system gnu)
+ #:use-module (gnu packages bdw-gc)
+ #:use-module (gnu packages boost)
+ #:use-module (gnu packages compression)
+ #:use-module (gnu packages crypto)
+ #:use-module (gnu packages curl)
+ #:use-module (gnu packages databases)
+ #:use-module (gnu packages gnuzilla)
+ #:use-module (gnu packages jemalloc)
+ #:use-module (gnu packages linux)
+ #:use-module (gnu packages openldap)
+ #:use-module (gnu packages pkg-config)
+ #:use-module (gnu packages python)
+ #:use-module (gnu packages tls)
+ #:use-module (gnu packages web)
+ #:use-module (gnu packages xml))
+
+(define-public ceph
+ (package
+ (name "ceph")
+ (version "10.2.5")
+ (source (origin
+ (method url-fetch)
+ (uri (string-append "https://download.ceph.com/tarballs/ceph-"
+ version ".tar.gz"))
+ (sha256
+ (base32
+ "1x6m69il34x4rjhybk5cpw4yiad4a193l9vgy57vidwfy5ql5pc2"))
+ (modules '((guix build utils)))
+ (snippet
+ '(begin
+ ;; Delete bundled software.
+ (delete-file-recursively "src/test/downloads") ; python-cram
+ (delete-file-recursively "src/rocksdb")
+ ;; TODO: unbundle gtest, civetweb, DPDK, SPDK, xxHash.
+ #t))))
+ (build-system gnu-build-system)
+ (arguments
+ `(#:configure-flags
+ (list (string-append "--exec_prefix=" (assoc-ref %outputs "out"))
+ (string-append "--libdir=" (assoc-ref %outputs "lib") "/lib")
+ (string-append "--includedir=" (assoc-ref %outputs "lib") "/include")
+ "--localstatedir=/var"
+ "--sysconfdir=/etc"
+ "--enable-static=no" ; TODO: separate output
+ "--with-man-pages"
+ (string-append "--with-systemd-unit-dir="
+ (assoc-ref %outputs "out") "/etc/systemd/system")
+ "--without-libxfs" ; TODO: enable when xfsprogs is added.
+ ;; Use jemalloc instead of tcmalloc.
+ "--with-jemalloc")
+ #:make-flags
+ ;; Pass sysconfdir here too so that the sample configuration files
+ ;; and directories are installed to the output instead of root level.
+ (list (string-append "sysconfdir=" (assoc-ref %outputs "out") "/etc")
+ (string-append "LDFLAGS=-Wl,-rpath="
+ (assoc-ref %outputs "lib") "/lib")
+ ;; The python libraries depend on the ceph libraries,
+ ;; so make sure they are in RUNPATH.
+ (string-append "PYTHON_LDFLAGS=-Wl,-rpath="
+ (assoc-ref %outputs "lib") "/lib"))
+ #:phases
+ (modify-phases %standard-phases
+ (add-after 'unpack 'patch-source
+ (lambda* (#:key inputs outputs #:allow-other-keys)
+ (substitute* "src/Makefile.in"
+ ;; By default, the PYTHONPATH is set to "src/pybind".
+ ;; Make sure our libraries are found too.
+ (("export PYTHONPATH=") (string-append "export PYTHONPATH="
+ (getenv "PYTHONPATH")
+ ":"))
+
+ ;; Replace 'rocksdb/librocksdb.a' with the rocksdb store path.
+ (("rocksdb/librocksdb\\.a") (string-append
+ (assoc-ref inputs "rocksdb:static")
+ "/lib/librocksdb.a"))
+ ;; The above string is used both as a make target and a compile
+ ;; flag. No-op the make target.
+ (("^.*cd rocksdb &&.*$") "\ttrue\n"))
+
+ (substitute* "src/test/run-cli-tests"
+ ;; Use our python-cram instead of the (un)bundled one.
+ (("CRAM_BIN=.*$")
+ (string-append "CRAM_BIN=" (which "cram") "\n")))
+
+ (substitute* "src/Makefile.in"
+ ;; Install python packages to default output.
+ (("\\$\\$root \\$\\$options") (string-append
+ "--root=/ --prefix="
+ (assoc-ref outputs "out"))))
+ #t)))
+ ;; XXX: Tests need more work. The bundled cram tarball (!) needs
+ ;; patching to work on Guix, and the system version does not support
+ ;; '--error-dir'. TODO: How to run other tests/targets?
+ #:tests? #f
+ ;; Tests uses lots of file descriptors and warns about using
+ ;; -j without raising limits on large systems.
+ #:parallel-tests? #f))
+ (outputs
+ '("out"
+ "lib"))
+ (native-inputs
+ `(("pkg-config" ,pkg-config)
+ ("rocksdb" ,rocksdb) ; for headers
+ ("python2-cram" ,python2-cram)
+ ("python2-cython" ,python2-cython)
+ ("python2-nose" ,python2-nose)
+ ("python2-pip" ,python2-pip)
+ ("python2-sphinx" ,python2-sphinx)
+ ("python2-tox" ,python2-tox)
+ ("python2-virtualenv" ,python2-virtualenv)))
+ (inputs
+ `(("boost" ,boost)
+ ("curl" ,curl)
+ ("expat" ,expat)
+ ("fcgi" ,fcgi)
+ ("fuse" ,fuse)
+ ("jemalloc" ,jemalloc)
+ ("keyutils" ,keyutils)
+ ("leveldb" ,leveldb)
+ ("libaio" ,libaio)
+ ("libatomic-ops" ,libatomic-ops)
+ ("lz4" ,lz4)
+ ("openldap" ,openldap)
+ ("openssl" ,openssl)
+ ("nss" ,nss)
+ ("python" ,python-2)
+ ("rocksdb:static" ,rocksdb "static")
+ ("snappy" ,snappy)
+ ("udev" ,eudev)
+ ("util-linux" ,util-linux)
+ ("zlib" ,zlib)))
+ (home-page "https://ceph.com")
+ (synopsis "Distributed object store and file system")
+ (description
+ "Ceph is a distributed storage system designed for reliability and
+performance. It provides network-based block devices (RBD), a POSIX
+compliant filesystem (CephFS), and offers compatibility with various
+storage protocols (S3, NFS, and others) through the RADOS gateway.")
+ ;; Ceph is licensed under LGPL2.1, but includes a number of components
+ ;; covered by other licenses. Consult COPYING for full information.
+ (license (list license:lgpl2.1
+ license:cc-by-sa3.0 ; documentation
+ license:bsd-2 ; xxHash
+ license:bsd-3
+ license:gpl3
+ license:gpl2
+ license:public-domain
+ license:boost1.0
+ license:expat))))
--
2.11.0
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 5/6] gnu: Add ceph.
2017-01-15 20:33 ` [PATCH 5/6] gnu: Add ceph Marius Bakke
@ 2017-01-17 22:47 ` Ludovic Courtès
2017-01-17 23:11 ` Marius Bakke
0 siblings, 1 reply; 19+ messages in thread
From: Ludovic Courtès @ 2017-01-17 22:47 UTC (permalink / raw)
To: Marius Bakke; +Cc: guix-devel
Marius Bakke <mbakke@fastmail.com> skribis:
> * gnu/packages/distributed-filesystems.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
[...]
> + (substitute* "src/Makefile.in"
> + ;; By default, the PYTHONPATH is set to "src/pybind".
> + ;; Make sure our libraries are found too.
> + (("export PYTHONPATH=") (string-append "export PYTHONPATH="
> + (getenv "PYTHONPATH")
> + ":"))
I’d move (string-append …) on the next line (for the following
substitutions as well.)
> + ;; Ceph is licensed under LGPL2.1, but includes a number of components
> + ;; covered by other licenses. Consult COPYING for full information.
> + (license (list license:lgpl2.1
> + license:cc-by-sa3.0 ; documentation
> + license:bsd-2 ; xxHash
> + license:bsd-3
> + license:gpl3
> + license:gpl2
> + license:public-domain
> + license:boost1.0
> + license:expat))))
No “or any later version” clause for the GNU licenses here?
Then apart from the wrapping that you mentioned in the cover letter and
other issues, that looks good. Great work!
Ludo’.
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 5/6] gnu: Add ceph.
2017-01-17 22:47 ` Ludovic Courtès
@ 2017-01-17 23:11 ` Marius Bakke
2017-01-18 21:27 ` Ludovic Courtès
0 siblings, 1 reply; 19+ messages in thread
From: Marius Bakke @ 2017-01-17 23:11 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: guix-devel
[-- Attachment #1: Type: text/plain, Size: 1941 bytes --]
Ludovic Courtès <ludo@gnu.org> writes:
>> + (substitute* "src/Makefile.in"
>> + ;; By default, the PYTHONPATH is set to "src/pybind".
>> + ;; Make sure our libraries are found too.
>> + (("export PYTHONPATH=") (string-append "export PYTHONPATH="
>> + (getenv "PYTHONPATH")
>> + ":"))
>
> I’d move (string-append …) on the next line (for the following
> substitutions as well.)
I tried that, but actually found this easier to read. :-)
Will double-check.
>> + ;; Ceph is licensed under LGPL2.1, but includes a number of components
>> + ;; covered by other licenses. Consult COPYING for full information.
>> + (license (list license:lgpl2.1
>> + license:cc-by-sa3.0 ; documentation
>> + license:bsd-2 ; xxHash
>> + license:bsd-3
>> + license:gpl3
>> + license:gpl2
>> + license:public-domain
>> + license:boost1.0
>> + license:expat))))
>
> No “or any later version” clause for the GNU licenses here?
Oops, yes, there are some gpl2+ files at least. But Ceph itself is
LGPL2.1 only AFAICT; the COPYING file has a debian-style copyright
format and adds "or later" on some components but not on '*'. No
copyright headers in the source files I've checked.
Unfortunately COPYING does not cover the bundled components, so I still
have some digging to do.
> Then apart from the wrapping that you mentioned in the cover letter and
> other issues, that looks good. Great work!
Thanks! I'm sure there is a lot of work remaining to make *everything*
work (e.g. 'ceph-deploy'), but will make sure it's possible to provision
a cluster the "old school" way at least :-)
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 5/6] gnu: Add ceph.
2017-01-17 23:11 ` Marius Bakke
@ 2017-01-18 21:27 ` Ludovic Courtès
0 siblings, 0 replies; 19+ messages in thread
From: Ludovic Courtès @ 2017-01-18 21:27 UTC (permalink / raw)
To: Marius Bakke; +Cc: guix-devel
Hi!
Marius Bakke <mbakke@fastmail.com> skribis:
> Ludovic Courtès <ludo@gnu.org> writes:
[...]
>>> + ;; Ceph is licensed under LGPL2.1, but includes a number of components
>>> + ;; covered by other licenses. Consult COPYING for full information.
>>> + (license (list license:lgpl2.1
>>> + license:cc-by-sa3.0 ; documentation
>>> + license:bsd-2 ; xxHash
>>> + license:bsd-3
>>> + license:gpl3
>>> + license:gpl2
>>> + license:public-domain
>>> + license:boost1.0
>>> + license:expat))))
>>
>> No “or any later version” clause for the GNU licenses here?
>
> Oops, yes, there are some gpl2+ files at least. But Ceph itself is
> LGPL2.1 only AFAICT; the COPYING file has a debian-style copyright
> format and adds "or later" on some components but not on '*'. No
> copyright headers in the source files I've checked.
>
> Unfortunately COPYING does not cover the bundled components, so I still
> have some digging to do.
OK.
>> other issues, that looks good. Great work!
>
> Thanks! I'm sure there is a lot of work remaining to make *everything*
> work (e.g. 'ceph-deploy'), but will make sure it's possible to provision
> a cluster the "old school" way at least :-)
Cool. :-)
I guess it can be fixed incrementally as you stumble upon problems.
Cheers,
Ludo’.
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 6/6] gnu: fio: Enable rbd support.
2017-01-15 20:33 [PATCH 0/6] gnu: Add ceph Marius Bakke
` (4 preceding siblings ...)
2017-01-15 20:33 ` [PATCH 5/6] gnu: Add ceph Marius Bakke
@ 2017-01-15 20:33 ` Marius Bakke
2017-01-17 22:48 ` Ludovic Courtès
5 siblings, 1 reply; 19+ messages in thread
From: Marius Bakke @ 2017-01-15 20:33 UTC (permalink / raw)
To: guix-devel; +Cc: Marius Bakke
* gnu/packages/benchmark.scm (fio)[inputs]: Add ceph:lib.
---
gnu/packages/benchmark.scm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/gnu/packages/benchmark.scm b/gnu/packages/benchmark.scm
index 465c81b43..6b3c28df3 100644
--- a/gnu/packages/benchmark.scm
+++ b/gnu/packages/benchmark.scm
@@ -22,6 +22,7 @@
#:use-module (guix download)
#:use-module (guix build-system gnu)
#:use-module (gnu packages compression)
+ #:use-module (gnu packages distributed-filesystems)
#:use-module (gnu packages linux))
(define-public fio
@@ -49,7 +50,8 @@
(zero? (system* "./configure"
(string-append "--prefix=" out)))))))))
(inputs
- `(("libaio" ,libaio)
+ `(("ceph" ,ceph "lib")
+ ("libaio" ,libaio)
("zlib" ,zlib)))
(home-page "https://github.com/axboe/fio")
(synopsis "Flexible I/O tester")
--
2.11.0
^ permalink raw reply related [flat|nested] 19+ messages in thread