[bug#30596] [PATCH 0/6] Update Ceph and dependencies

[bug#30596] [PATCH 0/6] Update Ceph and dependencies

[Marius Bakke]

This series updates Ceph to the latest LTS release.  It requires Boost
1.66(!), which in turn requires a CMake 3.11 release candidate for
"FindBoost.cmake" to work properly(!!).

I ran out of disk space when running the full RocksDB test suite, so I
adapted the check phase to only run a subset of the tests (approx 25%).

It also updates Snappy, since RocksDB now works with the latest version.

Marius Bakke (6):
  gnu: rocksdb: Update to 5.10.3.
  gnu: snappy: Update to 1.1.7.
  gnu: java-snappy: Update to 1.1.7.
  gnu: Add boost@1.66.0.
  gnu: Add cmake@3.11.
  gnu: ceph: Update to 12.2.3.

 gnu/local.mk                                       |  3 +-
 gnu/packages/boost.scm                             | 14 +++++
 gnu/packages/cmake.scm                             | 63 +++++++++++++++++++++
 gnu/packages/compression.scm                       | 43 +++++++++------
 gnu/packages/databases.scm                         | 29 +++++++---
 .../patches/ceph-disable-cpu-optimizations.patch   | 64 +++++++++-------------
 .../patches/ceph-disable-unittest-throttle.patch   | 52 ------------------
 gnu/packages/patches/ceph-rocksdb-compat.patch     | 63 +++++++++++++++++++++
 gnu/packages/patches/cmake-3.11-pthread-fix.patch  | 30 ++++++++++
 gnu/packages/storage.scm                           | 33 ++++++-----
 10 files changed, 264 insertions(+), 130 deletions(-)
 delete mode 100644 gnu/packages/patches/ceph-disable-unittest-throttle.patch
 create mode 100644 gnu/packages/patches/ceph-rocksdb-compat.patch
 create mode 100644 gnu/packages/patches/cmake-3.11-pthread-fix.patch

-- 
2.16.2

[bug#30596] [PATCH 1/6] gnu: rocksdb: Update to 5.10.3.

[Marius Bakke]

* gnu/packages/databases.scm (rocksdb): Update to 5.10.3.
[arguments]: Set #:test-target.  Add "USE_RTTI=1" to #:make-flags.  Use INVOKE
instead of SYSTEM*.  Replace 'check' phase and enable fixed test.
[license]: Update for changes in 5.5.4.
---
 gnu/packages/databases.scm | 29 ++++++++++++++++++++---------
 1 file changed, 20 insertions(+), 9 deletions(-)

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 29b6ca169..9f183b93e 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -17,7 +17,7 @@
 ;;; Copyright © 2016 Jan Nieuwenhuizen <janneke@gnu.org>
 ;;; Copyright © 2016 Andy Patterson <ajpatter@uwaterloo.ca>
 ;;; Copyright © 2016 Danny Milosavljevic <dannym+a@scratchpost.org>
-;;; Copyright © 2016, 2017 Marius Bakke <mbakke@fastmail.com>
+;;; Copyright © 2016, 2017, 2018 Marius Bakke <mbakke@fastmail.com>
 ;;; Copyright © 2017 Julien Lepiller <julien@lepiller.eu>
 ;;; Copyright © 2017 Thomas Danckaert <post@thomasdanckaert.be>
 ;;; Copyright © 2017 Jelle Licht <jlicht@fsfe.org>
@@ -840,7 +840,7 @@ types are supported, as is encryption.")
 (define-public rocksdb
   (package
     (name "rocksdb")
-    (version "5.2.1")
+    (version "5.10.3")
     (source (origin
               (method url-fetch)
               (uri (string-append "https://github.com/facebook/rocksdb"
@@ -848,7 +848,7 @@ types are supported, as is encryption.")
               (file-name (string-append name "-" version ".tar.gz"))
               (sha256
                (base32
-                "1v2q05bl56sfp51m09z7g6489hkfq4vf6b4qgfg3d96ylgmay9yb"))
+                "0kx3mn2ifl9wfqg49msgzpvmr0x8mbhfglf78639xwrdyi2f90ya"))
               (modules '((guix build utils)))
               (snippet
                '(begin
@@ -858,6 +858,8 @@ types are supported, as is encryption.")
     (build-system gnu-build-system)
     (arguments
      `(#:make-flags (list "CC=gcc"
+                          ;; Ceph requires that RTTI is enabled.
+                          "USE_RTTI=1"
                           (string-append "INSTALL_PATH="
                                          (assoc-ref %outputs "out")))
        ;; Many tests fail on 32-bit platforms. There are multiple reports about
@@ -888,20 +890,28 @@ types are supported, as is encryption.")
          (add-before 'check 'disable-failing-tests
            (lambda _
              (substitute* "Makefile"
-               ;; This test fails with GCC-5 and is unmaintained.
-               ;; https://github.com/facebook/rocksdb/issues/2148
-               (("^[[:blank:]]+spatial_db_test[[:blank:]]+\\\\") "\\")
                ;; These tests reliably fail due to "Too many open files".
                (("^[[:blank:]]+env_test[[:blank:]]+\\\\") "\\")
                (("^[[:blank:]]+persistent_cache_test[[:blank:]]+\\\\") "\\"))
              #t))
+         (replace 'check
+           (lambda* (#:key tests? #:allow-other-keys)
+             ;; Running the full test suite takes hours and require
+             ;; a lot of disk space.  Instead we only run a subset
+             ;; like the CI runners (see .travis.yml).
+             (if tests?
+                 (invoke "make" "V=1" "ROCKSDBTESTS_END=db_block_cache_test"
+                         "CC=gcc" "-j" (number->string (parallel-job-count))
+                         "all_but_some_tests" "check_some")
+                 (display "no test, no problem!\n"))
+             #t))
          (add-after 'check 'build-release-libraries
            ;; The default build target is a debug build for tests. The
            ;; install target depends on "shared_lib" and "static_lib"
            ;; targets for release builds so we build them here for clarity.
            ;; TODO: Add debug output.
            (lambda* (#:key (make-flags '()) #:allow-other-keys)
-             (zero? (apply system* "make" "shared_lib" make-flags)))))))
+             (apply invoke "make" "V=1" "shared_lib" make-flags))))))
     (native-inputs
      `(("parallel" ,parallel)
        ("perl" ,perl)
@@ -925,8 +935,9 @@ between @dfn{Write-Amplification-Factor} (WAF), @dfn{Read-Amplification-Factor}
 (RAF) and @dfn{Space-Amplification-Factor} (SAF).  It has multi-threaded
 compactions, making it specially suitable for storing multiple terabytes of
 data in a single database.  RocksDB is partially based on @code{LevelDB}.")
-    ;; RocksDB is BSD-3 and the JNI adapter is Apache 2.0.
-    (license (list license:bsd-3 license:asl2.0))))
+    ;; RocksDB is dual licensed under GPL2 and ASL 2.0.  Some header
+    ;; files carry the 3-clause BSD license.
+    (license (list license:gpl2 license:asl2.0 license:bsd-3))))
 
 (define-public sparql-query
   (package
-- 
2.16.2

[bug#30596] [PATCH 2/6] gnu: snappy: Update to 1.1.7.

[Marius Bakke]

* gnu/packages/compression.scm (snappy): Update to 1.1.7.
[source]: Change to git archive.  Set file-name.
[build-system]: Change to CMAKE-BUILD-SYSTEM.
[arguments]: Add #:configure-flags.
[description]: Fix infractions.
---
 gnu/packages/compression.scm | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index f25c4bba3..b7ae2b9ed 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -1070,21 +1070,23 @@ algorithm within the Numpy framework.")
 (define-public snappy
   (package
     (name "snappy")
-    (version "1.1.3")
+    (version "1.1.7")
     (source (origin
               (method url-fetch)
-              (uri (string-append
-                    "https://github.com/google/snappy/releases/download/"
-                    version "/" name "-" version ".tar.gz"))
+              (uri (string-append "https://github.com/google/snappy/archive/"
+                                  version ".tar.gz"))
+              (file-name (string-append "snappy-" version ".tar.gz"))
               (sha256
                (base32
-                "1wzf8yif5ym2gj52db6v5m1pxnmn258i38x7llk9x346y2nq47ig"))))
-    (build-system gnu-build-system)
+                "1m7rcdqzkys5lspj8jcsaah8w33zh28s771bw0ga2lgzfgl05yix"))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:configure-flags '("-DBUILD_SHARED_LIBS=ON")))
     (home-page "https://github.com/google/snappy")
     (synopsis "Fast compressor/decompressor")
-    (description "Snappy is a compression/decompression library. It does not
+    (description "Snappy is a compression/decompression library.  It does not
 aim for maximum compression, or compatibility with any other compression library;
-instead, it aims for very high speeds and reasonable compression. For instance,
+instead, it aims for very high speeds and reasonable compression.  For instance,
 compared to the fastest mode of zlib, Snappy is an order of magnitude faster
 for most inputs, but the resulting compressed files are anywhere from 20% to
 100% bigger.")
-- 
2.16.2

[bug#30596] [PATCH 3/6] gnu: java-snappy: Update to 1.1.7.

[Marius Bakke]

* gnu/packages/compression.scm (java-snappy): Update to 1.1.7.
[arguments]: Use INVOKE instead of SYSTEM*.  Avoid CMAKE dependency.  Delete
Hadoop test.  Enable fixed test.
[native-inputs]: Add JAVA-COMMONS-LANG and JAVA-COMMONS-IO.
---
 gnu/packages/compression.scm | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/compression.scm b/gnu/packages/compression.scm
index b7ae2b9ed..4e067c2a5 100644
--- a/gnu/packages/compression.scm
+++ b/gnu/packages/compression.scm
@@ -1132,7 +1132,7 @@ install: libbitshuffle.so
 (define-public java-snappy
   (package
     (name "java-snappy")
-    (version "1.1.4")
+    (version "1.1.7")
     (source (origin
               (method url-fetch)
               (uri (string-append "https://github.com/xerial/snappy-java/archive/"
@@ -1140,7 +1140,7 @@ install: libbitshuffle.so
               (file-name (string-append name "-" version ".tar.gz"))
               (sha256
                (base32
-                "1w58diryma7qz7aa24yv8shf3flxcbbw8jgcn2lih14wgmww58ww"))))
+                "0q4kxz2n97czf6g5gzq0d8yz22cgiaj7wp51rzsswh3bi99bpgg5"))))
     (build-system ant-build-system)
     (arguments
      `(#:jar-name "snappy.jar"
@@ -1180,7 +1180,9 @@ install: libbitshuffle.so
                (("NAME\\): \\$\\(SNAPPY_OBJ\\)")
                 "NAME): $(SNAPPY_OBJ)\n\t@mkdir -p $(@D)"))
              ;; Finally we can run the Makefile to build the dynamic library.
-             (zero? (system* "make" "native"))))
+             ;; Use the -nocmake target to avoid a dependency on cmake,
+             ;; which in turn requires the "git_unpacked" directory.
+             (invoke "make" "native-nocmake")))
          ;; Once we have built the shared library, we need to place it in the
          ;; "build" directory so it can be added to the jar file.
          (add-after 'build-jni 'copy-jni
@@ -1189,13 +1191,14 @@ install: libbitshuffle.so
                                "build/classes/org/xerial/snappy/native")))
          (add-before 'check 'fix-failing
            (lambda _
-             ;; This package assumes maven build, which puts results in "target".
-             ;; We put them in "build" instead, so fix that.
-             (substitute* "src/test/java/org/xerial/snappy/SnappyLoaderTest.java"
-               (("target/classes") "build/classes"))
-             ;; FIXME: probably an error
-             (substitute* "src/test/java/org/xerial/snappy/SnappyOutputStreamTest.java"
-               (("91080") "91013")))))))
+             (with-directory-excursion "src/test/java/org/xerial/snappy"
+               ;; This package assumes maven build, which puts results in "target".
+               ;; We put them in "build" instead, so fix that.
+               (substitute* "SnappyLoaderTest.java"
+                 (("target/classes") "build/classes"))
+               ;; This requires Hadoop, which is not in Guix yet.
+               (delete-file "SnappyHadoopCompatibleOutputStreamTest.java"))
+             #t)))))
     (inputs
      `(("osgi-framework" ,java-osgi-framework)))
     (propagated-inputs
@@ -1206,6 +1209,8 @@ install: libbitshuffle.so
        ("hamcrest" ,java-hamcrest-core)
        ("xerial-core" ,java-xerial-core)
        ("classworlds" ,java-plexus-classworlds)
+       ("commons-lang" ,java-commons-lang)
+       ("commons-io" ,java-commons-io)
        ("perl" ,perl)))
     (home-page "https://github.com/xerial/snappy-java")
     (synopsis "Compression/decompression algorithm in Java")
-- 
2.16.2

[bug#30596] [PATCH 4/6] gnu: Add boost@1.66.0.

[Marius Bakke]

* gnu/packages/boost.scm (boost-1.66): New public variable.
---
 gnu/packages/boost.scm | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/gnu/packages/boost.scm b/gnu/packages/boost.scm
index 91dd2f915..f4ba82810 100644
--- a/gnu/packages/boost.scm
+++ b/gnu/packages/boost.scm
@@ -110,6 +110,20 @@ across a broad spectrum of applications.")
     (license (license:x11-style "http://www.boost.org/LICENSE_1_0.txt"
                                 "Some components have other similar licences."))))
 
+(define-public boost-1.66
+  (package
+    (inherit boost)
+    (version "1.66.0")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "mirror://sourceforge/boost/boost/" version "/boost_"
+                    (string-map (lambda (x) (if (eq? x #\.) #\_ x)) version)
+                    ".tar.bz2"))
+              (sha256
+               (base32
+                "1aaw48cmimsskzgiclwn0iifp62a5iw9cbqrhfari876af1828ap"))))))
+
 (define-public mdds
   (package
     (name "mdds")
-- 
2.16.2

[bug#30596] [PATCH 5/6] gnu: Add cmake@3.11.

[Marius Bakke]

* gnu/packages/patches/cmake-3.11-pthread-fix.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register it.
* gnu/packages/cmake.scm (cmake-3.11): New public variable.
---
 gnu/local.mk                                      |  1 +
 gnu/packages/cmake.scm                            | 63 +++++++++++++++++++++++
 gnu/packages/patches/cmake-3.11-pthread-fix.patch | 30 +++++++++++
 3 files changed, 94 insertions(+)
 create mode 100644 gnu/packages/patches/cmake-3.11-pthread-fix.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index a10f3150e..75f275928 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -591,6 +591,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/clucene-pkgconfig.patch			\
   %D%/packages/patches/clx-remove-demo.patch			\
   %D%/packages/patches/cmake-fix-tests.patch			\
+  %D%/packages/patches/cmake-3.11-pthread-fix.patch		\
   %D%/packages/patches/coda-use-system-libs.patch		\
   %D%/packages/patches/cool-retro-term-dont-check-uninit-member.patch	\
   %D%/packages/patches/cool-retro-term-fix-array-size.patch	\
diff --git a/gnu/packages/cmake.scm b/gnu/packages/cmake.scm
index a4e2d38e3..801323c72 100644
--- a/gnu/packages/cmake.scm
+++ b/gnu/packages/cmake.scm
@@ -31,6 +31,7 @@
   #:use-module (gnu packages)
   #:use-module (gnu packages backup)
   #:use-module (gnu packages compression)
+  #:use-module (gnu packages crypto)
   #:use-module (gnu packages curl)
   #:use-module (gnu packages file)
   #:use-module (gnu packages libevent)
@@ -149,3 +150,65 @@ and workspaces that can be used in the compiler environment of your choice.")
                    license:bsd-2             ; cmlibarchive
                    license:expat             ; cmjsoncpp is dual MIT/public domain
                    license:public-domain)))) ; cmlibarchive/archive_getdate.c
+
+;; Recent Ceph requires Boost 1.66, which in turn requires CMake 3.11 for
+;; its updated "FindBoost.cmake" facility.
+(define-public cmake-3.11
+  (package
+    (inherit cmake)
+    (version "3.11.0-rc1")
+    (source (origin
+              (inherit (package-source cmake))
+              (uri (string-append "https://www.cmake.org/files/v"
+                                  (version-major+minor version)
+                                  "/cmake-" version ".tar.gz"))
+              (sha256
+               (base32
+                "0yz79h2yvqyhyzhr889mfadqq20am5g1z0aqz22kdia9jnxwl13g"))
+              (patches (append (origin-patches (package-source cmake))
+                               (search-patches "cmake-3.11-pthread-fix.patch")))
+              (snippet
+               '(begin
+                  ;; Drop bundled software.
+                  (with-directory-excursion "Utilities"
+                    (for-each delete-file-recursively
+                              '("cmbzip2"
+                                "cmcurl"
+                                "cmexpat"
+                                "cmliblzma"
+                                "cmzlib"))
+                    #t)))))
+    (build-system gnu-build-system)
+    (arguments
+     `(,@(substitute-keyword-arguments (package-arguments cmake)
+           ((#:phases phases)
+            `(modify-phases ,phases
+               (replace 'patch-bin-sh
+                 (lambda _
+                   ;; Replace "/bin/sh" by the right path in... a lot of
+                   ;; files.
+                   (substitute*
+                       '("Modules/CompilerId/Xcode-3.pbxproj.in"
+                         "Modules/CPack.RuntimeScript.in"
+                         "Source/cmakexbuild.cxx"
+                         "Source/cmGlobalXCodeGenerator.cxx"
+                         "Source/cmLocalUnixMakefileGenerator3.cxx"
+                         "Source/cmExecProgramCommand.cxx"
+                         "Utilities/Release/release_cmake.cmake"
+                         "Utilities/cmlibarchive/libarchive/archive_write_set_format_shar.c"
+                         "Tests/CMakeLists.txt"
+                         "Tests/RunCMake/File_Generate/RunCMakeTest.cmake")
+                     (("/bin/sh") (which "sh")))
+                   #t))
+               ;; Parallel tests are unsupported in newer releases.
+               (delete 'set-test-environment)
+               (add-before 'check 'disable-broken-test
+                 (lambda _
+                   ;; This test is broken with SOURCE_DATE_EPOCH since CMake 3.8.
+                   ;; https://gitlab.kitware.com/cmake/cmake/issues/17762
+                   (substitute* "Tests/CMakeTests/StringTest.cmake.in"
+                     ((".*TIMESTAMP-UnixTime.*") "\n"))
+                   #t)))))))
+    (inputs
+     `(("rhash" ,rhash)
+       ,@(package-inputs cmake)))))
diff --git a/gnu/packages/patches/cmake-3.11-pthread-fix.patch b/gnu/packages/patches/cmake-3.11-pthread-fix.patch
new file mode 100644
index 000000000..4f5ae5add
--- /dev/null
+++ b/gnu/packages/patches/cmake-3.11-pthread-fix.patch
@@ -0,0 +1,30 @@
+Fix pthread link failure when using system libuv.
+
+Copied from upstream source repository:
+https://gitlab.kitware.com/cmake/cmake/commit/94eee5e683b80e0ffd8e0c4fd3a45b03f0f37b18
+
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index e4d2a9a0a2..66239595ef 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -310,6 +310,8 @@ endmacro()
+ # Simply to improve readability of the main script.
+ #-----------------------------------------------------------------------
+ macro (CMAKE_BUILD_UTILITIES)
++  find_package(Threads)
++
+   #---------------------------------------------------------------------
+   # Create the kwsys library for CMake.
+   set(KWSYS_NAMESPACE cmsys)
+diff --git a/Source/CMakeLists.txt b/Source/CMakeLists.txt
+index e7c0732675..a0010a292b 100644
+--- a/Source/CMakeLists.txt
++++ b/Source/CMakeLists.txt
+@@ -765,6 +765,7 @@ target_link_libraries(CMakeLib cmsys
+   ${CMAKE_LIBUV_LIBRARIES}
+   ${CMAKE_LIBRHASH_LIBRARIES}
+   ${CMake_KWIML_LIBRARIES}
++  ${CMAKE_THREAD_LIBS_INIT}
+   )
+ 
+ if(CMAKE_SYSTEM_NAME STREQUAL "Linux" AND CMAKE_SYSTEM_PROCESSOR MATCHES "sparc")
-- 
2.16.2

[bug#30596] [PATCH 6/6] gnu: ceph: Update to 12.2.3.

[Marius Bakke]

* gnu/packages/patches/ceph-disable-cpu-optimizations.patch: Adjust to
upstream changes.
* gnu/packages/patches/ceph-disable-unittest-throttle.patch: Delete file.
* gnu/packages/patches/ceph-rocksdb-compat.patch: New file.
* gnu/local.mk (dist_patch_DATA): Adjust accordingly.
* gnu/packages/storage.scm (ceph): Update to 12.2.3.
[source](patches): Adjust.
[arguments]: Add "-DWITH_BABELTRACE=OFF" in #:configure-flags.  Build with
CMAKE-3.11.  Adjust file substitutions.  Wrap 'ceph' executable.
[inputs]: Add GPERF.  Replace BOOST with BOOST-1.66.  Add PYTHON2-PRETTYTABLE.
---
 gnu/local.mk                                       |  2 +-
 .../patches/ceph-disable-cpu-optimizations.patch   | 64 +++++++++-------------
 .../patches/ceph-disable-unittest-throttle.patch   | 52 ------------------
 gnu/packages/patches/ceph-rocksdb-compat.patch     | 63 +++++++++++++++++++++
 gnu/packages/storage.scm                           | 33 ++++++-----
 5 files changed, 111 insertions(+), 103 deletions(-)
 delete mode 100644 gnu/packages/patches/ceph-disable-unittest-throttle.patch
 create mode 100644 gnu/packages/patches/ceph-rocksdb-compat.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 75f275928..c80b2ac29 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -576,7 +576,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/cdparanoia-fpic.patch			\
   %D%/packages/patches/cdrtools-3.01-mkisofs-isoinfo.patch 	\
   %D%/packages/patches/ceph-disable-cpu-optimizations.patch	\
-  %D%/packages/patches/ceph-disable-unittest-throttle.patch	\
+  %D%/packages/patches/ceph-rocksdb-compat.patch		\
   %D%/packages/patches/ceph-skip-collect-sys-info-test.patch	\
   %D%/packages/patches/ceph-skip-unittest_blockdev.patch	\
   %D%/packages/patches/chmlib-inttypes.patch			\
diff --git a/gnu/packages/patches/ceph-disable-cpu-optimizations.patch b/gnu/packages/patches/ceph-disable-cpu-optimizations.patch
index f33eb629d..421b1d4fe 100644
--- a/gnu/packages/patches/ceph-disable-cpu-optimizations.patch
+++ b/gnu/packages/patches/ceph-disable-cpu-optimizations.patch
@@ -1,5 +1,4 @@
-Disable CPU optimizations not supported by all x86_64 systems. Also
-don't add anything for i686.
+Disable CPU optimizations not supported by all x86_64 systems.
 
 --- a/cmake/modules/SIMDExt.cmake	2017-03-23 22:22:58.254071694 +0100
 +++ b/cmake/modules/SIMDExt.cmake	2017-03-23 22:23:22.446848845 +0100
@@ -15,39 +14,30 @@ don't add anything for i686.
  #
  # SIMD_COMPILE_FLAGS
  #
-@@ -73,7 +68,7 @@
-     set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -mfpu=neon")
-   endif()
- 
--elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "i386|i686|amd64|x86_64|AMD64")
-+elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64|AMD64")
-   set(HAVE_INTEL 1)
-   CHECK_C_COMPILER_FLAG(-msse HAVE_INTEL_SSE)
-   if(HAVE_INTEL_SSE)
-@@ -83,26 +78,6 @@
-   if(HAVE_INTEL_SSE2)
-     set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -msse2")
-   endif()
--  CHECK_C_COMPILER_FLAG(-msse3 HAVE_INTEL_SSE3)
--  if(HAVE_INTEL_SSE3)
--    set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -msse3")
--  endif()
--  CHECK_C_COMPILER_FLAG(-mssse3 HAVE_INTEL_SSSE3)
--  if(HAVE_INTEL_SSSE3)
--    set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -mssse3")
--  endif()
--  CHECK_C_COMPILER_FLAG(-mpclmul HAVE_INTEL_PCLMUL)
--  if(HAVE_INTEL_PCLMUL)
--    set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -mpclmul")
--  endif()
--  CHECK_C_COMPILER_FLAG(-msse4.1 HAVE_INTEL_SSE4_1)
--  if(HAVE_INTEL_SSE4_1)
--    set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -msse4.1")
--  endif()
--  CHECK_C_COMPILER_FLAG(-msse4.2 HAVE_INTEL_SSE4_2)
--  if(HAVE_INTEL_SSE4_2)
--    set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -msse4.2")
--  endif()
+@@ -85,26 +80,6 @@
+       if(HAVE_INTEL_SSE2)
+         set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -msse2")
+       endif()
+-      CHECK_C_COMPILER_FLAG(-msse3 HAVE_INTEL_SSE3)
+-      if(HAVE_INTEL_SSE3)
+-        set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -msse3")
+-      endif()
+-      CHECK_C_COMPILER_FLAG(-mssse3 HAVE_INTEL_SSSE3)
+-      if(HAVE_INTEL_SSSE3)
+-        set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -mssse3")
+-      endif()
+-      CHECK_C_COMPILER_FLAG(-mpclmul HAVE_INTEL_PCLMUL)
+-      if(HAVE_INTEL_PCLMUL)
+-        set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -mpclmul")
+-      endif()
+-      CHECK_C_COMPILER_FLAG(-msse4.1 HAVE_INTEL_SSE4_1)
+-      if(HAVE_INTEL_SSE4_1)
+-        set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -msse4.1")
+-      endif()
+-      CHECK_C_COMPILER_FLAG(-msse4.2 HAVE_INTEL_SSE4_2)
+-      if(HAVE_INTEL_SSE4_2)
+-        set(SIMD_COMPILE_FLAGS "${SIMD_COMPILE_FLAGS} -msse4.2")
+-      endif()
+     endif(CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64|AMD64")
+   endif(CMAKE_SYSTEM_PROCESSOR MATCHES "i686|amd64|x86_64|AMD64")
  elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "(powerpc|ppc)64le")
-   set(HAVE_PPC64LE 1)
-   message(STATUS " we are ppc64le")
diff --git a/gnu/packages/patches/ceph-disable-unittest-throttle.patch b/gnu/packages/patches/ceph-disable-unittest-throttle.patch
deleted file mode 100644
index 08ae5f9e2..000000000
--- a/gnu/packages/patches/ceph-disable-unittest-throttle.patch
+++ /dev/null
@@ -1,52 +0,0 @@
-FIXME: This test broke after the gcc-5/glibc-2.25 core-updates merge.
-Not sure what's going on here, it hangs after spawning the first thread.
-
-diff --git a/src/test/common/Throttle.cc b/src/test/common/Throttle.cc
-index 5b6d73217d..40a477b2a3 100644
---- a/src/test/common/Throttle.cc
-+++ b/src/test/common/Throttle.cc
-@@ -216,44 +216,6 @@ TEST_F(ThrottleTest, wait) {
-   } while(!waited);
- }
- 
--TEST_F(ThrottleTest, destructor) {
--  Thread_get *t;
--  {
--    int64_t throttle_max = 10;
--    Throttle *throttle = new Throttle(g_ceph_context, "throttle", throttle_max);
--
--    ASSERT_FALSE(throttle->get(5));
--
--    t = new Thread_get(*throttle, 7);
--    t->create("t_throttle");
--    bool blocked;
--    useconds_t delay = 1;
--    do {
--      usleep(delay);
--      if (throttle->get_or_fail(1)) {
--	throttle->put(1);
--	blocked = false;
--      } else {
--	blocked = true;
--      }
--      delay *= 2;
--    } while(!blocked);
--    delete throttle;
--  }
--
--  { //
--    // The thread is left hanging, otherwise it will abort().
--    // Deleting the Throttle on which it is waiting creates a
--    // inconsistency that will be detected: the Throttle object that
--    // it references no longer exists.
--    //
--    pthread_t id = t->get_thread_id();
--    ASSERT_EQ(pthread_kill(id, 0), 0);
--    delete t;
--    ASSERT_EQ(pthread_kill(id, 0), 0);
--  }
--}
--
- std::pair<double, std::chrono::duration<double> > test_backoff(
-   double low_threshhold,
-   double high_threshhold,
diff --git a/gnu/packages/patches/ceph-rocksdb-compat.patch b/gnu/packages/patches/ceph-rocksdb-compat.patch
new file mode 100644
index 000000000..1a3c6b0b2
--- /dev/null
+++ b/gnu/packages/patches/ceph-rocksdb-compat.patch
@@ -0,0 +1,63 @@
+Fix compatibility with newer versions of RocksDB.
+
+Adapted from this upstream patch, with some additional changes for 12.2:
+https://github.com/ceph/ceph/commit/9d73a7121fdb1ae87cb1aa6f7d9d7a13f329ae68
+
+diff --git a/src/kv/RocksDBStore.cc b/src/kv/RocksDBStore.cc
+index 8660afe1886d..bc0de79e23cb 100644
+--- a/src/kv/RocksDBStore.cc
++++ b/src/kv/RocksDBStore.cc
+@@ -505,7 +505,7 @@
+   // considering performance overhead, default is disabled
+   if (g_conf->rocksdb_perf) {
+     rocksdb::SetPerfLevel(rocksdb::PerfLevel::kEnableTimeExceptForMutex);
+-    rocksdb::perf_context.Reset();
++    rocksdb::get_perf_context()->Reset();
+   }
+ 
+   RocksDBTransactionImpl * _t =
+@@ -532,13 +532,13 @@
+     utime_t write_wal_time;
+     utime_t write_pre_and_post_process_time;
+     write_wal_time.set_from_double(
+-	static_cast<double>(rocksdb::perf_context.write_wal_time)/1000000000);
++	static_cast<double>(rocksdb::get_perf_context()->write_wal_time)/1000000000);
+     write_memtable_time.set_from_double(
+-	static_cast<double>(rocksdb::perf_context.write_memtable_time)/1000000000);
++	static_cast<double>(rocksdb::get_perf_context()->write_memtable_time)/1000000000);
+     write_delay_time.set_from_double(
+-	static_cast<double>(rocksdb::perf_context.write_delay_time)/1000000000);
++	static_cast<double>(rocksdb::get_perf_context()->write_delay_time)/1000000000);
+     write_pre_and_post_process_time.set_from_double(
+-	static_cast<double>(rocksdb::perf_context.write_pre_and_post_process_time)/1000000000);
++	static_cast<double>(rocksdb::get_perf_context()->write_pre_and_post_process_time)/1000000000);
+     logger->tinc(l_rocksdb_write_memtable_time, write_memtable_time);
+     logger->tinc(l_rocksdb_write_delay_time, write_delay_time);
+     logger->tinc(l_rocksdb_write_wal_time, write_wal_time);
+@@ -558,7 +558,7 @@
+   // considering performance overhead, default is disabled
+   if (g_conf->rocksdb_perf) {
+     rocksdb::SetPerfLevel(rocksdb::PerfLevel::kEnableTimeExceptForMutex);
+-    rocksdb::perf_context.Reset();
++    rocksdb::get_perf_context()->Reset();
+   }
+ 
+   RocksDBTransactionImpl * _t =
+@@ -586,13 +586,13 @@
+     utime_t write_wal_time;
+     utime_t write_pre_and_post_process_time;
+     write_wal_time.set_from_double(
+-	static_cast<double>(rocksdb::perf_context.write_wal_time)/1000000000);
++	static_cast<double>(rocksdb::get_perf_context()->write_wal_time)/1000000000);
+     write_memtable_time.set_from_double(
+-	static_cast<double>(rocksdb::perf_context.write_memtable_time)/1000000000);
++	static_cast<double>(rocksdb::get_perf_context()->write_memtable_time)/1000000000);
+     write_delay_time.set_from_double(
+-	static_cast<double>(rocksdb::perf_context.write_delay_time)/1000000000);
++	static_cast<double>(rocksdb::get_perf_context()->write_delay_time)/1000000000);
+     write_pre_and_post_process_time.set_from_double(
+-	static_cast<double>(rocksdb::perf_context.write_pre_and_post_process_time)/1000000000);
++	static_cast<double>(rocksdb::get_perf_context()->write_pre_and_post_process_time)/1000000000);
+     logger->tinc(l_rocksdb_write_memtable_time, write_memtable_time);
+     logger->tinc(l_rocksdb_write_delay_time, write_delay_time);
+     logger->tinc(l_rocksdb_write_wal_time, write_wal_time);
diff --git a/gnu/packages/storage.scm b/gnu/packages/storage.scm
index bee349f6f..e6d566853 100644
--- a/gnu/packages/storage.scm
+++ b/gnu/packages/storage.scm
@@ -29,6 +29,7 @@
   #:use-module (gnu packages bdw-gc)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages check)
+  #:use-module (gnu packages cmake)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages crypto)
   #:use-module (gnu packages cryptsetup)
@@ -36,6 +37,7 @@
   #:use-module (gnu packages databases)
   #:use-module (gnu packages disk)
   #:use-module (gnu packages gnuzilla)
+  #:use-module (gnu packages gperf)
   #:use-module (gnu packages jemalloc)
   #:use-module (gnu packages linux)
   #:use-module (gnu packages lua)
@@ -50,18 +52,18 @@
 (define-public ceph
   (package
     (name "ceph")
-    (version "12.0.2")
+    (version "12.2.3")
     (source (origin
               (method url-fetch)
               (uri (string-append "https://download.ceph.com/tarballs/ceph-"
                                   version ".tar.gz"))
               (sha256
                (base32
-                "0l9v072ba28i984y5rwny9i11nmpyx5bl7awxg48plyadm2l3g14"))
+                "0ysfiicbmdgqd2ni7r7yviql0dvkwbadh4n7nl297wpk5bgjrk5y"))
               (patches
                (search-patches "ceph-skip-unittest_blockdev.patch"
                                "ceph-skip-collect-sys-info-test.patch"
-                               "ceph-disable-unittest-throttle.patch"
+                               "ceph-rocksdb-compat.patch"
                                "ceph-disable-cpu-optimizations.patch"))
               (modules '((guix build utils)))
               (snippet
@@ -74,7 +76,6 @@
                               ;"src/xxHash"
                               ;"src/zstd"
                               ;"src/civetweb"
-                              ;"src/Beast"
                               "src/test/downloads"
                               "src/dpdk"
                               "src/spdk"
@@ -105,6 +106,7 @@
                "-DWITH_SYSTEM_BOOST=ON"
                "-DWITH_PYTHON3=ON"
                ;; TODO: Enable these when available in Guix.
+               "-DWITH_BABELTRACE=OFF"
                "-DWITH_LTTNG=OFF"
                "-DWITH_XFS=OFF"
                "-DWITH_XIO=OFF"
@@ -114,6 +116,7 @@
        ;; <https://bugs.gnu.org/29674> for details. Disable tests until
        ;; resolved.
        #:tests? #f
+       #:cmake ,cmake-3.11
        #:phases
        (modify-phases %standard-phases
          (add-after 'unpack 'patch-source
@@ -152,21 +155,17 @@
 
                (substitute* "src/test/test_subprocess.cc"
                  (("/bin/sh") (which "sh")))
-               (substitute* "src/test/ceph_objectstore_tool.py"
+               (substitute* "qa/standalone/special/ceph_objectstore_tool.py"
                  (("/bin/rm") (which "rm")))
                (substitute* "src/ceph-disk/ceph_disk/main.py"
                  (("/bin/mount") "mount")
                  (("/bin/umount") "umount")
                  (("/sbin/blkid") (which "blkid"))
+                 (("'cryptsetup'") (string-append "'" (which "cryptsetup") "'"))
                  (("'sgdisk'") (string-append "'" (which "sgdisk") "'"))
                  (("'parted'") (string-append "'" (which "parted") "'"))
                  (("'udevadm'") (string-append "'" (which "udevadm") "'")))
 
-               (substitute* "src/ceph-disk-udev"
-                 (("/sbin/cryptsetup") (which "cryptsetup"))
-                 (("/usr/sbin/sgdisk") (which "sgdisk"))
-                 (("/usr/sbin/ceph-disk")
-                  (string-append out "/bin/ceph-disk")))
                (substitute* "udev/50-rbd.rules"
                  (("/usr/bin/ceph-rbdnamer")
                   (string-append out "/bin/ceph-rbdnamer")))
@@ -265,16 +264,22 @@
          (add-after 'install 'wrap-python-scripts
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((out (assoc-ref outputs "out"))
+                    (ceph (string-append out "/bin/ceph"))
                     (ceph-disk (string-append out "/bin/ceph-disk"))
                     (ceph-detect-init (string-append
                                        out "/bin/ceph-detect-init"))
+                    (sitedir (package)
+                             (string-append package
+                                            "/lib/python2.7/site-packages"))
+                    (prettytable (assoc-ref inputs "python2-prettytable"))
                     (PYTHONPATH (string-append
-                                 out "/lib/python2.7/site-packages")))
+                                 (sitedir out) ":"
+                                 (sitedir prettytable))))
 
                (for-each (lambda (executable)
                            (wrap-program executable
                              `("PYTHONPATH" ":" prefix (,PYTHONPATH))))
-                         (list ceph-disk ceph-detect-init))
+                         (list ceph ceph-disk ceph-detect-init))
                #t))))))
     (outputs
      '("out" "lib"))
@@ -310,12 +315,13 @@
        ("python2-testtools" ,python2-testtools)
        ("python2-tox" ,python2-tox)))
     (inputs
-     `(("boost" ,boost)
+     `(("boost" ,boost-1.66)
        ("curl" ,curl)
        ("cryptsetup" ,cryptsetup)
        ("expat" ,expat)
        ("fcgi" ,fcgi)
        ("fuse" ,fuse)
+       ("gperf" ,gperf)
        ("gptfdisk" ,gptfdisk)
        ("jemalloc" ,jemalloc)
        ("keyutils" ,keyutils)
@@ -329,6 +335,7 @@
        ("nss" ,nss)
        ("parted" ,parted)
        ("python@2" ,python-2)
+       ("python2-prettytable" ,python2-prettytable)      ;used by ceph_daemon.py
        ("python@3" ,python-3)
        ("rocksdb" ,rocksdb)
        ("snappy" ,snappy)
-- 
2.16.2

[bug#30596] [PATCH 1/6] gnu: rocksdb: Update to 5.10.3.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/databases.scm (rocksdb): Update to 5.10.3.
> [arguments]: Set #:test-target.  Add "USE_RTTI=1" to #:make-flags.  Use INVOKE
> instead of SYSTEM*.  Replace 'check' phase and enable fixed test.
> [license]: Update for changes in 5.5.4.

LGTM, thanks!

[bug#30596] [PATCH 2/6] gnu: snappy: Update to 1.1.7.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/compression.scm (snappy): Update to 1.1.7.
> [source]: Change to git archive.  Set file-name.
> [build-system]: Change to CMAKE-BUILD-SYSTEM.
> [arguments]: Add #:configure-flags.
> [description]: Fix infractions.

LGTM!

[bug#30596] [PATCH 3/6] gnu: java-snappy: Update to 1.1.7.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/compression.scm (java-snappy): Update to 1.1.7.
> [arguments]: Use INVOKE instead of SYSTEM*.  Avoid CMAKE dependency.  Delete
> Hadoop test.  Enable fixed test.
> [native-inputs]: Add JAVA-COMMONS-LANG and JAVA-COMMONS-IO.

LGTM!

[bug#30596] [PATCH 4/6] gnu: Add boost@1.66.0.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/boost.scm (boost-1.66): New public variable.

LGTM.

We’ll do the actual upgrade in ‘core-updates’, right?

Ludo’.

[bug#30596] [PATCH 5/6] gnu: Add cmake@3.11.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/patches/cmake-3.11-pthread-fix.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Register it.
> * gnu/packages/cmake.scm (cmake-3.11): New public variable.

[...]

> +;; Recent Ceph requires Boost 1.66, which in turn requires CMake 3.11 for
> +;; its updated "FindBoost.cmake" facility.
> +(define-public cmake-3.11

The actual upgrade is for ‘core-updates’ as well I guess?

> +    (arguments
> +     `(,@(substitute-keyword-arguments (package-arguments cmake)
> +           ((#:phases phases)

Or simply:

  (arguments
    (substitute-keyword-arguments …))

Otherwise LGTM, thanks!

Ludo’.

[bug#30596] [PATCH 6/6] gnu: ceph: Update to 12.2.3.

[Ludovic Courtès]

Marius Bakke <mbakke@fastmail.com> skribis:

> * gnu/packages/patches/ceph-disable-cpu-optimizations.patch: Adjust to
> upstream changes.
> * gnu/packages/patches/ceph-disable-unittest-throttle.patch: Delete file.
> * gnu/packages/patches/ceph-rocksdb-compat.patch: New file.
> * gnu/local.mk (dist_patch_DATA): Adjust accordingly.
> * gnu/packages/storage.scm (ceph): Update to 12.2.3.
> [source](patches): Adjust.
> [arguments]: Add "-DWITH_BABELTRACE=OFF" in #:configure-flags.  Build with
> CMAKE-3.11.  Adjust file substitutions.  Wrap 'ceph' executable.
> [inputs]: Add GPERF.  Replace BOOST with BOOST-1.66.  Add PYTHON2-PRETTYTABLE.

Are you sure gperf shouldn't be in ‘native-inputs’?

Otherwise LGTM.  That was not an easy one!

Thank you,
Ludo’.

[bug#30596] [PATCH 5/6] gnu: Add cmake@3.11.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> * gnu/packages/patches/cmake-3.11-pthread-fix.patch: New file.
>> * gnu/local.mk (dist_patch_DATA): Register it.
>> * gnu/packages/cmake.scm (cmake-3.11): New public variable.
>
> [...]
>
>> +;; Recent Ceph requires Boost 1.66, which in turn requires CMake 3.11 for
>> +;; its updated "FindBoost.cmake" facility.
>> +(define-public cmake-3.11
>
> The actual upgrade is for ‘core-updates’ as well I guess?

I think Arun had a patch for 3.10 on the way.  With luck, 3.11 gets
released in time too.

>
>> +    (arguments
>> +     `(,@(substitute-keyword-arguments (package-arguments cmake)
>> +           ((#:phases phases)
>
> Or simply:
>
>   (arguments
>     (substitute-keyword-arguments …))

Ahh, of course!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#30596: [PATCH 6/6] gnu: ceph: Update to 12.2.3.

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> * gnu/packages/patches/ceph-disable-cpu-optimizations.patch: Adjust to
>> upstream changes.
>> * gnu/packages/patches/ceph-disable-unittest-throttle.patch: Delete file.
>> * gnu/packages/patches/ceph-rocksdb-compat.patch: New file.
>> * gnu/local.mk (dist_patch_DATA): Adjust accordingly.
>> * gnu/packages/storage.scm (ceph): Update to 12.2.3.
>> [source](patches): Adjust.
>> [arguments]: Add "-DWITH_BABELTRACE=OFF" in #:configure-flags.  Build with
>> CMAKE-3.11.  Adjust file substitutions.  Wrap 'ceph' executable.
>> [inputs]: Add GPERF.  Replace BOOST with BOOST-1.66.  Add PYTHON2-PRETTYTABLE.
>
> Are you sure gperf shouldn't be in ‘native-inputs’?

Gperf is indeed a native input.  Thanks for reviewing!  Pushed to
master, with even newer versions of RocksDB, Ceph and CMake.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]