[bug#60406] [PATCH 0/2] Add DwarFS

[bug#60406] [PATCH 0/2] Add DwarFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

This adds the dwarfs package.

1. For libdwarf, Guix lint complains that the source archive fails to be
extracted, complaining that 3600 is not a valid date.  Another source
archive is fetched from a fallback source instead.

2. For dwarfs, I've disabled the tests because 1 test fails on 'modprobe
fuse'.  If someone knows how to disable a specific test in the
'googletest' framework, please let me know.

Pierre Neidhardt (2):
  gnu: Add libdwarf.
  gnu: Add dwarfs.

 gnu/packages/elf.scm          |  25 ++++++++-
 gnu/packages/file-systems.scm | 100 ++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+), 1 deletion(-)


base-commit: 77d4bff94c6918eb0c0ccd20ee610e6dfc823aec
-- 
2.38.1

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

[bug#60406] [PATCH 1/2] gnu: Add libdwarf.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1830 bytes --]


* gnu/packages/elf.scm (libdwarf): New variable.
---
 gnu/packages/elf.scm | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/elf.scm b/gnu/packages/elf.scm
index 8cc2ff3637..7f2865f17f 100644
--- a/gnu/packages/elf.scm
+++ b/gnu/packages/elf.scm
@@ -31,7 +31,7 @@ (define-module (gnu packages elf)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix build-system gnu)
-  #:use-module ((guix licenses) #:select (gpl3+ lgpl3+ lgpl2.0+))
+  #:use-module ((guix licenses) #:select (gpl3+ lgpl3+ lgpl2.0+ lgpl2.1 gpl2 bsd-2))
   #:use-module (gnu packages)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages compression)
@@ -282,3 +282,26 @@ (define-public patchelf
     (properties
      '((release-monitoring-url . "https://github.com/NixOS/patchelf/releases")))
     (license gpl3+)))
+
+(define-public libdwarf
+  (package
+    (name "libdwarf")
+    (version "0.5.0")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "https://www.prevanders.net/libdwarf-"
+                                  version ".tar.xz"))
+              (sha256
+               (base32
+                "1a8j05z0rxf5ab9p5av2hqz6kxl8hfgar8h1387a0zric0n85yhi"))))
+    (build-system gnu-build-system)
+    (arguments
+     '(#:configure-flags '("--enable-shared")))
+    (native-inputs (list python))
+    (inputs (list elfutils))
+    (home-page "https://www.prevanders.net/dwarf.html")
+    (synopsis "Handle DWARF debugging information")
+    (description "@code{libdwarf} is a library that handles the DWARF
+debugging information format.")
+    ;; See https://www.prevanders.net/dwarflicense.html:
+    (license (list lgpl2.1 gpl2 bsd-2))))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

[bug#60406] [PATCH 2/2] gnu: Add dwarfs.

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 5927 bytes --]


* gnu/packages/file-systems.scm (dwarfs): New variable.
---
 gnu/packages/file-systems.scm | 100 ++++++++++++++++++++++++++++++++++
 1 file changed, 100 insertions(+)

diff --git a/gnu/packages/file-systems.scm b/gnu/packages/file-systems.scm
index 3d45634169..e268c52f06 100644
--- a/gnu/packages/file-systems.scm
+++ b/gnu/packages/file-systems.scm
@@ -46,26 +46,36 @@ (define-module (gnu packages file-systems)
   #:use-module (gnu packages admin)
   #:use-module (gnu packages attr)
   #:use-module (gnu packages autotools)
+  #:use-module (gnu packages backup)
   #:use-module (gnu packages base)
   #:use-module (gnu packages bash)
   #:use-module (gnu packages bison)
+  #:use-module (gnu packages boost)
   #:use-module (gnu packages check)
   #:use-module (gnu packages compression)
+  #:use-module (gnu packages cpp)
   #:use-module (gnu packages crypto)
   #:use-module (gnu packages curl)
   #:use-module (gnu packages cyrus-sasl)
   #:use-module (gnu packages datastructures)
+  #:use-module (gnu packages digest)
   #:use-module (gnu packages documentation)
   #:use-module (gnu packages docbook)
+  #:use-module (gnu packages elf)
   #:use-module (gnu packages flex)
   #:use-module (gnu packages gawk)
   #:use-module (gnu packages glib)
   #:use-module (gnu packages gnupg)
   #:use-module (gnu packages golang)
   #:use-module (gnu packages guile)
+  #:use-module (gnu packages jemalloc)
   #:use-module (gnu packages kerberos)
+  #:use-module (gnu packages libevent)
   #:use-module (gnu packages libffi)
+  #:use-module (gnu packages libunwind)
   #:use-module (gnu packages linux)
+  #:use-module (gnu packages maths)
+  #:use-module (gnu packages man)
   #:use-module (gnu packages nfs)
   #:use-module (gnu packages onc-rpc)
   #:use-module (gnu packages openldap)
@@ -73,6 +83,8 @@ (define-module (gnu packages file-systems)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages photo)
   #:use-module (gnu packages pkg-config)
+  #:use-module (gnu packages popt)
+  #:use-module (gnu packages pretty-print)
   #:use-module (gnu packages python)
   #:use-module (gnu packages python-crypto)
   #:use-module (gnu packages python-web)
@@ -1790,3 +1802,91 @@ (define-public fuse-overlayfs
     (description "This package provides an implementation of overlay+shiftfs
 in FUSE for rootless containers.")
     (license license:gpl3)))
+
+(define-public dwarfs
+  (package
+    (name "dwarfs")
+    (version "0.6.2")
+    (source (origin
+              (method url-fetch)
+              ;; The release archive is needed so that version.h is included.
+              (uri (string-append "https://github.com/mhx/dwarfs/releases/download/v"
+                                  version "/dwarfs-" version ".tar.xz"))
+              (sha256
+               (base32
+                "1kncxf85gsj3anck8ccjmxn2azp5ifqbgkiky2kharmvphkbmfcv"))
+              (snippet
+               #~(begin
+                   (use-modules (guix build utils))
+                   ;; Prefer system libraries instead of submodules.
+                   ;; TODO: Package fbthrift.
+                   ;; TODO: Can we use Guix own folly?  There is no CMake option for it.
+                   ;; TODO: Package parallel-hashmap.
+                   (for-each delete-file-recursively
+                             '(;; "fbthrift"
+                               ;; "folly"
+                               ;; "parallel-hashmap"
+                               "xxHash"
+                               "zstd"))))))
+    (build-system cmake-build-system)
+    (arguments
+     '(#:tests? #f ; TODO: 1 test fails because 'modprobe fuse' needs privileged access.
+       #:configure-flags
+       (list "-DPREFER_SYSTEM_ZSTD=ON"
+             "-DPREFER_SYSTEM_XXHASH=ON"
+             "-DPREFER_SYSTEM_GTEST=ON"
+             "-DWITH_TESTS=ON"
+             ;; Disable man pages since ronn fails to run without hpricot.
+             "-DWITH_MAN_PAGES=OFF")))
+    (native-inputs
+     (list
+      ;; FIXME: Building with ronn fail because hpricot is missing.
+      ;; ronn
+      folly googletest libdwarf libevent pkg-config))
+    (inputs
+     (list
+      boost
+      double-conversion
+      fmt
+      fuse-3
+      gflags
+      jemalloc
+      libarchive
+      libunwind
+      lz4
+      openssl
+      xxhash
+      xz
+      `(,zstd "lib")))
+    (home-page "https://github.com/mhx/dwarfs")
+    (synopsis "Fast high compression read-only file system")
+    (description "DwarFS is a read-only file system with a focus on achieving
+very high compression ratios in particular for very redundant data.
+
+DwarFS also doesn't compromise on speed and for some cases it is on par with
+or performs better than SquashFS.  For the primary use case, DwarFS
+compression is an order of magnitude better than SquashFS compression, it's 6
+times faster to build the file system, it's typically faster to access files
+on DwarFS and it uses less CPU resources.
+
+Distinct features of DwarFS are:
+
+@itemize
+
+@item Clustering of files by similarity using a similarity hash function.  This
+makes it easier to exploit the redundancy across file boundaries.
+
+@item Segmentation analysis across file system blocks in order to reduce the
+size of the uncompressed file system.  This saves memory when using the
+compressed file system and thus potentially allows for higher cache hit rates
+as more data can be kept in the cache.
+
+@item Highly multi-threaded implementation.  Both the file system creation tool
+as well as the FUSE driver are able to make good use of the many cores of your
+system.
+
+@item Optional experimental Python scripting support to provide custom
+filtering and ordering functionality.
+
+@end itemize\n")
+    (license license:gpl3)))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

[bug#60406] [PATCH 0/2] Add DwarFS

[Simon Tournier]

Hi Pierre,

On jeu., 29 déc. 2022 at 15:50, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> 1. For libdwarf, Guix lint complains that the source archive fails to be
> extracted, complaining that 3600 is not a valid date.  Another source
> archive is fetched from a fallback source instead.

What is the issue the Git repository?

    https://github.com/davea42/libdwarf-code


Because that does not sound good: :-)

--8<---------------cut here---------------start------------->8---
$ guix download https://www.prevanders.net/libdwarf-0.5.0.tar.xz

Starting download of /tmp/guix-file.oWkq5d
From https://www.prevanders.net/libdwarf-0.5.0.tar.xz...
Bad Date header: 3600


Starting download of /tmp/guix-file.oWkq5d
From https://web.archive.org/web/20230112145352/https://www.prevanders.net/libdwarf-0.5.0.tar.xz...
following redirection to `https://web.archive.org/web/20221224112643/https://www.prevanders.net/libdwarf-0.5.0.tar.xz'...
 ….5.0.tar.xz  2.4MiB                                                                                              1.0MiB/s 00:02 [##################] 100.0%
/gnu/store/0xcriql6m20wcdzjv4a5fbcg7h0nmq8h-libdwarf-0.5.0.tar.xz
1a8j05z0rxf5ab9p5av2hqz6kxl8hfgar8h1387a0zric0n85yhi
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

[bug#60406] [PATCH 0/2] Add DwarFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 128 bytes --]

Oh right, I can just use the GitHub repo.
Thank you Simon! 
Problem 1 solved then.

I'll push with a note about the tests.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

[bug#60406] [PATCH 0/2] Add DwarFS

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 117 bytes --]

Pushed with

04ade01beb95da45d63ca1f58126508d0491d99d

and

3a1b18aa4540e6f96ded0a98dd907a8033262582

Thanks!
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]