[bug#68721] [PATCH 0/2] gnu: Add python-fastparquet.

[bug#68721] [PATCH 0/2] gnu: Add python-fastparquet.

[Troy Figiel]

This patch series adds python-fastparquet and is a bit unconvential,
because it depends on changes currently in the rust-team branch (through
python-cramjam, see #68665). Before we can merge these changes, the rust-team branch needs to be merged into master.

The base-commit is a commit on the rust-team branch, so be aware that you
might need to rebuild some other packages like python-pandas locally.

I decided to send these patches to guix-patches anyway, so they are at least out in the open.

Troy Figiel (2):
  gnu: Add python-cramjam.
  gnu: Add python-fastparquet.

 gnu/packages/databases.scm          | 67 ++++++++++++++++++++++++
 gnu/packages/python-compression.scm | 80 ++++++++++++++++++++++++++++-
 2 files changed, 146 insertions(+), 1 deletion(-)


base-commit: 3228c8176783db5627fd79d87652c67bfeb86512
-- 
2.42.0

[bug#68721] [PATCH 1/2] gnu: Add python-cramjam.

[Troy Figiel]

* gnu/packages/python-compression.scm (python-cramjam): New variable.
---
 gnu/packages/python-compression.scm | 80 ++++++++++++++++++++++++++++-
 1 file changed, 79 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-compression.scm b/gnu/packages/python-compression.scm
index 6f249dd3af..691fbd3065 100644
--- a/gnu/packages/python-compression.scm
+++ b/gnu/packages/python-compression.scm
@@ -7,7 +7,7 @@
 ;;; Copyright © 2020 Nicolas Goaziou <mail@nicolasgoaziou.fr>
 ;;; Copyright © 2020, 2022, 2023 Marius Bakke <marius@gnu.org>
 ;;; Copyright © 2021 Brendan Tildesley <mail@brendan.scot>
-;;; Copyright © 2023 Troy Figiel <troy@troyfigiel.com>
+;;; Copyright © 2023, 2024 Troy Figiel <troy@troyfigiel.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -30,12 +30,14 @@ (define-module (gnu packages python-compression)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix gexp)
+  #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system python)
   #:use-module (guix build-system pyproject)
   #:use-module (gnu packages)
   #:use-module (gnu packages libffi)
   #:use-module (gnu packages compression)
+  #:use-module (gnu packages crates-io)
   #:use-module (gnu packages check)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages pkg-config)
@@ -44,6 +46,7 @@ (define-module (gnu packages python-compression)
   #:use-module (gnu packages python-check)
   #:use-module (gnu packages python-crypto)
   #:use-module (gnu packages python-xyz)
+  #:use-module (gnu packages rust-apps)
   #:use-module (gnu packages sphinx))
 
 (define-public python-multivolumefile
@@ -73,6 +76,81 @@ (define-public python-multivolumefile
 were a single file.")
     (license license:lgpl2.1+)))
 
+(define-public python-cramjam
+  (package
+    (name "python-cramjam")
+    (version "2.7.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "cramjam" version))
+       (sha256
+        (base32 "1b69qlr0q7q3spa7zy55xc1dr5pjgsdavxx8ijhv2j60xqjbg7sp"))))
+    (build-system cargo-build-system)
+    (arguments
+     (list
+      #:imported-modules `(,@%cargo-build-system-modules
+                           ,@%pyproject-build-system-modules)
+      #:modules '((guix build cargo-build-system)
+                  ((guix build pyproject-build-system)
+                   #:prefix py:)
+                  (guix build utils))
+      #:phases #~(modify-phases %standard-phases
+                   ;; We use Maturin to build the project.
+                   (replace 'build
+                     (assoc-ref py:%standard-phases
+                                'build))
+                   ;; Before being able to run Python tests, we need to
+                   ;; install the module and add it to PYTHONPATH.
+                   (delete 'install)
+                   (add-after 'build 'install
+                     (assoc-ref py:%standard-phases
+                                'install))
+                   (add-after 'install 'add-install-to-pythonpath
+                     (assoc-ref py:%standard-phases
+                                'add-install-to-pythonpath))
+                   ;; Finally run the tests. Only Python tests are provided.
+                   (replace 'check
+                     (lambda* (#:key tests? inputs outputs #:allow-other-keys)
+                       (when tests?
+                         ;; Without the CI variable, tests are run in "local"
+                         ;; mode, which sets a deadline for hypothesis.  For a
+                         ;; deterministic build, we need to set CI.
+                         (setenv "CI" "1")
+                         (invoke "pytest" "-vv" "tests")))))
+      #:cargo-inputs `(("rust-brotli" ,rust-brotli-3)
+                       ("rust-bzip2" ,rust-bzip2-0.4)
+                       ("rust-flate2" ,rust-flate2-1)
+                       ("rust-lz4" ,rust-lz4-1)
+                       ("rust-pyo3" ,rust-pyo3-0.18)
+                       ("rust-snap" ,rust-snap-1)
+                       ("rust-zstd" ,rust-zstd-0.11))
+      #:install-source? #f))
+    (native-inputs (list maturin
+                         python-pytest
+                         python-pytest-xdist
+                         python-numpy
+                         python-hypothesis
+                         python-wrapper))
+    (home-page "https://github.com/milesgranger/cramjam")
+    (synopsis "Python bindings to compression algorithms in Rust")
+    (description
+     "This package provides thin Python bindings to compression and
+decomporession algorithms implemented in Rust.  This allows for using
+algorithms such as Snappy without additional system dependencies.  The
+following algorithms are available:
+
+@itemize
+@item Snappy
+@item Brotli
+@item Bzip2
+@item LZ4
+@item Gzip
+@item Deflate
+@item Zstd
+@end itemize")
+    (license license:expat)))
+
 (define-public python-pybcj
   (package
     (name "python-pybcj")
-- 
2.42.0

[bug#68721] [PATCH 2/2] gnu: Add python-fastparquet.

[Troy Figiel]

* gnu/packages/databases.scm (python-fastparquet): New variable.
---
 gnu/packages/databases.scm | 67 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index b56767d311..66f7f7d951 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -62,6 +62,7 @@
 ;;; Copyright © 2023 Felix Gruber <felgru@posteo.ne
 ;;; Copyright © 2023 Munyoki Kilyungi <me@bonfacemunyoki.com>
 ;;; Copyright © 2023 Giacomo Leidi <goodoldpaul@autistici.org>
+;;; Copyright © 2024 Troy Figiel <troy@troyfigiel.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -142,6 +143,7 @@ (define-module (gnu packages databases)
   #:use-module (gnu packages python)
   #:use-module (gnu packages python-build)
   #:use-module (gnu packages python-check)
+  #:use-module (gnu packages python-compression)
   #:use-module (gnu packages python-crypto)
   #:use-module (gnu packages python-science)
   #:use-module (gnu packages python-web)
@@ -4896,6 +4898,71 @@ (define-public python-pyarrow-0.16
 other traditional Python scientific computing packages.")
     (license license:asl2.0)))
 
+(define-public python-fastparquet
+  (package
+    (name "python-fastparquet")
+    (version "2023.10.1")
+    (source
+     (origin
+       ;; Fastparquet uses setuptools-scm to find the current version. This
+       ;; only works when we use the PyPI tarball, which does not contain
+       ;; tests. Instead, we use the git-fetch method and add the version back
+       ;; ourselves.
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/dask/fastparquet")
+             (commit version)))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32 "0msc2n4sjbq3h5pq6l94rfx27v0aqrk5cxbpg3yssr74gwx26h4r"))
+       (modules '((guix build utils)))
+       (snippet '(substitute* "setup.py"
+                   ;; Remove dependencies on git and setuptools-scm.
+                   (("^.*\"git\", \"status\".*$")
+                    "")
+                   (("'setuptools-scm>1.5.4',")
+                    "")
+                   ;; Guix is only compatible with a single version of numpy
+                   ;; at a time. We can safely remove this dependency.
+                   (("'oldest-supported-numpy'")
+                    "")))))
+    (build-system pyproject-build-system)
+    (arguments
+     (list
+      #:phases #~(modify-phases %standard-phases
+                   ;; Make sure to add back the missing version information
+                   ;; the build phase.
+                   (add-after 'unpack 'set-version
+                     (lambda _
+                       (call-with-output-file "fastparquet/_version.py"
+                         (lambda (port)
+                           (format port "__version__ = ~a"
+                                   (string-append "\""
+                                                  #$version "\""))))))
+                   ;; Cython extensions need to be built for the check phase.
+                   (add-before 'check 'build-cython-extensions
+                     (lambda _
+                       (invoke "python" "setup.py" "build_ext" "--inplace"))))))
+    (propagated-inputs (list python-cramjam python-fsspec python-numpy
+                             python-packaging python-pandas))
+    (native-inputs (list python-cython python-pytest-runner))
+    (home-page "https://github.com/dask/fastparquet")
+    (synopsis "Python implementation of the Parquet file format")
+    (description
+     "@code{fastparquet} is a Python implementation of the Parquet file
+format.  @code{fastparquet} is used implicitly by @code{dask}, @code{pandas}
+and @code{intake-parquet}.  It supports the following compression algorithms:
+
+@itemize
+@item Gzip
+@item Snappy
+@item Brotli
+@item LZ4
+@item Zstd
+@item LZO (optionally)
+@end itemize")
+    (license license:asl2.0)))
+
 (define-public python-crate
   (package
     (name "python-crate")
-- 
2.42.0

[bug#68721] [PATCH 0/2] gnu: Add python-fastparquet.

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 92 bytes --]

Hi,

As rust-team is merged, I prioritising this patch series to be reviewed.

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 227 bytes --]

[bug#68721] [PATCH 1/2] gnu: Add python-cramjam.

[Troy Figiel]

* gnu/packages/python-compression.scm (python-cramjam): New variable.
---
 gnu/packages/python-compression.scm | 80 ++++++++++++++++++++++++++++-
 1 file changed, 79 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python-compression.scm b/gnu/packages/python-compression.scm
index 6f249dd3af..691fbd3065 100644
--- a/gnu/packages/python-compression.scm
+++ b/gnu/packages/python-compression.scm
@@ -7,7 +7,7 @@
 ;;; Copyright © 2020 Nicolas Goaziou <mail@nicolasgoaziou.fr>
 ;;; Copyright © 2020, 2022, 2023 Marius Bakke <marius@gnu.org>
 ;;; Copyright © 2021 Brendan Tildesley <mail@brendan.scot>
-;;; Copyright © 2023 Troy Figiel <troy@troyfigiel.com>
+;;; Copyright © 2023, 2024 Troy Figiel <troy@troyfigiel.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -30,12 +30,14 @@ (define-module (gnu packages python-compression)
   #:use-module (guix packages)
   #:use-module (guix download)
   #:use-module (guix gexp)
+  #:use-module (guix build-system cargo)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system python)
   #:use-module (guix build-system pyproject)
   #:use-module (gnu packages)
   #:use-module (gnu packages libffi)
   #:use-module (gnu packages compression)
+  #:use-module (gnu packages crates-io)
   #:use-module (gnu packages check)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages pkg-config)
@@ -44,6 +46,7 @@ (define-module (gnu packages python-compression)
   #:use-module (gnu packages python-check)
   #:use-module (gnu packages python-crypto)
   #:use-module (gnu packages python-xyz)
+  #:use-module (gnu packages rust-apps)
   #:use-module (gnu packages sphinx))
 
 (define-public python-multivolumefile
@@ -73,6 +76,81 @@ (define-public python-multivolumefile
 were a single file.")
     (license license:lgpl2.1+)))
 
+(define-public python-cramjam
+  (package
+    (name "python-cramjam")
+    (version "2.7.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "cramjam" version))
+       (sha256
+        (base32 "1b69qlr0q7q3spa7zy55xc1dr5pjgsdavxx8ijhv2j60xqjbg7sp"))))
+    (build-system cargo-build-system)
+    (arguments
+     (list
+      #:imported-modules `(,@%cargo-build-system-modules
+                           ,@%pyproject-build-system-modules)
+      #:modules '((guix build cargo-build-system)
+                  ((guix build pyproject-build-system)
+                   #:prefix py:)
+                  (guix build utils))
+      #:phases #~(modify-phases %standard-phases
+                   ;; We use Maturin to build the project.
+                   (replace 'build
+                     (assoc-ref py:%standard-phases
+                                'build))
+                   ;; Before being able to run Python tests, we need to
+                   ;; install the module and add it to PYTHONPATH.
+                   (delete 'install)
+                   (add-after 'build 'install
+                     (assoc-ref py:%standard-phases
+                                'install))
+                   (add-after 'install 'add-install-to-pythonpath
+                     (assoc-ref py:%standard-phases
+                                'add-install-to-pythonpath))
+                   ;; Finally run the tests. Only Python tests are provided.
+                   (replace 'check
+                     (lambda* (#:key tests? inputs outputs #:allow-other-keys)
+                       (when tests?
+                         ;; Without the CI variable, tests are run in "local"
+                         ;; mode, which sets a deadline for hypothesis.  For a
+                         ;; deterministic build, we need to set CI.
+                         (setenv "CI" "1")
+                         (invoke "pytest" "-vv" "tests")))))
+      #:cargo-inputs `(("rust-brotli" ,rust-brotli-3)
+                       ("rust-bzip2" ,rust-bzip2-0.4)
+                       ("rust-flate2" ,rust-flate2-1)
+                       ("rust-lz4" ,rust-lz4-1)
+                       ("rust-pyo3" ,rust-pyo3-0.18)
+                       ("rust-snap" ,rust-snap-1)
+                       ("rust-zstd" ,rust-zstd-0.11))
+      #:install-source? #f))
+    (native-inputs (list maturin
+                         python-pytest
+                         python-pytest-xdist
+                         python-numpy
+                         python-hypothesis
+                         python-wrapper))
+    (home-page "https://github.com/milesgranger/cramjam")
+    (synopsis "Python bindings to compression algorithms in Rust")
+    (description
+     "This package provides thin Python bindings to compression and
+decomporession algorithms implemented in Rust.  This allows for using
+algorithms such as Snappy without additional system dependencies.  The
+following algorithms are available:
+
+@itemize
+@item Snappy
+@item Brotli
+@item Bzip2
+@item LZ4
+@item Gzip
+@item Deflate
+@item Zstd
+@end itemize")
+    (license license:expat)))
+
 (define-public python-pybcj
   (package
     (name "python-pybcj")
-- 
2.42.0

[bug#68721] [PATCH 2/2] gnu: Add python-fastparquet.

[Troy Figiel]

* gnu/packages/databases.scm (python-fastparquet): New variable.
---
 gnu/packages/databases.scm | 68 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index db4fe0b447..5ec0234ef1 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -62,6 +62,7 @@
 ;;; Copyright © 2023 Felix Gruber <felgru@posteo.ne
 ;;; Copyright © 2023 Munyoki Kilyungi <me@bonfacemunyoki.com>
 ;;; Copyright © 2023 Giacomo Leidi <goodoldpaul@autistici.org>
+;;; Copyright © 2024 Troy Figiel <troy@troyfigiel.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -143,6 +144,7 @@ (define-module (gnu packages databases)
   #:use-module (gnu packages python)
   #:use-module (gnu packages python-build)
   #:use-module (gnu packages python-check)
+  #:use-module (gnu packages python-compression)
   #:use-module (gnu packages python-crypto)
   #:use-module (gnu packages python-science)
   #:use-module (gnu packages python-web)
@@ -4984,6 +4986,72 @@ (define-public python-pyarrow-0.16
 other traditional Python scientific computing packages.")
     (license license:asl2.0)))
 
+(define-public python-fastparquet
+  (package
+    (name "python-fastparquet")
+    (version "2024.2.0")
+    (source
+     (origin
+       ;; Fastparquet uses setuptools-scm to find the current version. This
+       ;; only works when we use the PyPI tarball, which does not contain
+       ;; tests. Instead, we use the git-fetch method and add the version back
+       ;; ourselves.
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/dask/fastparquet")
+             (commit version)))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32 "0f32dj1xvd11l0siznqd33dpjlhg9siylcjcfkcdlqfcy45jfj3v"))))
+    (build-system pyproject-build-system)
+    (arguments
+     (list
+      #:phases #~(modify-phases %standard-phases
+                   ;; Make sure to add back the missing version information
+                   ;; the build phase.
+                   (add-after 'unpack 'set-version
+                     (lambda _
+                       (call-with-output-file "fastparquet/_version.py"
+                         (lambda (port)
+                           (format port "__version__ = ~a"
+                                   (string-append "\""
+                                                  #$version "\""))))))
+                   (add-after 'unpack 'relax-dependencies
+                     (lambda _
+                       (substitute* "setup.py"
+                         ;; Remove dependencies on git and setuptools-scm.
+                         (("^.*\"git\", \"status\".*$")
+                          "")
+                         (("'setuptools-scm>1.5.4',")
+                          "")
+                         ;; Guix is only compatible with a single version of numpy
+                         ;; at a time. We can safely remove this dependency.
+                         (("'oldest-supported-numpy'")
+                          ""))))
+                   ;; Cython extensions need to be built for the check phase.
+                   (add-before 'check 'build-cython-extensions
+                     (lambda _
+                       (invoke "python" "setup.py" "build_ext" "--inplace"))))))
+    (propagated-inputs (list python-cramjam python-fsspec python-numpy
+                             python-packaging python-pandas))
+    (native-inputs (list python-cython python-pytest-runner))
+    (home-page "https://github.com/dask/fastparquet")
+    (synopsis "Python implementation of the Parquet file format")
+    (description
+     "@code{fastparquet} is a Python implementation of the Parquet file
+format.  @code{fastparquet} is used implicitly by @code{dask}, @code{pandas}
+and @code{intake-parquet}.  It supports the following compression algorithms:
+
+@itemize
+@item Gzip
+@item Snappy
+@item Brotli
+@item LZ4
+@item Zstd
+@item LZO (optionally)
+@end itemize")
+    (license license:asl2.0)))
+
 (define-public python-crate
   (package
     (name "python-crate")
-- 
2.42.0

[bug#68721] [PATCH v2 0/2] gnu: Add python-fastparquet.

[Troy Figiel]

Hi Oleg,

I recently learned snippets should be used more conservatively. It makes sense
from the perspective of hacking on the source code locally. Therefore, I moved
the python-fastparquet snippet to a phase. I also updated python-fastparquet
to its most recent version.

Best wishes,

Troy

Troy Figiel (2):
  gnu: Add python-cramjam.
  gnu: Add python-fastparquet.

 gnu/packages/databases.scm          | 68 ++++++++++++++++++++++++
 gnu/packages/python-compression.scm | 80 ++++++++++++++++++++++++++++-
 2 files changed, 147 insertions(+), 1 deletion(-)


base-commit: 6f5ea7ac1acb3d1c53baf7620cca66cc87fe5a73
-- 
2.42.0

bug#68721: [PATCH 0/2] gnu: Add python-fastparquet.

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]


Hi,


Applied modifications [5/5]
- [X] Rename phase 'relax-dependencies -> 'relax-requirements for
  consistency with other python packages
- [X] Use 'pretend-version to set module version from git checkout
- [X] Adjust indentation style
- [X] Add optional python-lzo
- [X] Parallize tests

Pushed as bdecacafec..f00f56514d to master.

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]