[bug#36026] [PATCH 0/4] Update Conda

[bug#36026] [PATCH 0/4] Update Conda

[Pierre Neidhardt]

Recent conda needs a separate python-conda-package-handling library.
This library requires a libarchive built with zstd support.
So far so good, except that python-libarchive-c does not seem to support zstd in its source code.

I've reported the issue upstream
(https://github.com/conda/conda-package-handling/issues/14) but
according to them python-libarchive-c is fine.

I don't have enough Python knowledge to understand whether upstream is right
	or if there is indeed something fishy with python-libarchive-c.
Pierre Neidhardt (4):
  gnu: libarchive: Add zstd support.
  gnu: python-libarchive-c: Explicitly rely on latest libarchive.
  gnu: Add python-conda-package-handling.
  gnu: python-conda: Update to 4.7.0.

 gnu/packages/backup.scm             | 10 ++--
 gnu/packages/package-management.scm | 81 ++++++++++++++++-------------
 gnu/packages/python-xyz.scm         |  2 +-
 3 files changed, 54 insertions(+), 39 deletions(-)

-- 
2.21.0

[bug#36026] [PATCH 1/4] gnu: libarchive: Add zstd support.

[Pierre Neidhardt]

* gnu/packages/backup.scm (libarchive)[inputs]: Include zstd.
[arguments]: Build against zstd.
---
 gnu/packages/backup.scm | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/backup.scm b/gnu/packages/backup.scm
index b15c15ad46..da128a0901 100644
--- a/gnu/packages/backup.scm
+++ b/gnu/packages/backup.scm
@@ -216,7 +216,8 @@ backups (called chunks) to allow easy burning to CD/DVD.")
        ("lzo" ,lzo)
        ("bzip2" ,bzip2)
        ("libxml2" ,libxml2)
-       ("xz" ,xz)))
+       ("xz" ,xz)
+       ("zstd" ,zstd)))
     (arguments
      `(#:phases
        (modify-phases %standard-phases
@@ -245,7 +246,8 @@ backups (called chunks) to allow easy burning to CD/DVD.")
                     (libxml2 (assoc-ref inputs "libxml2"))
                     (xz      (assoc-ref inputs "xz"))
                     (zlib    (assoc-ref inputs "zlib"))
-                    (bzip2   (assoc-ref inputs "bzip2")))
+                    (bzip2   (assoc-ref inputs "bzip2"))
+                    (zstd   (assoc-ref inputs "zstd")))
                (substitute* (string-append lib "/pkgconfig/libarchive.pc")
                  (("-lnettle")
                   (string-append "-L" nettle "/lib -lnettle"))
@@ -256,7 +258,9 @@ backups (called chunks) to allow easy burning to CD/DVD.")
                  (("-lz")
                   (string-append "-L" zlib "/lib -lz"))
                  (("-lbz2")
-                  (string-append "-L" bzip2 "/lib -lbz2")))
+                  (string-append "-L" bzip2 "/lib -lbz2"))
+                 (("-lzstd")
+                  (string-append "-L" zstd "/lib -lzstd")))
                #t))))
 
        ;; libarchive/test/test_write_format_gnutar_filenames.c needs to be
-- 
2.21.0

[bug#36026] [PATCH 2/4] gnu: python-libarchive-c: Explicitly rely on latest libarchive.

[Pierre Neidhardt]

* gnu/packages/python-xyz.scm (python-libarchive-c)[inputs]: Do it.
---
 gnu/packages/python-xyz.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 5322b70759..9c9da1d678 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -7191,7 +7191,7 @@ a hash value.")
      `(("python-mock" ,python-mock)
        ("python-pytest" ,python-pytest)))
     (inputs
-     `(("libarchive" ,libarchive)))
+     `(("libarchive" ,libarchive-3.3.3)))
     (home-page "https://github.com/Changaco/python-libarchive-c")
     (synopsis "Python interface to libarchive")
     (description
-- 
2.21.0

[bug#36026] [PATCH 3/4] gnu: Add python-conda-package-handling.

[Pierre Neidhardt]

* gnu/packages/package-management.scm (python-conda-package-handling): New variable.
---
 gnu/packages/package-management.scm | 36 +++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index b4e1e077fe..28e23fd1d3 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -716,6 +716,42 @@ environments.")
 (define-public python2-anaconda-client
   (package-with-python2 python-anaconda-client))
 
+(define-public python-conda-package-handling
+  (package
+    (name "python-conda-package-handling")
+    (version "1.1.1")
+    (source
+     (origin
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/conda/conda-package-handling/")
+             (commit version)))
+       (file-name (git-file-name name version))
+       (sha256
+        (base32
+         "0qjraqb10bnnr329n68s97lkxh8mxchd6dmsynd0n6i9h49hhfgq"))))
+    (build-system python-build-system)
+    (inputs
+     `(("python-six" ,python-six)
+       ("python-libarchive-c" ,python-libarchive-c)
+       ("python-tqdm" ,python-tqdm)))
+    (native-inputs
+     `(("python-pytest" ,python-pytest)
+       ("python-pytest-cov" ,python-pytest-cov)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (replace 'check
+           (lambda _
+             ;; TODO: Tests fail.
+             (invoke "pytest" "-vv" "tests"))))))
+    (home-page "https://conda.io")
+    (synopsis "Create and extract conda packages of various formats")
+    (description
+     "This library is an abstraction of conda package handling and a tool for
+extracting, creating, and converting between formats.")
+    (license license:bsd-3)))
+
 (define-public python-conda
   (package
     (name "python-conda")
-- 
2.21.0

[bug#36026] [PATCH 4/4] gnu: python-conda: Update to 4.7.0.

[Pierre Neidhardt]

* gnu/packages/package-management.scm (python-conda): Update to 4.7.0.
---
 gnu/packages/package-management.scm | 45 +++++++----------------------
 1 file changed, 10 insertions(+), 35 deletions(-)

diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 28e23fd1d3..fb84f9abfc 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -755,16 +755,17 @@ extracting, creating, and converting between formats.")
 (define-public python-conda
   (package
     (name "python-conda")
-    (version "4.3.16")
+    (version "4.7.0")
     (source
      (origin
-       (method url-fetch)
-       (uri (string-append "https://github.com/conda/conda/archive/"
-                           version ".tar.gz"))
-       (file-name (string-append name "-" version ".tar.gz"))
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/conda/conda/")
+             (commit version)))
+       (file-name (git-file-name name version))
        (sha256
         (base32
-         "1jq8hyrc5npb5sf4vw6s6by4602yj8f79vzpbwdfgpkn02nfk1dv"))))
+         "0dzplykxrlri9a5c65bc48sdjpg2i50qzjf1qf9448qkghvmnmpr"))))
     (build-system python-build-system)
     (arguments
      `(#:phases
@@ -774,33 +775,6 @@ extracting, creating, and converting between formats.")
              (with-output-to-file "conda/.version"
                (lambda () (display ,version)))
              #t))
-         (add-before 'check 'remove-failing-tests
-           (lambda _
-             ;; These tests require internet/network access
-             (let ((network-tests '("test_cli.py"
-                                    "test_create.py"
-                                    "test_export.py"
-                                    "test_fetch.py"
-                                    "test_history.py"
-                                    "test_info.py"
-                                    "test_install.py"
-                                    "test_priority.py"
-                                    "conda_env/test_cli.py"
-                                    "conda_env/test_create.py"
-                                    "conda_env/specs/test_notebook.py"
-                                    "conda_env/utils/test_notebooks.py"
-                                    "core/test_index.py"
-                                    "core/test_repodata.py")))
-               (with-directory-excursion "tests"
-                 (for-each delete-file network-tests)
-
-                 ;; FIXME: This test creates a file, then deletes it and tests
-                 ;; that the file was deleted.  For some reason it fails when
-                 ;; building with guix, but does not when you run it in the
-                 ;; directory left when you build with the --keep-failed
-                 ;; option
-                 (delete-file "gateways/disk/test_delete.py")
-                 #t))))
          (replace 'check
            (lambda _
              (setenv "HOME" "/tmp")
@@ -813,8 +787,9 @@ extracting, creating, and converting between formats.")
        ("python-pytest" ,python-pytest)
        ("python-responses" ,python-responses)
        ("python-pyyaml" ,python-pyyaml)
-       ("python-anaconda-client" ,python-anaconda-client)))
-    (home-page "https://github.com/conda/conda")
+       ("python-anaconda-client" ,python-anaconda-client)
+       ("python-conda-package-handling" ,python-conda-package-handling)))
+    (home-page "https://conda.io")
     (synopsis "Cross-platform, OS-agnostic, system-level binary package manager")
     (description
      "Conda is a cross-platform, Python-agnostic binary package manager.  It
-- 
2.21.0

[bug#36026] [PATCH 4/4] gnu: python-conda: Update to 4.7.0.

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> * gnu/packages/package-management.scm (python-conda): Update to 4.7.0.
[…]
> -         (add-before 'check 'remove-failing-tests
> -           (lambda _
> -             ;; These tests require internet/network access
> -             (let ((network-tests '("test_cli.py"
> -                                    "test_create.py"
> -                                    "test_export.py"
> -                                    "test_fetch.py"
> -                                    "test_history.py"
> -                                    "test_info.py"
> -                                    "test_install.py"
> -                                    "test_priority.py"
> -                                    "conda_env/test_cli.py"
> -                                    "conda_env/test_create.py"
> -                                    "conda_env/specs/test_notebook.py"
> -                                    "conda_env/utils/test_notebooks.py"
> -                                    "core/test_index.py"
> -                                    "core/test_repodata.py")))
> -               (with-directory-excursion "tests"
> -                 (for-each delete-file network-tests)
> -
> -                 ;; FIXME: This test creates a file, then deletes it and tests
> -                 ;; that the file was deleted.  For some reason it fails when
> -                 ;; building with guix, but does not when you run it in the
> -                 ;; directory left when you build with the --keep-failed
> -                 ;; option
> -                 (delete-file "gateways/disk/test_delete.py")
> -                 #t))))

Do none of the tests now require network access?  Are the tests actually
still run?

--
Ricardo

[bug#36026] [PATCH 1/4] gnu: libarchive: Add zstd support.

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> * gnu/packages/backup.scm (libarchive)[inputs]: Include zstd.
> [arguments]: Build against zstd.

Here you’re modifying ‘libarchive’, which has tons of dependents.

If you instead modify ‘libarchive-3.3.3’, you’ll be able to apply it
directly on master, and follow up with a patch to apply the changes on
‘libarchive’ itself in ‘core-updates’.

WDYT?

Thanks,
Ludo’.

[bug#36026] [PATCH 0/4] Update Conda

[Ludovic Courtès]

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>   gnu: libarchive: Add zstd support.
>   gnu: python-libarchive-c: Explicitly rely on latest libarchive.
>   gnu: Add python-conda-package-handling.
>   gnu: python-conda: Update to 4.7.0.

Apart from the rebuild issue with patch #1 that needs to be addressed,
it LGTM!

Thanks,
Ludo’.

[bug#36026] [PATCH 0/4] Update Conda

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 520 bytes --]

Thanks for the clarifications around the libarchive business.

The problem is that this patch still does not work because
python-libarchive-c does not support zstd, which
python-conda-package-handling needs.  I don't understand how upstream
does this.  I've reported it upstream:

https://github.com/conda/conda-package-handling/issues/14

My knowledge of python is not enough to understand if upstream is wrong
or if I'm using python-libarchive-c incorrectly.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

[bug#36026] [PATCH 0/4] Update Conda

[Ricardo Wurmus]

Hi Pierre,

> Recent conda needs a separate python-conda-package-handling library.
> This library requires a libarchive built with zstd support.
> So far so good, except that python-libarchive-c does not seem to support zstd in its source code.
>
> I've reported the issue upstream
> (https://github.com/conda/conda-package-handling/issues/14) but
> according to them python-libarchive-c is fine.
>
> I don't have enough Python knowledge to understand whether upstream is right
> 	or if there is indeed something fishy with python-libarchive-c.
> Pierre Neidhardt (4):
>   gnu: libarchive: Add zstd support.
>   gnu: python-libarchive-c: Explicitly rely on latest libarchive.
>   gnu: Add python-conda-package-handling.
>   gnu: python-conda: Update to 4.7.0.

Does this actually work?  If it does: congratulations!

I tried to upgrade conda a while ago, but I got stuck:

   https://github.com/conda/conda/issues/8151

How did you overcome these problems?  I had to patch a lot of things and
even then couldn’t make it work.  Did I miss something obvious?

I’m glad you took care of this and I didn’t have to become the Conda
guy… :)

--
Ricardo

[bug#36026] [PATCH 0/4] Update Conda

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 401 bytes --]

Hi Ricardo,

Sorry to kill your enthusiasm, but nope, it does not work :/
The is an issue building the conda-package-handling dependency.  See the
prior discussion for the details.

So I haven't even reached to the issues you've hinted at on GitHub.
Don't know if they still apply with my patch, we can only figure it out
one step at a time :p

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#36026: Update Conda

[Ricardo Wurmus]

Commit 638ef1e81d upgrades Conda to 4.8.3.  This makes these patches
obsolete.

What a nightmare this was!

-- 
Ricardo