Be careful with PyPI

Be careful with PyPI

[zimoun]

Hi,

If the origin does not exist upstream, then Guix try other servers as
fallback.  For instance,

--8<---------------cut here---------------start------------->8---
Starting download of /gnu/store/lb0kb4c212f9f789ixd1c18bcm8qbsqi-Keras-2.11.0.tar.gz
From https://files.pythonhosted.org/packages/source/K/Keras/Keras-2.11.0.tar.gz...
download failed "https://files.pythonhosted.org/packages/source/K/Keras/Keras-2.11.0.tar.gz" 404 "Not Found"

Starting download of /gnu/store/lb0kb4c212f9f789ixd1c18bcm8qbsqi-Keras-2.11.0.tar.gz
From https://ci.guix.gnu.org/file/Keras-2.11.0.tar.gz/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh...
download failed "https://ci.guix.gnu.org/file/Keras-2.11.0.tar.gz/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh" 404 "Not Found"

Starting download of /gnu/store/lb0kb4c212f9f789ixd1c18bcm8qbsqi-Keras-2.11.0.tar.gz
From https://tarballs.nixos.org/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh...
following redirection to `https://tarballs.nixos.org/sha512/0d57cb8e0af234a619ba3dec245c2ab73cbd0566194ed6b59377231183b2ef916514ba87abe04ec7e518770c9e0cb157747db87cedf0ebeced4ae0f56be401c3'...
downloading from https://tarballs.nixos.org/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh ...
 1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh  748KiB                                                     67.1MiB/s 00:00 [##################] 100.0%
successfully built /gnu/store/75g4aq5b25g4j1qwd3ggl1sys24q63la-Keras-2.11.0.tar.gz.drv
--8<---------------cut here---------------end--------------->8---

One potential issue is that the tarballs.nixos.org is using the checksum
as lookup key.  Therefore, when modifying only the version and not the
checksum, the something is returned with an inconsistent name/content.

First, let get the source of the current Keras (v2.2.4).

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build python-keras -S
0.5 MB will be downloaded:
  /gnu/store/k2wxrqzmr29rjy1w5b0nhd4k36sv8szb-Keras-2.2.4.tar.xz
substituting /gnu/store/k2wxrqzmr29rjy1w5b0nhd4k36sv8szb-Keras-2.2.4.tar.xz...
downloading from https://ci.guix.gnu.org/nar/k2wxrqzmr29rjy1w5b0nhd4k36sv8szb-Keras-2.2.4.tar.xz ...
 Keras-2.2.4.tar.xz  521KiB                                                                                        3.6MiB/s 00:00 [##################] 100.0%

/gnu/store/k2wxrqzmr29rjy1w5b0nhd4k36sv8szb-Keras-2.2.4.tar.xz
--8<---------------cut here---------------end--------------->8---

Then, tweak only the version (update to 2.11.0) and get the source.

--8<---------------cut here---------------start------------->8---
$ git diff
diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index 75cc100851..2cbe0eddd6 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -2831,7 +2831,7 @@ (define-public python-keras-preprocessing
 (define-public python-keras
   (package
     (name "python-keras")
-    (version "2.2.4")
+    (version "2.11.0")
     (source
      (origin
        (method url-fetch)
$ ./pre-inst-env guix build python-keras -S
;;; note: source file /home/simon/src/guix/guix/gnu/packages/machine-learning.scm
;;;       newer than compiled /home/simon/src/guix/guix/gnu/packages/machine-learning.go
;;; note: source file /home/simon/src/guix/guix/gnu/packages/machine-learning.scm
;;;       newer than compiled /home/simon/src/guix/guix/gnu/packages/machine-learning.go
;;; note: source file /home/simon/src/guix/guix/gnu/packages/machine-learning.scm
;;;       newer than compiled /home/simon/.config/guix/profiles/emacs/emacs/lib/guile/3.0/site-ccache/gnu/packages/machine-learning.go
;;; note: source file /home/simon/src/guix/guix/gnu/packages/machine-learning.scm
;;;       newer than compiled /home/simon/.config/guix/profiles/emacs/emacs/lib/guile/3.0/site-ccache/gnu/packages/machine-learning.go
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations will be built:
  /gnu/store/by6pgdambmv4wrg2h40l287d7zhbwc17-Keras-2.11.0.tar.xz.drv
  /gnu/store/75g4aq5b25g4j1qwd3ggl1sys24q63la-Keras-2.11.0.tar.gz.drv
building /gnu/store/75g4aq5b25g4j1qwd3ggl1sys24q63la-Keras-2.11.0.tar.gz.drv...

Starting download of /gnu/store/lb0kb4c212f9f789ixd1c18bcm8qbsqi-Keras-2.11.0.tar.gz
From https://files.pythonhosted.org/packages/source/K/Keras/Keras-2.11.0.tar.gz...
download failed "https://files.pythonhosted.org/packages/source/K/Keras/Keras-2.11.0.tar.gz" 404 "Not Found"

Starting download of /gnu/store/lb0kb4c212f9f789ixd1c18bcm8qbsqi-Keras-2.11.0.tar.gz
From https://ci.guix.gnu.org/file/Keras-2.11.0.tar.gz/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh...
download failed "https://ci.guix.gnu.org/file/Keras-2.11.0.tar.gz/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh" 404 "Not Found"

Starting download of /gnu/store/lb0kb4c212f9f789ixd1c18bcm8qbsqi-Keras-2.11.0.tar.gz
From https://tarballs.nixos.org/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh...
following redirection to `https://tarballs.nixos.org/sha512/0d57cb8e0af234a619ba3dec245c2ab73cbd0566194ed6b59377231183b2ef916514ba87abe04ec7e518770c9e0cb157747db87cedf0ebeced4ae0f56be401c3'...
downloading from https://tarballs.nixos.org/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh ...
 1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh  748KiB                                                     67.1MiB/s 00:00 [##################] 100.0%
successfully built /gnu/store/75g4aq5b25g4j1qwd3ggl1sys24q63la-Keras-2.11.0.tar.gz.drv
building /gnu/store/by6pgdambmv4wrg2h40l287d7zhbwc17-Keras-2.11.0.tar.xz.drv...
Keras-2.2.4/
[...]
Keras-2.2.4/Keras.egg-info/dependency_links.txt
patching file tests/integration_tests/applications_test.py
Hunk #1 succeeded at 64 (offset 6 lines).
/gnu/store/g2ajyl8xk9aarxrgjbng2hkj3qm2v0z2-tar-1.34/bin/tar: Option --mtime: Treating date '@1' as 1970-01-01 00:00:01
Keras-2.2.4/
[...]
Keras-2.2.4/tests/test_multiprocessing.py
source is at 'Keras-2.2.4'
applying '/gnu/store/xbzvc0ij9z7d13mmjk3pzfsnfwsimlm7-python-keras-integration-test.patch'...
successfully built /gnu/store/by6pgdambmv4wrg2h40l287d7zhbwc17-Keras-2.11.0.tar.xz.drv
/gnu/store/pxj6cnk8bis14jiz79igmp0k2813v21d-Keras-2.11.0.tar.xz
--8<---------------cut here---------------end--------------->8---

It is not Keras-2.11.0 but Keras-2.2.4.

--8<---------------cut here---------------start------------->8---
$ sha256sum /gnu/store/k2wxrqzmr29rjy1w5b0nhd4k36sv8szb-Keras-2.2.4.tar.xz /gnu/store/pxj6cnk8bis14jiz79igmp0k2813v21d-Keras-2.11.0.tar.xz
09220b37e2a8dddcf9db1ea0a1d77d710bf084086fa9339e9278b30eac59b6b7  /gnu/store/k2wxrqzmr29rjy1w5b0nhd4k36sv8szb-Keras-2.2.4.tar.xz
09220b37e2a8dddcf9db1ea0a1d77d710bf084086fa9339e9278b30eac59b6b7  /gnu/store/pxj6cnk8bis14jiz79igmp0k2813v21d-Keras-2.11.0.tar.xz
--8<---------------cut here---------------end--------------->8---

Well, the core of the issue is that
https://files.pythonhosted.org/packages/source/K/Keras/Keras-2.11.0.tar.gz
does not exist, so the fallback uses the checksum (of Keras v2.2.4).

Note that Keras 2.11.0 is available on PyPI,

    https://pypi.org/project/keras/2.11.0/

but not the source.  They removed the source after 2.6.0,

    https://pypi.org/project/keras/2.6.0/#files

Arf!

Well, IMHO, we cannot do better except been really careful when fetching
from PyPI.  (And aside this kind of troubles [1]).

Cheers,
simon


1: <https://pytorch.org/blog/compromised-nightly-dependency/>

Re: Be careful with PyPI

[Leo Famulari]

On Fri, Jan 06, 2023 at 03:36:38PM +0100, zimoun wrote:
> If the origin does not exist upstream, then Guix try other servers as
> fallback.  For instance,
[...]
> downloading from https://tarballs.nixos.org/sha256/1j8bsqzh49vjdxy6l1k4iwax5vpjzniynyd041xjavdzvfii1dlh ...

> One potential issue is that the tarballs.nixos.org is using the checksum
> as lookup key.  Therefore, when modifying only the version and not the
> checksum, the something is returned with an inconsistent name/content.

Many of us discover this behaviour the hard way. It's not just about
PyPi: this can happen with any download, unless something changed.

Thanks for the detailed explanation!