bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

python-minimal@2.7.13 is failing to build on core-updates:

https://hydra.gnu.org/build/1912761

I saw this near the beginning of the install phase:

------
configure: error: cannot run /bin/sh
/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Modules/_ctypes/libffi/config.sub
Failed to configure _ctypes module

Python build finished, but the necessary bits to build these modules
were not found:
_bsddb             _curses            _curses_panel   
_sqlite3           _ssl               _tkinter        
bsddb185           dbm                dl              
gdbm               imageop            readline        
sunaudiodev        zlib                               
To find the necessary bits, look in setup.py in detect_modules() for the
module's name.
------

And at the end, it fails:

------
if test "xinstall" != "xno"  ; then \
	case install in \
		upgrade) ensurepip="--upgrade" ;; \
		install|*) ensurepip="" ;; \
	esac; \
	LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \
		$ensurepip --root=/ ; \
fi
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 205, in _main
    default_pip=args.default_pip,
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 108, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 31, in _run_pip
    import pip
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [Makefile:927: install] Error 1
phase `install' failed after 29.1 seconds
------

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Danny Milosavljevic]

Hmmm... Python 2.7.13 seems to bundle libffi-3.1. Do we want that?

Also, I checked libffi/configure: If CONFIG_SHELL was set, it would have picked it up.

Also, why does Python 2.7.13 try to configure things in the *install* phase ?

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

On Fri, Mar 17, 2017 at 03:08:51PM +0100, Danny Milosavljevic wrote:
> Hmmm... Python 2.7.13 seems to bundle libffi-3.1. Do we want that?

The python-minimal package *should* use that libffi. I'm not sure why
it's failing now.

> Also, I checked libffi/configure: If CONFIG_SHELL was set, it would have picked it up.

Do you mean if CONFIG_SHELL is not set, so the build process can't find
libffi?

There is also the question about zlib and the bundled pip: why is it
failing now?

I haven't yet found the changes between 2.7.12 and 2.7.13 that are
causing these issues.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Danny Milosavljevic]

Hi Leo,

>The python-minimal package *should* use that libffi. I'm not sure why it's failing now.

Yeah, sorry.  I checked the package definition now and it explicitly disables the system libffi.

>Do you mean if CONFIG_SHELL is not set, so the build process can't find libffi?

Yes, a workaround for the first problem is:

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index be452f062..978aeca47 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -192,7 +192,8 @@
                                   '("Lib/subprocess.py"
                                     "Lib/popen2.py"
                                     "Lib/distutils/tests/test_spawn.py"
-                                    "Lib/test/test_subprocess.py"))
+                                    "Lib/test/test_subprocess.py"
+                                    "Modules/_ctypes/libffi/configure"))
                (("/bin/sh") (which "sh")))
 
              ;; Use zero as the timestamp in .pyc files so that builds are

The reason why that helps is because the configure script contains embedded HERE scripts with their own shebangs which patch-shebang doesn't patch.

An alternative workaround is to use this instead:

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index be452f062..04c944d84 100644
--- a/gnu/packages/python.scm12/bin/shore-updates/guix$ /gnu/store/k7029k5va68lka
+++ b/gnu/packages/python.scm
@@ -229,6 +229,8 @@
            ;; before 1980".  Work around this by setting the file times in the
            ;; source tree to sometime in early 1980.
            (lambda _
+             (setenv "CONFIG_SHELL" (which "bash"))
+
              (let ((circa-1980 (* 10 366 24 60 60)))
                (ftw "." (lambda (file stat flag)
                           (utime file circa-1980 circa-1980)

>There is also the question about zlib and the bundled pip: why is it failing now?

The above doesn't fix the pip zlib problem.  It's very strange: The value of the environment variable C_INCLUDE_PATH includes bzip2 but not zlib.  Might make python unable to find zlib.

setup.py does some strange stuff with reading the zlib.h header file manually - maybe that broke.  Then it tries to find the library file for libz.  The value of the environment variable LIBRARY_PATH doesn't contain zlib either.

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Leo Famulari]

[-- Attachment #1.1: Type: text/plain, Size: 1232 bytes --]

On Sat, Mar 18, 2017 at 09:19:01PM +0100, Danny Milosavljevic wrote:
> An alternative workaround is to use this instead:
> 
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index be452f062..04c944d84 100644
> --- a/gnu/packages/python.scm12/bin/shore-updates/guix$ /gnu/store/k7029k5va68lka
> +++ b/gnu/packages/python.scm
> @@ -229,6 +229,8 @@
>             ;; before 1980".  Work around this by setting the file times in the
>             ;; source tree to sometime in early 1980.
>             (lambda _
> +             (setenv "CONFIG_SHELL" (which "bash"))
> +

I think this is the better solution to the libffi / ctypes issue. It
also fixes python-minimal@3 to use the bundled libffi. We added libffi
to python-minimal@3's inputs in
d0b73960db051baa468fc1a0fbab21fa9f89f885 to work around a similar issue.

In Python 3.7, the bundled libffi will be removed, and we will have to
build our minimal Python-3 variant with an external libffi package:

https://bugs.python.org/issue27976

Here's a patch that fixes this problem for python-minimal@2. I don't
want to change python-minimal@3 this late in the core-updates cycle; we
can do it in a python-updates branch after core-updates.

[-- Attachment #1.2: 0001-gnu-python-minimal-2-Fix-build-failure-due-to-missin.patch --]
[-- Type: text/plain, Size: 2725 bytes --]

From 9d7eef243fca8e53fb89bb17a999f0678d191caf Mon Sep 17 00:00:00 2001
From: Leo Famulari <leo@famulari.name>
Date: Sun, 19 Mar 2017 02:43:37 -0400
Subject: [PATCH] gnu: python-minimal@2: Fix build failure due to missing
 libffi.

* gnu/packages/python.scm (python-2)[arguments]: Add 'setenv' phase.
(python2-minimal)[inputs]: Add zlib.

Co-authored-by: Danny Milosavljevic <dannym@scratchpost.org>
---
 gnu/packages/python.scm | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index be452f062..95bf53c8f 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -17,7 +17,7 @@
 ;;; Copyright © 2015, 2016 Efraim Flashner <efraim@flashner.co.il>
 ;;; Copyright © 2015 Kyle Meyer <kyle@kyleam.com>
 ;;; Copyright © 2015, 2016 Chris Marusich <cmmarusich@gmail.com>
-;;; Copyright © 2016 Danny Milosavljevic <dannym+a@scratchpost.org>
+;;; Copyright © 2016, 2017 Danny Milosavljevic <dannym+a@scratchpost.org>
 ;;; Copyright © 2016 Lukas Gradl <lgradl@openmailbox.org>
 ;;; Copyright © 2016 Hartmut Goebel <h.goebel@crazy-compilers.com>
 ;;; Copyright © 2016 Daniel Pimentel <d4n1@d4n1.org>
@@ -223,6 +223,19 @@
              (setenv "HOME" (getcwd))
              #t))
           (add-after
+           'unpack 'setenv
+           (lambda _
+             ;; 'Modules/_ctypes/libffi/configure' contains scripts embedded in
+             ;; HEREDOCs whose shebangs aren't patched by the patch-shebang
+             ;; phase.  Setting this variables works around that shortcoming and
+             ;; allows us to keep using the bundled libffi in python-minimal and
+             ;; python2-minimal.
+             ;; XXX The bundled libffi will be removed in Python 3.7:
+             ;; https://bugs.python.org/issue27976
+             ;; https://github.com/python/cpython/commit/935043d1ac19d07ac48233c6175250e66a7bbebd
+             ;; https://github.com/python/cpython/commit/f40d4ddff3c800b3c956a5e8820aabe3aa87cddd
+             (setenv "CONFIG_SHELL" (which "sh"))))
+          (add-after
            'unpack 'set-source-file-times-to-1980
            ;; XXX One of the tests uses a ZIP library to pack up some of the
            ;; source tree, and fails with "ZIP does not support timestamps
@@ -363,7 +376,7 @@ data types.")
      (substitute-keyword-arguments (package-arguments python-2)
        ((#:configure-flags cf)
         `(append ,cf '("--without-system-ffi")))))
-    (inputs '())))                          ;none of the optional dependencies
+    (inputs `(("zlib" ,zlib)))))
 
 (define-public python-minimal
   (package (inherit python)
-- 
2.12.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Leo Famulari]

On Sat, Mar 18, 2017 at 09:19:01PM +0100, Danny Milosavljevic wrote:
> >There is also the question about zlib and the bundled pip: why is it
> >failing now?
> 
> The above doesn't fix the pip zlib problem.  It's very strange: The
> value of the environment variable C_INCLUDE_PATH includes bzip2 but
> not zlib.  Might make python unable to find zlib.
> 
> setup.py does some strange stuff with reading the zlib.h header file
> manually - maybe that broke.  Then it tries to find the library file
> for libz.  The value of the environment variable LIBRARY_PATH doesn't
> contain zlib either.

Ideally, the minimal Python variants could use the bundled zlib found at
'Modules/zlib'. We sort of gave up on this with python-minimal@3 the
last time we had this problem, but maybe we can solve it now :)

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Leo Famulari]

[-- Attachment #1.1: Type: text/plain, Size: 4145 bytes --]

On Sun, Mar 19, 2017 at 03:26:36PM -0400, Leo Famulari wrote:
> On Sat, Mar 18, 2017 at 09:19:01PM +0100, Danny Milosavljevic wrote:
> > >There is also the question about zlib and the bundled pip: why is it
> > >failing now?

Here are some build logs showing differences between master and
core-updates.

For 2.7.12, on the master branch, there is this:

------
if test "xinstall" != "xno"  ; then \
        case install in \
                upgrade) ensurepip="--upgrade" ;; \
                install|*) ensurepip="" ;; \
        esac; \
        LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.12.drv-0/Python-2.7.12 ./python -E -m ensurepip \
                $ensurepip --root=/ ; \
fi
Ignoring ensurepip failure: pip 8.1.1 requires SSL/TLS
phase `install' succeeded after 31.5 seconds
------

I guess we didn't notice the lack of TLS support because the bundled pip
is never able to reach the internet when used to build packages.

For 2.7.13 without zlib, on core-updates, it does this:

------
if test "xinstall" != "xno"  ; then \
        case install in \
                upgrade) ensurepip="--upgrade" ;; \
                install|*) ensurepip="" ;; \
        esac; \
        LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \
                $ensurepip --root=/ ; \
fi
Traceback (most recent call last):
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 205, in _main 
    default_pip=args.default_pip,
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 108, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13/Lib/ensurepip/__init__.py", line 31, in _run_pip
    import pip
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [Makefile:927: install] Error 1
phase `install' failed after 10.8 seconds
------

If you build 2.7.13 with zlib on core-updates (log not *not* attached):

------
if test "xinstall" != "xno"  ; then \                                                                                                                                                                   case install in \
                upgrade) ensurepip="--upgrade" ;; \
                install|*) ensurepip="" ;; \
        esac; \                                                                                                                                                                                         LD_LIBRARY_PATH=/tmp/guix-build-python-minimal-2.7.13.drv-0/Python-2.7.13 ./python -E -m ensurepip \                                                                                                    $ensurepip --root=/ ; \                                                                                                                                                         fi
Collecting setuptools                                                                                                                                                                           Collecting pip                                                                                                                                                                                  Installing collected packages: setuptools, pip                                                                                                                                                  Successfully installed pip-9.0.1 setuptools-28.8.0
phase `install' succeeded after 21.6 seconds                  
------

[-- Attachment #1.2: python-minimal-2.7.12.log.bz2 --]
[-- Type: application/octet-stream, Size: 60776 bytes --]

[-- Attachment #1.3: python-minimal-2.7.13-no-zlib.log.gz --]
[-- Type: application/x-gunzip, Size: 82309 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 113 bytes --]

Hi!

Adding libffi and zlib solves the problem, similar to python-minimal@3.

Go for it?  :-)

Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: the patch --]
[-- Type: text/x-patch, Size: 1272 bytes --]

From 1cb73a54813136632553f03063abdd5cd105a760 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
Date: Sun, 19 Mar 2017 22:10:12 +0100
Subject: [PATCH] gnu: python-minimal@2: Add dependencies on libffi and zlib.

* gnu/packages/python.scm (python2-minimal)[inputs]: Add LIBFFI and ZLIB.
[arguments]: Remove.
---
 gnu/packages/python.scm | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index be452f062..00598776a 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -359,11 +359,12 @@ data types.")
   (package (inherit python-2)
     (name "python-minimal")
     (outputs '("out"))
-    (arguments
-     (substitute-keyword-arguments (package-arguments python-2)
-       ((#:configure-flags cf)
-        `(append ,cf '("--without-system-ffi")))))
-    (inputs '())))                          ;none of the optional dependencies
+
+    ;; Keep zlib, which is used by 'pip' (via the 'zipimport' module), which
+    ;; is invoked upon 'make install'.  'pip' also expects 'ctypes' and thus
+    ;; libffi.
+    (inputs `(("libffi" ,libffi)
+              ("zlib" ,zlib)))))
 
 (define-public python-minimal
   (package (inherit python)
-- 
2.12.0

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> From 1cb73a54813136632553f03063abdd5cd105a760 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>
> Date: Sun, 19 Mar 2017 22:10:12 +0100
> Subject: [PATCH] gnu: python-minimal@2: Add dependencies on libffi and zlib.
>
> * gnu/packages/python.scm (python2-minimal)[inputs]: Add LIBFFI and ZLIB.
> [arguments]: Remove.

Pushed as 5927a843b4a91416b5be9e65c69616447f82e162 in core-updates,
following our discussion on IRC.

Thanks,
Ludo’.

bug#26136: core-updates: python-minimal@2.7.13 fails due to missing zlib

[Danny Milosavljevic]

Setting the environment variables

  CPPFLAGS="-I/gnu/store/y8c62fyb2x4g4qnl54pmlmid1fn8k3mx-zlib-1.2.8/include"
  LDFLAGS="-L/gnu/store/y8c62fyb2x4g4qnl54pmlmid1fn8k3mx-zlib-1.2.8/lib"

before running configure makes it work.

(I've also printed inc_dirs in setup.py without that - and zlib didn't appear)

So I guess the general question is how does Python set up inc_dirs ?

And is it normal that C_INCLUDE_PATH and LIBRARY_PATH doesn't contain zlib ? Apparently, since guix-master python2-minimal 2.7.12 doesn't contain it either -  and that one works...

So where would it get the other entries?