Python-build-system does not honour phases

Python-build-system does not honour phases

[Andreas Enge]

Apparently, the #:phases parameter has no influence on the python build
system.

For instance, when adding
(arguments
  `(#:phases '()))
the package still gets built with the %standard-phases.

I added a line
   #:phases ,phases
to
   (define builder
at line 131 of guix/build-system/python.scm.

Now I can empty out the phases.

But when I write something like
#:phases
  (alist-replace
   'install
   ...
   %standard-phases)
apparently the %standard-phases from the GNU build system are used,
as there is an error message that ./configure is not found.

Could maybe someone have a look and propose a solution?

Andreas

Re: Python-build-system does not honour phases

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

Andreas Enge <andreas@enge.fr> skribis:

> Apparently, the #:phases parameter has no influence on the python build
> system.
>
> For instance, when adding
> (arguments
>   `(#:phases '()))
> the package still gets built with the %standard-phases.
>
> I added a line
>    #:phases ,phases
> to
>    (define builder
> at line 131 of guix/build-system/python.scm.
>
> Now I can empty out the phases.

Yes, that’s the correct fix.

> But when I write something like
> #:phases
>   (alist-replace
>    'install
>    ...
>    %standard-phases)
> apparently the %standard-phases from the GNU build system are used,
> as there is an error message that ./configure is not found.
>
> Could maybe someone have a look and propose a solution?

The problem is that both the gnu-build-system and the
python-build-system were getting imported, and both export a
‘%standard-phases’.

The fix is to use only python-build-system:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 644 bytes --]

diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index b60adb1..ff34451 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -58,7 +58,6 @@
                                            (guix build gnu-build-system)
                                            (guix build utils)))
                        (modules '((guix build python-build-system)
-                                  (guix build gnu-build-system)
                                   (guix build utils))))
   "Build SOURCE using PYTHON, and with INPUTS.  This assumes that SOURCE
 provides a 'setup.py' file as its build system."

[-- Attachment #3: Type: text/plain, Size: 57 bytes --]


Could you check that and commit?

Thanks,
Ludo’.

Re: Python-build-system does not honour phases

[Andreas Enge]

On Mon, Sep 09, 2013 at 11:35:55PM +0200, Ludovic Courtès wrote:
> The problem is that both the gnu-build-system and the
> python-build-system were getting imported, and both export a
> ‘%standard-phases’.

That is what I thought. I tried to add a
  #:use-module ((guix build-system gnu)
                #:select (gnu-build-system))
but that was not enough.

> The fix is to use only python-build-system:

That also does not seem to work. I split out the packages into two modules,
python for the python binaries that use gnu-build-system, and python-modules
for the packages using python-build-system.

The same error occurs, (alist-replace %standard-build-system) ends up with
the configure from gnu-build-system. 

Could it have something to do with the way %standard-phases of python are
derived from those of gnu? Or that code is quoted and executed in a different
environment from where it is written?

Maybe we should try to use a variable name %python-standard-phases instead.

Andreas

Re: Python-build-system does not honour phases

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 2086 bytes --]

On Tue, Sep 10, 2013 at 10:26:55AM +0200, Andreas Enge wrote:
> Maybe we should try to use a variable name %python-standard-phases instead.

The attached patch to guix/build/python-build-system.scm does just this
and works. Would it make sense to push it?

The part of the patch adding python-setuptools is not finished yet.
Installation ends with the contradictory error message

running install
Checking .pth file support in /nix/store/dwfvjk9rii9xyshfd6snny590a82bbfq-python-setuptools-1.1.4/lib/python3.3/site-packages/
/nix/store/fqg4h8afvnh7c0l2pqrid0wsjldp03sh-python-wrapper-3.3.2/bin/python -E -c pass
TEST FAILED: /nix/store/dwfvjk9rii9xyshfd6snny590a82bbfq-python-setuptools-1.1.4/lib/python3.3/site-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /nix/store/dwfvjk9rii9xyshfd6snny590a82bbfq-python-setuptools-1.1.4/lib/python3.3/site-packages/

and your PYTHONPATH environment variable currently contains:

    '/nix/store/dwfvjk9rii9xyshfd6snny590a82bbfq-python-setuptools-1.1.4/lib/python3.3/site/packages/'

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations

Please make the appropriate changes for your system and try again.


From what I can tell, I followed exactly the second approach. 

Does any of the python specialists understand what is happening?

Andreas


[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 3395 bytes --]

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 3ff4da2..a8f8c9d 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -247,3 +247,44 @@ etc. ")
 
 (define-public python2-babel
   (package-with-python2 python-babel))
+
+(define-public python-setuptools
+  (package
+    (name "python-setuptools")
+    (version "1.1.4")
+    (source
+     (origin
+      (method url-fetch)
+      (uri (string-append "https://pypi.python.org/packages/source/s/setuptools/setuptools-"
+                          version ".tar.gz"))
+      (sha256
+       (base32
+        "0hl9sa5xr9bi2ifq51wy1bawsjv5nzvpbac7m9z1ciz778874csf"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:tests? #f ; FIXME: try to run tests
+       #:phases
+         (alist-replace
+          'install
+          (lambda* (#:key outputs #:allow-other-keys #:rest args)
+            (let* ((install (assoc-ref %python-standard-phases 'install))
+                   (out (assoc-ref outputs "out"))
+                   (path (string-append out "/lib/python3.3/site/packages/")))
+              (mkdir-p path)
+              (setenv "PYTHONPATH" path)
+              (format #t "XXX ~a XXX~%" (getenv "PYTHONPATH"))
+              (apply install args)))
+          %python-standard-phases)))
+    (home-page "https://pypi.python.org/pypi/setuptools")
+    (synopsis
+     "Library designed to facilitate packaging Python projects")
+    (description
+     "Setuptools is a fully-featured, stable library designed to facilitate
+packaging Python projects, where packaging includes:
+Python package and module definitions,
+distribution package metadata,
+test hooks,
+project installation,
+platform-specific details,
+Python 3 support.")
+    (license psfl)))
diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index f213a97..14cd086 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -19,14 +19,13 @@
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
 (define-module (guix build python-build-system)
-  #:use-module ((guix build gnu-build-system)
-                #:renamer (symbol-prefix-proc 'gnu:))
+  #:use-module (guix build gnu-build-system)
   #:use-module (guix build utils)
   #:use-module (ice-9 match)
   #:use-module (ice-9 ftw)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
-  #:export (%standard-phases
+  #:export (%python-standard-phases
             python-build))
 
 ;; Commentary:
@@ -91,7 +90,7 @@
                             files)))
               bindirs)))
 
-(define %standard-phases
+(define %python-standard-phases
   ;; 'configure' and 'build' phases are not needed.  Everything is done during
   ;; 'install'.
   (alist-cons-after
@@ -103,11 +102,11 @@
      'check check
      (alist-replace 'install install
                     (alist-delete 'configure
-                                               gnu:%standard-phases))))))
+                                  %standard-phases))))))
 
 (define* (python-build #:key inputs (phases %standard-phases)
                        #:allow-other-keys #:rest args)
   "Build the given Python package, applying all of PHASES in order."
-  (apply gnu:gnu-build #:inputs inputs #:phases phases args))
+  (apply gnu-build #:inputs inputs #:phases phases args))
 
 ;;; python-build-system.scm ends here

Re: Python-build-system does not honour phases

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

Andreas Enge <andreas@enge.fr> skribis:

> On Mon, Sep 09, 2013 at 11:35:55PM +0200, Ludovic Courtès wrote:
>> The problem is that both the gnu-build-system and the
>> python-build-system were getting imported, and both export a
>> ‘%standard-phases’.
>
> That is what I thought. I tried to add a
>   #:use-module ((guix build-system gnu)
>                 #:select (gnu-build-system))
> but that was not enough.

You’re mixing different things: the line above is on the host side,
whereas the patch I proposed changes the modules imported on the build
side.

>> The fix is to use only python-build-system:
>
> That also does not seem to work.

Could you be more specific?  :-)

With the patch I sent, the builder should start with:

  (begin
    (use-modules (guix build python-build-system)
                 (guix build utils))

    ...)

So the global variable named ‘%standard-phases’ here is necessarily that
of (guix build python-build-system).

Can you try this patch to check the value of ‘phases’?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 595 bytes --]

diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm
index 8429979..1b5eaa8 100644
--- a/guix/build/python-build-system.scm
+++ b/guix/build/python-build-system.scm
@@ -96,6 +96,6 @@
 (define* (python-build #:key inputs (phases %standard-phases)
                        #:allow-other-keys #:rest args)
   "Build the given Python package, applying all of PHASES in order."
-  (apply gnu:gnu-build #:inputs inputs #:phases phases args))
+  (apply gnu:gnu-build #:inputs inputs #:phases (pk 'the-python-phases phases) args))
 
 ;;; python-build-system.scm ends here

[-- Attachment #3: Type: text/plain, Size: 167 bytes --]


[...]

> Maybe we should try to use a variable name %python-standard-phases instead.

That would be cheating.  ;-)
And really, it’s unnecessary.

Ludo’.

Re: Python-build-system does not honour phases

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Tue, Sep 10, 2013 at 10:26:55AM +0200, Andreas Enge wrote:
>> Maybe we should try to use a variable name %python-standard-phases instead.
>
> The attached patch to guix/build/python-build-system.scm does just this
> and works. Would it make sense to push it?

No, I prefer to understand what’s going on and DTRT.

> The part of the patch adding python-setuptools is not finished yet.
> Installation ends with the contradictory error message

[...]

> You are attempting to install a package to a directory that is not
> on PYTHONPATH and which Python does not read ".pth" files from.  The
> installation directory you specified (via --install-dir, --prefix, or
> the distutils default setting) was:
>
>     /nix/store/dwfvjk9rii9xyshfd6snny590a82bbfq-python-setuptools-1.1.4/lib/python3.3/site-packages/
>
> and your PYTHONPATH environment variable currently contains:
>
>     '/nix/store/dwfvjk9rii9xyshfd6snny590a82bbfq-python-setuptools-1.1.4/lib/python3.3/site/packages/'

The error seems to be that we use ‘site-packages’ when it expects
‘site/packages’.

The latter is clearly wrong, but I couldn’t find where it comes from
(perhaps you fixed it in the meantime?).

Ludo’.

Re: Python-build-system does not honour phases

[Andreas Enge]

On Tue, Sep 10, 2013 at 07:48:30PM +0200, Ludovic Courtès wrote:
> You’re mixing different things: the line above is on the host side,
> whereas the patch I proposed changes the modules imported on the build
> side.

Okay, I get it!

> Can you try this patch to check the value of ‘phases’?

Without your previous patch, I get the following:

;;; (the-python-phases ((set-paths . #<procedure set-paths (#:key target inputs native-inputs search-paths native-search-paths)>) (unpack . #<procedure unpack (#:key source)>) (patch . #<procedure patch (#:key patches patch-flags)>) (patch-source-shebangs . #<procedure patch-source-shebangs (#:key source)>) (configure . #<procedure configure (#:key target native-inputs inputs outputs configure-flags out-of-source?)>) (patch-generated-file-shebangs . #<procedure patch-generated-file-shebangs rest>) (build . #<procedure build (#:key make-flags parallel-build?)>) (check . #<procedure check (#:key target make-flags tests? test-target parallel-tests?)>) (install . #<procedure 1c90070 at ice-9/eval.scm:264:7 %args>) (patch-shebangs . #<procedure patch-shebangs (#:key outputs patch-shebangs?)>) (
 strip . #<procedure strip (#:key target outputs strip-binaries? strip-command objcopy-command strip-flags strip-directories)>)))
WARNING: (guile-user): `%standard-phases' imported from both (guix build python-build-system) and (guix build gnu-build-system)


After applying your previous patch, I get this:

;;; (the-python-phases ((set-paths . #<procedure set-paths (#:key target inputs native-inputs search-paths native-search-paths)>) (unpack . #<procedure unpack (#:key source)>) (patch . #<procedure patch (#:key patches patch-flags)>) (patch-source-shebangs . #<procedure patch-source-shebangs (#:key source)>) (patch-generated-file-shebangs . #<procedure patch-generated-file-shebangs rest>) (build . #<procedure build empty>) (check . #<procedure check (#:key tests? test-target)>) (install . #<procedure b92230 at ice-9/eval.scm:264:7 %args>) (wrap . #<procedure wrap (#:key inputs outputs)>) (patch-shebangs . #<procedure patch-shebangs (#:key outputs patch-shebangs?)>) (strip . #<procedure strip (#:key target outputs strip-binaries? strip-command objcopy-command strip-flags strip-directories)>)
 ))


So indeed, your patch solves the confusion!

Thanks, I will push it then.

Andreas

Re: Python-build-system does not honour phases

[Andreas Enge]

On Tue, Sep 10, 2013 at 07:53:03PM +0200, Ludovic Courtès wrote:
> The error seems to be that we use ‘site-packages’ when it expects
> ‘site/packages’.
> The latter is clearly wrong, but I couldn’t find where it comes from
> (perhaps you fixed it in the meantime?).

Thanks for spotting this, it was a typo I made when changing the environment
variable!

I am pushing setuptools; the tests succeeded with Python 2.7.5, while there is
(at least) one that fails with 3.3.2. But I am curious whether this will allow
us to install further modules.

Andreas

Re: Python-build-system does not honour phases

[Andreas Enge]

So the next module python-dateutil works. But again,
   python setup.py install
complains that the path it wishes to install to,
   /nix/store/q637nhgrixha1f8cfl32l6gvviha737g-python2-dateutil-1.5/lib/python2.7/site-packages
does not exist and is not in PYTHONPATH.

So I added the following:
     `(#:phases
         (alist-replace
          'install
          (lambda* (#:key outputs #:allow-other-keys #:rest args)
            (let* ((install (assoc-ref %standard-phases 'install))
                   (out (assoc-ref outputs "out"))
                   (python (assoc-ref %build-inputs "python"))
                   (python-version (string-take (string-take-right python 5) 3))
                   (path (string-append out "/lib/python" python-version
                                        "/site-packages/")))
              (mkdir-p path)
              (setenv "PYTHONPATH" (string-append (getenv "PYTHONPATH") ":" path))
              (apply install args)))
          %standard-phases)))
which is (a refined version of) the expression for python-setuptools.

We need to factor this out. I suggest to do the following:
In the install phase, before running setup.py, we create the directory and
add it to the python path. But this would only be needed for programs creating
modules, and I suppose not for programs that only create executables.

I see two solutions:
- We create the directory anyway, and try to remove it after installation with
  "rmdir -p".
- Or more cleanly, we can add a variable #:module? (default: #f) and create
  the directory only if this variable is set to #t.

What do you think?

Andreas

Re: Python-build-system does not honour phases

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> So indeed, your patch solves the confusion!

Excellent, thanks!

Ludo’.

Re: Python-build-system does not honour phases

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> We need to factor this out. I suggest to do the following:
> In the install phase, before running setup.py, we create the directory and
> add it to the python path. But this would only be needed for programs creating
> modules, and I suppose not for programs that only create executables.
>
> I see two solutions:
> - We create the directory anyway, and try to remove it after installation with
>   "rmdir -p".
> - Or more cleanly, we can add a variable #:module? (default: #f) and create
>   the directory only if this variable is set to #t.
>
> What do you think?

It think it’s OK to create it anyway, and not even try to remove it,
because packages that use ‘python-build-system’ surely have Python
modules to install, no?

Ludo’.

Re: Python-build-system does not honour phases

[Andreas Enge]

On Tue, Sep 10, 2013 at 11:20:16PM +0200, Ludovic Courtès wrote:
> It think it’s OK to create it anyway, and not even try to remove it,
> because packages that use ‘python-build-system’ surely have Python
> modules to install, no?

Maybe. Bazaar does. Calibre creates elf executables, but also installs
python modules. So I will go ahead with this.

Andreas

Re: Python-build-system does not honour phases

[Andreas Enge]

On Tue, Sep 10, 2013 at 11:34:29PM +0200, Andreas Enge wrote:
> Maybe. Bazaar does. Calibre creates elf executables, but also installs
> python modules. So I will go ahead with this.

Okay, done in commit 824af8cadc1b4f1ac7a859f3d18cbe69b195a844, and used
for python(2)-setuptools and a new package already.

Andreas

Re: Python-build-system does not honour phases

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> On Tue, Sep 10, 2013 at 11:34:29PM +0200, Andreas Enge wrote:
>> Maybe. Bazaar does. Calibre creates elf executables, but also installs
>> python modules. So I will go ahead with this.
>
> Okay, done in commit 824af8cadc1b4f1ac7a859f3d18cbe69b195a844, and used
> for python(2)-setuptools and a new package already.

Nice, thanks!

Ludo’.