Should python-build-system packages have native-inputs?

Should python-build-system packages have native-inputs?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2596 bytes --]

Hi Guix,

I've noticed that a fair number of packages in gnu/packages/python.scm
using the python-build-system declare native-inputs.  I suspect that in
every case, these should actually just be inputs.  I also suspect that
this is benign, except perhaps for the fact that it may confuse
Pythonistas who (like myself) initially started out by looking at these
packages as examples of how to get started defining packages in Guix.

The python-build-system's "lower" procedure (in
guix/build-system/python.scm) explicitly forbids cross-compilation:

--8<---------------cut here---------------start------------->8---
(define* (lower name
                #:key source inputs native-inputs outputs system target
                (python (default-python))
                #:allow-other-keys
                #:rest arguments)
  "Return a bag for NAME."
  (define private-keywords
    '(#:source #:target #:python #:inputs #:native-inputs))

  (and (not target)                               ;XXX: no cross-compilation
       (bag
         (name name)
         (system system)
         (host-inputs `(,@(if source
                              `(("source" ,source))
                              '())
                        ,@inputs

                        ;; Keep the standard inputs of 'gnu-build-system'.
                        ,@(standard-packages)))
         (build-inputs `(("python" ,python)
                         ,@native-inputs))
         (outputs outputs)
         (build python-build)
         (arguments (strip-keyword-arguments private-keywords arguments)))))
--8<---------------cut here---------------end--------------->8---

As for the native-inputs, they get stored in the bag's build-inputs,
which eventually find their way to the "inputs" keyword argument used on
the build side by the various build phases.  In fact, the inputs,
propagated-inputs, and native-inputs of any package that uses the
python-build-system are all put into this "inputs" keyword argument.

With this in mind, I have two questions:

* Should we change these native-inputs to inputs to prevent confusion?
  I can personally vouch for the fact that the presence of native-inputs
  in python-build-system packages confused the heck out of me at first!

* Are there any circumstances under which it actually WOULD make sense
  to cross-compile a Python package?

For now, I think the answers to these questions are "sure" and "probably
not", respectively.  I'm very curious to hear your thoughts about the
second question, in particular!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Should python-build-system packages have native-inputs?

[Fis Trivial]

Chris Marusich writes:

> Hi Guix,
>
> I've noticed that a fair number of packages in gnu/packages/python.scm
> using the python-build-system declare native-inputs.  I suspect that in
> every case, these should actually just be inputs.  I also suspect that
> this is benign, except perhaps for the fact that it may confuse
> Pythonistas who (like myself) initially started out by looking at these
> packages as examples of how to get started defining packages in Guix.
>
> The python-build-system's "lower" procedure (in
> guix/build-system/python.scm) explicitly forbids cross-compilation:
>
> --8<---------------cut here---------------start------------->8---
> (define* (lower name
>                 #:key source inputs native-inputs outputs system target
>                 (python (default-python))
>                 #:allow-other-keys
>                 #:rest arguments)
>   "Return a bag for NAME."
>   (define private-keywords
>     '(#:source #:target #:python #:inputs #:native-inputs))
>
>   (and (not target)                               ;XXX: no cross-compilation
>        (bag
>          (name name)
>          (system system)
>          (host-inputs `(,@(if source
>                               `(("source" ,source))
>                               '())
>                         ,@inputs
>
>                         ;; Keep the standard inputs of 'gnu-build-system'.
>                         ,@(standard-packages)))
>          (build-inputs `(("python" ,python)
>                          ,@native-inputs))
>          (outputs outputs)
>          (build python-build)
>          (arguments (strip-keyword-arguments private-keywords arguments)))))
> --8<---------------cut here---------------end--------------->8---
>
> As for the native-inputs, they get stored in the bag's build-inputs,
> which eventually find their way to the "inputs" keyword argument used on
> the build side by the various build phases.  In fact, the inputs,
> propagated-inputs, and native-inputs of any package that uses the
> python-build-system are all put into this "inputs" keyword argument.
>
> With this in mind, I have two questions:
>
> * Should we change these native-inputs to inputs to prevent confusion?
>   I can personally vouch for the fact that the presence of native-inputs
>   in python-build-system packages confused the heck out of me at first!
>
> * Are there any circumstances under which it actually WOULD make sense
>   to cross-compile a Python package?
>
> For now, I think the answers to these questions are "sure" and "probably
> not", respectively.  I'm very curious to hear your thoughts about the
> second question, in particular!

I'm confused, some native-inputs are for testing. They shouldn't be
installed in normal case.

Re: Should python-build-system packages have native-inputs?

[Hartmut Goebel]

Am 28.04.2018 um 08:50 schrieb Chris Marusich:
> * Should we change these native-inputs to inputs to prevent confusion?
>   I can personally vouch for the fact that the presence of native-inputs
>   in python-build-system packages confused the heck out of me at first!
 
As Fis already wrote:  These native-inputs are for testing and shouldn't
be installed in normal case. Please see "Python Modules" in the manual:

Python packages required only at build time---e.g., those listed with
the @code{setup_requires} keyword in @file{setup.py}---or only for
testing---e.g., those in @code{tests_require}---go into
@code{native-inputs}.  The rationale is that (1) they do not need to be
propagated because they are not needed at run time, and (2) in a
cross-compilation context, it's the ``native'' input that we'd want.

> * Are there any circumstances under which it actually WOULD make sense
>   to cross-compile a Python package?

Of course: Pure-python packages should be able to be cross-compiled
without any problems, sicne the bytes-code is the same for all
platforms. And for extension modules it would allow compiling on a
faster environment (e.g. x86 vs. ARMv4).

(I was not aware of python packages are not cross-compiled, thus I can
only guess the reason why this is not possible: Python distutils may not
be able to *cross*-compile extension modules. Maybe we could work on this.)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Should python-build-system packages have native-inputs?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 4085 bytes --]

Hi Fis and Hartmut,

Thank you for the quick response!

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> As Fis already wrote:  These native-inputs are for testing and shouldn't
> be installed in normal case.

It's true that for some of the packages that use the
python-build-system, we have been putting the dependencies required for
testing (such as python-pytest) into the package's native-inputs.
However, whether such dependencies are inputs or native-inputs does not
matter.  Because the python-build-system never cross-compiles, all of
the inputs, propagated-inputs, and native-inputs will be included in the
single list that gets passed to each of the build phases via the
#:inputs keyword argument.  You can verify this yourself by inserting
debug statements in the build phases.

In other words, it doesn't matter if we put python-pytest in a package's
inputs or its native-inputs.  The end result is the same.  If the
python-build-system actually did support cross-compilation, then this
might be a different story.  However, the python-build-system doesn't
cross-compile.  As a result, native-inputs and inputs are treated the
same in all of the phases defined in guix/build/python-build-system.scm.

> Please see "Python Modules" in the manual:
>
> Python packages required only at build time---e.g., those listed with
> the @code{setup_requires} keyword in @file{setup.py}---or only for
> testing---e.g., those in @code{tests_require}---go into
> @code{native-inputs}.  The rationale is that (1) they do not need to be
> propagated because they are not needed at run time, and (2) in a
> cross-compilation context, it's the ``native'' input that we'd want.

Thank you for mentioning the manual; I had forgotten that we include
explicit guidance for Python modules.  I've just reviewed the "Python
Modules" section.  I think we should not be advising people to use
native-inputs in packages that use the python-build-system.  There is no
meaningful difference between "native-inputs" and "inputs" in this case,
so asking people to contemplate the difference is like asking them a
kōan.  It's just going to cause confusion.

This is confusing.  And that is precisely why I think we should stop
declaring native-inputs for packages that use the python-build-system.
My understanding is that the concept of "native-inputs" for a package
only makes sense when that package uses a build system that can
cross-compile, such as the gnu-build-system.  Because the
python-build-system never cross-compiles, it doesn't make sense to
declare native-inputs for a package that uses the python-build-system.
Instead, those dependencies should just be declared as inputs.

>> * Are there any circumstances under which it actually WOULD make sense
>>   to cross-compile a Python package?
>
> Of course: Pure-python packages should be able to be cross-compiled
> without any problems, sicne the bytes-code is the same for all
> platforms.

I'm not sure that's the same thing as cross compilation.  When cross
compiling a program for a different architecture, the output of the
build is different for each architecture.  If Python's bytecode is the
same for all platforms, then it sounds like no cross-compilation is
necessary, which suggests that the notion of "cross compilation" does
not make sense for Python code.  Did I misinterpret what you meant?

> And for extension modules it would allow compiling on a faster
> environment (e.g. x86 vs. ARMv4).
>
> (I was not aware of python packages are not cross-compiled, thus I can
> only guess the reason why this is not possible: Python distutils may not
> be able to *cross*-compile extension modules. Maybe we could work on this.)

I am curious about extension modules.  I understand they are tied
closely to the underlying architecture, but I have little experience
with them, so I'm not sure how they relate to cross compilation.  In any
case, it doesn't change the fact that today, the python-build-system
does not cross-compile.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Should python-build-system packages have native-inputs?

[Hartmut Goebel]

[-- Attachment #1.1: Type: text/plain, Size: 2190 bytes --]

Am 28.04.2018 um 12:11 schrieb Chris Marusich:

I understand your concerns, and I understand why this is hard to get for
a Pythonista. But this is exactly why we added this section to the manual.

> Because the python-build-system never cross-compiles, 

This is an implementation detail which might might change. And if we
remove all inputs now, we need to add again them later. This is a lot of
work, I know since I've cleaned this up for all Python modules. IMHO
it's not a good idea for drop this knowledge from the code.

> If the
> python-build-system actually did support cross-compilation, then this
> might be a different story.

Maybe this is going to change somewhen :-) We should aim to the top, not
the status quo :-)



> My understanding is that the concept of "native-inputs" for a package
> only makes sense when that package uses a build system that can
> cross-compile,

This is my understanding, too. But the python-build-system might be able
to cross-compile somewhen and then this information is essential.

>> And for extension modules it would allow compiling on a faster
>> environment (e.g. x86 vs. ARMv4).
>>
>> (I was not aware of python packages are not cross-compiled, thus I can
>> only guess the reason why this is not possible: Python distutils may not
>> be able to *cross*-compile extension modules. Maybe we could work on this.)
> I am curious about extension modules.  I understand they are tied
> closely to the underlying architecture, but I have little experience
> with them, so I'm not sure how they relate to cross compilation.

Extension modules are simply modules or libraries  written in C/C++ or
other languages. Even modules written in Cython would be counted in
here, since they are translated to C and then compiled into platform
dependent code.

> In any
> case, it doesn't change the fact that today, the python-build-system
> does not cross-compile.

In any case, this is a current limitation only :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Should python-build-system packages have native-inputs?

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

>> the python-build-system does not cross-compile.
>
> In any case, this is a current limitation only :-)

I see. If we actually do plan on implementing some kind of
cross-compilation support for the python-build-system, then I can
understand why it makes sense to proactively put into the native-inputs
those things which might possibly need to be native-inputs when that
time comes.  Can we at least mention in the manual that the
python-build-system doesn't currently cross-compile, so native-inputs
will be treated the same as inputs for now, but we still recommend
putting the "build-only dependencies" in native-inputs in order to
future-proof our package definitions?  That alone would have helped
clarify things for me when I was starting out.

After reading about Python extension modules a little more, it seems
that they can in fact be cross-compiled.  I didn't look into Python 2,
and I don't know what it would take to enable such cross-compilation in
the python-build-system.

For those following along, here are some related links.

An explanation of Python extension modules:
https://docs.python.org/3/extending/building.html#distributing-your-extension-modules

Some Python open bugs that mention cross-compilation:
https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus&%40sort=-activity&%40group=priority&%40filter=status&status=-1%2C1%2C3&%40search_text=cross-compile&submit=search+in+open+issues

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Should python-build-system packages have native-inputs?

[Mark H Weaver]

Fis Trivial <ybbs.daans@hotmail.com> writes:
> I'm confused, some native-inputs are for testing. They shouldn't be
> installed in normal case.

native-inputs are not only for testing.  More generally, when
cross-compiling, any programs that must be run on the build machine must
be 'native-inputs'.  Any programs that must be run on the target machine
must be 'inputs'.

       Mark

Re: Should python-build-system packages have native-inputs?

[Mark H Weaver]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> Am 28.04.2018 um 12:11 schrieb Chris Marusich:
>
>> Because the python-build-system never cross-compiles, 
>
> This is an implementation detail which might might change. And if we
> remove all inputs now, we need to add again them later. This is a lot of
> work, I know since I've cleaned this up for all Python modules. IMHO
> it's not a good idea for drop this knowledge from the code.

I agree.

      Mark