Numpy failures

Numpy failures

[Andreas Enge]

python-numpy-bootstrap currently fails its tests on hydra, which entails a
bunch of other failures. On my own x86_64 machines, the build succeeds,
however.

Andreas

Re: Numpy failures

[Federico Beffa]

Andreas Enge <andreas@enge.fr> writes:

> python-numpy-bootstrap currently fails its tests on hydra, which entails a
> bunch of other failures. On my own x86_64 machines, the build succeeds,
> however.

We believe the reason being the fact that hydra doesn't handle the flag
'#:substitutable?' properly. As a result we have the following
situation:

1. hydra builds a version of ATLAS optimized for its CPU locally.

2. 'python-numpy-bootstrap' and co., on some architectures, probably get
an incompatible version of ATLAS and therefore fail to pass some tests.

You can check in the build log of 'python-numpy-bootstrap' that ATLAS,
despite the flas, is substituted (no local build on the slave).

At some point we should fix the support for '#:substitutable?' on hydra
(or the upcoming 'guix publish').

Regards,
Fede

bug#19805: Numpy failures

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> Andreas Enge <andreas@enge.fr> writes:
>
>> python-numpy-bootstrap currently fails its tests on hydra, which entails a
>> bunch of other failures. On my own x86_64 machines, the build succeeds,
>> however.
>
> We believe the reason being the fact that hydra doesn't handle the flag
> '#:substitutable?' properly. As a result we have the following
> situation:
>
> 1. hydra builds a version of ATLAS optimized for its CPU locally.
>
> 2. 'python-numpy-bootstrap' and co., on some architectures, probably get
> an incompatible version of ATLAS and therefore fail to pass some tests.
>
> You can check in the build log of 'python-numpy-bootstrap' that ATLAS,
> despite the flas, is substituted (no local build on the slave).

Oh.  Actually, slaves do not use substitutes at all, but they receive
missing store items from hydra.gnu.org when builds are offloaded.  So I
suppose they receive something built on hydra.gnu.org, which may be
incompatible.

> At some point we should fix the support for '#:substitutable?' on hydra

That probably means that #:substitutable? should be propagated–i.e.,
that anything depending on ATLAS should not be substituted.

Thanks for investigating!

Ludo’.

Re: bug#19805: Numpy failures

[Ludovic Courtès]

Federico Beffa <beffa@ieee.org> skribis:

> Andreas Enge <andreas@enge.fr> writes:
>
>> python-numpy-bootstrap currently fails its tests on hydra, which entails a
>> bunch of other failures. On my own x86_64 machines, the build succeeds,
>> however.
>
> We believe the reason being the fact that hydra doesn't handle the flag
> '#:substitutable?' properly. As a result we have the following
> situation:
>
> 1. hydra builds a version of ATLAS optimized for its CPU locally.
>
> 2. 'python-numpy-bootstrap' and co., on some architectures, probably get
> an incompatible version of ATLAS and therefore fail to pass some tests.
>
> You can check in the build log of 'python-numpy-bootstrap' that ATLAS,
> despite the flas, is substituted (no local build on the slave).

Oh.  Actually, slaves do not use substitutes at all, but they receive
missing store items from hydra.gnu.org when builds are offloaded.  So I
suppose they receive something built on hydra.gnu.org, which may be
incompatible.

> At some point we should fix the support for '#:substitutable?' on hydra

That probably means that #:substitutable? should be propagated–i.e.,
that anything depending on ATLAS should not be substituted.

Thanks for investigating!

Ludo’.

bug#19805: Numpy failures

[Federico Beffa]

On Tue, Feb 10, 2015 at 12:07 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>> At some point we should fix the support for '#:substitutable?' on hydra
>
> That probably means that #:substitutable? should be propagated–i.e.,
> that anything depending on ATLAS should not be substituted.

I guess so. Is there a way to specify this in a package or does it
require changes to the guix code?

Regards,
Fede

Re: bug#19805: Numpy failures

[Federico Beffa]

On Tue, Feb 10, 2015 at 12:07 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>> At some point we should fix the support for '#:substitutable?' on hydra
>
> That probably means that #:substitutable? should be propagated–i.e.,
> that anything depending on ATLAS should not be substituted.

I guess so. Is there a way to specify this in a package or does it
require changes to the guix code?

Regards,
Fede

bug#19805: Numpy failures

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Federico Beffa <beffa@ieee.org> skribis:
>
>> At some point we should fix the support for '#:substitutable?' on hydra
>
> That probably means that #:substitutable? should be propagated–i.e.,
> that anything depending on ATLAS should not be substituted.

As I just wrote in the "texlive failure" thread, setting
#:substitutable? #f might make sense for texlive.  However, in that case
we certainly would not want to propagate that setting.

So, instead of propagating it, how about simply arranging to not send
such packages to the build slaves, instead forcing them to build it
locally on their own?

I suppose this would cause the texlive build log to be prepended to some
other build log, which is not so good, but that could also be dealt with
in various ways, e.g. by changing the "send dependency to slave" logic
for non-substitutable packages to build the package on that particular
slave.

What do you think?

      Mark

Re: bug#19805: Numpy failures

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> Federico Beffa <beffa@ieee.org> skribis:
>
>> At some point we should fix the support for '#:substitutable?' on hydra
>
> That probably means that #:substitutable? should be propagated–i.e.,
> that anything depending on ATLAS should not be substituted.

As I just wrote in the "texlive failure" thread, setting
#:substitutable? #f might make sense for texlive.  However, in that case
we certainly would not want to propagate that setting.

So, instead of propagating it, how about simply arranging to not send
such packages to the build slaves, instead forcing them to build it
locally on their own?

I suppose this would cause the texlive build log to be prepended to some
other build log, which is not so good, but that could also be dealt with
in various ways, e.g. by changing the "send dependency to slave" logic
for non-substitutable packages to build the package on that particular
slave.

What do you think?

      Mark

bug#19805: Numpy failures

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Federico Beffa <beffa@ieee.org> skribis:
>>
>>> At some point we should fix the support for '#:substitutable?' on hydra
>>
>> That probably means that #:substitutable? should be propagated–i.e.,
>> that anything depending on ATLAS should not be substituted.
>
> As I just wrote in the "texlive failure" thread, setting
> #:substitutable? #f might make sense for texlive.  However, in that case
> we certainly would not want to propagate that setting.
>
> So, instead of propagating it, how about simply arranging to not send
> such packages to the build slaves, instead forcing them to build it
> locally on their own?

I think there are two properties of interest: “substitutability”, and
“offloadability”.

What do we want for ATLAS, for NumPy, and for TeX Live?

However, note that we can’t currently distinguish between these two
properties: <http://bugs.gnu.org/18747>.

Thanks,
Ludo’.

Re: bug#19805: Numpy failures

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Federico Beffa <beffa@ieee.org> skribis:
>>
>>> At some point we should fix the support for '#:substitutable?' on hydra
>>
>> That probably means that #:substitutable? should be propagated–i.e.,
>> that anything depending on ATLAS should not be substituted.
>
> As I just wrote in the "texlive failure" thread, setting
> #:substitutable? #f might make sense for texlive.  However, in that case
> we certainly would not want to propagate that setting.
>
> So, instead of propagating it, how about simply arranging to not send
> such packages to the build slaves, instead forcing them to build it
> locally on their own?

I think there are two properties of interest: “substitutability”, and
“offloadability”.

What do we want for ATLAS, for NumPy, and for TeX Live?

However, note that we can’t currently distinguish between these two
properties: <http://bugs.gnu.org/18747>.

Thanks,
Ludo’.

bug#19805: Numpy failures

[Ludovic Courtès]

This bug is moot now that NumPy uses OpenBLAS instead of ATLAS (commit
dbdfe515), so closing it.

The suggestion about propagating the #:substitutable? flag at
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=19805#11> remains valid,
but it’s a separate issue.

Ludo’.