Re: 01/01: gnu: boost: Update to 1.58.0.

Re: 01/01: gnu: boost: Update to 1.58.0.

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> commit 16de69c950c473cc3d386cd7569bd1d6051614b6
> Author: Andreas Enge <andreas@enge.fr>
> Date:   Wed Jul 8 23:18:40 2015 +0200
>
>     gnu: boost: Update to 1.58.0.
>     
>     * gnu/packages/boost.scm (boost): Update to 1.58.0.

This fails to build on mips64el, where boost-1.57.0 worked.

  http://hydra.gnu.org/build/563826

It appears to be because of the -m32 compiler option, which is Intel
specific.  Can you take a look?

      Mark

Re: 01/01: gnu: boost: Update to 1.58.0.

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> Andreas Enge <andreas@enge.fr> writes:
>
>> commit 16de69c950c473cc3d386cd7569bd1d6051614b6
>> Author: Andreas Enge <andreas@enge.fr>
>> Date:   Wed Jul 8 23:18:40 2015 +0200
>>
>>     gnu: boost: Update to 1.58.0.
>>     
>>     * gnu/packages/boost.scm (boost): Update to 1.58.0.
>
> This fails to build on mips64el, where boost-1.57.0 worked.
>
>   http://hydra.gnu.org/build/563826
>
> It appears to be because of the -m32 compiler option, which is Intel
> specific.  Can you take a look?

Boost turns out to be an important package.  The fallout from this is
412 newly failed builds on mips64el:

  http://hydra.gnu.org/eval/105333#tabs-now-fail
  http://hydra.gnu.org/eval/105355?compare=105351#tabs-now-fail

If we can't fix this soon, I think we should revert the boost update.
What do you think?

      Mark

Re: 01/01: gnu: boost: Update to 1.58.0.

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 5165 bytes --]

On Thu, Jul 09, 2015 at 06:00:39PM -0400, Mark H Weaver wrote:
> Boost turns out to be an important package.  The fallout from this is
> 412 newly failed builds on mips64el:
>   http://hydra.gnu.org/eval/105333#tabs-now-fail

I do not quite understand this number, it should be 65 according to the
first link. But anyway, it is an important package.

I noticed the problem this morning when looking at hydra, and did a quick
check, by grepping for "-m32". It appears in
boost_1_58_0/tools/build/src/tools/gcc.jam:

rule setup-address-model ( targets * : sources * : properties * )
{
    local model = [ feature.get-values address-model : $(properties) ] ;
    if $(model)
    {
        local option ;
        local os = [ feature.get-values target-os : $(properties) ] ;
        if $(os) = aix
...

        else if $(os) = hpux
...
        else
        {
            local arch = [ feature.get-values architecture : $(properties) ] ;
            if $(arch) != arm
            {
                if $(model) = 32
                {
                    option = -m32 ;
                }
                else if $(model) = 64
                {
                    option = -m64 ;
                }
            }
            # For darwin, the model can be 32_64. darwin.jam will handle that
            # on its own.
        }
        OPTIONS on $(targets) += $(option) ;
    }
}

So it might be enough to exclude mips (or something starting with "mips")
at the same time as arm.

On the other hand, I just checked that the same expressions were already
there in 1.57, so this cannot be the problem.

And on yet another hand, I made a recursive diff of boost 1.57 and 1.58
and grepped for '\-m32', without success. So the change must be more subtle.

The build log for 1.58 contains the following:
starting phase `build'
Performing configuration checks

    - 32-bit                   : yes
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - x86                      : no
    - combined                 : no

Building the Boost C++ Libraries.


    - has_icu builds           : no
warning: Graph library does not contain MPI-based parallel components.
note: to enable them, add "using mpi ;" to your user-config.jam
    - zlib                     : no
    - iconv (libc)             : no
    - iconv (separate)         : no
    - icu                      : no
    - icu (lib64)              : no
- Boost.Locale needs either iconv or ICU library to be built.
- Boost.Locale needs either iconv or ICU library to be built.
- Boost.Locale needs either iconv or ICU library to be built.
- Boost.Locale needs either iconv or ICU library to be built.
- Boost.Locale needs either iconv or ICU library to be built.
- Boost.Locale needs either iconv or ICU library to be built.
    - x86                      : no
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - lockfree boost::atomic_flag : no
    - gcc visibility           : no
    - long double support      : no

For the last (working) build of boost 1.57, the corresponding passage was:
starting phase `build'

Building the Boost C++ Libraries.


Performing configuration checks

    - has_icu builds           : no
warning: Graph library does not contain MPI-based parallel components.
note: to enable them, add "using mpi ;" to your user-config.jam
    - zlib                     : yes
    - iconv (libc)             : yes
    - icu                      : no
    - icu (lib64)              : no
    - x86                      : no
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - lockfree boost::atomic_flag : yes
    - gcc visibility           : yes
    - long double support      : yes

The new line "32-bit: yes" looks suspicious. Strangely enough, it was
already present in boost 1.57 for i686, and similar lines ("32-bit: no
64-bit: yes") for x86_64. So I think that the logic for checking for
32 bits changed.


When I try to build by hand (with the toolchain from debian jessie, not guix)
on my mips machine, the first few lines are
Performing configuration checks

    - 32-bit                   : yes 
    - arm                      : no 
    - mips1                    : yes

Building the Boost C++ Libraries.

This is already different; on hydra, the mips architecture is apparently
not recognised. But I also get the additional "-m32" flag and the subsequent
build failure.

I tried to replace in the above file gcc.jam the line
            if $(arch) != arm
by
            if $(arch) = x86

The build starts and compiles a few files without problem.

I need to stop now, but can make more thorough tests tomorrow, unless
you feel like giving the attached patch a try on mips and x86, which
could be helpful: The last time I updated my guix on mips was before the
core-updates merge, so it may be a long while until I catch up. If you
wish, you may also revert the boost upgrade for the time being.

Andreas


[-- Attachment #2: boost-mips-avoid-m32.patch --]
[-- Type: text/plain, Size: 527 bytes --]

diff -u -r boost_1_58_0.orig/tools/build/src/tools/gcc.jam boost_1_58_0/tools/build/src/tools/gcc.jam
--- boost_1_58_0.orig/tools/build/src/tools/gcc.jam	2015-04-04 19:25:07.000000000 +0200
+++ boost_1_58_0/tools/build/src/tools/gcc.jam	2015-07-10 01:08:19.822733823 +0200
@@ -451,7 +451,7 @@
         else
         {
             local arch = [ feature.get-values architecture : $(properties) ] ;
-            if $(arch) != arm
+            if $(arch) = x86
             {
                 if $(model) = 32
                 {

Re: 01/01: gnu: boost: Update to 1.58.0.

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> On Thu, Jul 09, 2015 at 06:00:39PM -0400, Mark H Weaver wrote:
>> Boost turns out to be an important package.  The fallout from this is
>> 412 newly failed builds on mips64el:
>>   http://hydra.gnu.org/eval/105333#tabs-now-fail
>
> I do not quite understand this number, it should be 65 according to the
> first link. But anyway, it is an important package.

I'm not 100% sure what's happening either, but more packages are
becoming broken over time.  I think it has to do with the fact that
'git' is one of the broken packages, and other packages that fetch their
source code using 'git' are becoming broken whenever Guix decides it's
time to try re-downloading the source, e.g.:

  http://hydra.gnu.org/eval/105357#tabs-now-fail
  http://hydra.gnu.org/eval/105355?filter=.mips64el-linux&compare=105351#tabs-now-fail

Anyway, thanks for investigating this, but it seems that we still don't
understand what's going on, and meanwhile our MIPS port is extremely
broken.

I've reverted the patch.  After we have a solution to this problem, we
can build it in a separate branch.  I think we should have done this
anyway, since updating Boost entails a lot of rebuilds, and has a
history of being problematic on non-Intel platforms.

Does that make sense?

      Mark

Re: 01/01: gnu: boost: Update to 1.58.0.

[Andreas Enge]

On Fri, Jul 10, 2015 at 12:52:24PM -0400, Mark H Weaver wrote:
> I'm not 100% sure what's happening either, but more packages are
> becoming broken over time.  I think it has to do with the fact that
> 'git' is one of the broken packages, and other packages that fetch their
> source code using 'git' are becoming broken whenever Guix decides it's
> time to try re-downloading the source, e.g.:

Okay, that is an interesting explanation!

> I've reverted the patch.  After we have a solution to this problem, we
> can build it in a separate branch.  I think we should have done this
> anyway, since updating Boost entails a lot of rebuilds, and has a
> history of being problematic on non-Intel platforms.

With only 69 dependent packages, it did not look like a big risk!
It just built with the patch on my mips machine:
   Performing configuration checks

    - 32-bit                   : yes
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - x86                      : no
    - combined                 : no
I still find it suspicious that it is not recognised as "mips1"; it may
have to do with the different ABIs, since when I build it on debian,
it says "mips1 : yes".

I will push this to a wip-boost branch, and try to build a dependent package
locally. I wonder if I should base wip-boost on openssl-update; but with
only 69 dependent packages (if the count is true), it probably does not
matter.

Andreas



> 
> Does that make sense?
> 
>       Mark

Re: 01/01: gnu: boost: Update to 1.58.0.

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> On Fri, Jul 10, 2015 at 12:52:24PM -0400, Mark H Weaver wrote:
>> I'm not 100% sure what's happening either, but more packages are
>> becoming broken over time.  I think it has to do with the fact that
>> 'git' is one of the broken packages, and other packages that fetch their
>> source code using 'git' are becoming broken whenever Guix decides it's
>> time to try re-downloading the source, e.g.:
>
> Okay, that is an interesting explanation!
>
>> I've reverted the patch.  After we have a solution to this problem, we
>> can build it in a separate branch.  I think we should have done this
>> anyway, since updating Boost entails a lot of rebuilds, and has a
>> history of being problematic on non-Intel platforms.
>
> With only 69 dependent packages, it did not look like a big risk!

It's definitely more than that.  As I recall, Hydra had to do on the
order of 600 rebuilds after your boost update.  This is a case where
"guix refresh -l" is way off.

> It just built with the patch on my mips machine:
>    Performing configuration checks
>
>     - 32-bit                   : yes
>     - arm                      : no
>     - mips1                    : no
>     - power                    : no
>     - sparc                    : no
>     - x86                      : no
>     - combined                 : no
> I still find it suspicious that it is not recognised as "mips1"; it may
> have to do with the different ABIs, since when I build it on debian,
> it says "mips1 : yes".

Yes, it might have to do with that.  Debian uses the O32 ABI.

> I will push this to a wip-boost branch, and try to build a dependent package
> locally. I wonder if I should base wip-boost on openssl-update; but with
> only 69 dependent packages (if the count is true), it probably does not
> matter.

Can we do this on core-updates instead?  I'm currently working on a
core-updates branch.  I will push it soon, after I've done some basic
testing on it.

Hydra is already overloaded trying to rebuild the openssl-update jobset,
and now it will have more to do since I reverted boost on master and
rebased openssl-update on that.  I want to get openssl-update built
ASAP.

What do you think?

      Mark

Re: 01/01: gnu: boost: Update to 1.58.0.

[Andreas Enge]

Hi Mark,

On Fri, Jul 10, 2015 at 04:03:50PM -0400, Mark H Weaver wrote:
> Can we do this on core-updates instead?  I'm currently working on a
> core-updates branch.  I will push it soon, after I've done some basic
> testing on it.

I would prefer a separate branch - with hydra lagging behind, we tend
to build only the core packages on core-updates, so I think we would not
even see the result of a boost update.

On Fri, Jul 10, 2015 at 05:28:10PM -0400, Mark H Weaver wrote:
> I would also prefer to prioritize getting this openssl security fix
> deployed, if you don't mind.  Toward that end, I'm going to delete the
> jobset on Hydra for now.  If you disagree, can we talk about it?

No problem; I was just in a hurry because I am leaving for a week.
I just created a clean wip-boost branch with the two commits (boost update
and patch) applied on top of the current master. Once openssl-update has
built, could you please rebase wip-boost and try to have it built on hydra?

Thanks a lot!

Andreas

Re: 01/01: gnu: boost: Update to 1.58.0.

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> Hi Mark,
>
> On Fri, Jul 10, 2015 at 04:03:50PM -0400, Mark H Weaver wrote:
>> Can we do this on core-updates instead?  I'm currently working on a
>> core-updates branch.  I will push it soon, after I've done some basic
>> testing on it.
>
> I would prefer a separate branch - with hydra lagging behind, we tend
> to build only the core packages on core-updates, so I think we would not
> even see the result of a boost update.

Okay.

> On Fri, Jul 10, 2015 at 05:28:10PM -0400, Mark H Weaver wrote:
>> I would also prefer to prioritize getting this openssl security fix
>> deployed, if you don't mind.  Toward that end, I'm going to delete the
>> jobset on Hydra for now.  If you disagree, can we talk about it?
>
> No problem; I was just in a hurry because I am leaving for a week.
> I just created a clean wip-boost branch with the two commits (boost update
> and patch) applied on top of the current master. Once openssl-update has
> built, could you please rebase wip-boost and try to have it built on hydra?

Will do.

    Thanks,
      Mark

Re: 01/01: gnu: boost: Update to 1.58.0.

[Mark H Weaver]

Andreas Enge <andreas@enge.fr> writes:

> I will push this to a wip-boost branch, and try to build a dependent package
> locally. I wonder if I should base wip-boost on openssl-update; but with
> only 69 dependent packages (if the count is true), it probably does not
> matter.

I see that you pushed this branch and asked Hydra to build it.  First, I
don't think that we should be building a new branch against a version of
openssl with a known security flaw.  It will only have to be immediately
rebuilt again.

I would also prefer to prioritize getting this openssl security fix
deployed, if you don't mind.  Toward that end, I'm going to delete the
jobset on Hydra for now.  If you disagree, can we talk about it?

      Mark