Let's fix core-updates!

Let's fix core-updates!

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

Hi everyone!

Currently, 13% of builds on core-updates fail:

  https://hydra.gnu.org/jobset/gnu/core-updates

We need to fix this to help Ricardo prepare for the next release.
Questions:

  1) When is core-updates "done"?  Do we merge once we're below a
     specific failure rate, once specific bugs have been fixed, or a
     combination of the two?

  2) How shall we prioritize and divvy up work for fixing the failures?
     I'm guessing people just need to volunteer and start debugging!

  3) Are there any tools to help us understand what the failures might
     have in common?  E.g., if half the failures occur because a package
     deep in the dependency graph fails to build, clearly that package
     should be prioritized for fixing.  I suppose we'll learn about
     commonalities as we go, but it'd be nice if there were a tool that
     might help us understand what to focus on first.

  4) What other bugs/features need to be addressed to un-block release?

I know that we want to update the default JDK used by Java packages from
7 to 8, but there are probably more important tasks to finish up, also.

Let's get started!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Let's fix core-updates!

[Pjotr Prins]

On Sat, Feb 10, 2018 at 03:51:59AM +0100, Chris Marusich wrote:
> Hi everyone!
> 
> Currently, 13% of builds on core-updates fail:
> 
>   https://hydra.gnu.org/jobset/gnu/core-updates

It is quite a list. I think we should purge packages that have been
failing for a longish time - apparently no one cares about those.

Maybe list the more important ones in debbugs as a first step and tag
them as core-updates-fail or something? Or does that overflow debbugs?
Main reason is that we now track things in two places. On debbugs you
can assign it.

Pj.

Re: Let's fix core-updates!

[Ricardo Wurmus]

Hi Chris,

thank you for taking the initiative!

> Currently, 13% of builds on core-updates fail:
>
>   https://hydra.gnu.org/jobset/gnu/core-updates

This may be a better URL:

   https://hydra.gnu.org/eval/109908?full=1&compare=master

It is a list of build failures that happen on core-updates but not on
master.  While the growing list of packages that fail to build is a
concern it is tangential to getting core-updates merged.

>   1) When is core-updates "done"?  Do we merge once we're below a
>      specific failure rate, once specific bugs have been fixed, or a
>      combination of the two?

We usually just declare it done when there are no more “big failures”.
Looking at the list of ~500 failures I see that we still have failures
that are responsible for a lot of other failures.

Take “sbcl” for example.  It is marked red, which means that it failed
not because one of its dependencies failed to build, but because there’s
something wrong with it.  Sbcl is needed for all the “sbcl-*” packages,
which are also needed for “uglify-js”, which in turn is needed for
pretty much all of the failing “r-*” packages and for all of the “js-*”
packages.

So, fixing the “sbcl” package appears to be important.

The next thing that stands out to me is the long list of “java-*”
packages that are marked brown, indicating that a dependency failed to
build.  Turns out that “java-hamcrest-all” fails to build.  The build
error is … strange.  But maybe we can ignore the Java stuff if we can
merge the dedicated Java updates branch right after merging
core-updates.

>   2) How shall we prioritize and divvy up work for fixing the failures?
>      I'm guessing people just need to volunteer and start debugging!

Only the red packages really need investigation.

I’ll take a look at these packages:

- augeas
- knights
- python-ipy
- sbcl

For sbcl we may just be able to upgrade to the latest version.  I’m
going to give that a try.

>   3) Are there any tools to help us understand what the failures might
>      have in common?  E.g., if half the failures occur because a package
>      deep in the dependency graph fails to build, clearly that package
>      should be prioritized for fixing.  I suppose we'll learn about
>      commonalities as we go, but it'd be nice if there were a tool that
>      might help us understand what to focus on first.

I don’t know of any.  We can look at the logs of brown packages to see
if we can detect any common dependency failures (as with sbcl above) and
we can look at the output of guix graph.  But there’s nothing else I can
think of.

>   4) What other bugs/features need to be addressed to un-block release?

The next release primarily depends on core-updates.  As a bonus we can
fix a couple more bugs in the bug tracker.  I don’t expect us to cut a
new release right after the merge of core-updates, but that’s the
biggest part of it.

> I know that we want to update the default JDK used by Java packages from
> 7 to 8, but there are probably more important tasks to finish up, also.

Yes, I’d like to get this branch merged before the new release, because
it’s not far from completion.

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Let's fix core-updates!

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

>> Currently, 13% of builds on core-updates fail:
>>
>>   https://hydra.gnu.org/jobset/gnu/core-updates
>
> This may be a better URL:
>
>    https://hydra.gnu.org/eval/109908?full=1&compare=master

If you cannot access this because hydra is timing out, then try this
copy:

    https://elephly.net/downies/hydra.mhtml

-- 
Ricardo
  
GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Let's fix core-updates!

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 4997 bytes --]

Chris Marusich <cmmarusich@gmail.com> writes:

> Hi everyone!
>
> Currently, 13% of builds on core-updates fail:
>
>   https://hydra.gnu.org/jobset/gnu/core-updates
>
> We need to fix this to help Ricardo prepare for the next release.
> Questions:
>
>   1) When is core-updates "done"?  Do we merge once we're below a
>      specific failure rate, once specific bugs have been fixed, or a
>      combination of the two?

Typically a combination.  Packages with many dependents usually get
fixed first, then once we're down to mostly "leaf" packages, if they are
not hugely popular (say GNOME, or SLiM), we can merge and deal with the
remainder on 'master'.

Right now, the biggest blockers are:

* Shepherd on armhf: <https://hydra.gnu.org/job/gnu/core-updates/shepherd-0.3.2.armhf-linux>

It's an indeterministic failure, but happens nearly every build.  This
is fixed in Shepherd git, but not yet in Guix.  Once the Hydra build
succeeds, we need to restart the builds for all of Shepherds dependent
packages in order to get substitutes for them.

At this point, we might consider applying the upstream fix directly for
armhf only.  Then we don't have to manually restart the dependent jobs,
since that would create a new derivation and Hydra will make new jobs.

* ruby-sqlite3: <https://hydra.gnu.org/job/gnu/core-updates/ruby-sqlite3-1.3.13.x86_64-linux>

I filed an issue upstream here:
<https://github.com/sparklemotion/sqlite3-ruby/issues/226>.  It seems
harmless, so I guess we can s/integer/INTEGER/ and move on.

* SBCL: <https://hydra.gnu.org/job/gnu/core-updates/sbcl-1.3.7.x86_64-linux>

I don't know anything about this package, so not sure what to do about
it.  It has 45 dependents, including MATE, so merging without it is not
great.  I tried updating to the latest version, which made the build
succeed, but then many of the dependent packages failed.

* python-django: <https://hydra.gnu.org/job/gnu/core-updates/python-django-1.10.8.x86_64-linux>

This version of Django is fairly old and riddled with security holes, so
I'm willing to "punt" on this and let Django users fix it when it hits
'master'.  Here I also tried updating to the latest LTS, but got some
odd and very difficult test failures.

In addition, there are many "leaf" Python packages that are failing,
which is typical when we go from 3.5->3.6.  Updating usually solves it.

Note that we have a bug with python2-flake8: if you see a python2
variant failing due to missing "enum34", it will have to be added as a
propagated-input until we can fix it in flake8 proper.  See these
commits:

<https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates&id=01af1e11a67466d5f6338bdb207ff99ef5cf094e>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates&id=b7049b2e23a514a1d9c131af484e808ee0ee4226>
<https://git.savannah.gnu.org/cgit/guix.git/commit/?h=core-updates&id=53f826cd0f429864d46fc3bf6305c14356d0b2ad>

Most of the currently failing Python packages can be fixed after the
merge IMO, as they don't have a lot of dependents.  But don't let that
stop anyone from fixing them now :-)

>   2) How shall we prioritize and divvy up work for fixing the failures?
>      I'm guessing people just need to volunteer and start debugging!

I encourage everyone to check the Hydra core-updates page every now and
then, press "compare" and choose "master" and go on the "currently
failing" tab.  If you recognize any of the failing packages, please try
to fix them :-)

Here is the comparison for the latest evaluation (this might not load
since Hydra is currently very busy):
<https://hydra.gnu.org/eval/109908?compare=109907&full=1#tabs-now-fail>.

>   3) Are there any tools to help us understand what the failures might
>      have in common?  E.g., if half the failures occur because a package
>      deep in the dependency graph fails to build, clearly that package
>      should be prioritized for fixing.  I suppose we'll learn about
>      commonalities as we go, but it'd be nice if there were a tool that
>      might help us understand what to focus on first.

On the "currently failing" tab, only those with a bright red icon are
failing.  Most of the jobs have an orange icon which means a dependency
failed.  Clicking on the job will show you which dependency that is.

>   4) What other bugs/features need to be addressed to un-block release?

I believe most of the "hard" problems this round have been addressed, so
now we just need the fixes mentioned above.  That said, please try to
pull core-updates on your systems to make sure :-)

> I know that we want to update the default JDK used by Java packages from
> 7 to 8, but there are probably more important tasks to finish up, also.

We don't have a lot of Java dependents still, so IMO this can be done on
'master' (if everything builds) or maybe 'staging' for testing all arches.

> Let's get started!

I'm going on a road trip for a few days, so thanks for picking up this
thread.  I hope this branch is merged when I get back!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Let's fix core-updates!

[Ricardo Wurmus]

Marius Bakke <mbakke@fastmail.com> writes:

> * SBCL: <https://hydra.gnu.org/job/gnu/core-updates/sbcl-1.3.7.x86_64-linux>
>
> I don't know anything about this package, so not sure what to do about
> it.  It has 45 dependents, including MATE, so merging without it is not
> great.  I tried updating to the latest version, which made the build
> succeed, but then many of the dependent packages failed.

Oh, I updated sbcl on core-updates before seeing your message.
I’ll take a look at the dependent packages tomorrow.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Let's fix core-updates!

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> Marius Bakke <mbakke@fastmail.com> writes:
>
>> * SBCL: <https://hydra.gnu.org/job/gnu/core-updates/sbcl-1.3.7.x86_64-linux>
>>
>> I don't know anything about this package, so not sure what to do about
>> it.  It has 45 dependents, including MATE, so merging without it is not
>> great.  I tried updating to the latest version, which made the build
>> succeed, but then many of the dependent packages failed.
>
> Oh, I updated sbcl on core-updates before seeing your message.
> I’ll take a look at the dependent packages tomorrow.

Actually, I just built r-shiny (which depends on uglify-js, which
depends on sbcl) without problems.  Also built sbcl-stumpwm, so I think
sbcl is really fine now.

-- 
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Let's fix core-updates!

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]

On Sat, Feb 10, 2018 at 11:26:03PM +0100, Ricardo Wurmus wrote:
> 
> Ricardo Wurmus <rekado@elephly.net> writes:
> 
> > Marius Bakke <mbakke@fastmail.com> writes:
> >
> >> * SBCL: <https://hydra.gnu.org/job/gnu/core-updates/sbcl-1.3.7.x86_64-linux>
> >>
> >> I don't know anything about this package, so not sure what to do about
> >> it.  It has 45 dependents, including MATE, so merging without it is not
> >> great.  I tried updating to the latest version, which made the build
> >> succeed, but then many of the dependent packages failed.
> >
> > Oh, I updated sbcl on core-updates before seeing your message.
> > I’ll take a look at the dependent packages tomorrow.
> 
> Actually, I just built r-shiny (which depends on uglify-js, which
> depends on sbcl) without problems.  Also built sbcl-stumpwm, so I think
> sbcl is really fine now.
> 

sbcl currently doesn't build on aarch64, but this is no change from
current master. clisp is about to get a version bump but we can deal
with that when its actually released. Sounds like this part is good.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]