“Which important packages fail to build?”

“Which important packages fail to build?”

[Ludovic Courtès]

Hello Guix!

I’ve just added a new ‘--coverage’ option to ‘guix weather’.  The goal
is to answer the question: “which important packages fail to build?”, or
at least, “which important packages have no substitutes?”.  I believe
this is what we want to know in particular to determine whether a branch
can be merged.  Demonstration:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env  guix weather --substitute-urls=https://ci.guix.info -c 10
computing 8,983 package derivations for x86_64-linux...
looking for 9,343 store items on https://ci.guix.info...
updating substitutes from 'https://ci.guix.info'... 100.0%
https://ci.guix.info
  64.7% substitutes available (6,048 out of 9,343)
  18,642.5 MiB of nars (compressed)
  54,068.6 MiB on disk (uncompressed)
  0.018 seconds per request (164.6 seconds in total)
  56.8 requests per second
  'https://ci.guix.info/api/queue?nr=1000' returned 504 ("Gateway Time-out")
updating substitutes from 'https://ci.guix.info'... 100.0%
2502 packages are missing from 'https://ci.guix.info' for 'x86_64-linux', among which:
    58  kcoreaddons@5.49.0      /gnu/store/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
    47  kcoreaddons@5.49.0      /gnu/store/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
    46  qgpgme@1.11.1   /gnu/store/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1
    37  perl-http-cookiejar@0.008       /gnu/store/2npd0vs3ipyqms06kbgh7ialp9n0fr6m-perl-http-cookiejar-0.008
    30  qgpgme@1.11.1   /gnu/store/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1
    26  ocaml4.02-ppx-deriving@4.1      /gnu/store/siv42br9h06i1zhmmj6s2qian45zacgm-ocaml4.02-ppx-deriving-4.1
    18  ruby-ansi@1.5.0 /gnu/store/p5adbp7bhvjzpjxr8brnfm52imqc4f07-ruby-ansi-1.5.0
    16  cl-rt@1990.12.19        /gnu/store/vs9ddw7avffmnzxcknvyazay29vihy28-cl-rt-1990.12.19
    16  ecl-rt@1990.12.19       /gnu/store/765znwnf4bxdaz82akx60pkx2ac59fyn-ecl-rt-1990.12.19
    13  ecl-trivial-gray-streams@0.0.0-1.0483ade        /gnu/store/lyj6w0p9v3kfkzmar1pd46qlg36zp7dd-ecl-trivial-gray-streams-0.0.0-1.0483ade
    11  ruby-brass@1.2.1        /gnu/store/jxf63lbwv2rg95pf5zmznjzv8s367xqc-ruby-brass-1.2.1
    11  ruby-rspec-expectations@2.14.5  /gnu/store/6dacx02mn2skwbiajfwnx1q0fk5iw05y-ruby-rspec-expectations-2.14.5
    11  ruby-rspec-mocks@2.14.6 /gnu/store/9lirirccwslkghpps1al23m7pdi5yw1b-ruby-rspec-mocks-2.14.6
    11  ruby-rspec-core@2.14.8  /gnu/store/xrpaql4jldvvd08im37x8k42crsxpip0-ruby-rspec-core-2.14.8
    11  ecl-slynk-boot0@1.0.0-beta-2.cbf84c3    /gnu/store/l23p319knfzygsc38zxf7l5g892bqy1z-ecl-slynk-boot0-1.0.0-beta-2.cbf84c3
    11  python2-oslotest@3.4.0  /gnu/store/7f87w09iinxv06dld394x7w47chcy3ip-python2-oslotest-3.4.0
    10  pt-scotch@6.0.5a        /gnu/store/gw4mzjlwlf010ijhyznhr5sjmpxizl9v-pt-scotch-6.0.5a
--8<---------------cut here---------------end--------------->8---

What we see here is that on current master, kcoreaddons and (presumably)
the 115 packages that depend on it¹ have no substitutes.  Likewise for
qgpgme and its 76 dependents.

As it turns out, these two packages fail to build:

  https://berlin.guixsd.org/log/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
  https://berlin.guixsd.org/log/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1

If you fix them, you’ll unlock no less than 191 packages and surely
fellow hackers will acclaim you when you arrive at the Guix Days.  :-)

In some cases, like ‘ocaml4.02-ppx-deriving’, it seems the package
simply hasn’t been built, for obscure reasons that build farm admins
should investigate.

Anyway, I think we should start looking at info, in particular so we can
finally get our act together and merge ‘staging’!

Feedback welcome!

Ludo’.

¹ Why do kcoreaddons and qgpgme appear twice?  Because there are two
  distinct (in the sense of ‘eq?’) package objects for each of these,
  even though they map to the same derivation.  The redundant package is
  introduced by ‘package-input-rewriting’ in kde-frameworks.scm.  Oh well!

Re: “Which important packages fail to build?”

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> I’ve just added a new ‘--coverage’ option to ‘guix weather’.  The goal
> is to answer the question: “which important packages fail to build?”, or
> at least, “which important packages have no substitutes?”.  I believe
> this is what we want to know in particular to determine whether a branch
> can be merged.

Very nice!  This will be very useful.

> As it turns out, these two packages fail to build:
>
>   https://berlin.guixsd.org/log/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
>   https://berlin.guixsd.org/log/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1

I just fixed the qgpgme test failure on master and staging.

-- 
Ricardo

Re: “Which important packages fail to build?”

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 100 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Feedback welcome!

This looks great, thanks Ludo!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: “Which important packages fail to build?”

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

>> As it turns out, these two packages fail to build:
>>
>>   https://berlin.guixsd.org/log/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
>>   https://berlin.guixsd.org/log/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1
>
> I just fixed the qgpgme test failure on master and staging.

Woohoo!  Now *you* will be acclaimed when we meet in Brussels!

Ludo’.

Re: “Which important packages fail to build?”

[Joshua Branson]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello Guix!
>
> I’ve just added a new ‘--coverage’ option to ‘guix weather’.  The goal
> is to answer the question: “which important packages fail to build?”, or
> at least, “which important packages have no substitutes?”.  I believe
> this is what we want to know in particular to determine whether a branch
> can be merged.  Demonstration:
>
> $ ./pre-inst-env  guix weather --substitute-urls=https://ci.guix.info -c 10
> computing 8,983 package derivations for x86_64-linux...
> looking for 9,343 store items on https://ci.guix.info...
> updating substitutes from 'https://ci.guix.info'... 100.0%
> https://ci.guix.info
>   64.7% substitutes available (6,048 out of 9,343)
>   18,642.5 MiB of nars (compressed)
>   54,068.6 MiB on disk (uncompressed)
>   0.018 seconds per request (164.6 seconds in total)
>   56.8 requests per second
>   'https://ci.guix.info/api/queue?nr=1000' returned 504 ("Gateway Time-out")
> updating substitutes from 'https://ci.guix.info'... 100.0%
> 2502 packages are missing from 'https://ci.guix.info' for 'x86_64-linux', among which:
>     58  kcoreaddons@5.49.0      /gnu/store/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
>     47  kcoreaddons@5.49.0      /gnu/store/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
>     46  qgpgme@1.11.1   /gnu/store/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1
>     37  perl-http-cookiejar@0.008       /gnu/store/2npd0vs3ipyqms06kbgh7ialp9n0fr6m-perl-http-cookiejar-0.008
>     30  qgpgme@1.11.1   /gnu/store/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1
>     26  ocaml4.02-ppx-deriving@4.1      /gnu/store/siv42br9h06i1zhmmj6s2qian45zacgm-ocaml4.02-ppx-deriving-4.1
>     18  ruby-ansi@1.5.0 /gnu/store/p5adbp7bhvjzpjxr8brnfm52imqc4f07-ruby-ansi-1.5.0
>     16  cl-rt@1990.12.19        /gnu/store/vs9ddw7avffmnzxcknvyazay29vihy28-cl-rt-1990.12.19
>     16  ecl-rt@1990.12.19       /gnu/store/765znwnf4bxdaz82akx60pkx2ac59fyn-ecl-rt-1990.12.19
>     13  ecl-trivial-gray-streams@0.0.0-1.0483ade        /gnu/store/lyj6w0p9v3kfkzmar1pd46qlg36zp7dd-ecl-trivial-gray-streams-0.0.0-1.0483ade
>     11  ruby-brass@1.2.1        /gnu/store/jxf63lbwv2rg95pf5zmznjzv8s367xqc-ruby-brass-1.2.1
>     11  ruby-rspec-expectations@2.14.5  /gnu/store/6dacx02mn2skwbiajfwnx1q0fk5iw05y-ruby-rspec-expectations-2.14.5
>     11  ruby-rspec-mocks@2.14.6 /gnu/store/9lirirccwslkghpps1al23m7pdi5yw1b-ruby-rspec-mocks-2.14.6
>     11  ruby-rspec-core@2.14.8  /gnu/store/xrpaql4jldvvd08im37x8k42crsxpip0-ruby-rspec-core-2.14.8
>     11  ecl-slynk-boot0@1.0.0-beta-2.cbf84c3    /gnu/store/l23p319knfzygsc38zxf7l5g892bqy1z-ecl-slynk-boot0-1.0.0-beta-2.cbf84c3
>     11  python2-oslotest@3.4.0  /gnu/store/7f87w09iinxv06dld394x7w47chcy3ip-python2-oslotest-3.4.0
>     10  pt-scotch@6.0.5a        /gnu/store/gw4mzjlwlf010ijhyznhr5sjmpxizl9v-pt-scotch-6.0.5a

As an aside, should Icecat be on this list?  I keep having to use "guix package
--do-not-upgrade=icecat -u", because Icecat takes forever to build, and
I'm not certain that I have enough memory to build it.

However,

#BEGIN_SRC sh
icecat --version
#END_SRC

#+:RESULTS:
GNU IceCat 60.3.0


BUT why does guix package -u try to rebuild Icecat?

#BEGIN_SRC sh
guix package -u
guix package: warning: package 'jmacs' no longer exists
substitute: updating substitutes from 'https://ci.guix.info'... 100.0%
building /gnu/store/r583x6jd6qlb7im42c0awmj6jvi1g13c-icecat-60.3.0-gnu1.drv...
#END_SRC


P.S. jmacs is my attempt to have a "custom" Emacs.  It's basically Emacs
with additional modules installed.

Thanks,

Joshua

--
Joshua Branson
Sent from Emacs and Gnus

Re: “Which important packages fail to build?”

[John Soo]

Hi,

I think icecat is ok. I’m probably wrong about this but what I understand is that icecat is being rebuilt after the rust bootstrap got merged. Upgrading icecat now requires building several versions of the rust compiler. I hope a substitute becomes available soon since it really does take a long time. I have a pretty nice set of hardware and it took nearly a day. I can’t imagine on an older machine.

- John

> On Jan 26, 2019, at 8:57 AM, Joshua Branson <jbranso@dismail.de> wrote:
> 
> Ludovic Courtès <ludo@gnu.org> writes:
> 
>> Hello Guix!
>> 
>> I’ve just added a new ‘--coverage’ option to ‘guix weather’.  The goal
>> is to answer the question: “which important packages fail to build?”, or
>> at least, “which important packages have no substitutes?”.  I believe
>> this is what we want to know in particular to determine whether a branch
>> can be merged.  Demonstration:
>> 
>> $ ./pre-inst-env  guix weather --substitute-urls=https://ci.guix.info -c 10
>> computing 8,983 package derivations for x86_64-linux...
>> looking for 9,343 store items on https://ci.guix.info...
>> updating substitutes from 'https://ci.guix.info'... 100.0%
>> https://ci.guix.info
>>  64.7% substitutes available (6,048 out of 9,343)
>>  18,642.5 MiB of nars (compressed)
>>  54,068.6 MiB on disk (uncompressed)
>>  0.018 seconds per request (164.6 seconds in total)
>>  56.8 requests per second
>>  'https://ci.guix.info/api/queue?nr=1000' returned 504 ("Gateway Time-out")
>> updating substitutes from 'https://ci.guix.info'... 100.0%
>> 2502 packages are missing from 'https://ci.guix.info' for 'x86_64-linux', among which:
>>    58  kcoreaddons@5.49.0      /gnu/store/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
>>    47  kcoreaddons@5.49.0      /gnu/store/8v69bzi29rla9q952ajgpsvsb7k955l6-kcoreaddons-5.49.0
>>    46  qgpgme@1.11.1   /gnu/store/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1
>>    37  perl-http-cookiejar@0.008       /gnu/store/2npd0vs3ipyqms06kbgh7ialp9n0fr6m-perl-http-cookiejar-0.008
>>    30  qgpgme@1.11.1   /gnu/store/rgvjzck002diandc1nhkw0dy1p9zqaw2-qgpgme-1.11.1
>>    26  ocaml4.02-ppx-deriving@4.1      /gnu/store/siv42br9h06i1zhmmj6s2qian45zacgm-ocaml4.02-ppx-deriving-4.1
>>    18  ruby-ansi@1.5.0 /gnu/store/p5adbp7bhvjzpjxr8brnfm52imqc4f07-ruby-ansi-1.5.0
>>    16  cl-rt@1990.12.19        /gnu/store/vs9ddw7avffmnzxcknvyazay29vihy28-cl-rt-1990.12.19
>>    16  ecl-rt@1990.12.19       /gnu/store/765znwnf4bxdaz82akx60pkx2ac59fyn-ecl-rt-1990.12.19
>>    13  ecl-trivial-gray-streams@0.0.0-1.0483ade        /gnu/store/lyj6w0p9v3kfkzmar1pd46qlg36zp7dd-ecl-trivial-gray-streams-0.0.0-1.0483ade
>>    11  ruby-brass@1.2.1        /gnu/store/jxf63lbwv2rg95pf5zmznjzv8s367xqc-ruby-brass-1.2.1
>>    11  ruby-rspec-expectations@2.14.5  /gnu/store/6dacx02mn2skwbiajfwnx1q0fk5iw05y-ruby-rspec-expectations-2.14.5
>>    11  ruby-rspec-mocks@2.14.6 /gnu/store/9lirirccwslkghpps1al23m7pdi5yw1b-ruby-rspec-mocks-2.14.6
>>    11  ruby-rspec-core@2.14.8  /gnu/store/xrpaql4jldvvd08im37x8k42crsxpip0-ruby-rspec-core-2.14.8
>>    11  ecl-slynk-boot0@1.0.0-beta-2.cbf84c3    /gnu/store/l23p319knfzygsc38zxf7l5g892bqy1z-ecl-slynk-boot0-1.0.0-beta-2.cbf84c3
>>    11  python2-oslotest@3.4.0  /gnu/store/7f87w09iinxv06dld394x7w47chcy3ip-python2-oslotest-3.4.0
>>    10  pt-scotch@6.0.5a        /gnu/store/gw4mzjlwlf010ijhyznhr5sjmpxizl9v-pt-scotch-6.0.5a
> 
> As an aside, should Icecat be on this list?  I keep having to use "guix package
> --do-not-upgrade=icecat -u", because Icecat takes forever to build, and
> I'm not certain that I have enough memory to build it.
> 
> However,
> 
> #BEGIN_SRC sh
> icecat --version
> #END_SRC
> 
> #+:RESULTS:
> GNU IceCat 60.3.0
> 
> 
> BUT why does guix package -u try to rebuild Icecat?
> 
> #BEGIN_SRC sh
> guix package -u
> guix package: warning: package 'jmacs' no longer exists
> substitute: updating substitutes from 'https://ci.guix.info'... 100.0%
> building /gnu/store/r583x6jd6qlb7im42c0awmj6jvi1g13c-icecat-60.3.0-gnu1.drv...
> #END_SRC
> 
> 
> P.S. jmacs is my attempt to have a "custom" Emacs.  It's basically Emacs
> with additional modules installed.
> 
> Thanks,
> 
> Joshua
> 
> --
> Joshua Branson
> Sent from Emacs and Gnus
> 

Re: “Which important packages fail to build?”

[Ricardo Wurmus]

Hi Joshua,

> As an aside, should Icecat be on this list?

This list is generated automatically.  Icecat is a leaf package —
nothing else depends on it.

> I keep having to use "guix package
> --do-not-upgrade=icecat -u", because Icecat takes forever to build, and
> I'm not certain that I have enough memory to build it.
>
> However,
>
> #BEGIN_SRC sh
> icecat --version
> #END_SRC
>
> #+:RESULTS:
> GNU IceCat 60.3.0
>
>
> BUT why does guix package -u try to rebuild Icecat?

Having a name and a version is not sufficient.  Guix is only concerned
with derivations and their contents.  So if any dependency of Icecat has
changed this causes a rebuild of Icecat, even if you already happen to
have a different variant of it that shares the name and version string.

--
Ricardo

Re: “Which important packages fail to build?”

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

>> As an aside, should Icecat be on this list?
>
> This list is generated automatically.  Icecat is a leaf package —
> nothing else depends on it.

It’s true that some leaf packages are “more important” than others, and
IceCat is one of them, but we usually check for these manually.

This list is about giving an overview of the state of the whole package
collection.

Ludo’.