Planning for a release, for real

Planning for a release, for real

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 3027 bytes --]

Hello Guix!

Will Guix’s 10th year be a release year?  I hope so!

We need to plan and coordinate.  Releases have to be a group effort;
some of the most important work won’t be coding but coordination.
Coordination is key.  I don’t think I should be spearheading that
effort, but I’m happy to be part of it.

Who’s ready to commit time towards that goal for the coming weeks?

Here’s a list of things to do to get there:

  • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
    a couple of weeks ago, but then I lost track of it.  Marius?

    We need a ‘staging’ champion to keep track of what’s left to be
    done, reports progress, pings people, etc.  That person does not
    have to be hacking like crazy, on the contrary!

  • Get base binaries on all supported architectures in a timely
    fashion, or drop some of the architectures.

    Namely, ‘make assert-binaries-available’ is currently failing.  It
    uses a manifest that encodes what we consider to be the basic
    requirements for each architecture; it’s not demanding for
    aarch64-linux, even less for armhf-linux and i586-gnu—yet we’re not
    meeting these criteria yet.

    We need to look at missing substitutes, address build issues and
    build farm issues that cause them until we get to zero failures.  If
    after some effort we fail to get to zero, then we should consider
    dropping architectures (I’m looking at armhf-linux and i586-gnu
    specifically).

    Again we need a champion to keep track of this and ping people so we
    make progress!

  • Address the blockers of <https://issues.guix.gnu.org/53214>, most of
    which are issues in the installer.

  • Freeze strings: enter a period where translatable strings in the
    code and in the manual must not be changed so translators have a
    chance to keep up.  Julien, how would you like to do that?  Weblate
    has given us more flexibility it seems.

  • Publish a release candidate and call for testing of the installer in
    particular.  Fix bugs, loop.

  • Update NEWS (mostly done already!), prepare a blog post listing the
    highlights and linking to the relevant material.  (See
    <https://guix.gnu.org/en/blog/tags/releases/> for inspiration.)

I’d like us to do this with an eye of getting better organized, which
involves defining roles such as that of “release managers”.

The NixOS folks handle this in a way that I find inspiring, with
rotating release manager responsibilities, a schedule announced upfront,
and a detailed description of the process:

  https://github.com/NixOS/nixpkgs/issues/193585
  https://nixos.github.io/release-wiki/Home.html

We have
<https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/release.org>
but it’s low-level and dates back to a time where release were a
one-person activity.  Time for a change.

So, who’s in?  Let’s get our act together!

Ludo’.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 853 bytes --]

Re: Planning for a release, for real

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 3653 bytes --]

I'll take care of the cranslations (notifying translators, ensuring string freeze is respected, …)

We need to be careful not to start the stsing freeze step too early. Last time (or previous?) we started a week before the scheduled release date, but the schedule slipped by a few weeks and we had some pressure in the pipeline because some patches could not be applied because of string changes.

Let's try to have a better vision on the planning this time :)

Le 6 octobre 2022 16:50:22 GMT+02:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hello Guix!
>
>Will Guix’s 10th year be a release year?  I hope so!
>
>We need to plan and coordinate.  Releases have to be a group effort;
>some of the most important work won’t be coding but coordination.
>Coordination is key.  I don’t think I should be spearheading that
>effort, but I’m happy to be part of it.
>
>Who’s ready to commit time towards that goal for the coming weeks?
>
>Here’s a list of things to do to get there:
>
>  • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
>    a couple of weeks ago, but then I lost track of it.  Marius?
>
>    We need a ‘staging’ champion to keep track of what’s left to be
>    done, reports progress, pings people, etc.  That person does not
>    have to be hacking like crazy, on the contrary!
>
>  • Get base binaries on all supported architectures in a timely
>    fashion, or drop some of the architectures.
>
>    Namely, ‘make assert-binaries-available’ is currently failing.  It
>    uses a manifest that encodes what we consider to be the basic
>    requirements for each architecture; it’s not demanding for
>    aarch64-linux, even less for armhf-linux and i586-gnu—yet we’re not
>    meeting these criteria yet.
>
>    We need to look at missing substitutes, address build issues and
>    build farm issues that cause them until we get to zero failures.  If
>    after some effort we fail to get to zero, then we should consider
>    dropping architectures (I’m looking at armhf-linux and i586-gnu
>    specifically).
>
>    Again we need a champion to keep track of this and ping people so we
>    make progress!
>
>  • Address the blockers of <https://issues.guix.gnu.org/53214>, most of
>    which are issues in the installer.
>
>  • Freeze strings: enter a period where translatable strings in the
>    code and in the manual must not be changed so translators have a
>    chance to keep up.  Julien, how would you like to do that?  Weblate
>    has given us more flexibility it seems.
>
>  • Publish a release candidate and call for testing of the installer in
>    particular.  Fix bugs, loop.
>
>  • Update NEWS (mostly done already!), prepare a blog post listing the
>    highlights and linking to the relevant material.  (See
>    <https://guix.gnu.org/en/blog/tags/releases/> for inspiration.)
>
>I’d like us to do this with an eye of getting better organized, which
>involves defining roles such as that of “release managers”.
>
>The NixOS folks handle this in a way that I find inspiring, with
>rotating release manager responsibilities, a schedule announced upfront,
>and a detailed description of the process:
>
>  https://github.com/NixOS/nixpkgs/issues/193585
>  https://nixos.github.io/release-wiki/Home.html
>
>We have
><https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/release.org>
>but it’s low-level and dates back to a time where release were a
>one-person activity.  Time for a change.
>
>So, who’s in?  Let’s get our act together!
>
>Ludo’.

[-- Attachment #2: Type: text/html, Size: 4327 bytes --]

Re: Planning for a release, for real

[Ludovic Courtès]

Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

> I'll take care of the cranslations (notifying translators, ensuring string freeze is respected, …)

Great, thanks.

> We need to be careful not to start the stsing freeze step too early. Last time (or previous?) we started a week before the scheduled release date, but the schedule slipped by a few weeks and we had some pressure in the pipeline because some patches could not be applied because of string changes.
>
> Let's try to have a better vision on the planning this time :)

Right!  How long do you think we need for the string freeze?

I feel that thanks to Weblate translations are continuously updated, so
maybe things can be frozen for a shorter period of time?

Thanks,
Ludo’.

Re: Planning for a release, for real

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

It depends on the language, I'd say a week is good, maybe less as long as we can include a week-end (maybe it's personal, but I find more time to contribute in the week-ends).

Le 7 octobre 2022 11:49:21 GMT+02:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hi,
>
>Julien Lepiller <julien@lepiller.eu> skribis:
>
>> I'll take care of the cranslations (notifying translators, ensuring string freeze is respected, …)
>
>Great, thanks.
>
>> We need to be careful not to start the stsing freeze step too early. Last time (or previous?) we started a week before the scheduled release date, but the schedule slipped by a few weeks and we had some pressure in the pipeline because some patches could not be applied because of string changes.
>>
>> Let's try to have a better vision on the planning this time :)
>
>Right!  How long do you think we need for the string freeze?
>
>I feel that thanks to Weblate translations are continuously updated, so
>maybe things can be frozen for a shorter period of time?
>
>Thanks,
>Ludo’.

[-- Attachment #2: Type: text/html, Size: 1591 bytes --]

Re: Planning for a release, for real

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 985 bytes --]

On 06-10-2022 16:50, Ludovic Courtès wrote:
> Hello Guix!
> 
> Will Guix’s 10th year be a release year?  I hope so!
> 
> We need to plan and coordinate.  Releases have to be a group effort;
> some of the most important work won’t be coding but coordination.
> Coordination is key.  I don’t think I should be spearheading that
> effort, but I’m happy to be part of it.
> 
> Who’s ready to commit time towards that goal for the coming weeks?
> 
> Here’s a list of things to do to get there: [...]

I have some unapplied security patches (from before the latest release 
(1.3.0) (!)) (more precisely, some patches that prepare for actually 
being able to write the security patches, once the preparation patches 
are merged, the actual security fixes should be relatively simple); I 
think it would be rather bad for known security fixes to not be applied 
for a whole release, especially given how long release cycles are in Guix.

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: Planning for a release, for real

[Ludovic Courtès]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> I have some unapplied security patches (from before the latest release
> (1.3.0) (!)) (more precisely, some patches that prepare for actually
> being able to write the security patches, once the preparation patches
> are merged, the actual security fixes should be relatively simple); I
> think it would be rather bad for known security fixes to not be
> applied for a whole release, especially given how long release cycles
> are in Guix.

Could you provide links to these?

If they depend on ‘openat’ in Guile, that won’t be an option
unfortunately since we’re not going to upgrade the ‘guile’ package
during that time frame.

Thanks,
Ludo’.

Re: Planning for a release, for real

[Maxime Devos]

[-- Attachment #1.1.1: Type: text/plain, Size: 1115 bytes --]



On 07-10-2022 11:50, Ludovic Courtès wrote:
> Hi,
> 
> Maxime Devos <maximedevos@telenet.be> skribis:
> 
>> I have some unapplied security patches (from before the latest release
>> (1.3.0) (!)) (more precisely, some patches that prepare for actually
>> being able to write the security patches, once the preparation patches
>> are merged, the actual security fixes should be relatively simple); I
>> think it would be rather bad for known security fixes to not be
>> applied for a whole release, especially given how long release cycles
>> are in Guix.
> 
> Could you provide links to these?

'These' are the ‘openat’ patches:

> If they depend on ‘openat’ in Guile, that won’t be an option
> unfortunately since we’re not going to upgrade the ‘guile’ package
> during that time frame.

However, upgrading the Guile package is not required, I proposed making 
a separate ‘guile-with-openat’ package (= current Guile + some patches), 
which can then be used by the service code: 
<https://issues.guix.gnu.org/54485> (without world rebuilds).

Greetings,
Maxime.

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 929 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Supported architectures

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3285 bytes --]

On Thu, Oct 06, 2022 at 04:50:22PM +0200, Ludovic Courtès wrote:
> Hello Guix!
> 
> Will Guix’s 10th year be a release year?  I hope so!
> 
> We need to plan and coordinate.  Releases have to be a group effort;
> some of the most important work won’t be coding but coordination.
> Coordination is key.  I don’t think I should be spearheading that
> effort, but I’m happy to be part of it.
> 
> Who’s ready to commit time towards that goal for the coming weeks?
> 
> Here’s a list of things to do to get there:
> 
>   • Get base binaries on all supported architectures in a timely
>     fashion, or drop some of the architectures.
> 
>     Namely, ‘make assert-binaries-available’ is currently failing.  It
>     uses a manifest that encodes what we consider to be the basic
>     requirements for each architecture; it’s not demanding for
>     aarch64-linux, even less for armhf-linux and i586-gnu—yet we’re not
>     meeting these criteria yet.
> 
>     We need to look at missing substitutes, address build issues and
>     build farm issues that cause them until we get to zero failures.  If
>     after some effort we fail to get to zero, then we should consider
>     dropping architectures (I’m looking at armhf-linux and i586-gnu
>     specifically).
> 
> So, who’s in?  Let’s get our act together!
> 
> Ludo’.

Firstly, I'd like to mention that we, in general, have a minimum system
requirement of 2GB of RAM, and IIRC there aren't a lot of armhf boards
out there which have that much. We do have a difference between building
natively and cross building / building with '--target'.

I'd like to comment on armhf for a moment. My memory is a but rusty, but
I'm pretty sure that in December of 2021 mesa was bumped from 21.2.x to
21.3.x, and at that time it stopped building on/for armhf. I noticed in
May of 2022 (5 months later) and got the build working again. That we
went 5 months without anyone saying anything in bug reports that mesa
wasn't building shows that either everyone who is using it is using
software that doesn't use mesa, or we really don't have any armhf-linux
users. I'm not advocating dropping the architecture, but it does feel
like we're already at a best-effort level with it. As far as the pieces
needed for bootstrapping aarch64 software (go and probably others),
those get built anyway as needed by aarch64, so there's no worry about
losing support for those software bits.

i586-gnu: Do we have a mini guide on how to setup a build environment?
Something like "add the childhurd service and the secrets service, with
these bits and you're all set to go"? I don't mind poking builds from
time to time, but I'm not sure about how to set it up.

aarch64-linux: I tried a while ago to fix a bunch of the failed builds
on ci.guix.gnu.org and I think I made it worse. Right now there are many
build failures and pending builds. I might see about canceling some of
them and then restarting individual builds to try to increase coverage
again.



-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Supported architectures

[Ludovic Courtès]

Hi,

Efraim Flashner <efraim@flashner.co.il> skribis:

> I'd like to comment on armhf for a moment. My memory is a but rusty, but
> I'm pretty sure that in December of 2021 mesa was bumped from 21.2.x to
> 21.3.x, and at that time it stopped building on/for armhf. I noticed in
> May of 2022 (5 months later) and got the build working again. That we
> went 5 months without anyone saying anything in bug reports that mesa
> wasn't building shows that either everyone who is using it is using
> software that doesn't use mesa, or we really don't have any armhf-linux
> users. I'm not advocating dropping the architecture, but it does feel
> like we're already at a best-effort level with it. As far as the pieces
> needed for bootstrapping aarch64 software (go and probably others),
> those get built anyway as needed by aarch64, so there's no worry about
> losing support for those software bits.

We’re definitely on a best-effort basis.  I think we should try and see
if we can make the armhf-linux selection that’s in
‘etc/release-manifests.scm’ build (I think it doesn’t involve Mesa).  If
we realize it’s out of reach, then we mark armhf-linux as in the same
state as mips64el-linux.

> i586-gnu: Do we have a mini guide on how to setup a build environment?
> Something like "add the childhurd service and the secrets service, with
> these bits and you're all set to go"? I don't mind poking builds from
> time to time, but I'm not sure about how to set it up.

Set up a childhurd and offload to it:

  https://guix.gnu.org/manual/devel/en/html_node/Virtualization-Services.html#index-childhurd_002c-offloading

The main issue at the moment is lack of substitutes, which is due to the
fact that childhurds won’t start on the build machines behind ci.guix.
I’m investigating this here:

  https://issues.guix.gnu.org/58320

Hopefully we’ll have a bug fix or a workaround soon.

> aarch64-linux: I tried a while ago to fix a bunch of the failed builds
> on ci.guix.gnu.org and I think I made it worse. Right now there are many
> build failures and pending builds. I might see about canceling some of
> them and then restarting individual builds to try to increase coverage
> again.

Right, currently there’s something wrong going on with aarch64-linux
builds on ci.guix, which Maxim and I have been looking at.  In
particular, some of the build machines are unreachable right now and may
need a kick.

The starting point for someone willing to help is this dashboard:

--8<---------------cut here---------------start------------->8---
$ make assert-binaries-available 
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
computing 400 package derivations for x86_64-linux...
looking for 512 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org ☀
  91.6% substitutes available (469 out of 512)
  at least 3,699.6 MiB of nars (compressed)
  6,450.3 MiB on disk (uncompressed)
  0.021 seconds per request (1.1 seconds in total)
  47.1 requests per second

  7.0% (3 out of 43) of the missing items are queued
  at least 1,000 queued builds
      powerpc64le-linux: 699 (69.9%)
      aarch64-linux: 301 (30.1%)
  build rate: 11.31 builds per hour
      i686-linux: 2.31 builds per hour
      x86_64-linux: 7.08 builds per hour
      aarch64-linux: 2.45 builds per hour
      powerpc64le-linux: 0.49 builds per hour
      armhf-linux: 0.02 builds per hour

Substitutes are missing for the following items:
  /gnu/store/pjxakb6iqarajwj5xxbl4xibd40zrwzs-gettext-0.21-doc                   x86_64-linux
  /gnu/store/04wlmb4mlbhm3hwpp7cwzk45cc8wr7bm-gettext-0.21                       x86_64-linux
  /gnu/store/8xjxp2gmg1l3db1n18iq0y8mgilf7faf-zlib-1.2.11-static                 x86_64-linux
  /gnu/store/xdvgaj90p1lr4wj0lvsn2m0sqcxs2zsk-xf86-video-sis-0.12.0              i686-linux
  /gnu/store/9gl116gdip5zkpzijk07j3kh3hkj21np-grub-hybrid-2.06                   i686-linux
  /gnu/store/55yxbr2fywkd6x3pzqhh0z22cl6kf3s3-libreoffice-7.3.5.2                i686-linux
  /gnu/store/v99pba09zv9m9wl3dgnrq4i26fkrragz-xf86-video-vesa-2.5.0              x86_64-linux
  /gnu/store/z47kyvcrxxq4xwczwlbnap9kfvvfigib-gnome-user-docs-42.0               x86_64-linux
  /gnu/store/475m6q7hp7a1gw98ki9l4g04gmvsa75y-xz-5.2.5                           i586-gnu
  /gnu/store/8fhdpb76nqy3v22jr33j18i1k07rl5n1-xz-5.2.5-static                    i586-gnu
  /gnu/store/6dvavfwg4zdih3rlvac4qqkx46my8gl6-tar-1.34                           i586-gnu
  /gnu/store/sa1ay72axmi9g75sll2wy9cqpfldfy40-gcc-toolchain-12.2.0-debug         i586-gnu
  /gnu/store/qxwclv8hx9z3gqwjil4hpwkwndh6f2zm-gcc-toolchain-12.2.0               i586-gnu
  /gnu/store/dypv1jgfzxlkbpp36z393xbdqal1v591-gcc-toolchain-12.2.0-static        i586-gnu
  /gnu/store/7bx9jykip9lc13yn2bck1m4q8ccds1mz-make-4.3-debug                     i586-gnu
  /gnu/store/422i4q46cisabwsxrs7raf67awwwzsys-make-4.3                           i586-gnu
  /gnu/store/f8jsczp72i49c79rjf8nv2q6jskqa5vy-gawk-5.1.0                         i586-gnu
  /gnu/store/d646qvpcdi0l9r2mqhqkxkrgwm0b50qh-findutils-4.8.0                    i586-gnu
  /gnu/store/zb0zbds0k2vjnln88dp4paldghl2mdwv-grep-3.6                           i586-gnu
  /gnu/store/62hb8sk7vnz26flasklrm0x0yh5pdnq4-coreutils-8.32-debug               i586-gnu
  /gnu/store/fmk805j58dig4076wy8q6fj1w47jxaw1-coreutils-8.32                     i586-gnu
  /gnu/store/g9jzhmhd841aqmmz2q7xqxq8v4269n9p-guix-1.3.0-30.17134b9              armhf-linux
  /gnu/store/nz1rw5cfrh4z3bl7fm2qsvxxpl955cqh-guile-3.0.8-debug                  armhf-linux
  /gnu/store/zmk1kmfk7wxm5w3ambajgnx7b0s5iq84-guile-3.0.8                        armhf-linux
  /gnu/store/26yb2pj71wg9cywmhpmsf6n1d81i43c5-python-3.9.9-idle                  armhf-linux
  /gnu/store/dh5rr8gd148afs3jzijs8i9gfwwi6igz-python-3.9.9                       armhf-linux
  /gnu/store/x0yzk738mm4if6kbc8i8q7x3ajz2rd27-python-3.9.9-tk                    armhf-linux
  /gnu/store/n7zl19njilpgisd80f63vwsli1gf8ykf-vim-9.0.0594                       armhf-linux
  /gnu/store/d4ab2pl3b3c7z9gcl1cb9c1xljh56wpj-emacs-no-x-28.1                    armhf-linux
  /gnu/store/7hwwpiplw46a3wjsfhl47g2d18zlyc26-openssh-8.9p1                      armhf-linux
  /gnu/store/5n3n87ybpmy3sh4k68xqv661wzs8zhs1-nss-certs-3.71                     armhf-linux
  /gnu/store/5nn8q80kywqvpzkhafpv3lppfbm5wm7n-bootstrap-tarballs-0               armhf-linux
  /gnu/store/vi0rxvwjkyds5dg5plqhzkrd3h0hmg96-guix-1.3.0-30.17134b9              i686-linux
  /gnu/store/7wx2f445piapz2mwii1mip9gsaxhmkqj-guix-1.3.0-30.17134b9              powerpc64le-linux
  /gnu/store/kpda3vy8qi2zv2lz7l9g736dv6w63yq0-vim-9.0.0594                       powerpc64le-linux
  /gnu/store/pgm8608mzhwxn86q48lpb77vp4pxp6g3-python-3.9.9-idle                  aarch64-linux
  /gnu/store/89qy7yb6c0wszq0686xcq4n66h5gy7mb-vim-9.0.0594                       aarch64-linux
  /gnu/store/fjqj99dav5wdqnmv2pdkr9y285sd5aam-emacs-28.1                         aarch64-linux
  /gnu/store/01k1v00g7vc7n90y5yr9bacrnr3ml46p-nss-certs-3.71                     aarch64-linux
  /gnu/store/pfk09jhc4fqalkv6bbv0cv7j00whydzm-gcc-toolchain-12.2.0-debug         aarch64-linux
  /gnu/store/kjcr6zmkh0gdraclp5v5kqqqsy4hdx9h-gcc-toolchain-12.2.0               aarch64-linux
  /gnu/store/ylhykr5g3yvbdarwb0h7smhhx2wga89m-gcc-toolchain-12.2.0-static        aarch64-linux
  /gnu/store/v14vx389rwshm5chr5llbrnjyrvgxbp7-bootstrap-tarballs-0               aarch64-linux
--8<---------------cut here---------------end--------------->8---

From there one can try to build those things that are failing and
gradually fix things.  For example, if you want to see why Vim is
missing on aarch64-linux, you can run:

  guix build $(guix gc --derivers /gnu/store/89qy7yb6c0wszq0686xcq4n66h5gy7mb-vim-9.0.0594)

(This is assuming you’re on an aarch64-linux machine or that you set up
offloading or transparent emulation with binfmt_misc.)

Sometimes things build but substitutes are missing, in which case it
ends up being infrastructure work.

Looking at these issues can be tedious, so let’s join forces and share
our progress on IRC to begin with!

Thanks,
Ludo’.

Re: Supported architectures

[Csepp]

Efraim Flashner <efraim@flashner.co.il> writes:

> Firstly, I'd like to mention that we, in general, have a minimum system
> requirement of 2GB of RAM, and IIRC there aren't a lot of armhf boards
> out there which have that much. We do have a difference between building
> natively and cross building / building with '--target'.

This really needs to be lowered IMHO.  2GB being the minimum should be
treated as a bug.

> I'd like to comment on armhf for a moment. My memory is a but rusty, but
> I'm pretty sure that in December of 2021 mesa was bumped from 21.2.x to
> 21.3.x, and at that time it stopped building on/for armhf. I noticed in
> May of 2022 (5 months later) and got the build working again. That we
> went 5 months without anyone saying anything in bug reports that mesa
> wasn't building shows that either everyone who is using it is using
> software that doesn't use mesa, or we really don't have any armhf-linux
> users. I'm not advocating dropping the architecture, but it does feel
> like we're already at a best-effort level with it. As far as the pieces
> needed for bootstrapping aarch64 software (go and probably others),
> those get built anyway as needed by aarch64, so there's no worry about
> losing support for those software bits.

Personally I'm not using Guix on my armhf machines *because* armhf is
buggy.  I would certainly love to use it, but right now postmarketOS is
sooo much better on armhf.  It would be great to be in a position where
I could submit bug reports from armhf but just doing it on i686 is
already a challenge.

Re: Supported architectures

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1978 bytes --]

On 2022-10-07, Efraim Flashner wrote:
> On Thu, Oct 06, 2022 at 04:50:22PM +0200, Ludovic Courtès wrote:

> Firstly, I'd like to mention that we, in general, have a minimum system
> requirement of 2GB of RAM, and IIRC there aren't a lot of armhf boards
> out there which have that much. We do have a difference between building
> natively and cross building / building with '--target'.
>
> I'd like to comment on armhf for a moment. My memory is a but rusty, but
> I'm pretty sure that in December of 2021 mesa was bumped from 21.2.x to
> 21.3.x, and at that time it stopped building on/for armhf. I noticed in
> May of 2022 (5 months later) and got the build working again. That we
> went 5 months without anyone saying anything in bug reports that mesa
> wasn't building shows that either everyone who is using it is using
> software that doesn't use mesa, or we really don't have any armhf-linux
> users. I'm not advocating dropping the architecture, but it does feel
> like we're already at a best-effort level with it. As far as the pieces
> needed for bootstrapping aarch64 software (go and probably others),
> those get built anyway as needed by aarch64, so there's no worry about
> losing support for those software bits.

FWIW, on Debian guix 1.3.0 is currently at risk due to armhf and
i386/i686-linux missings builds due to test suite failures. It has been
hard to keep up with Guix in Debian, especially supporting "obscure"
platforms...

Though I'm guessing there may be reluctance to drop support for
i686-linux... I much more frequently encounter test suite failures when
building it, at least on Debian. It is surely less well supported than
x86_64-linux.

But dropping armhf and i386 at the moment looks ... helpful from a
Debian packaging perspective... :/

I'll try to package a git snapshot of guix and see if that fares any
better and hopefully find and fix issues before the next guix release!

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Supported architectures

[Ludovic Courtès]

Howdy!

Vagrant Cascadian <vagrant@debian.org> skribis:

> FWIW, on Debian guix 1.3.0 is currently at risk due to armhf and
> i386/i686-linux missings builds due to test suite failures. It has been
> hard to keep up with Guix in Debian, especially supporting "obscure"
> platforms...
>
> Though I'm guessing there may be reluctance to drop support for
> i686-linux... I much more frequently encounter test suite failures when
> building it, at least on Debian. It is surely less well supported than
> x86_64-linux.
>
> But dropping armhf and i386 at the moment looks ... helpful from a
> Debian packaging perspective... :/
>
> I'll try to package a git snapshot of guix and see if that fares any
> better and hopefully find and fix issues before the next guix release!

That’d be great!

Please report test failures that you find; we may be seeing them as well
in Guix.  :-)

Ludo’.

Re: Planning for a release, for real

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2962 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> We need to plan and coordinate.  Releases have to be a group effort;
> some of the most important work won’t be coding but coordination.
> Coordination is key.  I don’t think I should be spearheading that
> effort, but I’m happy to be part of it.
>
> Who’s ready to commit time towards that goal for the coming weeks?
>
> Here’s a list of things to do to get there:
>
>   • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
>     a couple of weeks ago, but then I lost track of it.  Marius?
>
>     We need a ‘staging’ champion to keep track of what’s left to be
>     done, reports progress, pings people, etc.  That person does not
>     have to be hacking like crazy, on the contrary!

I'd like to get qa.guix.gnu.org to the point where it's useful for
getting branches merged. Currently, it's possible to submit builds for
branches, which is happening currently for staging.

While that's great, the substitute availability for the branch is still
poor [1], the builds are happening, but there's just not enough hardware
behind bordeaux.guix.gnu.org currently to keep up with both the large
number of builds on the master branch as well as building staging
quickly.

1: http://data.qa.guix.gnu.org/repository/2/branch/staging/latest-processed-revision/package-substitute-availability

I know that's not what you're asking for here, but I think a big problem
when it comes to merging branches is that of checking what's broken, and
that's what I'd like to make easier.

>   • Get base binaries on all supported architectures in a timely
>     fashion, or drop some of the architectures.
>
>     Namely, ‘make assert-binaries-available’ is currently failing.  It
>     uses a manifest that encodes what we consider to be the basic
>     requirements for each architecture; it’s not demanding for
>     aarch64-linux, even less for armhf-linux and i586-gnu—yet we’re not
>     meeting these criteria yet.
>
>     We need to look at missing substitutes, address build issues and
>     build farm issues that cause them until we get to zero failures.  If
>     after some effort we fail to get to zero, then we should consider
>     dropping architectures (I’m looking at armhf-linux and i586-gnu
>     specifically).
>
>     Again we need a champion to keep track of this and ping people so we
>     make progress!

Is there a reason why ‘make assert-binaries-available’ is just checking
ci.guix.gnu.org? One of the main reasons why bordeaux.guix.gnu.org
exists is to address the lack of substitutes, and if you do check the
mainifest against bordeaux.guix.gnu.org you do get a slightly higher
percentage.

i586-gnu is a shared problem though, I'll try and see if I can get some
childhurd VMs working, although I'm not sure this'll be timely enough
for the upcoming release.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Planning for a release, for real

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> We need to plan and coordinate.  Releases have to be a group effort;
>> some of the most important work won’t be coding but coordination.
>> Coordination is key.  I don’t think I should be spearheading that
>> effort, but I’m happy to be part of it.
>>
>> Who’s ready to commit time towards that goal for the coming weeks?
>>
>> Here’s a list of things to do to get there:
>>
>>   • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
>>     a couple of weeks ago, but then I lost track of it.  Marius?
>>
>>     We need a ‘staging’ champion to keep track of what’s left to be
>>     done, reports progress, pings people, etc.  That person does not
>>     have to be hacking like crazy, on the contrary!
>
> I'd like to get qa.guix.gnu.org to the point where it's useful for
> getting branches merged. Currently, it's possible to submit builds for
> branches, which is happening currently for staging.
>
> While that's great, the substitute availability for the branch is still
> poor [1], the builds are happening, but there's just not enough hardware
> behind bordeaux.guix.gnu.org currently to keep up with both the large
> number of builds on the master branch as well as building staging
> quickly.
>
> 1: http://data.qa.guix.gnu.org/repository/2/branch/staging/latest-processed-revision/package-substitute-availability
>
> I know that's not what you're asking for here, but I think a big problem
> when it comes to merging branches is that of checking what's broken, and
> that's what I'd like to make easier.

Yes, definitely!

> Is there a reason why ‘make assert-binaries-available’ is just checking
> ci.guix.gnu.org?

It’s mostly that ‘guix weather --display-missing’ is not really helpful
when passed more than one URL: it reports all the things that are
missing on at least one of the URLs (this is what the comment in
‘Makefile.am’ suggests).

The goal of this command is to check that “basic packages” build and are
available, and we can expect basic packages to be available on both
build farms so I think it’s OK to check just one of them.  (When I first
wrote that manifest, I naively thought we’d be at 100% at any time since
this is really the baseline…)

> i586-gnu is a shared problem though, I'll try and see if I can get some
> childhurd VMs working, although I'm not sure this'll be timely enough
> for the upcoming release.

For i586-gnu the expectation is really low right now: see
‘%base-packages/hurd’ in ‘etc/release-manifest.scm’.

Ludo’.

Re: Planning for a release, for real

[zimoun]

Hi,

On jeu., 06 oct. 2022 at 16:50, Ludovic Courtès <ludo@gnu.org> wrote:

>   • Update NEWS (mostly done already!), prepare a blog post listing the
>     highlights and linking to the relevant material.  (See
>     <https://guix.gnu.org/en/blog/tags/releases/> for inspiration.)

Just a reminder about ’guix environment’:

--8<---------------cut here---------------start------------->8---
@quotation Deprecation warning
The @command{guix environment} command is deprecated in favor of
@command{guix shell}, which performs similar functions but is more
convenient to use.  @xref{Invoking guix shell}.

Being deprecated, @command{guix environment} is slated for eventual
removal, but the Guix project is committed to keeping it until May 1st,
2023.  Please get in touch with us at @email{guix-devel@@gnu.org} if you
would like to discuss it.
@end quotation
--8<---------------cut here---------------end--------------->8---

Therefore, it would appear helpful to mention that for this future
release.  Because May 1st, 2023 is… soon! :-)


Cheers,
simon

Release progress, week 1

[Ludovic Courtès]

Hello!

Release progress: week 1.

Ludovic Courtès <ludo@gnu.org> skribis:

> Here’s a list of things to do to get there:
>
>   • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
>     a couple of weeks ago, but then I lost track of it.  Marius?

Marius, any update?

Chris, does data.guix.gnu.org have enough info to provide insights?

>   • Get base binaries on all supported architectures in a timely
>     fashion, or drop some of the architectures.

There was progress on i686-linux since last week:

  https://issues.guix.gnu.org/58366
  https://issues.guix.gnu.org/58352

I’m also making slow progress on making childhurds usable on the ci.guix
infrastructure (and probably elsewhere) for i586-gnu builds:

  https://issues.guix.gnu.org/58320

‘make assert-binaries-available’ reports 91.9% right now (commit
8b192c5550213911f930594f4fd7386f36618237).

>   • Address the blockers of <https://issues.guix.gnu.org/53214>, most of
>     which are issues in the installer.

Mathieu fixed <https://issues.guix.gnu.org/57232> today.

Discussions are ongoing regarding <https://issues.guix.gnu.org/55360>.

How about meeting on #guix on IRC tomorrow Friday at 2PM CEST (UTC+2) to
discuss the issues above and distribute work among ourselves?  Who’s in?

Ludo’.

Re: Release progress, week 1

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1968 bytes --]

On Thu, Oct 13, 2022 at 05:19:48PM +0200, Ludovic Courtès wrote:
> Hello!
> 
> Release progress: week 1.
> 
> Ludovic Courtès <ludo@gnu.org> skribis:
> 
> > Here’s a list of things to do to get there:
> >
> >   • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
> >     a couple of weeks ago, but then I lost track of it.  Marius?
> 
> Marius, any update?

There's a commit by mhw I'd like to get in, it causes a rebuild of the
rust bootstrap but makes it take far less resources to build.

> Chris, does data.guix.gnu.org have enough info to provide insights?
> 
> >   • Get base binaries on all supported architectures in a timely
> >     fashion, or drop some of the architectures.
> 
> There was progress on i686-linux since last week:
> 
>   https://issues.guix.gnu.org/58366
>   https://issues.guix.gnu.org/58352
> 
> I’m also making slow progress on making childhurds usable on the ci.guix
> infrastructure (and probably elsewhere) for i586-gnu builds:
> 
>   https://issues.guix.gnu.org/58320
> 
> ‘make assert-binaries-available’ reports 91.9% right now (commit
> 8b192c5550213911f930594f4fd7386f36618237).

I'm seeing 92.5% on staging, so if we're still planning on merging that
first IMO that branch is a bit more important.

> >   • Address the blockers of <https://issues.guix.gnu.org/53214>, most of
> >     which are issues in the installer.
> 
> Mathieu fixed <https://issues.guix.gnu.org/57232> today.
> 
> Discussions are ongoing regarding <https://issues.guix.gnu.org/55360>.
> 
> How about meeting on #guix on IRC tomorrow Friday at 2PM CEST (UTC+2) to
> discuss the issues above and distribute work among ourselves?  Who’s in?
> 
> Ludo’.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Release progress, week 1

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2532 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Release progress: week 1.
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Here’s a list of things to do to get there:
>>
>>   • Merge ‘staging’ (?).  What’s the status of that one, it seemed ready
>>     a couple of weeks ago, but then I lost track of it.  Marius?
>
> Marius, any update?
>
> Chris, does data.guix.gnu.org have enough info to provide insights?

Unfortunately not, you can see the substitute availability here [1]. Not
enough has been built yet to do a comprehensive comparison.

1: https://data.qa.guix.gnu.org/repository/2/branch/staging/latest-processed-revision/package-substitute-availability

The builds are happening, and I've tweaked some prioritisation to try
and speed things up for x86_64-linux and aarch64-linux, but
fundamentally the only way to really speed things up is add more
hardware.

>>   • Get base binaries on all supported architectures in a timely
>>     fashion, or drop some of the architectures.
>
> There was progress on i686-linux since last week:
>
>   https://issues.guix.gnu.org/58366
>   https://issues.guix.gnu.org/58352
>
> I’m also making slow progress on making childhurds usable on the ci.guix
> infrastructure (and probably elsewhere) for i586-gnu builds:
>
>   https://issues.guix.gnu.org/58320
>
> ‘make assert-binaries-available’ reports 91.9% right now (commit
> 8b192c5550213911f930594f4fd7386f36618237).

I've also looked again at childhurds, it was actually surprisingly easy
to get some running (I don't think I ran in to the same problem as
you). I'm not sure why I didn't set some up sooner.

Anyway, I also spotted the guix package failing to build for
powerpc64le-linux. That seemed to be a problem with the linux-libre
packages lacking powerpc64le-linux as a supported system, so I pushed
[2] to fix that. Hopefully when the guix package is next updated, it'll
build on powerpc64le-linux.

2: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=166faae8d8aea499be7985255c4d47e36095a6bd

>>   • Address the blockers of <https://issues.guix.gnu.org/53214>, most of
>>     which are issues in the installer.
>
> Mathieu fixed <https://issues.guix.gnu.org/57232> today.
>
> Discussions are ongoing regarding <https://issues.guix.gnu.org/55360>.
>
> How about meeting on #guix on IRC tomorrow Friday at 2PM CEST (UTC+2) to
> discuss the issues above and distribute work among ourselves?  Who’s in?

I'll try to be around :)

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Release progress, week 2

[Ludovic Courtès]

Hello!

Release progress: week 2.

  • ‘staging’ branch → merged!

    Apparently there’s a regression with Rust no longer being buildable
    on aarch64-linux, but I can’t find a bug report.  Marius?

  • ‘make assert-binaries-available’ reports 92.7% coverage, slightly
    more than last week, and despite the ‘staging’ merge; details below.

  • Architectures:

     - armhf-linux is disabled on ci.guix due to improper offloading
       setup (probably along the lines of
       <https://issues.guix.gnu.org/53463>).  Should we try and reenable
       it, or should we drop it?

     - powerpc64le-linux is disabled on ci.guix since today
       (maintenance.git commit
       d641115e20973731555b586985fa81fbe293aeca).  However it did work
       until recently and we have one machine to offload to.  Should we
       fix it or drop it?  Mathieu?

     - i586-gnu is disabled due to <https://issues.guix.gnu.org/58320>.
       No fix yet, so I may resort to installing the workdaround so we
       can try and build things there.   I believe Chris didn’t hit this
       bug when setting up childhurds on Intel hardware behind
       bordeaux.guix, so substitutes should be coming there.

  • Installer:

     - ‘guix system init’ will print progress bars instead of dots
       again (patch available): <https://issues.guix.gnu.org/58375>.

  • Release issue <https://issues.guix.gnu.org/53214> still blocked
    by 8 open bugs that we should review!

  • Others:

     - Bugs related to grafts were reported and fixed this week:
         https://issues.guix.gnu.org/58419
         https://issues.guix.gnu.org/58567

     - A memory leak of shepherd that manifests on berlin is being
       investigated: <https://issues.guix.gnu.org/58631>.

Let’s coordinate and focus on these issues!

Ludo’.

--8<---------------cut here---------------start------------->8---
$ make assert-binaries-available 
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
Compiling Scheme modules...
computing 401 package derivations for x86_64-linux...
looking for 508 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org ☀
  92.7% substitutes available (471 out of 508)
  at least 4,332.1 MiB of nars (compressed)
  6,445.4 MiB on disk (uncompressed)
  0.026 seconds per request (1.1 seconds in total)
  38.4 requests per second

  0.0% (0 out of 37) of the missing items are queued
  at least 1,000 queued builds
      x86_64-linux: 1000 (100.0%)
  build rate: 15.77 builds per hour
      i686-linux: 2.96 builds per hour
      x86_64-linux: 9.88 builds per hour
      aarch64-linux: 3.71 builds per hour
      armhf-linux: 0.04 builds per hour

Substitutes are missing for the following items:
  /gnu/store/885iln2zpcxqvbr35d54bqhzs7l9djmq-libreoffice-7.3.5.2                i686-linux
  /gnu/store/3fbzxn7bmlx7f250n6wdk56fl633giha-mate-1.24.1                        i686-linux
  /gnu/store/475m6q7hp7a1gw98ki9l4g04gmvsa75y-xz-5.2.5                           i586-gnu
  /gnu/store/8fhdpb76nqy3v22jr33j18i1k07rl5n1-xz-5.2.5-static                    i586-gnu
  /gnu/store/6dvavfwg4zdih3rlvac4qqkx46my8gl6-tar-1.34                           i586-gnu
  /gnu/store/sa1ay72axmi9g75sll2wy9cqpfldfy40-gcc-toolchain-12.2.0-debug         i586-gnu
  /gnu/store/qxwclv8hx9z3gqwjil4hpwkwndh6f2zm-gcc-toolchain-12.2.0               i586-gnu
  /gnu/store/dypv1jgfzxlkbpp36z393xbdqal1v591-gcc-toolchain-12.2.0-static        i586-gnu
  /gnu/store/7bx9jykip9lc13yn2bck1m4q8ccds1mz-make-4.3-debug                     i586-gnu
  /gnu/store/422i4q46cisabwsxrs7raf67awwwzsys-make-4.3                           i586-gnu
  /gnu/store/f8jsczp72i49c79rjf8nv2q6jskqa5vy-gawk-5.1.0                         i586-gnu
  /gnu/store/d646qvpcdi0l9r2mqhqkxkrgwm0b50qh-findutils-4.8.0                    i586-gnu
  /gnu/store/zb0zbds0k2vjnln88dp4paldghl2mdwv-grep-3.6                           i586-gnu
  /gnu/store/62hb8sk7vnz26flasklrm0x0yh5pdnq4-coreutils-8.32-debug               i586-gnu
  /gnu/store/fmk805j58dig4076wy8q6fj1w47jxaw1-coreutils-8.32                     i586-gnu
  /gnu/store/kgz0xm0c05ys92nkg07l7lbbikrx7hia-guix-1.3.0-31.3170843              armhf-linux
  /gnu/store/nz1rw5cfrh4z3bl7fm2qsvxxpl955cqh-guile-3.0.8-debug                  armhf-linux
  /gnu/store/zmk1kmfk7wxm5w3ambajgnx7b0s5iq84-guile-3.0.8                        armhf-linux
  /gnu/store/26yb2pj71wg9cywmhpmsf6n1d81i43c5-python-3.9.9-idle                  armhf-linux
  /gnu/store/dh5rr8gd148afs3jzijs8i9gfwwi6igz-python-3.9.9                       armhf-linux
  /gnu/store/x0yzk738mm4if6kbc8i8q7x3ajz2rd27-python-3.9.9-tk                    armhf-linux
  /gnu/store/5p9jplb7hzci9nrpk4nxqa7qlyfb6wji-vim-9.0.0719                       armhf-linux
  /gnu/store/kxzqk19zm8y8dchcgx0izhwhfmzxwdmi-emacs-no-x-28.2                    armhf-linux
  /gnu/store/3ss4kln2a69xfja55wbi46pr1nsjbngr-openssh-8.9p1                      armhf-linux
  /gnu/store/s95hh0h7zak67iwhq8aavl3np53ri9y7-nss-certs-3.81                     armhf-linux
  /gnu/store/5nn8q80kywqvpzkhafpv3lppfbm5wm7n-bootstrap-tarballs-0               armhf-linux
  /gnu/store/z74rg03jdf18byyin6ygggw5q77mk1mn-guix-1.3.0-31.3170843              powerpc64le-linux
  /gnu/store/ycl224w6nz4dj2rnb7mcki4p5w46cnfp-vim-9.0.0719                       powerpc64le-linux
  /gnu/store/f2mw6w0nk0j670qvlw2z72mvrwm5575w-emacs-28.2                         powerpc64le-linux
  /gnu/store/8psdslg1p8g814ih6sd1xrxvx39bf9v4-openssh-8.9p1                      powerpc64le-linux
  /gnu/store/5i4v6wbdlj4y9dpfp15jnrnphg0x84py-guix-1.3.0-31.3170843              aarch64-linux
  /gnu/store/pgm8608mzhwxn86q48lpb77vp4pxp6g3-python-3.9.9-idle                  aarch64-linux
  /gnu/store/v4hgmb3k2l60yy8vzl3h1wvp5fa15db7-emacs-28.2                         aarch64-linux
  /gnu/store/pfk09jhc4fqalkv6bbv0cv7j00whydzm-gcc-toolchain-12.2.0-debug         aarch64-linux
  /gnu/store/kjcr6zmkh0gdraclp5v5kqqqsy4hdx9h-gcc-toolchain-12.2.0               aarch64-linux
  /gnu/store/ylhykr5g3yvbdarwb0h7smhhx2wga89m-gcc-toolchain-12.2.0-static        aarch64-linux
  /gnu/store/v14vx389rwshm5chr5llbrnjyrvgxbp7-bootstrap-tarballs-0               aarch64-linux
make: *** [Makefile:7146: assert-binaries-available] Error 1
--8<---------------cut here---------------end--------------->8---

Re: Release progress, week 2

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 8255 bytes --]

On Thu, Oct 20, 2022 at 03:49:00PM +0200, Ludovic Courtès wrote:
> Hello!
> 
> Release progress: week 2.
> 
>   • ‘staging’ branch → merged!
> 
>     Apparently there’s a regression with Rust no longer being buildable
>     on aarch64-linux, but I can’t find a bug report.  Marius?

I can answer this one.

I'm not sure there is a bug report, I didn't see it either. It looks
like when I bumped rust-bootstrap from 1.39 to 1.54 we lost aarch64
support. I've bumped mrustc on staging and successfully performed a
qemu-binfmt build of rust-bootstrap for aarch64 on my x86_64 box. I was
then able to use that to build rust-1.55 on actual aarch64 hardware, so
I assume it's good, I just don't have the hardware to build
rust-bootstrap for aarch64 natively.

With the bump to mrustc I also pushed (a modified version of) Mark's
patches to reduce the resource load of building the rust bootstrap
chain.

Direct links to the build status of rust on staging:
https://ci.guix.gnu.org/build/1632586/details for x86_64
https://ci.guix.gnu.org/build/1628677/details for aarch64

I also tossed in a patch to reduce the verbosity of unpacking rust
crates during the 'configure phase of cargo-build-system.

>   • ‘make assert-binaries-available’ reports 92.7% coverage, slightly
>     more than last week, and despite the ‘staging’ merge; details below.
> 
>   • Architectures:
> 
>      - armhf-linux is disabled on ci.guix due to improper offloading
>        setup (probably along the lines of
>        <https://issues.guix.gnu.org/53463>).  Should we try and reenable
>        it, or should we drop it?
> 
>      - powerpc64le-linux is disabled on ci.guix since today
>        (maintenance.git commit
>        d641115e20973731555b586985fa81fbe293aeca).  However it did work
>        until recently and we have one machine to offload to.  Should we
>        fix it or drop it?  Mathieu?

What happened to guixp9? I recently used GUIX_DAEMON_SOCKET to have it
build vim, emacs and openssh to help make assert-binaries-available, for
when those are next offloaded to it.

>      - i586-gnu is disabled due to <https://issues.guix.gnu.org/58320>.
>        No fix yet, so I may resort to installing the workdaround so we
>        can try and build things there.   I believe Chris didn’t hit this
>        bug when setting up childhurds on Intel hardware behind
>        bordeaux.guix, so substitutes should be coming there.
> 
>   • Installer:
> 
>      - ‘guix system init’ will print progress bars instead of dots
>        again (patch available): <https://issues.guix.gnu.org/58375>.
> 
>   • Release issue <https://issues.guix.gnu.org/53214> still blocked
>     by 8 open bugs that we should review!
> 
>   • Others:
> 
>      - Bugs related to grafts were reported and fixed this week:
>          https://issues.guix.gnu.org/58419
>          https://issues.guix.gnu.org/58567
> 
>      - A memory leak of shepherd that manifests on berlin is being
>        investigated: <https://issues.guix.gnu.org/58631>.
> 
> Let’s coordinate and focus on these issues!
> 
> Ludo’.
> 
> --8<---------------cut here---------------start------------->8---
> $ make assert-binaries-available 
> Compiling Scheme modules...
> Compiling Scheme modules...
> Compiling Scheme modules...
> Compiling Scheme modules...
> computing 401 package derivations for x86_64-linux...
> looking for 508 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org ☀
>   92.7% substitutes available (471 out of 508)
>   at least 4,332.1 MiB of nars (compressed)
>   6,445.4 MiB on disk (uncompressed)
>   0.026 seconds per request (1.1 seconds in total)
>   38.4 requests per second
> 
>   0.0% (0 out of 37) of the missing items are queued
>   at least 1,000 queued builds
>       x86_64-linux: 1000 (100.0%)
>   build rate: 15.77 builds per hour
>       i686-linux: 2.96 builds per hour
>       x86_64-linux: 9.88 builds per hour
>       aarch64-linux: 3.71 builds per hour
>       armhf-linux: 0.04 builds per hour
> 
> Substitutes are missing for the following items:
>   /gnu/store/885iln2zpcxqvbr35d54bqhzs7l9djmq-libreoffice-7.3.5.2                i686-linux
>   /gnu/store/3fbzxn7bmlx7f250n6wdk56fl633giha-mate-1.24.1                        i686-linux

mate is blocked by some java-12 unpack bug, through a series of packages
to font-abattis-cantarell. I think libreoffice is a plain FTBFS that no
one has sorted out yet.

>   /gnu/store/475m6q7hp7a1gw98ki9l4g04gmvsa75y-xz-5.2.5                           i586-gnu
>   /gnu/store/8fhdpb76nqy3v22jr33j18i1k07rl5n1-xz-5.2.5-static                    i586-gnu
>   /gnu/store/6dvavfwg4zdih3rlvac4qqkx46my8gl6-tar-1.34                           i586-gnu
>   /gnu/store/sa1ay72axmi9g75sll2wy9cqpfldfy40-gcc-toolchain-12.2.0-debug         i586-gnu
>   /gnu/store/qxwclv8hx9z3gqwjil4hpwkwndh6f2zm-gcc-toolchain-12.2.0               i586-gnu
>   /gnu/store/dypv1jgfzxlkbpp36z393xbdqal1v591-gcc-toolchain-12.2.0-static        i586-gnu
>   /gnu/store/7bx9jykip9lc13yn2bck1m4q8ccds1mz-make-4.3-debug                     i586-gnu
>   /gnu/store/422i4q46cisabwsxrs7raf67awwwzsys-make-4.3                           i586-gnu
>   /gnu/store/f8jsczp72i49c79rjf8nv2q6jskqa5vy-gawk-5.1.0                         i586-gnu
>   /gnu/store/d646qvpcdi0l9r2mqhqkxkrgwm0b50qh-findutils-4.8.0                    i586-gnu
>   /gnu/store/zb0zbds0k2vjnln88dp4paldghl2mdwv-grep-3.6                           i586-gnu
>   /gnu/store/62hb8sk7vnz26flasklrm0x0yh5pdnq4-coreutils-8.32-debug               i586-gnu
>   /gnu/store/fmk805j58dig4076wy8q6fj1w47jxaw1-coreutils-8.32                     i586-gnu
>   /gnu/store/kgz0xm0c05ys92nkg07l7lbbikrx7hia-guix-1.3.0-31.3170843              armhf-linux
>   /gnu/store/nz1rw5cfrh4z3bl7fm2qsvxxpl955cqh-guile-3.0.8-debug                  armhf-linux
>   /gnu/store/zmk1kmfk7wxm5w3ambajgnx7b0s5iq84-guile-3.0.8                        armhf-linux
>   /gnu/store/26yb2pj71wg9cywmhpmsf6n1d81i43c5-python-3.9.9-idle                  armhf-linux
>   /gnu/store/dh5rr8gd148afs3jzijs8i9gfwwi6igz-python-3.9.9                       armhf-linux
>   /gnu/store/x0yzk738mm4if6kbc8i8q7x3ajz2rd27-python-3.9.9-tk                    armhf-linux
>   /gnu/store/5p9jplb7hzci9nrpk4nxqa7qlyfb6wji-vim-9.0.0719                       armhf-linux
>   /gnu/store/kxzqk19zm8y8dchcgx0izhwhfmzxwdmi-emacs-no-x-28.2                    armhf-linux
>   /gnu/store/3ss4kln2a69xfja55wbi46pr1nsjbngr-openssh-8.9p1                      armhf-linux
>   /gnu/store/s95hh0h7zak67iwhq8aavl3np53ri9y7-nss-certs-3.81                     armhf-linux
>   /gnu/store/5nn8q80kywqvpzkhafpv3lppfbm5wm7n-bootstrap-tarballs-0               armhf-linux
>   /gnu/store/z74rg03jdf18byyin6ygggw5q77mk1mn-guix-1.3.0-31.3170843              powerpc64le-linux
>   /gnu/store/ycl224w6nz4dj2rnb7mcki4p5w46cnfp-vim-9.0.0719                       powerpc64le-linux
>   /gnu/store/f2mw6w0nk0j670qvlw2z72mvrwm5575w-emacs-28.2                         powerpc64le-linux
>   /gnu/store/8psdslg1p8g814ih6sd1xrxvx39bf9v4-openssh-8.9p1                      powerpc64le-linux
>   /gnu/store/5i4v6wbdlj4y9dpfp15jnrnphg0x84py-guix-1.3.0-31.3170843              aarch64-linux
>   /gnu/store/pgm8608mzhwxn86q48lpb77vp4pxp6g3-python-3.9.9-idle                  aarch64-linux
>   /gnu/store/v4hgmb3k2l60yy8vzl3h1wvp5fa15db7-emacs-28.2                         aarch64-linux
>   /gnu/store/pfk09jhc4fqalkv6bbv0cv7j00whydzm-gcc-toolchain-12.2.0-debug         aarch64-linux
>   /gnu/store/kjcr6zmkh0gdraclp5v5kqqqsy4hdx9h-gcc-toolchain-12.2.0               aarch64-linux
>   /gnu/store/ylhykr5g3yvbdarwb0h7smhhx2wga89m-gcc-toolchain-12.2.0-static        aarch64-linux
>   /gnu/store/v14vx389rwshm5chr5llbrnjyrvgxbp7-bootstrap-tarballs-0               aarch64-linux
> make: *** [Makefile:7146: assert-binaries-available] Error 1
> --8<---------------cut here---------------end--------------->8---
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Rust on aarch64-linux

[Ludovic Courtès]

Hello,

Efraim Flashner <efraim@flashner.co.il> skribis:

> I'm not sure there is a bug report, I didn't see it either. It looks
> like when I bumped rust-bootstrap from 1.39 to 1.54 we lost aarch64
> support. I've bumped mrustc on staging and successfully performed a
> qemu-binfmt build of rust-bootstrap for aarch64 on my x86_64 box. I was
> then able to use that to build rust-1.55 on actual aarch64 hardware, so
> I assume it's good, I just don't have the hardware to build
> rust-bootstrap for aarch64 natively.

So the presumed fix involves bumping rust-bootstrap from 1.54 to 1.55,
is that right?

That means we’ll have to rebuild on all architectures.  This is
happening here:

  https://ci.guix.gnu.org/eval/739823

Could you monitor it?  If things go well, can we aim for a merge by next
Thursday?

Thanks,
Ludo’.

Re: Rust on aarch64-linux

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1699 bytes --]

On Fri, Oct 21, 2022 at 10:51:59AM +0200, Ludovic Courtès wrote:
> Hello,
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > I'm not sure there is a bug report, I didn't see it either. It looks
> > like when I bumped rust-bootstrap from 1.39 to 1.54 we lost aarch64
> > support. I've bumped mrustc on staging and successfully performed a
> > qemu-binfmt build of rust-bootstrap for aarch64 on my x86_64 box. I was
> > then able to use that to build rust-1.55 on actual aarch64 hardware, so
> > I assume it's good, I just don't have the hardware to build
> > rust-bootstrap for aarch64 natively.
> 
> So the presumed fix involves bumping rust-bootstrap from 1.54 to 1.55,
> is that right?

Not quite. We keep rust-bootstrap at 1.54, but we bump the mrustc commit
we use from the v0.10 tag to its current master (about 50 commits).
Among the commits there is one that doesn't get aarch64 to try to spit
out assembly, (or illegal assembly or something) and then we just have
to build out again.

> That means we’ll have to rebuild on all architectures.  This is
> happening here:
> 
>   https://ci.guix.gnu.org/eval/739823
> 
> Could you monitor it?  If things go well, can we aim for a merge by next
> Thursday?
> 
> Thanks,
> Ludo’.

rust-1.60 built just fine on x86_64, and it looks like of the ~6700
packages built 567 failed, with another ~3400 waiting to be built.

I also haven't seen any movement on aarch64 in the build farm.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Rust on aarch64-linux

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]

On Fri, Oct 21, 2022 at 10:51:59AM +0200, Ludovic Courtès wrote:
> Hello,
> 
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > I'm not sure there is a bug report, I didn't see it either. It looks
> > like when I bumped rust-bootstrap from 1.39 to 1.54 we lost aarch64
> > support. I've bumped mrustc on staging and successfully performed a
> > qemu-binfmt build of rust-bootstrap for aarch64 on my x86_64 box. I was
> > then able to use that to build rust-1.55 on actual aarch64 hardware, so
> > I assume it's good, I just don't have the hardware to build
> > rust-bootstrap for aarch64 natively.
> 
> So the presumed fix involves bumping rust-bootstrap from 1.54 to 1.55,
> is that right?
> 
> That means we’ll have to rebuild on all architectures.  This is
> happening here:
> 
>   https://ci.guix.gnu.org/eval/739823
> 
> Could you monitor it?  If things go well, can we aim for a merge by next
> Thursday?

I think it looks good, but I'm not sure how to compare it to master.
perhaps some Magic View™ using the guix data service?

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Rust on aarch64-linux

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

On Sat, Oct 22, 2022 at 11:22:50PM +0300, Efraim Flashner wrote:
> On Fri, Oct 21, 2022 at 10:51:59AM +0200, Ludovic Courtès wrote:
> > Hello,
> > 
> > Efraim Flashner <efraim@flashner.co.il> skribis:
> > 
> > > I'm not sure there is a bug report, I didn't see it either. It looks
> > > like when I bumped rust-bootstrap from 1.39 to 1.54 we lost aarch64
> > > support. I've bumped mrustc on staging and successfully performed a
> > > qemu-binfmt build of rust-bootstrap for aarch64 on my x86_64 box. I was
> > > then able to use that to build rust-1.55 on actual aarch64 hardware, so
> > > I assume it's good, I just don't have the hardware to build
> > > rust-bootstrap for aarch64 natively.
> > 
> > So the presumed fix involves bumping rust-bootstrap from 1.54 to 1.55,
> > is that right?
> > 
> > That means we’ll have to rebuild on all architectures.  This is
> > happening here:
> > 
> >   https://ci.guix.gnu.org/eval/739823
> > 
> > Could you monitor it?  If things go well, can we aim for a merge by next
> > Thursday?
> 
> I think it looks good, but I'm not sure how to compare it to master.
> perhaps some Magic View™ using the guix data service?

I merged master into staging again and took a look at the build
failures. They seemed alright (and I restarted a few) and then merged
staging into master. Everything looks good from cuirass in relation to
the merging.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Status of armhf-linux and powerpc64le-linux

[Ludovic Courtès]

Moin!

Mathieu Othacehe <othacehe@gnu.org> skribis:

>>      - armhf-linux is disabled on ci.guix due to improper offloading
>>        setup (probably along the lines of
>>        <https://issues.guix.gnu.org/53463>).  Should we try and reenable
>>        it, or should we drop it?
>>
>>      - powerpc64le-linux is disabled on ci.guix since today
>>        (maintenance.git commit
>>        d641115e20973731555b586985fa81fbe293aeca).  However it did work
>>        until recently and we have one machine to offload to.  Should we
>>        fix it or drop it?  Mathieu?
>
> Yeah, we only have a single machine to offload to and each time it is
> not reachable, the "guix" specification fails on Cuirass.

How frequently does that machine become unreachable?

Its uptime right now is “only” 51 days, but it seems to have been
reliably building things so far (surprisingly so!).

> That's because we need to offload to a powerpc64le-linux machine in
> order to evaluate the guix derivation for that specific architecture
> (that's true for all the other architectures).

Maybe we should arrange to be more resilient to transient build machine
outage.

For that we need redundancy; we have it for ARM, but not for POWER9.  A
simple way to get redundancy today would be to set up transparent
emulation for POWER9 on one of the x86_64 boxes.  That’ll be
inefficient, but that’ll let Cuirass survive transient failures of that
one POWER9 box.

WDYT?

Longer-term, people interested in POWER9 should look into:

  • Purchasing, setting up, and hosting POWER9 hardware (funds held at
    the FSF are probably sufficient for that!).

  • And/or: getting in touch with companies who could sponsor us by
    providing hardware (the AArch64 port was started thanks to a
    donation by ARM).

In Cuirass, we should arrange to support partial evaluations or
per-system evaluations so that a single missing offload machine doesn’t
cause the whole evaluation to fail.

> Given the lack of workers for powerpc64le-linux I think we should drop
> it.

We can do that, but I find embarrassing to drop the architecture after
all the work people have put it “just” because of infrastructure issues.

> Regarding armhf-linux we can in theory rely on the overdrives but we
> are already struggling on aarch64-linux, we I think we should also
> drop it for now.

In theory, ci.guix has at least 3 Honeycombs (2 are currently offline)
and 2 Overdrives, so it’s not that bad, and they don’t seem to be all
that busy.

So you’re right in a way, but at the same time this seems to be an
infrastructure issue.

> Focusing on x86_64-linux, i686-linux and aarch64-linux for this release
> seems more pragmatic.

That’s radical, but maybe that’s the most reasonable option.

How about a plan like this: until next Thursday, we try to address the
infrastructure issues discussed above to estimate feasibility.  Then we
decide on the way forward.  WDYT?

If we end up dropping architectures, we’ll have to:

  1. Update the documentation (and eventually the web site).

  2. Offer a clear plan as to what it would take to reinstate those
     architectures, and probably define clear criteria for architecture
     support going forward.

Thanks,
Ludo’.

Re: Status of armhf-linux and powerpc64le-linux

[Mathieu Othacehe]

Hey,

> How frequently does that machine become unreachable?
>
> Its uptime right now is “only” 51 days, but it seems to have been
> reliably building things so far (surprisingly so!).

Oh so it must be available from the Cuirass point of view but not from
the guix offload point of view. I'll try to fix it today.

> In Cuirass, we should arrange to support partial evaluations or
> per-system evaluations so that a single missing offload machine doesn’t
> cause the whole evaluation to fail.

We can define a guix specification for x86_64-linux, i686-linux and
aarch64-linux and a different one for powerpc64le-linux. That can be
done really quickly. That can break some mechanisms relying on the fact
that the guix specification name is "guix" though.

> That’s radical, but maybe that’s the most reasonable option.
>
> How about a plan like this: until next Thursday, we try to address the
> infrastructure issues discussed above to estimate feasibility.  Then we
> decide on the way forward.  WDYT?

Alright, let's try that :). I agree that it is a pity not to release for
those architectures but on the other hand, we cannot offer fresh
substitutes reliably for them.

The recent outages of ci.guix.gnu.org have shown once again that the
infrastructure is maybe one of the most important Guix aspect. Without
it, Guix is almost unusable, in particular on architectures for which
it's hard to find powerful hardware.

Given the limited amount of people willing to help for
powerpc64le-linux and armhf-linux and the limited amount of hardware
resources available for those, I think it could be reasonable to focus
on a smaller set of architectures and provide stronger guarantees on
those in term of substitutes availability.

But let's discuss that again next week depending on our progress.

Thanks,

Mathieu

Re: Status of armhf-linux and powerpc64le-linux

[Tobias Platen]

On Fri, 2022-10-21 at 10:43 +0200, Ludovic Courtès wrote:
> Moin!
> 
> Mathieu Othacehe <othacehe@gnu.org> skribis:
> 
> > >      - armhf-linux is disabled on ci.guix due to improper
> > > offloading
> > >        setup (probably along the lines of
> > >        <https://issues.guix.gnu.org/53463>).  Should we try and
> > > reenable
> > >        it, or should we drop it?
> > > 
> > >      - powerpc64le-linux is disabled on ci.guix since today
> > >        (maintenance.git commit
> > >        d641115e20973731555b586985fa81fbe293aeca).  However it did
> > > work
> > >        until recently and we have one machine to offload to. 
> > > Should we
> > >        fix it or drop it?  Mathieu?
> > 
> > Yeah, we only have a single machine to offload to and each time it
> > is
> > not reachable, the "guix" specification fails on Cuirass.
> 
> How frequently does that machine become unreachable?
> 
> Its uptime right now is “only” 51 days, but it seems to have been
> reliably building things so far (surprisingly so!).
> 
> > That's because we need to offload to a powerpc64le-linux machine in
> > order to evaluate the guix derivation for that specific
> > architecture
> > (that's true for all the other architectures).
> 
> Maybe we should arrange to be more resilient to transient build
> machine
> outage.
> 
> For that we need redundancy; we have it for ARM, but not for POWER9. 
> A
> simple way to get redundancy today would be to set up transparent
> emulation for POWER9 on one of the x86_64 boxes.  That’ll be
> inefficient, but that’ll let Cuirass survive transient failures of
> that
> one POWER9 box.
> 
> WDYT?
> 
> Longer-term, people interested in POWER9 should look into:
> 
>   • Purchasing, setting up, and hosting POWER9 hardware (funds held
> at
>     the FSF are probably sufficient for that!).
Vikings also offers hosting of POWER9 hardware.
> 
>   • And/or: getting in touch with companies who could sponsor us by
>     providing hardware (the AArch64 port was started thanks to a
>     donation by ARM).
> 
> In Cuirass, we should arrange to support partial evaluations or
> per-system evaluations so that a single missing offload machine
> doesn’t
> cause the whole evaluation to fail.
> 
> > Given the lack of workers for powerpc64le-linux I think we should
> > drop
> > it.
> 
> We can do that, but I find embarrassing to drop the architecture
> after
> all the work people have put it “just” because of infrastructure
> issues.
> 
> > Regarding armhf-linux we can in theory rely on the overdrives but
> > we
> > are already struggling on aarch64-linux, we I think we should also
> > drop it for now.
> 
> In theory, ci.guix has at least 3 Honeycombs (2 are currently
> offline)
> and 2 Overdrives, so it’s not that bad, and they don’t seem to be all
> that busy.
> 
> So you’re right in a way, but at the same time this seems to be an
> infrastructure issue.
> 
> > Focusing on x86_64-linux, i686-linux and aarch64-linux for this
> > release
> > seems more pragmatic.
> 
> That’s radical, but maybe that’s the most reasonable option.
> 
> How about a plan like this: until next Thursday, we try to address
> the
> infrastructure issues discussed above to estimate feasibility.  Then
> we
> decide on the way forward.  WDYT?
> 
> If we end up dropping architectures, we’ll have to:
> 
>   1. Update the documentation (and eventually the web site).
> 
>   2. Offer a clear plan as to what it would take to reinstate those
>      architectures, and probably define clear criteria for
> architecture
>      support going forward.
> 
> Thanks,
> Ludo’.
> 

Re: Release progress, week 2

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1716 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

>   • Architectures:
>
>      - armhf-linux is disabled on ci.guix due to improper offloading
>        setup (probably along the lines of
>        <https://issues.guix.gnu.org/53463>).  Should we try and reenable
>        it, or should we drop it?
>
>      - powerpc64le-linux is disabled on ci.guix since today
>        (maintenance.git commit
>        d641115e20973731555b586985fa81fbe293aeca).  However it did work
>        until recently and we have one machine to offload to.  Should we
>        fix it or drop it?  Mathieu?

bordeaux.guix.gnu.org should have good substitute availability for both
of these architectures.

>      - i586-gnu is disabled due to <https://issues.guix.gnu.org/58320>.
>        No fix yet, so I may resort to installing the workdaround so we
>        can try and build things there.   I believe Chris didn’t hit this
>        bug when setting up childhurds on Intel hardware behind
>        bordeaux.guix, so substitutes should be coming there.

I've also got some childhurds running. I think there were some build
coordinator issues that meant some of the agents were getting stuck, but
I think I'm making progress with those now.

In terms of actually building things, I think there are some problems
building grep and coreutils. I've been looking at the hello package [1]
and the build for i586-gnu [2]. You can see the required failed builds
on the build page [2].

1: https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package/hello/2.12.1?locale=en_US.UTF-8
2: https://data.guix.gnu.org/build-server/2/build?build_server_build_id=b4b1ddb6-b2b2-454e-89b9-10d7ff61498c

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Release progress, week 2, release manifest, what builds are failing?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 7197 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> $ make assert-binaries-available 
> Compiling Scheme modules...
> Compiling Scheme modules...
> Compiling Scheme modules...
> Compiling Scheme modules...
> computing 401 package derivations for x86_64-linux...
> looking for 508 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org ☀
>   92.7% substitutes available (471 out of 508)
>   at least 4,332.1 MiB of nars (compressed)
>   6,445.4 MiB on disk (uncompressed)
>   0.026 seconds per request (1.1 seconds in total)
>   38.4 requests per second
>
>   0.0% (0 out of 37) of the missing items are queued
>   at least 1,000 queued builds
>       x86_64-linux: 1000 (100.0%)
>   build rate: 15.77 builds per hour
>       i686-linux: 2.96 builds per hour
>       x86_64-linux: 9.88 builds per hour
>       aarch64-linux: 3.71 builds per hour
>       armhf-linux: 0.04 builds per hour
>
> Substitutes are missing for the following items:
>   /gnu/store/885iln2zpcxqvbr35d54bqhzs7l9djmq-libreoffice-7.3.5.2                i686-linux
>   /gnu/store/3fbzxn7bmlx7f250n6wdk56fl633giha-mate-1.24.1                        i686-linux
>   /gnu/store/475m6q7hp7a1gw98ki9l4g04gmvsa75y-xz-5.2.5                           i586-gnu
>   /gnu/store/8fhdpb76nqy3v22jr33j18i1k07rl5n1-xz-5.2.5-static                    i586-gnu
>   /gnu/store/6dvavfwg4zdih3rlvac4qqkx46my8gl6-tar-1.34                           i586-gnu
>   /gnu/store/sa1ay72axmi9g75sll2wy9cqpfldfy40-gcc-toolchain-12.2.0-debug         i586-gnu
>   /gnu/store/qxwclv8hx9z3gqwjil4hpwkwndh6f2zm-gcc-toolchain-12.2.0               i586-gnu
>   /gnu/store/dypv1jgfzxlkbpp36z393xbdqal1v591-gcc-toolchain-12.2.0-static        i586-gnu
>   /gnu/store/7bx9jykip9lc13yn2bck1m4q8ccds1mz-make-4.3-debug                     i586-gnu
>   /gnu/store/422i4q46cisabwsxrs7raf67awwwzsys-make-4.3                           i586-gnu
>   /gnu/store/f8jsczp72i49c79rjf8nv2q6jskqa5vy-gawk-5.1.0                         i586-gnu
>   /gnu/store/d646qvpcdi0l9r2mqhqkxkrgwm0b50qh-findutils-4.8.0                    i586-gnu
>   /gnu/store/zb0zbds0k2vjnln88dp4paldghl2mdwv-grep-3.6                           i586-gnu
>   /gnu/store/62hb8sk7vnz26flasklrm0x0yh5pdnq4-coreutils-8.32-debug               i586-gnu
>   /gnu/store/fmk805j58dig4076wy8q6fj1w47jxaw1-coreutils-8.32                     i586-gnu
>   /gnu/store/kgz0xm0c05ys92nkg07l7lbbikrx7hia-guix-1.3.0-31.3170843              armhf-linux
>   /gnu/store/nz1rw5cfrh4z3bl7fm2qsvxxpl955cqh-guile-3.0.8-debug                  armhf-linux
>   /gnu/store/zmk1kmfk7wxm5w3ambajgnx7b0s5iq84-guile-3.0.8                        armhf-linux
>   /gnu/store/26yb2pj71wg9cywmhpmsf6n1d81i43c5-python-3.9.9-idle                  armhf-linux
>   /gnu/store/dh5rr8gd148afs3jzijs8i9gfwwi6igz-python-3.9.9                       armhf-linux
>   /gnu/store/x0yzk738mm4if6kbc8i8q7x3ajz2rd27-python-3.9.9-tk                    armhf-linux
>   /gnu/store/5p9jplb7hzci9nrpk4nxqa7qlyfb6wji-vim-9.0.0719                       armhf-linux
>   /gnu/store/kxzqk19zm8y8dchcgx0izhwhfmzxwdmi-emacs-no-x-28.2                    armhf-linux
>   /gnu/store/3ss4kln2a69xfja55wbi46pr1nsjbngr-openssh-8.9p1                      armhf-linux
>   /gnu/store/s95hh0h7zak67iwhq8aavl3np53ri9y7-nss-certs-3.81                     armhf-linux
>   /gnu/store/5nn8q80kywqvpzkhafpv3lppfbm5wm7n-bootstrap-tarballs-0               armhf-linux
>   /gnu/store/z74rg03jdf18byyin6ygggw5q77mk1mn-guix-1.3.0-31.3170843              powerpc64le-linux
>   /gnu/store/ycl224w6nz4dj2rnb7mcki4p5w46cnfp-vim-9.0.0719                       powerpc64le-linux
>   /gnu/store/f2mw6w0nk0j670qvlw2z72mvrwm5575w-emacs-28.2                         powerpc64le-linux
>   /gnu/store/8psdslg1p8g814ih6sd1xrxvx39bf9v4-openssh-8.9p1                      powerpc64le-linux
>   /gnu/store/5i4v6wbdlj4y9dpfp15jnrnphg0x84py-guix-1.3.0-31.3170843              aarch64-linux
>   /gnu/store/pgm8608mzhwxn86q48lpb77vp4pxp6g3-python-3.9.9-idle                  aarch64-linux
>   /gnu/store/v4hgmb3k2l60yy8vzl3h1wvp5fa15db7-emacs-28.2                         aarch64-linux
>   /gnu/store/pfk09jhc4fqalkv6bbv0cv7j00whydzm-gcc-toolchain-12.2.0-debug         aarch64-linux
>   /gnu/store/kjcr6zmkh0gdraclp5v5kqqqsy4hdx9h-gcc-toolchain-12.2.0               aarch64-linux
>   /gnu/store/ylhykr5g3yvbdarwb0h7smhhx2wga89m-gcc-toolchain-12.2.0-static        aarch64-linux
>   /gnu/store/v14vx389rwshm5chr5llbrnjyrvgxbp7-bootstrap-tarballs-0               aarch64-linux
> make: *** [Makefile:7146: assert-binaries-available] Error 1

So, now that bordeaux.guix.gnu.org has caught up with the changes from
the staging merge, checking the release manifest against it hopefully
just gives outputs that are missing because of build failures.

I've just pushed a change to the release manifest to tweak it's handling
of armhf-linux. The emacs package doesn't need replacing, but the guix
package doesn't build for armhf-linux (and I guess that's not getting
fixed), so that's now removed.

I've also just pushed an update to the guix package, and I'm pretty sure
this'll fix the build issue on powerpc64le-linux.

With those changes, that should mean there's just two sets of things
missing substitutes.

For i686-linux, that's these things:

/gnu/store/abmv94z7szi683d7k7wg9c058bdyjxc8-network-manager-applet-1.28.0
/gnu/store/885iln2zpcxqvbr35d54bqhzs7l9djmq-libreoffice-7.3.5.2
/gnu/store/x3vgcm9fqhkd272fb8azyp8kby66nm6j-enlightenment-0.25.4
/gnu/store/3dng8r8ydla5sw515rb252mvl9xrvc42-mate-1.24.1

If you ask data.guix.gnu.org about each output (e.g. [1]) it should show
you the build (which will probably be in the scheduled state). If you
click on that build, you should see the "Required failed builds".

1: https://data.guix.gnu.org/gnu/store/abmv94z7szi683d7k7wg9c058bdyjxc8-network-manager-applet-1.28.0

I've done that for the above outputs, and here are the derivations that
are failing to build:

/gnu/store/msq74p2bd4g99n2x2wl85pjwc51pp82f-gst-plugins-bad-1.20.3.drv
/gnu/store/5hbj16adwxi09bm7qm82i6x7hm2ybnqs-python-gpg-1.10.0.drv
/gnu/store/gkwiqqrjpaq1pbss980l71bd717wzj41-libreoffice-7.3.5.2.drv
/gnu/store/j2pifji7gn8spqcikjj2mgs59850rlxr-gnome-keyring-42.1.drv
/gnu/store/8q1qfhps0l9kgik9rz2rx3s0syk56ckn-python-afdko-3.9.1.drv

For i586-gnu, here is the simplified list of missing outputs.

/gnu/store/475m6q7hp7a1gw98ki9l4g04gmvsa75y-xz-5.2.5
/gnu/store/6dvavfwg4zdih3rlvac4qqkx46my8gl6-tar-1.34
/gnu/store/qxwclv8hx9z3gqwjil4hpwkwndh6f2zm-gcc-toolchain-12.2.0
/gnu/store/422i4q46cisabwsxrs7raf67awwwzsys-make-4.3
/gnu/store/f8jsczp72i49c79rjf8nv2q6jskqa5vy-gawk-5.1.0
/gnu/store/d646qvpcdi0l9r2mqhqkxkrgwm0b50qh-findutils-4.8.0
/gnu/store/zb0zbds0k2vjnln88dp4paldghl2mdwv-grep-3.6
/gnu/store/fmk805j58dig4076wy8q6fj1w47jxaw1-coreutils-8.32

As far as I can see, the blocking builds are just coreutils and grep:

/gnu/store/ff595b1xwwa7qhrh7zhpzkf23bs5whbs-coreutils-8.32.drv
/gnu/store/yl2w258h95j1gcz3k21y0llgajy8wd5x-grep-3.6.drv


So yeah, if anyone is up for looking at build failures and trying to fix
them, these are some good derivations to look at!

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Release progress, week 2, release manifest, what builds are failing: gst-plugins-bad

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> /gnu/store/msq74p2bd4g99n2x2wl85pjwc51pp82f-gst-plugins-bad-1.20.3.drv

I spotted ci.guix.gnu.org actually has a substitute for this. Turns out
it's just very flaky. bordeaux.guix.gnu.org has now attempted to build
it 9 times, and only 1 build succeeded:

https://data.guix.gnu.org/gnu/store/msq74p2bd4g99n2x2wl85pjwc51pp82f-gst-plugins-bad-1.20.3.drv

Anyway, I've sent a patch that disables the flaky test:

  https://issues.guix.gnu.org/58773

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]