Core-updates after the staging merge

Core-updates after the staging merge

[Andreas Enge]

Hello all,

the merge of staging to master, and the subsequent merge of master to
core-updates did break a few things; but on the positive side, we are
halfway there with getting rid of the staging and core-updates branches ;-)
CI has almost caught up on x86_64; looking at the dashboard at
   https://ci.guix.gnu.org/eval/402403/dashboard
shows much more red than on master, so we will need to do some more work
before the merge. Since it is ordered alphabetically and our package names
often start by the language, red streaks often indicate problems with
a given programming language. There are things to work on for most of the
teams!

- rust: This one bootstrapped up to the latest version! There are a few
  red dots for individual packages.
- python: It is mostly green, but with lots of red, including at least
  one package that prevents icecat from building. I would call for careful
  updates and bug fixes to move on; leaf packages can be updated to the
  latest version, but for important intermediate packages please be
  careful.
- Go looks good.
- Java is red, but apparently still bootstrapping; everything hinges on
  icedtea@2.6, which is still scheduled for build:
     https://ci.guix.gnu.org/build/1007080/details
  So we need to wait, and potentially restart jobs that are marked as
  failed, but may actually build once their inputs are there.
- Everything common lisp related is red (cl-*, sbcl-*), but maybe there
  is a similar thing going; or maybe not, since clisp is built.
- R is red, but for the time being held up by texlive packages still being
  built.
- ghc is taking a long time...

A single package holds up a lot (gtk+ and so on): openldap. It fails after
about a minute of compiling like so:
  starting phase `provide-libldap_r'
  error: in phase 'provide-libldap_r': uncaught exception:
  unbound-variable #f "Unbound variable: ~S" (ungexp) #f
  phase `provide-libldap_r' failed after 0.0 seconds
This looks more like a typo than anything else.


On i686, we are still stuck by wget.


On aarch64, I think we are mainly stuck by lack of build power. I cancelled
a few build jobs on CI belonging to older git commits, but there is not much
we can do apart from bringing back build machines.

On powerpc, my cancelling of old jobs has apparently also cancelled newer
jobs that are actually the same; I am trying to restart them all, but am
not sure if I succeed. Apologies! As written yesterday, CI does not seem
to use all the build power at our disposal for this architecture.

Andreas

[bug#62863] Openldap in core-updates

[Andreas Enge]

Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
> A single package holds up a lot (gtk+ and so on): openldap.

It was pointed out to me on IRC that there are already two fixes available
on our issue tracker! I applied one of them and also removed the additional
phase as suggested there.

In a separate commit, I also updated to 2.6.4 and removed the non-hidden
openldap-for-linphone variable, which had the confusing effect that the
command line would refer to this package (so "guix refresh -l openldap"
showed almost no dependents, for instance).

If any of these pose problems, we should selectively revert part of the
commits; but I am rather optimistic, as both versions create a library
with the same soname.

Thanks to all involved!

Andreas

wget on i686 in core-updates

[Andreas Enge]

Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
> On i686, we are still stuck by wget.

HEAD of git solves the problem, I will try to propose a solution later.

Andreas

Re: wget on i686 in core-updates

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

Am Sun, Apr 16, 2023 at 04:00:15PM +0200 schrieb Andreas Enge:
> HEAD of git solves the problem, I will try to propose a solution later.

Due to gnulib handled as a submodule, this turned out to be quite tricky
(for details, see the IRC logs with my discussion with jlicht).
The solution is to create a tarball after checking out the gnulib
submodule corresponding to the latest wget git commit and after running
./bootstrap.

As a stop-gap measure, I suggest to use a self-hosted tarball as in the
attached commit. I have good hope that we will soon see a 1.23.4 release,
see https://lists.gnu.org/archive/html/bug-wget/2023-04/msg00002.html .

What do you think?

Andreas


[-- Attachment #2: 0001-gnu-wget-Update-to-1.21.3.24.patch --]
[-- Type: text/plain, Size: 1392 bytes --]

From 424ee72ffbdf9a79eee4dd4aac43404167e66083 Mon Sep 17 00:00:00 2001
From: Andreas Enge <andreas@enge.fr>
Date: Sun, 16 Apr 2023 20:52:35 +0200
Subject: [PATCH] gnu: wget: Update to 1.21.3.24.

This update to a non-release version fixes a build failure on i686-linux.

* gnu/packages/wget.scm (wget): Update to 1.21.3.24.
[origin]: Use a self-hosted tarball created from the latest git commit.
---
 gnu/packages/wget.scm | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/wget.scm b/gnu/packages/wget.scm
index 083cf27212..17459344c0 100644
--- a/gnu/packages/wget.scm
+++ b/gnu/packages/wget.scm
@@ -46,15 +46,16 @@ (define-module (gnu packages wget)
 (define-public wget
   (package
     (name "wget")
-    (version "1.21.3")
+    (version "1.21.3.24")
     (source
      (origin
       (method url-fetch)
-      (uri (string-append "mirror://gnu/wget/wget-"
-                          version ".tar.lz"))
+      ;;(uri (string-append "mirror://gnu/wget/wget-"
+      ;;                    version ".tar.lz"))
+      (uri "https://www.multiprecision.org/wget-1.21.3.24-2b723.tar.lz")
       (sha256
        (base32
-        "19afmyr1i3zwdwr8wkyz8q6z5764ik3dm87as194g78l8xggplnv"))))
+        "17ip94mvax83h0gh4905jqc64g5qf3vgxr3bj9gn02pijjm5lzbp"))))
     (build-system gnu-build-system)
     (inputs
      (list gnutls libidn2 libpsl))
-- 
2.39.2

Re: wget on i686 in core-updates

[Andreas Enge]

Am Sun, Apr 16, 2023 at 08:57:48PM +0200 schrieb Andreas Enge:
> As a stop-gap measure, I suggest to use a self-hosted tarball as in the
> attached commit. I have good hope that we will soon see a 1.23.4 release,
> see https://lists.gnu.org/archive/html/bug-wget/2023-04/msg00002.html .

I just pushed the commit. There should be a new wget release this week:
    https://lists.gnu.org/archive/html/bug-wget/2023-04/msg00005.html
and then we can push the update. Having wget will enable us to see where
more work is needed on i686.

Andreas

Re: Core-updates after the staging merge

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

Andreas Enge <andreas@enge.fr> skribis:

> Hello all,
>
> the merge of staging to master, and the subsequent merge of master to
> core-updates did break a few things; but on the positive side, we are
> halfway there with getting rid of the staging and core-updates branches ;-)
> CI has almost caught up on x86_64; looking at the dashboard at
>    https://ci.guix.gnu.org/eval/402403/dashboard
> shows much more red than on master, so we will need to do some more work
> before the merge. Since it is ordered alphabetically and our package names
> often start by the language, red streaks often indicate problems with
> a given programming language. There are things to work on for most of the
> teams!
>
> [...]
> - Everything common lisp related is red (cl-*, sbcl-*), but maybe there
>   is a similar thing going; or maybe not, since clisp is built.
> [...]

Hi,
I tried to build the 1611 dependents of sbcl on x86-64, and most of them
build fine. I get only 6 failures, some of them because some
dependencies like mysql or supercollider are failing to build.
So overall, Common Lisp packages on core-updates are in a pretty good
shape.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 08:18:00AM +0000 schrieb Guillaume Le Vaillant:
> I tried to build the 1611 dependents of sbcl on x86-64, and most of them
> build fine. I get only 6 failures, some of them because some
> dependencies like mysql or supercollider are failing to build.
> So overall, Common Lisp packages on core-updates are in a pretty good
> shape.

Great, thanks for testing! Indeed it looks as if the buildfarm just
needed to catch up.

Andreas

Re: Core-updates after the staging merge

[Andreas Enge]

Hello,

just a quick update after a night of building on CI.
Things look generally quite good on x86_64; some things are being rebuilt
due to the recent wget update, and that should hopefully also sort out
i686 rather quickly.

Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
> - python: It is mostly green, but with lots of red, including at least
>   one package that prevents icecat from building. I would call for careful
>   updates and bug fixes to move on; leaf packages can be updated to the
>   latest version, but for important intermediate packages please be
>   careful.

Python probably is the place where work is needed most, the strip in the
dashboard looks like it has measles. Fixing one package there could bring
us a long way (or not - we are in plain hand resolution of dependency
problems there, one update could also hide another one). Sometimes it is
just a matter of trying an update to see whether a package and its
dependents build, so even someone who knows nothing about Python, but
has a web browser pointed at https://pypi.org/, can do some useful work
(I am speaking from experience).

> - ghc is taking a long time...

This is a real nightmare. ghc@9.2 on x86_64 has been building for 6 hours
in a row:
   https://ci.guix.gnu.org/build/984270/details
and we need to build many versions one after the other during bootstrapping.
While not blocking the core-updates merge, maybe the ghc team can try and
see whether we can cut corners. Or maybe drop tests on intermediate versions
that are only required for bootstrapping?

On aarch64 and powerpc, we are still stuck by CI problems.

Andreas

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 11:03:25AM +0200 schrieb Andreas Enge:
> - ghc is taking a long time...

Actually ghc@9.0 fails its tests, which are written in Python (it looks
like ".abc" needs to be added to collections); but since it is not needed
for bootstrapping any more, maybe we could drop it?

Well, there is the dependent package ghc-next@9.4.4 that would need to be
built from 8.10 also, or maybe 9.2. Some work for the haskell team ;-)

Andreas

Re: Core-updates after the staging merge

[Simon Tournier]

Hi,

On lun., 17 avril 2023 at 11:56, Andreas Enge <andreas@enge.fr> wrote:
> Am Mon, Apr 17, 2023 at 11:03:25AM +0200 schrieb Andreas Enge:
>> - ghc is taking a long time...
>
> Actually ghc@9.0 fails its tests, which are written in Python (it looks
> like ".abc" needs to be added to collections); but since it is not needed
> for bootstrapping any more, maybe we could drop it?

Well, the current Haskell bootstrap chain is:

   7.8.4
-> 7.10.3
-> 8.0.2
-> 8.4.4
-> 8.6.5
-> 8.8.4
-> 8.10.7
-> 9.0.2; 9.2.5
-> 9.4.4 (next)

and instead we could try this shorter one:

   7.8.4
-> 8.0.2 (needs >= 7.10)
-> 8.4.4 (needs >= 8.0)
-> 8.8.4 (needs >= 8.4)
-> 9.2.5 (needs >= 8.6)

WDYT?  If no objection, I will give a try.



Moreover, the recipe for GHC looks like:

--8<---------------cut here---------------start------------->8---
(define-public ghc-9.0
  (package
    (inherit ghc-8.10)
    (name "ghc")
    (version "9.0.2")
    (source (origin
              (method url-fetch)
              (uri (string-append "https://www.haskell.org/ghc/dist/" version
                                  "/ghc-" version "-src.tar.xz"))
              (sha256
               (base32
                "15wii8can2r3dcl6jjmd50h2jvn7rlmn05zb74d2scj6cfwl43hl"))))
    (native-inputs
     `(;; GHC 9.0.2 must be built with GHC >= 8.8
       ("ghc-bootstrap" ,ghc-8.10)
       ("ghc-testsuite"
        ,(origin
           (method url-fetch)
           (uri (string-append
                  "https://www.haskell.org/ghc/dist/"
                  version "/ghc-" version "-testsuite.tar.xz"))
           (sha256
            (base32
             "1m5fzhr4gjn9ni8gxx7ag3fkbw1rspjzgv39mnfb0nkm5mw70v3s"))))
       ,@(filter (match-lambda
                   (("ghc-bootstrap" . _) #f)
                   (("ghc-testsuite" . _) #f)
                   (_ #t))
                 (package-native-inputs ghc-8.10))))
    (native-search-paths
     (list (search-path-specification
            (variable "GHC_PACKAGE_PATH")
            (files (list (string-append "lib/ghc-" version)))
            (file-pattern ".*\\.conf\\.d$")
            (file-type 'directory))))))
--8<---------------cut here---------------end--------------->8---

and it is similar for ’ghc-8.10’ and other, well something like:

    (native-inputs
     `(
       ("ghc-bootstrap" ,ghc-x.y)
       ("ghc-testsuite"
        ,(origin
           (method url-fetch)
           (uri (string-append
                  "https://www.haskell.org/ghc/dist/"
                  version "/ghc-" version "-testsuite.tar.xz"))

where ghc-x.y is a full GHC containing the complete testsuite.  I
propose here to replace by ’ghc-x.y/bootstrap’ where the tests are
turned off for this ghc-x.y/bootstrap.  We would not have to wait hours
and hours… Hum, I do not know if it is a good idea. :-)

I mean I propose to have both: ghc-x.y (with tests) and
ghc.x.y/bootstrap (without tests), it would ease the maintenance of the
Haskell ecosystem on several architectures.  WDYT?

Cheers,
simon

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 02:19:43PM +0200 schrieb Simon Tournier:
> and instead we could try this shorter one:
>    7.8.4
> -> 8.0.2 (needs >= 7.10)

Here it looks like we still need 7.10.

> where ghc-x.y is a full GHC containing the complete testsuite.  I
> propose here to replace by ’ghc-x.y/bootstrap’ where the tests are
> turned off for this ghc-x.y/bootstrap.  We would not have to wait hours
> and hours… Hum, I do not know if it is a good idea. :-)

Well, it would mean that bootstrapping would somehow happen in parallel
to building the full ghc versions with tests. But is there any use for
keeping these old versions around except for bootstrapping? If not, it
might be enough to enable tests only for the last, user facing version.

Andreas

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 11:03:25AM +0200 schrieb Andreas Enge:
> On aarch64 and powerpc, we are still stuck by CI problems.

Things improve! I could reenable sjd-p9 by a little "guix gc". Thanks to
Ricardo for walking me through a few cuirass steps! So we have four more
build slots on powerpc, five times what we had before. I still do not see
why only one of the two slots on guixp9 is building.

And a second aarch64 machine is online, with people working on others.

Andreas

Re: Core-updates after the staging merge

[Simon Tournier]

On Mon, 17 Apr 2023 at 14:39, Andreas Enge <andreas@enge.fr> wrote:
>
> Am Mon, Apr 17, 2023 at 02:19:43PM +0200 schrieb Simon Tournier:
> > and instead we could try this shorter one:
> >    7.8.4
> > -> 8.0.2 (needs >= 7.10)
>
> Here it looks like we still need 7.10.

Sorry for the typo:

-> 8.0.1 (needs >= 7.8)

Bottom of <https://www.haskell.org/ghc/download_ghc_8_0_1.html>: "The
source distribution needs an installed GHC (version 7.8 at least)."

Contrary to 8.0.2;
<https://www.haskell.org/ghc/download_ghc_8_0_2.html>: "The source
distribution needs an installed GHC (version 7.10 at least)."

Cheers,
simon

Re: Core-updates after the staging merge

[Lars-Dominik Braun]

Hi Simon,

> and instead we could try this shorter one:
> 
>    7.8.4
> -> 8.0.2 (needs >= 7.10)
> -> 8.4.4 (needs >= 8.0)
> -> 8.8.4 (needs >= 8.4)
> -> 9.2.5 (needs >= 8.6)
> 
> WDYT?  If no objection, I will give a try.

note that the information on haskell.org is not always accurate and thus
this shorter chain may not actually work. Please give it a try and send
a patch.

> I mean I propose to have both: ghc-x.y (with tests) and
> ghc.x.y/bootstrap (without tests), it would ease the maintenance of the
> Haskell ecosystem on several architectures.  WDYT?

I have a bad feeling about turning the testsuites of intermediate versions
off. Yes, they take a long time, but then they also ensure the resulting
compiler actually works (with high confidence) and we don’t silently
propagate issues into the next one.

Cheers,
Lars

Re: Core-updates after the staging merge

[Simon Tournier]

Hi,

On lun., 17 avril 2023 at 16:12, Lars-Dominik Braun <lars@6xq.net> wrote:

> note that the information on haskell.org is not always accurate and thus
> this shorter chain may not actually work. Please give it a try and send
> a patch.

If I read correctly, the current chain is:

           7.8.4
        -> 7.10.3
        -> 8.0.2
        -> 8.4.4
        -> 8.6.5
        -> 8.8.4
        -> 8.10.7
        -> 9.0.2; 9.2.5
        -> 9.4.4 (next)

where 9.2.5 is the one used by haskell-build-system.  Instead, this
chain,

           7.8.4  (needs 7.4 at least)
        -> 8.0.1  (needs 7.8 at least)
        -> 8.4.4  (needs 8.0 at least)
        -> 8.6.5  (needs 8.2 at least)
        -> 8.10.7  (needs 8.6 at least)
        -> 9.2.5  (needs 8.10 at least)

builds for me.  So It removes 7.10.3 and 8.8.4.


That said, note that the current binary root is 7.8.4 with the hope to
join with the current,

           4.08.2 (needs GCC and outputs of previous GHC)
        -> 6.0    (needs 4.08 at least)
        -> 6.6    (needs 5.04 at least)
        -> 6.10.4 (needs 6.6  at least)

Well, joining 9.2.5 to 4.08.2 could be done using intermediary versions:

-> 6.10.4 (needs 6.6  at least)  | -> 6.10.4 (needs 6.6  at least)
   6.12.3 (needs 6.8  at least)  |    7.2.2  (needs 6.10 at least)
   7.4.2  (needs 6.12 at least)  |    7.6.3  (needs 7.0  at least)
   7.8.4  (needs 7.4 at least)   | -> 7.10.3 (needs 7.6  at least)
-> 8.0.1  (needs 7.8 at least)   |    8.2.2  (needs 7.10 at least)
-> 8.4.4  (needs 8.0 at least)   | -> 8.6.5  (needs 8.2  at least)
-> 8.6.5  (needs 8.2 at least)   | -> 8.10.7 (needs 8.6 at least)
-> 8.10.7 (needs 8.6 at least)   | -> 9.2.5  (needs 8.10 at least)
-> 9.2.5  (needs 8.10 at least)

Well, one version would be win when using 7.10.3 (modulo inaccurate
information on Haskell website :-)).


All in all, I am proposing to send a patch for the first path for this
core-updates cycle and postpone this other path – not doable for this
cycle; I will resume this story later.


Andreas, core-updates is frozen but is the former shorter bootstrap
chain acceptable?


>> I mean I propose to have both: ghc-x.y (with tests) and
>> ghc.x.y/bootstrap (without tests), it would ease the maintenance of the
>> Haskell ecosystem on several architectures.  WDYT?
>
> I have a bad feeling about turning the testsuites of intermediate versions
> off. Yes, they take a long time, but then they also ensure the resulting
> compiler actually works (with high confidence) and we don’t silently
> propagate issues into the next one.

Well, I am convinced that we are doing the optimal way.  For instance,
the run is sequencial,

--8<---------------cut here---------------start------------->8---
       #:test-target "test"
       ;; We get a smaller number of test failures by disabling parallel test
       ;; execution.
       #:parallel-tests? #f
--8<---------------cut here---------------end--------------->8---

Then, for another example, the GHC testsuite is distributed separately,

    https://downloads.haskell.org/~ghc/8.10.7/ghc-8.10.7-src.tar.xz
    https://downloads.haskell.org/~ghc/8.10.7/ghc-8.10.7-testsuite.tar.xz

And Debian seems considering the GHC testsuite as a package,

    https://tracker.debian.org/pkg/ghc-testsuite

Maybe we could do that.  It would allow to catch errors and not wait
ages after each GHC bootstrap toolchain completes all its testsuite.

WDYT?


Cheers,
simon

Re: Core-updates after the staging merge

[Maxim Cournoyer]

Hi,

Andreas Enge <andreas@enge.fr> writes:

> Hello,
>
> just a quick update after a night of building on CI.
> Things look generally quite good on x86_64; some things are being rebuilt
> due to the recent wget update, and that should hopefully also sort out
> i686 rather quickly.
>
> Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
>> - python: It is mostly green, but with lots of red, including at least
>>   one package that prevents icecat from building. I would call for careful
>>   updates and bug fixes to move on; leaf packages can be updated to the
>>   latest version, but for important intermediate packages please be
>>   careful.
>
> Python probably is the place where work is needed most, the strip in the
> dashboard looks like it has measles. Fixing one package there could bring
> us a long way (or not - we are in plain hand resolution of dependency
> problems there, one update could also hide another one). Sometimes it is
> just a matter of trying an update to see whether a package and its
> dependents build, so even someone who knows nothing about Python, but
> has a web browser pointed at https://pypi.org/, can do some useful work
> (I am speaking from experience).

I think python is starting to look good here, after a few upgrades.
It'll cause a rebuild of at least GTK+, so I'll push it a bit later
today (during the European night).

-- 
Thanks,
Maxim

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 07:47:16PM +0200 schrieb Simon Tournier:
> All in all, I am proposing to send a patch for the first path for this
> core-updates cycle and postpone this other path – not doable for this
> cycle; I will resume this story later.
> Andreas, core-updates is frozen but is the former shorter bootstrap
> chain acceptable?

Well, I would see this as rather an action for a later feature branch.
Except for removing 9.0 by building 9.4 with 9.2, since 9.0 does not
build now anyway.

Andreas

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 02:03:14PM -0400 schrieb Maxim Cournoyer:
> I think python is starting to look good here, after a few upgrades.
> It'll cause a rebuild of at least GTK+, so I'll push it a bit later
> today (during the European night).

Nice! I just merged master back, hopefully this is all compatible with
your changes (and hopefully I did not introduce many errors...).

Andreas

Re: Core-updates after the staging merge

[Lars-Dominik Braun]

Hi Andreas,

> Well, I would see this as rather an action for a later feature branch.
> Except for removing 9.0 by building 9.4 with 9.2, since 9.0 does not
> build now anyway.

shouldn’t this snippet from 8.10 also work for 9.0?

           (modules '((guix build utils)))
           (snippet
            ;; collections.Iterable was moved to collections.abc in Python 3.10.
            '(substitute* "testsuite/driver/testlib.py"
               (("collections\\.Iterable")
                "collections.abc.Iterable")))))

Cheers,
Lars

Re: Core-updates after the staging merge

[John Kehayias]

Hi Andreas and Guixers,

Again, thanks Andreas for all your work on core-updates!

On Mon, Apr 17, 2023 at 11:03 AM, Andreas Enge wrote:

> Hello,
>
> just a quick update after a night of building on CI.
> Things look generally quite good on x86_64; some things are being rebuilt
> due to the recent wget update, and that should hopefully also sort out
> i686 rather quickly.
>

Packages I use all seem to be good too, except wine as there is a
samba failure for i686. Pending fix: <https://issues.guix.gnu.org/62894>

> Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
>> - python: It is mostly green, but with lots of red, including at least
>>   one package that prevents icecat from building. I would call for careful
>>   updates and bug fixes to move on; leaf packages can be updated to the
>>   latest version, but for important intermediate packages please be
>>   careful.
>
> Python probably is the place where work is needed most, the strip in the
> dashboard looks like it has measles. Fixing one package there could bring
> us a long way (or not - we are in plain hand resolution of dependency
> problems there, one update could also hide another one). Sometimes it is
> just a matter of trying an update to see whether a package and its
> dependents build, so even someone who knows nothing about Python, but
> has a web browser pointed at <https://pypi.org/>, can do some useful work
> (I am speaking from experience).
>

I took a stab at a few random packages and ended up finding out that
python-urllib3 needed an update due to our updated
python-cryptography. Only saw this through test failures of leaf
packages rather than itself (noted in upstream update that failures
with newer cryptography tend to point people to the wrong packages).
While urllib3 has quite a few dependents (> 500 rebuilds on update)
and this may break more than it fixes, I pushed the latest version. A
few other updates/build fixes for the packages I was trying as well.

In the chain of oauth/google packages I was looking at, I was left at
python-google-auth-1 (older version) as failing tests even adding in
the missing python-mock. At quick glance I'm guessing due to the newer
python-cryptography. I haven't had a chance to see how we want to work
around that, or if the older versions of the few dependents should be
kept. It is only a few packages anyway, so maybe bigger fish to hunt
first.

I'll check in tomorrow on how the urllib update built on Cuirass and
see if I can pick off any other easy python-* fixes.

Oh, looks like no x86_64 builds currently on Cuirass...hopefully
temporary?

John

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 09:01:20PM +0200 schrieb Lars-Dominik Braun:
> shouldn’t this snippet from 8.10 also work for 9.0?
>            (modules '((guix build utils)))
>            (snippet
>             ;; collections.Iterable was moved to collections.abc in Python 3.10.
>             '(substitute* "testsuite/driver/testlib.py"
>                (("collections\\.Iterable")
>                 "collections.abc.Iterable")))))

Probably so! I will let you decide whether to apply it or to drop the 9.0
version, both are fine from the core-updates point of view.

Andreas

Re: Core-updates after the staging merge

[Andreas Enge]

Am Tue, Apr 18, 2023 at 05:04:35AM +0000 schrieb John Kehayias:
> I took a stab at a few random packages and ended up finding out that
> python-urllib3 needed an update due to our updated
> python-cryptography. Only saw this through test failures of leaf
> packages rather than itself (noted in upstream update that failures
> with newer cryptography tend to point people to the wrong packages).
> While urllib3 has quite a few dependents (> 500 rebuilds on update)
> and this may break more than it fixes, I pushed the latest version. A
> few other updates/build fixes for the packages I was trying as well.

Thanks for working on these packages! I tend to always update to the newest
version, but with Python this seems to be much more tricky when depending
packages do not work with the latest and greatest. Good luck in finding
combinations that work well together!

Andreas

Latest news on core-updates

[Andreas Enge]

Hello all,

just a quick update to share good news and to heap praise on people who are
not rewarded by seeing their name in a git commit.

Am Mon, Apr 17, 2023 at 11:03:25AM +0200 schrieb Andreas Enge:
> On aarch64 and powerpc, we are still stuck by CI problems.

Thanks to tireless work by Ricardo, Ludovic, Julien, Christopher and myself
enough machines for these architectures are running in the build farm to
hopefully clear the backlog. We went from 1 to 5 aarch machines (or from 4
to 14 build slots), and from 1 to 2 powerpc64le machines (or from 2 to 8
build slots).

The dash boards are filling up, and on x86_64 and i686 the situation looks
comparable to master. On powerpc64le, I have been restarting cancelled jobs
with a little script, and the red dots are also becoming less. On aarch64,
I am waiting for the backlog from master to clear up before trying something
similar. I think the situation on these architectures is better than it
looks on the dashboard.

I hope we will be able to merge core-updates after next weekend; also for
personal reasons, since I will not be able to spend much time on it any
more after that.

So please help us in a last sprint to get core-updates into shape!

Andreas

Re: Latest news on core-updates

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]


Andreas Enge <andreas@enge.fr> writes:

> I hope we will be able to merge core-updates after next weekend; also for
> personal reasons, since I will not be able to spend much time on it any
> more after that.

After next weekend (around the end of April) sounds good to me.

The bordeaux build farm has pretty much caught up after the staging
merge which should mean there's some capacity to attempt to build
core-updates changes prior to the merge around next weekend.

I haven't restarted submitting builds for core-updates given I think
there's still some big changes going to land, but in any case, it would
be good to get this going once there are no planned big changes (aka the
branch is frozen). Getting the builds to happen currently requires a
code change in the qa-frontpage, but I can do this, I just need someone
to let me know when I should.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Latest news on core-updates

[Andreas Enge]

Am Wed, Apr 19, 2023 at 11:58:03AM +0100 schrieb Christopher Baines:
> I haven't restarted submitting builds for core-updates given I think
> there's still some big changes going to land, but in any case, it would
> be good to get this going once there are no planned big changes (aka the
> branch is frozen).

I am not sure what "frozen" means exactly; I would say we must not make
commits to core-updates unless they repair a broken package, or maybe help
repair broken packages further down towards the leaves. (And this has been
true for a while now.) And maybe we should go further and refrain from
repairing things in core-updates that are already broken on master, at
least if they are relatively close to the root of the dependency graph.
These can be done later in feature branches; I think we have the infra-
structure to shoulder bigger changes outside of core-updates.

On the other hand, this report is a bit worrying:
   [core-updates] locales not installed properly
   https://issues.guix.gnu.org/62934
It could potentially mean changes to glibc, which would throw us back
a week.

Andreas

Re: Latest news on core-updates

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Andreas Enge <andreas@enge.fr> writes:
>
>> I hope we will be able to merge core-updates after next weekend; also for
>> personal reasons, since I will not be able to spend much time on it any
>> more after that.
>
> After next weekend (around the end of April) sounds good to me.
>
> The bordeaux build farm has pretty much caught up after the staging
> merge which should mean there's some capacity to attempt to build
> core-updates changes prior to the merge around next weekend.
>
> I haven't restarted submitting builds for core-updates given I think
> there's still some big changes going to land, but in any case, it would
> be good to get this going once there are no planned big changes (aka the
> branch is frozen). Getting the builds to happen currently requires a
> code change in the qa-frontpage, but I can do this, I just need someone
> to let me know when I should.

Andreas suggested on IRC that I get things building, so I've configured
the qa-frontpage to start submitting builds for core-updates again.

You can track the progress here [1] of course, and if you want to get an
idea if any of the builds are currently happening, you can get some
information on what builds are assigned to what agents here [2]
(providing the experimental interface hasn't crashed).

1: https://qa.guix.gnu.org/branch/core-updates
2: https://bordeaux.guix.gnu.org/activity

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Latest news on core-updates

[Andreas Enge]

Am Thu, Apr 20, 2023 at 02:56:48PM +0100 schrieb Christopher Baines:
> Andreas suggested on IRC that I get things building, so I've configured
> the qa-frontpage to start submitting builds for core-updates again.
> You can track the progress here [1] of course, and if you want to get an
> idea if any of the builds are currently happening, you can get some
> information on what builds are assigned to what agents here [2]
> (providing the experimental interface hasn't crashed).

Thanks! I see numbers going up on powerpc :-)
[2] times out, however.

Andreas

Re: Latest news on core-updates

[Andreas Enge]

Hello Maxim,

Am Wed, Apr 19, 2023 at 02:41:31PM +0200 schrieb Andreas Enge:
> I am not sure what "frozen" means exactly; I would say we must not make
> commits to core-updates unless they repair a broken package, or maybe help
> repair broken packages further down towards the leaves.

your recent docbook commits probably do not fit this definition, much less
so than the suggested valgrind update that we nevertheless postponed until
after the merge. They cause an extraordinary number of rebuilds. I hope they
are mostly harmless and do not cause additional build failures, so keeping
them in is probably easier than reverting them.

But let us from now on stick with commits that make important packages work,
or packages very close to the leaves.

I suggest the following timetable:
- Hard freeze now, only important broken packages or packages close to the
  leaves may be changed.
- Over the weekend, update your systems and profiles and test and repair
  problems that appear.
- If no show-stopper occurs, merge core-updates into master on Tuesday.

Andreas

Re: Latest news on core-updates

[Simon Tournier]

Hi Andreas,

On Wed, 19 Apr 2023 at 12:48, Andreas Enge <andreas@enge.fr> wrote:

> I hope we will be able to merge core-updates after next weekend; also for
> personal reasons, since I will not be able to spend much time on it any
> more after that.

I do not know if something is twisted with the report [1] of Cuirass but
many things seems missing.  Right?

Therefore, if many things are still missing, I suggest to apply #62967
[2].  It removes the annoyance you spotted out in #62954 [3].  And as I
explained in [4], it will be safe because ’valgrind’ appears only in the
testsuite of ’lz4’ and ’valgrind’ is not even run there.

Doing so, it will fix failures for powerpc64le and will increase the
coverage.

WDYT?


1: https://ci.guix.gnu.org/eval/417699/dashboard
2: https://issues.guix.gnu.org/62967
3 https://issues.guix.gnu.org/62954
4: https://issues.guix.gnu.org/62954#3


Moreover, I have revamped the GHC bootstrap chain.

 1. it cuts the number of Haskell compilers to build,
 2. it separates building the compiler itself and running the testsuite,
 3. it introduces ghc-toolchain that glue one compiler and its testsuite.

Well, #2 removing failure for i686 [5] since the failure of the
testsuite of 8.10.7 is not blocking for 9.2.5.

However, the patches needs more polishing and then because of ’pandoc’
depending on the GHC bootstrap, it would a large rebuild.

Since Haskell is also broken on master for i686, I guess the option is
to apply on some “feature™ branch” after the merge, right?


5: https://ci.guix.gnu.org/build/953241/details

Cheers,
simon

Re: Latest news on core-updates

[Andreas Enge]

Am Fri, Apr 21, 2023 at 10:20:18AM +0200 schrieb Simon Tournier:
> Since Haskell is also broken on master for i686, I guess the option is
> to apply on some “feature™ branch” after the merge, right?

Yes indeed.

> Therefore, if many things are still missing, I suggest to apply #62967
> [2].  It removes the annoyance you spotted out in #62954 [3].  And as I
> explained in [4], it will be safe because ’valgrind’ appears only in the
> testsuite of ’lz4’ and ’valgrind’ is not even run there.
> Doing so, it will fix failures for powerpc64le and will increase the
> coverage.

CI just disappeared, so I cannot check. If r-minimal builds for powerpc64le
on master, then it is justifiable to fix the regression (leaving open the
possibility of an immediate revert if the patch introduces problems else-
where, which I do not expect). Otherwise I would postpone it until after
the merge.

Andreas

Re: Latest news on core-updates

[Simon Tournier]

Hi,

On ven., 21 avril 2023 at 10:47, Andreas Enge <andreas@enge.fr> wrote:

>> Therefore, if many things are still missing, I suggest to apply #62967
>> [2].  It removes the annoyance you spotted out in #62954 [3].  And as I
>> explained in [4], it will be safe because ’valgrind’ appears only in the
>> testsuite of ’lz4’ and ’valgrind’ is not even run there.
>> Doing so, it will fix failures for powerpc64le and will increase the
>> coverage.
>
> CI just disappeared, so I cannot check. If r-minimal builds for powerpc64le
> on master, then it is justifiable to fix the regression (leaving open the
> possibility of an immediate revert if the patch introduces problems else-
> where, which I do not expect). Otherwise I would postpone it until after
> the merge.

Yes r-minimal builds for powerpc64le-linux on master:

    https://ci.guix.gnu.org/build/1050833/details

--8<---------------cut here---------------start------------->8---
simon@pfiuh07$ guix weather -s powerpc64le-linux r-minimal
computing 1 package derivations for powerpc64le-linux...
looking for 1 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org ☀
  100.0% substitutes available (1 out of 1)
  at least 49,1 MiB of nars (compressed)
  52,7 MiB on disk (uncompressed)
  2,529 seconds per request (2,5 seconds in total)
  0,4 requests per second

  at least 1 000 queued builds
      aarch64-linux: 819 (81.9%)
      x86_64-linux: 117 (11.7%)
      powerpc64le-linux: 7 (.7%)
      i686-linux: 57 (5.7%)
  build rate: 189.97 builds per hour
      i686-linux: 138.13 builds per hour
      x86_64-linux: 27.34 builds per hour
      powerpc64le-linux: 21.66 builds per hour
      aarch64-linux: 5.56 builds per hour
looking for 1 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org ☀
  100.0% substitutes available (1 out of 1)
  24,2 MiB of nars (compressed)
  52,7 MiB on disk (uncompressed)
  0,852 seconds per request (0,9 seconds in total)
  1,2 requests per second
  'https://bordeaux.guix.gnu.org/api/queue?nr=1000' returned 502 ("Bad Gateway")
--8<---------------cut here---------------end--------------->8---

Note that the patch removes the dependency between subversion and
valgrind.  Therefore, as you noted, many TeXlive packages could also
build.

Well, I do not see any cons. :-)  The only difference for ’lz4’ is the
path of the store item; here without and with the removal of valgrind:

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference /gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3 /gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3
diff -r --no-dereference /gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/lib/pkgconfig/liblz4.pc /gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/lib/pkgconfig/liblz4.pc
5,7c5,7
< prefix=/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3
< libdir=/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/lib
< includedir=/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/include
---
> prefix=/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3
> libdir=/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/lib
> includedir=/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/include
13,14c13,14
< Libs: -L/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/lib -llz4
< Cflags: -I/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/include
---
> Libs: -L/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/lib -llz4
> Cflags: -I/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/include
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

Re: Latest news on core-updates

[Maxim Cournoyer]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> Hello Maxim,
>
> Am Wed, Apr 19, 2023 at 02:41:31PM +0200 schrieb Andreas Enge:
>> I am not sure what "frozen" means exactly; I would say we must not make
>> commits to core-updates unless they repair a broken package, or maybe help
>> repair broken packages further down towards the leaves.
>
> your recent docbook commits probably do not fit this definition, much less
> so than the suggested valgrind update that we nevertheless postponed until
> after the merge. They cause an extraordinary number of rebuilds. I hope they
> are mostly harmless and do not cause additional build failures, so keeping
> them in is probably easier than reverting them.

10,000 total rebuilds for Cuirass is about 2500 packages per platforms.
It's large, but not 'extraordinary' large :-).  8 hours later we're back
to our current 50% substitute coverage target.

> But let us from now on stick with commits that make important packages work,
> or packages very close to the leaves.
>
> I suggest the following timetable:
> - Hard freeze now, only important broken packages or packages close to the
>   leaves may be changed.
> - Over the weekend, update your systems and profiles and test and repair
>   problems that appear.
> - If no show-stopper occurs, merge core-updates into master on Tuesday.

OK, thanks for making this clear!  I'll refrain from changes causing big
rebuilds from now on, unless they are truly unavoidable.  Sounds like a
good plan.

Cheers,

-- 
Maxim

Re: Latest news on core-updates

[Katherine Cox-Buday]

On 4/19/23 4:48 AM, Andreas Enge wrote:

> just a quick update to share good news and to heap praise on people who are
> not rewarded by seeing their name in a git commit.

Consider my praise added to the heap (and hopefully not GCed). As 
always, I am deeply appreciative to all of our maintainers.

Thank you, thank you, thank you!

--
Katherine

Re: Latest news on core-updates

[Katherine Cox-Buday]

On 4/19/23 4:48 AM, Andreas Enge wrote:

> just a quick update to share good news and to heap praise on people who are
> not rewarded by seeing their name in a git commit.

Consider my praise added to the heap (and hopefully not GCed). As 
always, I am deeply appreciative to all of our maintainers.

Thank you, thank you, thank you!

--
Katherine

Re: Latest news on core-updates

[reza.housseini]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

Consider my praise added as well! Was there already a discussion about adding liberapay, to at least have some monetary compensation for the hard and necessary work done by all these volunteers?

On April 21, 2023 6:12:55 PM GMT+02:00, Katherine Cox-Buday <cox.katherine.e@gmail.com> wrote:
>On 4/19/23 4:48 AM, Andreas Enge wrote:
>
>> just a quick update to share good news and to heap praise on people who are
>> not rewarded by seeing their name in a git commit.
>
>Consider my praise added to the heap (and hopefully not GCed). As always, I am deeply appreciative to all of our maintainers.
>
>Thank you, thank you, thank you!
>
>--
>Katherine
>

-- Sent from /e/ Mail.

[-- Attachment #2: Type: text/html, Size: 1086 bytes --]

Re: Core-updates after the staging merge

[Lars-Dominik Braun]

Hi Andreas,

> Probably so! I will let you decide whether to apply it or to drop the 9.0
> version, both are fine from the core-updates point of view.

I fixed 9.0 in commit dc9c09023a5258de035424169b8e804acfd38cb2, but 9.4
still fails :( Will investigate.

Lars

Re: Latest news on core-updates

[Andreas Enge]

Am Fri, Apr 21, 2023 at 07:04:19PM +0200 schrieb reza.housseini@gmail.com:
> Consider my praise added as well! Was there already a discussion about adding
> liberapay, to at least have some monetary compensation for the hard and
> necessary work done by all these volunteers?

Thanks for the thanks! Well, the volunteers are volunteers; if you want
to donate to Guix (so far, mainly for covering infrastructure costs and
Outreachy internships), you can do so at Guix Europe or the FSF.

Andreas