Core-updates after the staging merge

Core-updates after the staging merge

[Andreas Enge]

Hello all,

the merge of staging to master, and the subsequent merge of master to
core-updates did break a few things; but on the positive side, we are
halfway there with getting rid of the staging and core-updates branches ;-)
CI has almost caught up on x86_64; looking at the dashboard at
   https://ci.guix.gnu.org/eval/402403/dashboard
shows much more red than on master, so we will need to do some more work
before the merge. Since it is ordered alphabetically and our package names
often start by the language, red streaks often indicate problems with
a given programming language. There are things to work on for most of the
teams!

- rust: This one bootstrapped up to the latest version! There are a few
  red dots for individual packages.
- python: It is mostly green, but with lots of red, including at least
  one package that prevents icecat from building. I would call for careful
  updates and bug fixes to move on; leaf packages can be updated to the
  latest version, but for important intermediate packages please be
  careful.
- Go looks good.
- Java is red, but apparently still bootstrapping; everything hinges on
  icedtea@2.6, which is still scheduled for build:
     https://ci.guix.gnu.org/build/1007080/details
  So we need to wait, and potentially restart jobs that are marked as
  failed, but may actually build once their inputs are there.
- Everything common lisp related is red (cl-*, sbcl-*), but maybe there
  is a similar thing going; or maybe not, since clisp is built.
- R is red, but for the time being held up by texlive packages still being
  built.
- ghc is taking a long time...

A single package holds up a lot (gtk+ and so on): openldap. It fails after
about a minute of compiling like so:
  starting phase `provide-libldap_r'
  error: in phase 'provide-libldap_r': uncaught exception:
  unbound-variable #f "Unbound variable: ~S" (ungexp) #f
  phase `provide-libldap_r' failed after 0.0 seconds
This looks more like a typo than anything else.


On i686, we are still stuck by wget.


On aarch64, I think we are mainly stuck by lack of build power. I cancelled
a few build jobs on CI belonging to older git commits, but there is not much
we can do apart from bringing back build machines.

On powerpc, my cancelling of old jobs has apparently also cancelled newer
jobs that are actually the same; I am trying to restart them all, but am
not sure if I succeed. Apologies! As written yesterday, CI does not seem
to use all the build power at our disposal for this architecture.

Andreas

[bug#62863] Openldap in core-updates

[Andreas Enge]

Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
> A single package holds up a lot (gtk+ and so on): openldap.

It was pointed out to me on IRC that there are already two fixes available
on our issue tracker! I applied one of them and also removed the additional
phase as suggested there.

In a separate commit, I also updated to 2.6.4 and removed the non-hidden
openldap-for-linphone variable, which had the confusing effect that the
command line would refer to this package (so "guix refresh -l openldap"
showed almost no dependents, for instance).

If any of these pose problems, we should selectively revert part of the
commits; but I am rather optimistic, as both versions create a library
with the same soname.

Thanks to all involved!

Andreas

wget on i686 in core-updates

[Andreas Enge]

Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
> On i686, we are still stuck by wget.

HEAD of git solves the problem, I will try to propose a solution later.

Andreas

Re: wget on i686 in core-updates

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

Am Sun, Apr 16, 2023 at 04:00:15PM +0200 schrieb Andreas Enge:
> HEAD of git solves the problem, I will try to propose a solution later.

Due to gnulib handled as a submodule, this turned out to be quite tricky
(for details, see the IRC logs with my discussion with jlicht).
The solution is to create a tarball after checking out the gnulib
submodule corresponding to the latest wget git commit and after running
./bootstrap.

As a stop-gap measure, I suggest to use a self-hosted tarball as in the
attached commit. I have good hope that we will soon see a 1.23.4 release,
see https://lists.gnu.org/archive/html/bug-wget/2023-04/msg00002.html .

What do you think?

Andreas


[-- Attachment #2: 0001-gnu-wget-Update-to-1.21.3.24.patch --]
[-- Type: text/plain, Size: 1392 bytes --]

From 424ee72ffbdf9a79eee4dd4aac43404167e66083 Mon Sep 17 00:00:00 2001
From: Andreas Enge <andreas@enge.fr>
Date: Sun, 16 Apr 2023 20:52:35 +0200
Subject: [PATCH] gnu: wget: Update to 1.21.3.24.

This update to a non-release version fixes a build failure on i686-linux.

* gnu/packages/wget.scm (wget): Update to 1.21.3.24.
[origin]: Use a self-hosted tarball created from the latest git commit.
---
 gnu/packages/wget.scm | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/wget.scm b/gnu/packages/wget.scm
index 083cf27212..17459344c0 100644
--- a/gnu/packages/wget.scm
+++ b/gnu/packages/wget.scm
@@ -46,15 +46,16 @@ (define-module (gnu packages wget)
 (define-public wget
   (package
     (name "wget")
-    (version "1.21.3")
+    (version "1.21.3.24")
     (source
      (origin
       (method url-fetch)
-      (uri (string-append "mirror://gnu/wget/wget-"
-                          version ".tar.lz"))
+      ;;(uri (string-append "mirror://gnu/wget/wget-"
+      ;;                    version ".tar.lz"))
+      (uri "https://www.multiprecision.org/wget-1.21.3.24-2b723.tar.lz")
       (sha256
        (base32
-        "19afmyr1i3zwdwr8wkyz8q6z5764ik3dm87as194g78l8xggplnv"))))
+        "17ip94mvax83h0gh4905jqc64g5qf3vgxr3bj9gn02pijjm5lzbp"))))
     (build-system gnu-build-system)
     (inputs
      (list gnutls libidn2 libpsl))
-- 
2.39.2

Re: wget on i686 in core-updates

[Andreas Enge]

Am Sun, Apr 16, 2023 at 08:57:48PM +0200 schrieb Andreas Enge:
> As a stop-gap measure, I suggest to use a self-hosted tarball as in the
> attached commit. I have good hope that we will soon see a 1.23.4 release,
> see https://lists.gnu.org/archive/html/bug-wget/2023-04/msg00002.html .

I just pushed the commit. There should be a new wget release this week:
    https://lists.gnu.org/archive/html/bug-wget/2023-04/msg00005.html
and then we can push the update. Having wget will enable us to see where
more work is needed on i686.

Andreas

Re: Core-updates after the staging merge

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

Andreas Enge <andreas@enge.fr> skribis:

> Hello all,
>
> the merge of staging to master, and the subsequent merge of master to
> core-updates did break a few things; but on the positive side, we are
> halfway there with getting rid of the staging and core-updates branches ;-)
> CI has almost caught up on x86_64; looking at the dashboard at
>    https://ci.guix.gnu.org/eval/402403/dashboard
> shows much more red than on master, so we will need to do some more work
> before the merge. Since it is ordered alphabetically and our package names
> often start by the language, red streaks often indicate problems with
> a given programming language. There are things to work on for most of the
> teams!
>
> [...]
> - Everything common lisp related is red (cl-*, sbcl-*), but maybe there
>   is a similar thing going; or maybe not, since clisp is built.
> [...]

Hi,
I tried to build the 1611 dependents of sbcl on x86-64, and most of them
build fine. I get only 6 failures, some of them because some
dependencies like mysql or supercollider are failing to build.
So overall, Common Lisp packages on core-updates are in a pretty good
shape.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 08:18:00AM +0000 schrieb Guillaume Le Vaillant:
> I tried to build the 1611 dependents of sbcl on x86-64, and most of them
> build fine. I get only 6 failures, some of them because some
> dependencies like mysql or supercollider are failing to build.
> So overall, Common Lisp packages on core-updates are in a pretty good
> shape.

Great, thanks for testing! Indeed it looks as if the buildfarm just
needed to catch up.

Andreas

Re: Core-updates after the staging merge

[Andreas Enge]

Hello,

just a quick update after a night of building on CI.
Things look generally quite good on x86_64; some things are being rebuilt
due to the recent wget update, and that should hopefully also sort out
i686 rather quickly.

Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
> - python: It is mostly green, but with lots of red, including at least
>   one package that prevents icecat from building. I would call for careful
>   updates and bug fixes to move on; leaf packages can be updated to the
>   latest version, but for important intermediate packages please be
>   careful.

Python probably is the place where work is needed most, the strip in the
dashboard looks like it has measles. Fixing one package there could bring
us a long way (or not - we are in plain hand resolution of dependency
problems there, one update could also hide another one). Sometimes it is
just a matter of trying an update to see whether a package and its
dependents build, so even someone who knows nothing about Python, but
has a web browser pointed at https://pypi.org/, can do some useful work
(I am speaking from experience).

> - ghc is taking a long time...

This is a real nightmare. ghc@9.2 on x86_64 has been building for 6 hours
in a row:
   https://ci.guix.gnu.org/build/984270/details
and we need to build many versions one after the other during bootstrapping.
While not blocking the core-updates merge, maybe the ghc team can try and
see whether we can cut corners. Or maybe drop tests on intermediate versions
that are only required for bootstrapping?

On aarch64 and powerpc, we are still stuck by CI problems.

Andreas

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 11:03:25AM +0200 schrieb Andreas Enge:
> - ghc is taking a long time...

Actually ghc@9.0 fails its tests, which are written in Python (it looks
like ".abc" needs to be added to collections); but since it is not needed
for bootstrapping any more, maybe we could drop it?

Well, there is the dependent package ghc-next@9.4.4 that would need to be
built from 8.10 also, or maybe 9.2. Some work for the haskell team ;-)

Andreas

Re: Core-updates after the staging merge

[Simon Tournier]

Hi,

On lun., 17 avril 2023 at 11:56, Andreas Enge <andreas@enge.fr> wrote:
> Am Mon, Apr 17, 2023 at 11:03:25AM +0200 schrieb Andreas Enge:
>> - ghc is taking a long time...
>
> Actually ghc@9.0 fails its tests, which are written in Python (it looks
> like ".abc" needs to be added to collections); but since it is not needed
> for bootstrapping any more, maybe we could drop it?

Well, the current Haskell bootstrap chain is:

   7.8.4
-> 7.10.3
-> 8.0.2
-> 8.4.4
-> 8.6.5
-> 8.8.4
-> 8.10.7
-> 9.0.2; 9.2.5
-> 9.4.4 (next)

and instead we could try this shorter one:

   7.8.4
-> 8.0.2 (needs >= 7.10)
-> 8.4.4 (needs >= 8.0)
-> 8.8.4 (needs >= 8.4)
-> 9.2.5 (needs >= 8.6)

WDYT?  If no objection, I will give a try.



Moreover, the recipe for GHC looks like:

--8<---------------cut here---------------start------------->8---
(define-public ghc-9.0
  (package
    (inherit ghc-8.10)
    (name "ghc")
    (version "9.0.2")
    (source (origin
              (method url-fetch)
              (uri (string-append "https://www.haskell.org/ghc/dist/" version
                                  "/ghc-" version "-src.tar.xz"))
              (sha256
               (base32
                "15wii8can2r3dcl6jjmd50h2jvn7rlmn05zb74d2scj6cfwl43hl"))))
    (native-inputs
     `(;; GHC 9.0.2 must be built with GHC >= 8.8
       ("ghc-bootstrap" ,ghc-8.10)
       ("ghc-testsuite"
        ,(origin
           (method url-fetch)
           (uri (string-append
                  "https://www.haskell.org/ghc/dist/"
                  version "/ghc-" version "-testsuite.tar.xz"))
           (sha256
            (base32
             "1m5fzhr4gjn9ni8gxx7ag3fkbw1rspjzgv39mnfb0nkm5mw70v3s"))))
       ,@(filter (match-lambda
                   (("ghc-bootstrap" . _) #f)
                   (("ghc-testsuite" . _) #f)
                   (_ #t))
                 (package-native-inputs ghc-8.10))))
    (native-search-paths
     (list (search-path-specification
            (variable "GHC_PACKAGE_PATH")
            (files (list (string-append "lib/ghc-" version)))
            (file-pattern ".*\\.conf\\.d$")
            (file-type 'directory))))))
--8<---------------cut here---------------end--------------->8---

and it is similar for ’ghc-8.10’ and other, well something like:

    (native-inputs
     `(
       ("ghc-bootstrap" ,ghc-x.y)
       ("ghc-testsuite"
        ,(origin
           (method url-fetch)
           (uri (string-append
                  "https://www.haskell.org/ghc/dist/"
                  version "/ghc-" version "-testsuite.tar.xz"))

where ghc-x.y is a full GHC containing the complete testsuite.  I
propose here to replace by ’ghc-x.y/bootstrap’ where the tests are
turned off for this ghc-x.y/bootstrap.  We would not have to wait hours
and hours… Hum, I do not know if it is a good idea. :-)

I mean I propose to have both: ghc-x.y (with tests) and
ghc.x.y/bootstrap (without tests), it would ease the maintenance of the
Haskell ecosystem on several architectures.  WDYT?

Cheers,
simon

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 02:19:43PM +0200 schrieb Simon Tournier:
> and instead we could try this shorter one:
>    7.8.4
> -> 8.0.2 (needs >= 7.10)

Here it looks like we still need 7.10.

> where ghc-x.y is a full GHC containing the complete testsuite.  I
> propose here to replace by ’ghc-x.y/bootstrap’ where the tests are
> turned off for this ghc-x.y/bootstrap.  We would not have to wait hours
> and hours… Hum, I do not know if it is a good idea. :-)

Well, it would mean that bootstrapping would somehow happen in parallel
to building the full ghc versions with tests. But is there any use for
keeping these old versions around except for bootstrapping? If not, it
might be enough to enable tests only for the last, user facing version.

Andreas

Re: Core-updates after the staging merge

[Simon Tournier]

On Mon, 17 Apr 2023 at 14:39, Andreas Enge <andreas@enge.fr> wrote:
>
> Am Mon, Apr 17, 2023 at 02:19:43PM +0200 schrieb Simon Tournier:
> > and instead we could try this shorter one:
> >    7.8.4
> > -> 8.0.2 (needs >= 7.10)
>
> Here it looks like we still need 7.10.

Sorry for the typo:

-> 8.0.1 (needs >= 7.8)

Bottom of <https://www.haskell.org/ghc/download_ghc_8_0_1.html>: "The
source distribution needs an installed GHC (version 7.8 at least)."

Contrary to 8.0.2;
<https://www.haskell.org/ghc/download_ghc_8_0_2.html>: "The source
distribution needs an installed GHC (version 7.10 at least)."

Cheers,
simon

Re: Core-updates after the staging merge

[Lars-Dominik Braun]

Hi Simon,

> and instead we could try this shorter one:
> 
>    7.8.4
> -> 8.0.2 (needs >= 7.10)
> -> 8.4.4 (needs >= 8.0)
> -> 8.8.4 (needs >= 8.4)
> -> 9.2.5 (needs >= 8.6)
> 
> WDYT?  If no objection, I will give a try.

note that the information on haskell.org is not always accurate and thus
this shorter chain may not actually work. Please give it a try and send
a patch.

> I mean I propose to have both: ghc-x.y (with tests) and
> ghc.x.y/bootstrap (without tests), it would ease the maintenance of the
> Haskell ecosystem on several architectures.  WDYT?

I have a bad feeling about turning the testsuites of intermediate versions
off. Yes, they take a long time, but then they also ensure the resulting
compiler actually works (with high confidence) and we don’t silently
propagate issues into the next one.

Cheers,
Lars

Re: Core-updates after the staging merge

[Simon Tournier]

Hi,

On lun., 17 avril 2023 at 16:12, Lars-Dominik Braun <lars@6xq.net> wrote:

> note that the information on haskell.org is not always accurate and thus
> this shorter chain may not actually work. Please give it a try and send
> a patch.

If I read correctly, the current chain is:

           7.8.4
        -> 7.10.3
        -> 8.0.2
        -> 8.4.4
        -> 8.6.5
        -> 8.8.4
        -> 8.10.7
        -> 9.0.2; 9.2.5
        -> 9.4.4 (next)

where 9.2.5 is the one used by haskell-build-system.  Instead, this
chain,

           7.8.4  (needs 7.4 at least)
        -> 8.0.1  (needs 7.8 at least)
        -> 8.4.4  (needs 8.0 at least)
        -> 8.6.5  (needs 8.2 at least)
        -> 8.10.7  (needs 8.6 at least)
        -> 9.2.5  (needs 8.10 at least)

builds for me.  So It removes 7.10.3 and 8.8.4.


That said, note that the current binary root is 7.8.4 with the hope to
join with the current,

           4.08.2 (needs GCC and outputs of previous GHC)
        -> 6.0    (needs 4.08 at least)
        -> 6.6    (needs 5.04 at least)
        -> 6.10.4 (needs 6.6  at least)

Well, joining 9.2.5 to 4.08.2 could be done using intermediary versions:

-> 6.10.4 (needs 6.6  at least)  | -> 6.10.4 (needs 6.6  at least)
   6.12.3 (needs 6.8  at least)  |    7.2.2  (needs 6.10 at least)
   7.4.2  (needs 6.12 at least)  |    7.6.3  (needs 7.0  at least)
   7.8.4  (needs 7.4 at least)   | -> 7.10.3 (needs 7.6  at least)
-> 8.0.1  (needs 7.8 at least)   |    8.2.2  (needs 7.10 at least)
-> 8.4.4  (needs 8.0 at least)   | -> 8.6.5  (needs 8.2  at least)
-> 8.6.5  (needs 8.2 at least)   | -> 8.10.7 (needs 8.6 at least)
-> 8.10.7 (needs 8.6 at least)   | -> 9.2.5  (needs 8.10 at least)
-> 9.2.5  (needs 8.10 at least)

Well, one version would be win when using 7.10.3 (modulo inaccurate
information on Haskell website :-)).


All in all, I am proposing to send a patch for the first path for this
core-updates cycle and postpone this other path – not doable for this
cycle; I will resume this story later.


Andreas, core-updates is frozen but is the former shorter bootstrap
chain acceptable?


>> I mean I propose to have both: ghc-x.y (with tests) and
>> ghc.x.y/bootstrap (without tests), it would ease the maintenance of the
>> Haskell ecosystem on several architectures.  WDYT?
>
> I have a bad feeling about turning the testsuites of intermediate versions
> off. Yes, they take a long time, but then they also ensure the resulting
> compiler actually works (with high confidence) and we don’t silently
> propagate issues into the next one.

Well, I am convinced that we are doing the optimal way.  For instance,
the run is sequencial,

--8<---------------cut here---------------start------------->8---
       #:test-target "test"
       ;; We get a smaller number of test failures by disabling parallel test
       ;; execution.
       #:parallel-tests? #f
--8<---------------cut here---------------end--------------->8---

Then, for another example, the GHC testsuite is distributed separately,

    https://downloads.haskell.org/~ghc/8.10.7/ghc-8.10.7-src.tar.xz
    https://downloads.haskell.org/~ghc/8.10.7/ghc-8.10.7-testsuite.tar.xz

And Debian seems considering the GHC testsuite as a package,

    https://tracker.debian.org/pkg/ghc-testsuite

Maybe we could do that.  It would allow to catch errors and not wait
ages after each GHC bootstrap toolchain completes all its testsuite.

WDYT?


Cheers,
simon

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 07:47:16PM +0200 schrieb Simon Tournier:
> All in all, I am proposing to send a patch for the first path for this
> core-updates cycle and postpone this other path – not doable for this
> cycle; I will resume this story later.
> Andreas, core-updates is frozen but is the former shorter bootstrap
> chain acceptable?

Well, I would see this as rather an action for a later feature branch.
Except for removing 9.0 by building 9.4 with 9.2, since 9.0 does not
build now anyway.

Andreas

Re: Core-updates after the staging merge

[Lars-Dominik Braun]

Hi Andreas,

> Well, I would see this as rather an action for a later feature branch.
> Except for removing 9.0 by building 9.4 with 9.2, since 9.0 does not
> build now anyway.

shouldn’t this snippet from 8.10 also work for 9.0?

           (modules '((guix build utils)))
           (snippet
            ;; collections.Iterable was moved to collections.abc in Python 3.10.
            '(substitute* "testsuite/driver/testlib.py"
               (("collections\\.Iterable")
                "collections.abc.Iterable")))))

Cheers,
Lars

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 09:01:20PM +0200 schrieb Lars-Dominik Braun:
> shouldn’t this snippet from 8.10 also work for 9.0?
>            (modules '((guix build utils)))
>            (snippet
>             ;; collections.Iterable was moved to collections.abc in Python 3.10.
>             '(substitute* "testsuite/driver/testlib.py"
>                (("collections\\.Iterable")
>                 "collections.abc.Iterable")))))

Probably so! I will let you decide whether to apply it or to drop the 9.0
version, both are fine from the core-updates point of view.

Andreas

Re: Core-updates after the staging merge

[Lars-Dominik Braun]

Hi Andreas,

> Probably so! I will let you decide whether to apply it or to drop the 9.0
> version, both are fine from the core-updates point of view.

I fixed 9.0 in commit dc9c09023a5258de035424169b8e804acfd38cb2, but 9.4
still fails :( Will investigate.

Lars

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 11:03:25AM +0200 schrieb Andreas Enge:
> On aarch64 and powerpc, we are still stuck by CI problems.

Things improve! I could reenable sjd-p9 by a little "guix gc". Thanks to
Ricardo for walking me through a few cuirass steps! So we have four more
build slots on powerpc, five times what we had before. I still do not see
why only one of the two slots on guixp9 is building.

And a second aarch64 machine is online, with people working on others.

Andreas

Re: Core-updates after the staging merge

[Maxim Cournoyer]

Hi,

Andreas Enge <andreas@enge.fr> writes:

> Hello,
>
> just a quick update after a night of building on CI.
> Things look generally quite good on x86_64; some things are being rebuilt
> due to the recent wget update, and that should hopefully also sort out
> i686 rather quickly.
>
> Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
>> - python: It is mostly green, but with lots of red, including at least
>>   one package that prevents icecat from building. I would call for careful
>>   updates and bug fixes to move on; leaf packages can be updated to the
>>   latest version, but for important intermediate packages please be
>>   careful.
>
> Python probably is the place where work is needed most, the strip in the
> dashboard looks like it has measles. Fixing one package there could bring
> us a long way (or not - we are in plain hand resolution of dependency
> problems there, one update could also hide another one). Sometimes it is
> just a matter of trying an update to see whether a package and its
> dependents build, so even someone who knows nothing about Python, but
> has a web browser pointed at https://pypi.org/, can do some useful work
> (I am speaking from experience).

I think python is starting to look good here, after a few upgrades.
It'll cause a rebuild of at least GTK+, so I'll push it a bit later
today (during the European night).

-- 
Thanks,
Maxim

Re: Core-updates after the staging merge

[Andreas Enge]

Am Mon, Apr 17, 2023 at 02:03:14PM -0400 schrieb Maxim Cournoyer:
> I think python is starting to look good here, after a few upgrades.
> It'll cause a rebuild of at least GTK+, so I'll push it a bit later
> today (during the European night).

Nice! I just merged master back, hopefully this is all compatible with
your changes (and hopefully I did not introduce many errors...).

Andreas

Re: Core-updates after the staging merge

[John Kehayias]

Hi Andreas and Guixers,

Again, thanks Andreas for all your work on core-updates!

On Mon, Apr 17, 2023 at 11:03 AM, Andreas Enge wrote:

> Hello,
>
> just a quick update after a night of building on CI.
> Things look generally quite good on x86_64; some things are being rebuilt
> due to the recent wget update, and that should hopefully also sort out
> i686 rather quickly.
>

Packages I use all seem to be good too, except wine as there is a
samba failure for i686. Pending fix: <https://issues.guix.gnu.org/62894>

> Am Sun, Apr 16, 2023 at 01:09:02PM +0200 schrieb Andreas Enge:
>> - python: It is mostly green, but with lots of red, including at least
>>   one package that prevents icecat from building. I would call for careful
>>   updates and bug fixes to move on; leaf packages can be updated to the
>>   latest version, but for important intermediate packages please be
>>   careful.
>
> Python probably is the place where work is needed most, the strip in the
> dashboard looks like it has measles. Fixing one package there could bring
> us a long way (or not - we are in plain hand resolution of dependency
> problems there, one update could also hide another one). Sometimes it is
> just a matter of trying an update to see whether a package and its
> dependents build, so even someone who knows nothing about Python, but
> has a web browser pointed at <https://pypi.org/>, can do some useful work
> (I am speaking from experience).
>

I took a stab at a few random packages and ended up finding out that
python-urllib3 needed an update due to our updated
python-cryptography. Only saw this through test failures of leaf
packages rather than itself (noted in upstream update that failures
with newer cryptography tend to point people to the wrong packages).
While urllib3 has quite a few dependents (> 500 rebuilds on update)
and this may break more than it fixes, I pushed the latest version. A
few other updates/build fixes for the packages I was trying as well.

In the chain of oauth/google packages I was looking at, I was left at
python-google-auth-1 (older version) as failing tests even adding in
the missing python-mock. At quick glance I'm guessing due to the newer
python-cryptography. I haven't had a chance to see how we want to work
around that, or if the older versions of the few dependents should be
kept. It is only a few packages anyway, so maybe bigger fish to hunt
first.

I'll check in tomorrow on how the urllib update built on Cuirass and
see if I can pick off any other easy python-* fixes.

Oh, looks like no x86_64 builds currently on Cuirass...hopefully
temporary?

John

Re: Core-updates after the staging merge

[Andreas Enge]

Am Tue, Apr 18, 2023 at 05:04:35AM +0000 schrieb John Kehayias:
> I took a stab at a few random packages and ended up finding out that
> python-urllib3 needed an update due to our updated
> python-cryptography. Only saw this through test failures of leaf
> packages rather than itself (noted in upstream update that failures
> with newer cryptography tend to point people to the wrong packages).
> While urllib3 has quite a few dependents (> 500 rebuilds on update)
> and this may break more than it fixes, I pushed the latest version. A
> few other updates/build fixes for the packages I was trying as well.

Thanks for working on these packages! I tend to always update to the newest
version, but with Python this seems to be much more tricky when depending
packages do not work with the latest and greatest. Good luck in finding
combinations that work well together!

Andreas

Latest news on core-updates

[Andreas Enge]

Hello all,

just a quick update to share good news and to heap praise on people who are
not rewarded by seeing their name in a git commit.

Am Mon, Apr 17, 2023 at 11:03:25AM +0200 schrieb Andreas Enge:
> On aarch64 and powerpc, we are still stuck by CI problems.

Thanks to tireless work by Ricardo, Ludovic, Julien, Christopher and myself
enough machines for these architectures are running in the build farm to
hopefully clear the backlog. We went from 1 to 5 aarch machines (or from 4
to 14 build slots), and from 1 to 2 powerpc64le machines (or from 2 to 8
build slots).

The dash boards are filling up, and on x86_64 and i686 the situation looks
comparable to master. On powerpc64le, I have been restarting cancelled jobs
with a little script, and the red dots are also becoming less. On aarch64,
I am waiting for the backlog from master to clear up before trying something
similar. I think the situation on these architectures is better than it
looks on the dashboard.

I hope we will be able to merge core-updates after next weekend; also for
personal reasons, since I will not be able to spend much time on it any
more after that.

So please help us in a last sprint to get core-updates into shape!

Andreas

Re: Latest news on core-updates

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]


Andreas Enge <andreas@enge.fr> writes:

> I hope we will be able to merge core-updates after next weekend; also for
> personal reasons, since I will not be able to spend much time on it any
> more after that.

After next weekend (around the end of April) sounds good to me.

The bordeaux build farm has pretty much caught up after the staging
merge which should mean there's some capacity to attempt to build
core-updates changes prior to the merge around next weekend.

I haven't restarted submitting builds for core-updates given I think
there's still some big changes going to land, but in any case, it would
be good to get this going once there are no planned big changes (aka the
branch is frozen). Getting the builds to happen currently requires a
code change in the qa-frontpage, but I can do this, I just need someone
to let me know when I should.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Latest news on core-updates

[Andreas Enge]

Am Wed, Apr 19, 2023 at 11:58:03AM +0100 schrieb Christopher Baines:
> I haven't restarted submitting builds for core-updates given I think
> there's still some big changes going to land, but in any case, it would
> be good to get this going once there are no planned big changes (aka the
> branch is frozen).

I am not sure what "frozen" means exactly; I would say we must not make
commits to core-updates unless they repair a broken package, or maybe help
repair broken packages further down towards the leaves. (And this has been
true for a while now.) And maybe we should go further and refrain from
repairing things in core-updates that are already broken on master, at
least if they are relatively close to the root of the dependency graph.
These can be done later in feature branches; I think we have the infra-
structure to shoulder bigger changes outside of core-updates.

On the other hand, this report is a bit worrying:
   [core-updates] locales not installed properly
   https://issues.guix.gnu.org/62934
It could potentially mean changes to glibc, which would throw us back
a week.

Andreas

Re: Latest news on core-updates

[Andreas Enge]

Hello Maxim,

Am Wed, Apr 19, 2023 at 02:41:31PM +0200 schrieb Andreas Enge:
> I am not sure what "frozen" means exactly; I would say we must not make
> commits to core-updates unless they repair a broken package, or maybe help
> repair broken packages further down towards the leaves.

your recent docbook commits probably do not fit this definition, much less
so than the suggested valgrind update that we nevertheless postponed until
after the merge. They cause an extraordinary number of rebuilds. I hope they
are mostly harmless and do not cause additional build failures, so keeping
them in is probably easier than reverting them.

But let us from now on stick with commits that make important packages work,
or packages very close to the leaves.

I suggest the following timetable:
- Hard freeze now, only important broken packages or packages close to the
  leaves may be changed.
- Over the weekend, update your systems and profiles and test and repair
  problems that appear.
- If no show-stopper occurs, merge core-updates into master on Tuesday.

Andreas

Re: Latest news on core-updates

[Maxim Cournoyer]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> Hello Maxim,
>
> Am Wed, Apr 19, 2023 at 02:41:31PM +0200 schrieb Andreas Enge:
>> I am not sure what "frozen" means exactly; I would say we must not make
>> commits to core-updates unless they repair a broken package, or maybe help
>> repair broken packages further down towards the leaves.
>
> your recent docbook commits probably do not fit this definition, much less
> so than the suggested valgrind update that we nevertheless postponed until
> after the merge. They cause an extraordinary number of rebuilds. I hope they
> are mostly harmless and do not cause additional build failures, so keeping
> them in is probably easier than reverting them.

10,000 total rebuilds for Cuirass is about 2500 packages per platforms.
It's large, but not 'extraordinary' large :-).  8 hours later we're back
to our current 50% substitute coverage target.

> But let us from now on stick with commits that make important packages work,
> or packages very close to the leaves.
>
> I suggest the following timetable:
> - Hard freeze now, only important broken packages or packages close to the
>   leaves may be changed.
> - Over the weekend, update your systems and profiles and test and repair
>   problems that appear.
> - If no show-stopper occurs, merge core-updates into master on Tuesday.

OK, thanks for making this clear!  I'll refrain from changes causing big
rebuilds from now on, unless they are truly unavoidable.  Sounds like a
good plan.

Cheers,

-- 
Maxim

Re: Latest news on core-updates

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]


Christopher Baines <mail@cbaines.net> writes:

> Andreas Enge <andreas@enge.fr> writes:
>
>> I hope we will be able to merge core-updates after next weekend; also for
>> personal reasons, since I will not be able to spend much time on it any
>> more after that.
>
> After next weekend (around the end of April) sounds good to me.
>
> The bordeaux build farm has pretty much caught up after the staging
> merge which should mean there's some capacity to attempt to build
> core-updates changes prior to the merge around next weekend.
>
> I haven't restarted submitting builds for core-updates given I think
> there's still some big changes going to land, but in any case, it would
> be good to get this going once there are no planned big changes (aka the
> branch is frozen). Getting the builds to happen currently requires a
> code change in the qa-frontpage, but I can do this, I just need someone
> to let me know when I should.

Andreas suggested on IRC that I get things building, so I've configured
the qa-frontpage to start submitting builds for core-updates again.

You can track the progress here [1] of course, and if you want to get an
idea if any of the builds are currently happening, you can get some
information on what builds are assigned to what agents here [2]
(providing the experimental interface hasn't crashed).

1: https://qa.guix.gnu.org/branch/core-updates
2: https://bordeaux.guix.gnu.org/activity

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Latest news on core-updates

[Andreas Enge]

Am Thu, Apr 20, 2023 at 02:56:48PM +0100 schrieb Christopher Baines:
> Andreas suggested on IRC that I get things building, so I've configured
> the qa-frontpage to start submitting builds for core-updates again.
> You can track the progress here [1] of course, and if you want to get an
> idea if any of the builds are currently happening, you can get some
> information on what builds are assigned to what agents here [2]
> (providing the experimental interface hasn't crashed).

Thanks! I see numbers going up on powerpc :-)
[2] times out, however.

Andreas

Re: Latest news on core-updates

[Simon Tournier]

Hi Andreas,

On Wed, 19 Apr 2023 at 12:48, Andreas Enge <andreas@enge.fr> wrote:

> I hope we will be able to merge core-updates after next weekend; also for
> personal reasons, since I will not be able to spend much time on it any
> more after that.

I do not know if something is twisted with the report [1] of Cuirass but
many things seems missing.  Right?

Therefore, if many things are still missing, I suggest to apply #62967
[2].  It removes the annoyance you spotted out in #62954 [3].  And as I
explained in [4], it will be safe because ’valgrind’ appears only in the
testsuite of ’lz4’ and ’valgrind’ is not even run there.

Doing so, it will fix failures for powerpc64le and will increase the
coverage.

WDYT?


1: https://ci.guix.gnu.org/eval/417699/dashboard
2: https://issues.guix.gnu.org/62967
3 https://issues.guix.gnu.org/62954
4: https://issues.guix.gnu.org/62954#3


Moreover, I have revamped the GHC bootstrap chain.

 1. it cuts the number of Haskell compilers to build,
 2. it separates building the compiler itself and running the testsuite,
 3. it introduces ghc-toolchain that glue one compiler and its testsuite.

Well, #2 removing failure for i686 [5] since the failure of the
testsuite of 8.10.7 is not blocking for 9.2.5.

However, the patches needs more polishing and then because of ’pandoc’
depending on the GHC bootstrap, it would a large rebuild.

Since Haskell is also broken on master for i686, I guess the option is
to apply on some “feature™ branch” after the merge, right?


5: https://ci.guix.gnu.org/build/953241/details

Cheers,
simon

Re: Latest news on core-updates

[Andreas Enge]

Am Fri, Apr 21, 2023 at 10:20:18AM +0200 schrieb Simon Tournier:
> Since Haskell is also broken on master for i686, I guess the option is
> to apply on some “feature™ branch” after the merge, right?

Yes indeed.

> Therefore, if many things are still missing, I suggest to apply #62967
> [2].  It removes the annoyance you spotted out in #62954 [3].  And as I
> explained in [4], it will be safe because ’valgrind’ appears only in the
> testsuite of ’lz4’ and ’valgrind’ is not even run there.
> Doing so, it will fix failures for powerpc64le and will increase the
> coverage.

CI just disappeared, so I cannot check. If r-minimal builds for powerpc64le
on master, then it is justifiable to fix the regression (leaving open the
possibility of an immediate revert if the patch introduces problems else-
where, which I do not expect). Otherwise I would postpone it until after
the merge.

Andreas

Re: Latest news on core-updates

[Simon Tournier]

Hi,

On ven., 21 avril 2023 at 10:47, Andreas Enge <andreas@enge.fr> wrote:

>> Therefore, if many things are still missing, I suggest to apply #62967
>> [2].  It removes the annoyance you spotted out in #62954 [3].  And as I
>> explained in [4], it will be safe because ’valgrind’ appears only in the
>> testsuite of ’lz4’ and ’valgrind’ is not even run there.
>> Doing so, it will fix failures for powerpc64le and will increase the
>> coverage.
>
> CI just disappeared, so I cannot check. If r-minimal builds for powerpc64le
> on master, then it is justifiable to fix the regression (leaving open the
> possibility of an immediate revert if the patch introduces problems else-
> where, which I do not expect). Otherwise I would postpone it until after
> the merge.

Yes r-minimal builds for powerpc64le-linux on master:

    https://ci.guix.gnu.org/build/1050833/details

--8<---------------cut here---------------start------------->8---
simon@pfiuh07$ guix weather -s powerpc64le-linux r-minimal
computing 1 package derivations for powerpc64le-linux...
looking for 1 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org ☀
  100.0% substitutes available (1 out of 1)
  at least 49,1 MiB of nars (compressed)
  52,7 MiB on disk (uncompressed)
  2,529 seconds per request (2,5 seconds in total)
  0,4 requests per second

  at least 1 000 queued builds
      aarch64-linux: 819 (81.9%)
      x86_64-linux: 117 (11.7%)
      powerpc64le-linux: 7 (.7%)
      i686-linux: 57 (5.7%)
  build rate: 189.97 builds per hour
      i686-linux: 138.13 builds per hour
      x86_64-linux: 27.34 builds per hour
      powerpc64le-linux: 21.66 builds per hour
      aarch64-linux: 5.56 builds per hour
looking for 1 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org ☀
  100.0% substitutes available (1 out of 1)
  24,2 MiB of nars (compressed)
  52,7 MiB on disk (uncompressed)
  0,852 seconds per request (0,9 seconds in total)
  1,2 requests per second
  'https://bordeaux.guix.gnu.org/api/queue?nr=1000' returned 502 ("Bad Gateway")
--8<---------------cut here---------------end--------------->8---

Note that the patch removes the dependency between subversion and
valgrind.  Therefore, as you noted, many TeXlive packages could also
build.

Well, I do not see any cons. :-)  The only difference for ’lz4’ is the
path of the store item; here without and with the removal of valgrind:

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference /gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3 /gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3
diff -r --no-dereference /gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/lib/pkgconfig/liblz4.pc /gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/lib/pkgconfig/liblz4.pc
5,7c5,7
< prefix=/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3
< libdir=/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/lib
< includedir=/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/include
---
> prefix=/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3
> libdir=/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/lib
> includedir=/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/include
13,14c13,14
< Libs: -L/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/lib -llz4
< Cflags: -I/gnu/store/3sabpv5k19lnifda5ihk7340xm3idxi2-lz4-1.9.3/include
---
> Libs: -L/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/lib -llz4
> Cflags: -I/gnu/store/1j0jq03930q4c7570145vjiaa8pnajz4-lz4-1.9.3/include
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

Re: Latest news on core-updates

[Katherine Cox-Buday]

On 4/19/23 4:48 AM, Andreas Enge wrote:

> just a quick update to share good news and to heap praise on people who are
> not rewarded by seeing their name in a git commit.

Consider my praise added to the heap (and hopefully not GCed). As 
always, I am deeply appreciative to all of our maintainers.

Thank you, thank you, thank you!

--
Katherine

Re: Latest news on core-updates

[Katherine Cox-Buday]

On 4/19/23 4:48 AM, Andreas Enge wrote:

> just a quick update to share good news and to heap praise on people who are
> not rewarded by seeing their name in a git commit.

Consider my praise added to the heap (and hopefully not GCed). As 
always, I am deeply appreciative to all of our maintainers.

Thank you, thank you, thank you!

--
Katherine

Re: Latest news on core-updates

[reza.housseini]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

Consider my praise added as well! Was there already a discussion about adding liberapay, to at least have some monetary compensation for the hard and necessary work done by all these volunteers?

On April 21, 2023 6:12:55 PM GMT+02:00, Katherine Cox-Buday <cox.katherine.e@gmail.com> wrote:
>On 4/19/23 4:48 AM, Andreas Enge wrote:
>
>> just a quick update to share good news and to heap praise on people who are
>> not rewarded by seeing their name in a git commit.
>
>Consider my praise added to the heap (and hopefully not GCed). As always, I am deeply appreciative to all of our maintainers.
>
>Thank you, thank you, thank you!
>
>--
>Katherine
>

-- Sent from /e/ Mail.

[-- Attachment #2: Type: text/html, Size: 1086 bytes --]

Re: Latest news on core-updates

[Andreas Enge]

Am Fri, Apr 21, 2023 at 07:04:19PM +0200 schrieb reza.housseini@gmail.com:
> Consider my praise added as well! Was there already a discussion about adding
> liberapay, to at least have some monetary compensation for the hard and
> necessary work done by all these volunteers?

Thanks for the thanks! Well, the volunteers are volunteers; if you want
to donate to Guix (so far, mainly for covering infrastructure costs and
Outreachy internships), you can do so at Guix Europe or the FSF.

Andreas

i686 core-updates failure.

[Kaelyn]

Hi,

I've been working on getting wine to build on core-updates, with the most recent blocker being python-numpy failing two tests on i686 (one of which is apparently too big for 32-bit systems, based on the comment in the Gentoo ebuild of python-numpy where they skip the test). I just poked through the CI dashboard for the latest core-updates evaluation to see if it was hitting the same failure (it was), and discovered an odd chain of dependencies. Before going into the chain, I want to mention that wine has 6 failing dependencies[1], one of which is pulseaudio, two of which include pulseaudio as part of why they failed, and two include dependency failures that are on pulseaudio's dependency chain (the 6th failed dependency is due to qtbase which I didn't look into; none are direct failures).

On to the odd chain:
* The sole failed dependency of pulseaudio is bluez
* bluez's sole failed dependency is libical
* libical's sole failed dependency is gtk-doc (also a failed dependency of two other wine dependencies)
* gtk-doc's sole failed dependency is dblatex
* dblatex's sole failed dependency is inkscape(?!?!?)
* inkscape's sole failed dependency is python-numpy, which is failing two tests.

I call the dependency chain odd in that pulseaudio--which is near the heart of a lot of Linux software that supports audio--is failing because inkscape--a graphical vector editor--is a dependency of a tool designed to extract API documentation from source code.

For the dependency chain, I have no real thoughts, goals, or proposed changes other than surprise that pulseaudio fails to build because a GUI SVG editor failed to build.

For the actual failures in python-numpy[2]:
1. TestUfunc::test_identityless_reduction_huge_array in numpy/core/tests/test_ufunc.py: as I alluded to earlier, the Gentoo ebuild[3] disables this test with a comment that the test is "too large for 32-bit platforms".
2. TestKind::test_all in numpy/f2py/tests/test_kind.py: I'm not sure why this test is failing or what to do about it, since it seems to be seeing incorrect values for objects coming from Fortran code. Perhaps a type size mismatch for 32-bit platforms, but hard for me to say since I don't know Fortran and I haven't had luck finding other reports of that test failing.

Cheers,
Kaelyn


[1] https://ci.guix.gnu.org/build/843194/details
[2] https://ci.guix.gnu.org/build/712672/log/raw
[3] https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/numpy/numpy-1.24.0.ebuild#n142

wget (was Re: i686 core-updates failure.)

[Simon Tournier]

Hi,

Fixing ’wget’ for i686 would help for Java and Julia.  Well, currently
’wget’ is broken on core-updates for i686,

    https://ci.guix.gnu.org/build/709528/details

And the 5 error seem similar (missing file?), i.e., read,

--8<---------------cut here---------------start------------->8---
FAIL: Test-hsts
===============

Setting --no-config (noconfig) to 1
Setting --hsts-file (hstsfile) to /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv
Setting --ca-certificate (cacertificate) to /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/certs/ca-cert.pem
DEBUG output created by Wget 1.21.3 on linux-gnu.

Reading HSTS entries from /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv
URI encoding = 'ANSI_X3.4-1968'
converted 'http://localhost:41133/hw' (ANSI_X3.4-1968) -> 'http://localhost:41133/hw' (UTF-8)
URL transformed to HTTPS due to an HSTS policy
Converted file name 'hw' (UTF-8) -> 'hw' (ANSI_X3.4-1968)
--2023-04-06 18:17:39--  https://localhost:41133/hw
Loaded CA certificate '/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/certs/ca-cert.pem'
Certificates loaded: 1
Resolving localhost (localhost)... 127.0.0.1
Caching localhost => 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:41133... connected.
Created socket 3.
Releasing 0x080e1190 (new refcount 1).
The certificate has not yet been activated
Saving HSTS entries to /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv
1
Running Test Test-hsts.py
/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/src/wget --debug --no-config --hsts-file=/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv --ca-certificate=/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/certs/ca-cert.pem http://localhost:41133/hw 
['/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/src/wget', '--debug', '--no-config', '--hsts-file=/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv', '--ca-certificate=/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/certs/ca-cert.pem', 'http://localhost:41133/hw']
{'HOME': '/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/Test-hsts.py-test'}
Error: Expected file hw not found..
Traceback (most recent call last):
  File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/./Test-hsts.py", line 76, in <module>
    err = test.begin()
  File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/test/http_test.py", line 41, in begin
    self.do_test()
  File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/test/base_test.py", line 198, in do_test
    self.post_hook_call()
  File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/test/base_test.py", line 217, in post_hook_call
    self.hook_call(self.post_configs, 'Post Test Function')
  File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/test/base_test.py", line 207, in hook_call
    conf.find_conf(conf_name)(conf_arg)(self)
  File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/conf/expected_files.py", line 55, in __call__
    raise TestFailed('Expected file %s not found.' % file.name)
exc.test_failed.TestFailed: Expected file hw not found.
FAIL Test-hsts.py (exit status: 1)
--8<---------------cut here---------------end--------------->8---


Any idea?


Cheers,
simon

Re: wget (was Re: i686 core-updates failure.)

[Maxim Cournoyer]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> Fixing ’wget’ for i686 would help for Java and Julia.  Well, currently
> ’wget’ is broken on core-updates for i686,
>
>     https://ci.guix.gnu.org/build/709528/details
>
> And the 5 error seem similar (missing file?), i.e., read,
>
> FAIL: Test-hsts
> ===============
>
> Setting --no-config (noconfig) to 1
> Setting --hsts-file (hstsfile) to /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv
> Setting --ca-certificate (cacertificate) to /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/certs/ca-cert.pem
> DEBUG output created by Wget 1.21.3 on linux-gnu.
>
> Reading HSTS entries from /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv
> URI encoding = 'ANSI_X3.4-1968'
> converted 'http://localhost:41133/hw' (ANSI_X3.4-1968) -> 'http://localhost:41133/hw' (UTF-8)
> URL transformed to HTTPS due to an HSTS policy
> Converted file name 'hw' (UTF-8) -> 'hw' (ANSI_X3.4-1968)
> --2023-04-06 18:17:39--  https://localhost:41133/hw
> Loaded CA certificate '/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/certs/ca-cert.pem'
> Certificates loaded: 1
> Resolving localhost (localhost)... 127.0.0.1
> Caching localhost => 127.0.0.1
> Connecting to localhost (localhost)|127.0.0.1|:41133... connected.
> Created socket 3.
> Releasing 0x080e1190 (new refcount 1).
> The certificate has not yet been activated
> Saving HSTS entries to /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv
> 1
> Running Test Test-hsts.py
> /tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/src/wget --debug --no-config --hsts-file=/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv --ca-certificate=/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/certs/ca-cert.pem http://localhost:41133/hw 
> ['/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/src/wget', '--debug', '--no-config', '--hsts-file=/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/.wget-hsts-testenv', '--ca-certificate=/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/certs/ca-cert.pem', 'http://localhost:41133/hw']
> {'HOME': '/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/Test-hsts.py-test'}
> Error: Expected file hw not found..
> Traceback (most recent call last):
>   File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/./Test-hsts.py", line 76, in <module>
>     err = test.begin()
>   File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/test/http_test.py", line 41, in begin
>     self.do_test()
>   File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/test/base_test.py", line 198, in do_test
>     self.post_hook_call()
>   File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/test/base_test.py", line 217, in post_hook_call
>     self.hook_call(self.post_configs, 'Post Test Function')
>   File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/test/base_test.py", line 207, in hook_call
>     conf.find_conf(conf_name)(conf_arg)(self)
>   File "/tmp/guix-build-wget-1.21.3.drv-0/wget-1.21.3/testenv/conf/expected_files.py", line 55, in __call__
>     raise TestFailed('Expected file %s not found.' % file.name)
> exc.test_failed.TestFailed: Expected file hw not found.
> FAIL Test-hsts.py (exit status: 1)
>
>
> Any idea?

None, but are there wget uptsream reports about the problem?

-- 
Thanks,
Maxim

Re: wget (was Re: i686 core-updates failure.)

[Andreas Enge]

Am Fri, Apr 14, 2023 at 08:51:18PM -0400 schrieb Maxim Cournoyer:
> None, but are there wget uptsream reports about the problem?

I do not see anything at
   https://gitlab.com/gnuwget/wget

Andreas

Re: wget (was Re: i686 core-updates failure.)

[Kaelyn]

Hi,

------- Original Message -------
On Saturday, April 15th, 2023 at 10:43 AM, Andreas Enge <andreas@enge.fr> wrote:

> 
> Am Fri, Apr 14, 2023 at 08:51:18PM -0400 schrieb Maxim Cournoyer:
> 
> > None, but are there wget uptsream reports about the problem?
> 
> 
> I do not see anything at
> https://gitlab.com/gnuwget/wget
> 
> Andreas

Doing a search of the error message, I just found https://savannah.gnu.org/bugs/?62110; the subject is about the HSTS tests being broken 32-bit big endian devices, but comment #3 mentions the tests failing on i686, with the same error we are seeing. It looks like https://gitlab.com/gnuwget/wget/-/merge_requests/31 was merged to fix the issue about a month after 1.21.3 was released (I haven't tested yet since the package definition downloads a release tarball instead of building from git).

Cheers,
Kaelyn

Re: wget (was Re: i686 core-updates failure.)

[Kaelyn]

------- Original Message -------
On Saturday, April 15th, 2023 at 4:37 PM, Kaelyn <kaelyn.alexi@protonmail.com> wrote:


> 
> 
> Hi,
> 
> ------- Original Message -------
> On Saturday, April 15th, 2023 at 10:43 AM, Andreas Enge andreas@enge.fr wrote:
> 
> > Am Fri, Apr 14, 2023 at 08:51:18PM -0400 schrieb Maxim Cournoyer:
> > 
> > > None, but are there wget uptsream reports about the problem?
> > 
> > I do not see anything at
> > https://gitlab.com/gnuwget/wget
> > 
> > Andreas
> 
> 
> Doing a search of the error message, I just found https://savannah.gnu.org/bugs/?62110; the subject is about the HSTS tests being broken 32-bit big endian devices, but comment #3 mentions the tests failing on i686, with the same error we are seeing. It looks like https://gitlab.com/gnuwget/wget/-/merge_requests/31 was merged to fix the issue about a month after 1.21.3 was released (I haven't tested yet since the package definition downloads a release tarball instead of building from git).

I tried to update the package definition to be able to build from git but it became a much bigger rabbit hole than I have the energy for at the moment. I also tried to cherry-pick the commit from the merge request without luck. At least until a new release of wget occurs (and is confirmed to build on i686), I feel like the most expedient solution may be to revert commit 9cd22702b8 which updated wget from 1.21.1 to 1.21.3. I tried building 1.21.2 for i686 and hit the same test failures, but have confirmed that 1.21.1 builds just fine for i686 on core-updates

Cheers,
Kaelyn

Re: wget on i686 in core-updates

[Andreas Enge]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

Hello,

Am Sat, Apr 15, 2023 at 06:25:52PM +0000 schrieb Kaelyn:
> I tried to update the package definition to be able to build from git but it became a much bigger rabbit hole than I have the energy for at the moment. I also tried to cherry-pick the commit from the merge request without luck.

I just downloaded the commit, which is in uniform diff format and can serve
as a patch to be applied to the tarball. The result is attached, but it
does not work: The tests fail as before with the "file not found" messages.

But thanks for digging through the wget issues!

Andreas


[-- Attachment #2: 0001-gnu-wget-Add-patch-needed-for-i686.patch --]
[-- Type: text/plain, Size: 11436 bytes --]

From 0991a0daf74c0a3754618f99b7a7cb812debfa1a Mon Sep 17 00:00:00 2001
From: Andreas Enge <andreas@enge.fr>
Date: Sat, 15 Apr 2023 22:52:39 +0200
Subject: [PATCH] gnu: wget: Add patch needed for i686.

* gnu/packages/patches/wget-hsts-portability.patch: New file.
* gnu/local.mk (dist_patch_DATA): Register patch.
* gnu/packages/wget.scm (wget)[origin]: Apply patch.
---
 gnu/local.mk                                  |   1 +
 .../patches/wget-hsts-portability.patch       | 223 ++++++++++++++++++
 gnu/packages/wget.scm                         |   4 +-
 3 files changed, 227 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/wget-hsts-portability.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 73756a8c49..8dd0c45cea 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -2021,6 +2021,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/webrtc-audio-processing-big-endian.patch	\
   %D%/packages/patches/webrtc-for-telegram-desktop-fix-gcc12-cstdint.patch   \
   %D%/packages/patches/websocketpp-fix-for-cmake-3.15.patch	\
+  %D%/packages/patches/wget-hsts-portability.patch              \
   %D%/packages/patches/wmctrl-64-fix.patch			\
   %D%/packages/patches/wmfire-update-for-new-gdk-versions.patch	\
   %D%/packages/patches/wordnet-CVE-2008-2149.patch			\
diff --git a/gnu/packages/patches/wget-hsts-portability.patch b/gnu/packages/patches/wget-hsts-portability.patch
new file mode 100644
index 0000000000..62dabd7fae
--- /dev/null
+++ b/gnu/packages/patches/wget-hsts-portability.patch
@@ -0,0 +1,223 @@
+From 9a3479a23c15cd7234a54296ae50c48f29c427ec Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?Tim=20R=C3=BChsen?= <tim.ruehsen@gmx.de>
+Date: Sun, 20 Mar 2022 12:18:20 +0100
+Subject: [PATCH] Fix HSTS portability by using int64_t instead of time_t.
+
+* src/hsts.c: Use int64_t instead of time_t.
+* src/http.c: Use int64_t for parsing Strict-Transport-Security.
+---
+ src/hsts.c | 41 ++++++++++++++++++++---------------------
+ src/hsts.h |  2 +-
+ src/http.c | 24 ++++++++++++------------
+ 3 files changed, 33 insertions(+), 34 deletions(-)
+
+diff --git a/src/hsts.c b/src/hsts.c
+index 0a014401..72c5e936 100644
+--- a/src/hsts.c
++++ b/src/hsts.c
+@@ -61,8 +61,8 @@ struct hsts_kh {
+ };
+ 
+ struct hsts_kh_info {
+-  time_t created;
+-  time_t max_age;
++  int64_t created;
++  int64_t max_age;
+   bool include_subdomains;
+ };
+ 
+@@ -166,7 +166,7 @@ end:
+ static bool
+ hsts_new_entry_internal (hsts_store_t store,
+                          const char *host, int port,
+-                         time_t created, time_t max_age,
++                         int64_t created, int64_t max_age,
+                          bool include_subdomains,
+                          bool check_validity,
+                          bool check_expired,
+@@ -216,21 +216,21 @@ bail:
+ static bool
+ hsts_add_entry (hsts_store_t store,
+                 const char *host, int port,
+-                time_t max_age, bool include_subdomains)
++                int64_t max_age, bool include_subdomains)
+ {
+-  time_t t = time (NULL);
++  int64_t t = (int64_t) time (NULL);
+ 
+   /* It might happen time() returned -1 */
+-  return (t == (time_t)(-1) ?
++  return (t == -1) ?
+       false :
+-      hsts_new_entry_internal (store, host, port, t, max_age, include_subdomains, false, true, false));
++      hsts_new_entry_internal (store, host, port, t, max_age, include_subdomains, false, true, false);
+ }
+ 
+ /* Creates a new entry, unless an identical one already exists. */
+ static bool
+ hsts_new_entry (hsts_store_t store,
+                 const char *host, int port,
+-                time_t created, time_t max_age,
++                int64_t created, int64_t max_age,
+                 bool include_subdomains)
+ {
+   return hsts_new_entry_internal (store, host, port, created, max_age, include_subdomains, true, true, true);
+@@ -245,7 +245,7 @@ hsts_remove_entry (hsts_store_t store, struct hsts_kh *kh)
+ static bool
+ hsts_store_merge (hsts_store_t store,
+                   const char *host, int port,
+-                  time_t created, time_t max_age,
++                  int64_t created, int64_t max_age,
+                   bool include_subdomains)
+ {
+   enum hsts_kh_match match_type = NO_MATCH;
+@@ -276,11 +276,11 @@ hsts_read_database (hsts_store_t store, FILE *fp, bool merge_with_existing_entri
+   size_t len = 0;
+   int items_read;
+   bool result = false;
+-  bool (*func)(hsts_store_t, const char *, int, time_t, time_t, bool);
++  bool (*func)(hsts_store_t, const char *, int, int64_t, int64_t, bool);
+ 
+   char host[256];
+   int port;
+-  time_t created, max_age;
++  int64_t created, max_age;
+   int include_subdomains;
+ 
+   func = (merge_with_existing_entries ? hsts_store_merge : hsts_new_entry);
+@@ -326,10 +326,9 @@ hsts_store_dump (hsts_store_t store, FILE *fp)
+       struct hsts_kh *kh = (struct hsts_kh *) it.key;
+       struct hsts_kh_info *khi = (struct hsts_kh_info *) it.value;
+ 
+-      if (fprintf (fp, "%s\t%d\t%d\t%lu\t%lu\n",
++      if (fprintf (fp, "%s\t%d\t%d\t%" PRId64 "\t%" PRId64 "\n",
+                    kh->host, kh->explicit_port, khi->include_subdomains,
+-                   (unsigned long) khi->created,
+-                   (unsigned long) khi->max_age) < 0)
++                   khi->created, khi->max_age) < 0)
+         {
+           logprintf (LOG_ALWAYS, "Could not write the HSTS database correctly.\n");
+           break;
+@@ -439,7 +438,7 @@ hsts_match (hsts_store_t store, struct url *u)
+ bool
+ hsts_store_entry (hsts_store_t store,
+                   enum url_scheme scheme, const char *host, int port,
+-                  time_t max_age, bool include_subdomains)
++                  int64_t max_age, bool include_subdomains)
+ {
+   bool result = false;
+   enum hsts_kh_match match = NO_MATCH;
+@@ -464,9 +463,9 @@ hsts_store_entry (hsts_store_t store,
+                * 'created' field too. The RFC also states that we have to
+                * update the entry each time we see HSTS header.
+                * See also Section 11.2. */
+-              time_t t = time (NULL);
++              int64_t t = (int64_t) time (NULL);
+ 
+-              if (t != (time_t)(-1) && t != entry->created)
++              if (t != -1 && t != entry->created)
+                 {
+                   entry->created = t;
+                   entry->max_age = max_age;
+@@ -792,7 +791,7 @@ test_hsts_read_database (void)
+   hsts_store_t table;
+   char *file = NULL;
+   FILE *fp = NULL;
+-  time_t created = time(NULL) - 10;
++  int64_t created = time(NULL) - 10;
+ 
+   if (opt.homedir)
+     {
+@@ -801,9 +800,9 @@ test_hsts_read_database (void)
+       if (fp)
+         {
+           fputs ("# dummy comment\n", fp);
+-          fprintf (fp, "foo.example.com\t0\t1\t%lu\t123\n",(unsigned long) created);
+-          fprintf (fp, "bar.example.com\t0\t0\t%lu\t456\n", (unsigned long) created);
+-          fprintf (fp, "test.example.com\t8080\t0\t%lu\t789\n", (unsigned long) created);
++          fprintf (fp, "foo.example.com\t0\t1\t%" PRId64 "\t123\n", created);
++          fprintf (fp, "bar.example.com\t0\t0\t%" PRId64 "\t456\n", created);
++          fprintf (fp, "test.example.com\t8080\t0\t%" PRId64 "\t789\n", created);
+           fclose (fp);
+ 
+           table = hsts_store_open (file);
+diff --git a/src/hsts.h b/src/hsts.h
+index 6ecd5060..be048944 100644
+--- a/src/hsts.h
++++ b/src/hsts.h
+@@ -46,7 +46,7 @@ bool hsts_store_has_changed (hsts_store_t);
+ 
+ bool hsts_store_entry (hsts_store_t,
+                        enum url_scheme, const char *, int,
+-                       time_t, bool);
++                       int64_t, bool);
+ bool hsts_match (hsts_store_t, struct url *);
+ 
+ #endif /* HAVE_HSTS */
+diff --git a/src/http.c b/src/http.c
+index f61c99a7..87b51b00 100644
+--- a/src/http.c
++++ b/src/http.c
+@@ -1300,7 +1300,7 @@ parse_content_disposition (const char *hdr, char **filename)
+ 
+ #ifdef HAVE_HSTS
+ static bool
+-parse_strict_transport_security (const char *header, time_t *max_age, bool *include_subdomains)
++parse_strict_transport_security (const char *header, int64_t *max_age, bool *include_subdomains)
+ {
+   param_token name, value;
+   const char *c_max_age = NULL;
+@@ -1330,7 +1330,7 @@ parse_strict_transport_security (const char *header, time_t *max_age, bool *incl
+            * Also, time_t is normally defined as a long, so this should not break.
+            */
+           if (max_age)
+-            *max_age = (time_t) strtol (c_max_age, NULL, 10);
++            *max_age = (int64_t) strtoll (c_max_age, NULL, 10);
+           if (include_subdomains)
+             *include_subdomains = is;
+ 
+@@ -3184,9 +3184,6 @@ gethttp (const struct url *u, struct url *original_url, struct http_stat *hs,
+ #else
+   extern hsts_store_t hsts_store;
+ #endif
+-  const char *hsts_params;
+-  time_t max_age;
+-  bool include_subdomains;
+ #endif
+ 
+   int sock = -1;
+@@ -3674,21 +3671,24 @@ gethttp (const struct url *u, struct url *original_url, struct http_stat *hs,
+ #ifdef HAVE_HSTS
+   if (opt.hsts && hsts_store)
+     {
+-      hsts_params = resp_header_strdup (resp, "Strict-Transport-Security");
++      int64_t max_age;
++      const char *hsts_params = resp_header_strdup (resp, "Strict-Transport-Security");
++      bool include_subdomains;
++
+       if (parse_strict_transport_security (hsts_params, &max_age, &include_subdomains))
+         {
+           /* process strict transport security */
+           if (hsts_store_entry (hsts_store, u->scheme, u->host, u->port, max_age, include_subdomains))
+-            DEBUGP(("Added new HSTS host: %s:%u (max-age: %lu, includeSubdomains: %s)\n",
++            DEBUGP(("Added new HSTS host: %s:%" PRIu32 " (max-age: %" PRId64 ", includeSubdomains: %s)\n",
+                    u->host,
+-                   (unsigned) u->port,
+-                   (unsigned long) max_age,
++                   (uint32_t) u->port,
++                   max_age,
+                    (include_subdomains ? "true" : "false")));
+           else
+-            DEBUGP(("Updated HSTS host: %s:%u (max-age: %lu, includeSubdomains: %s)\n",
++            DEBUGP(("Updated HSTS host: %s:%" PRIu32 " (max-age: %" PRId64 ", includeSubdomains: %s)\n",
+                    u->host,
+-                   (unsigned) u->port,
+-                   (unsigned long) max_age,
++                   (uint32_t) u->port,
++                   max_age,
+                    (include_subdomains ? "true" : "false")));
+         }
+       xfree (hsts_params);
+-- 
+GitLab
+
diff --git a/gnu/packages/wget.scm b/gnu/packages/wget.scm
index 083cf27212..6774fd477f 100644
--- a/gnu/packages/wget.scm
+++ b/gnu/packages/wget.scm
@@ -6,6 +6,7 @@
 ;;; Copyright © 2018–2021 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2020 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;; Copyright © 2021 Michael Rohleder <mike@rohleder.de>
+;;; Copyright © 2023 Andreas Enge <andreas@enge.fr>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -54,7 +55,8 @@ (define-public wget
                           version ".tar.lz"))
       (sha256
        (base32
-        "19afmyr1i3zwdwr8wkyz8q6z5764ik3dm87as194g78l8xggplnv"))))
+        "19afmyr1i3zwdwr8wkyz8q6z5764ik3dm87as194g78l8xggplnv"))
+      (patches (search-patches "wget-hsts-portability.patch"))))
     (build-system gnu-build-system)
     (inputs
      (list gnutls libidn2 libpsl))
-- 
2.39.2