Julia packages on build farms?

Julia packages on build farms?

[zimoun]

Hi,

I am confused by the missing Julia substitutes on Berlin (ci.guix).  And
I miss why the are missing – another topic; what recently changed on
master that could trigger this rebuild?

That’s said, I do not understand the result of “guix weather” and “guix
build”.  Let pick an example.  Using Guix 06493e7, the package
julia-staticarrays is unavailable on Berlin (ci.guix) and available on
Bordeaux (bordeaux.guix).  So far, so good!

--8<---------------cut here---------------start------------->8---
$ guix weather julia-staticarrays
computing 1 package derivations for x86_64-linux...
looking for 1 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  0.0% substitutes available (0 out of 1)
  unknown substitute sizes
  0,0 MiB on disk (uncompressed)

  0.0% (0 out of 1) of the missing items are queued
  at least 1 000 queued builds
      x86_64-linux: 374 (37.4%)
      i686-linux: 341 (34.1%)
      powerpc64le-linux: 260 (26.0%)
      aarch64-linux: 25 (2.5%)
  build rate: 128.06 builds per hour
      x86_64-linux: 59.81 builds per hour
      i686-linux: 55.88 builds per hour
      powerpc64le-linux: 13.49 builds per hour
      aarch64-linux: 42.47 builds per hour
looking for 1 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  100.0% substitutes available (1 out of 1)
  0,5 MiB of nars (compressed)
  4,7 MiB on disk (uncompressed)
  (continuous integration information unavailable)
--8<---------------cut here---------------end--------------->8---

Now, I get:

--8<---------------cut here---------------start------------->8---
$ guix build julia-staticarrays --substitute-urls=https://ci.guix.gnu.org -n
The following derivation would be built:
  /gnu/store/cinn2zy26791mp4d3qv2gda635a5a2r3-julia-staticarrays-1.2.13.drv

$ guix build julia-staticarrays --substitute-urls=https://bordeaux.guix.gnu.org -n
The following derivation would be built:
  /gnu/store/cinn2zy26791mp4d3qv2gda635a5a2r3-julia-staticarrays-1.2.13.drv
--8<---------------cut here---------------end--------------->8---

where the former is expected, not the latter.  Why is the package
locally built when “guix weather” says the substitutes is available?

What am I doing wrong?


Cheers,
simon

Re: Julia packages on build farms?

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3173 bytes --]

On Wed, Jul 06, 2022 at 01:25:59PM +0200, zimoun wrote:
> Hi,
> 
> I am confused by the missing Julia substitutes on Berlin (ci.guix).  And
> I miss why the are missing – another topic; what recently changed on
> master that could trigger this rebuild?
> 
> That’s said, I do not understand the result of “guix weather” and “guix
> build”.  Let pick an example.  Using Guix 06493e7, the package
> julia-staticarrays is unavailable on Berlin (ci.guix) and available on
> Bordeaux (bordeaux.guix).  So far, so good!
> 
> --8<---------------cut here---------------start------------->8---
> $ guix weather julia-staticarrays
> computing 1 package derivations for x86_64-linux...
> looking for 1 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   0.0% substitutes available (0 out of 1)
>   unknown substitute sizes
>   0,0 MiB on disk (uncompressed)
> 
>   0.0% (0 out of 1) of the missing items are queued
>   at least 1 000 queued builds
>       x86_64-linux: 374 (37.4%)
>       i686-linux: 341 (34.1%)
>       powerpc64le-linux: 260 (26.0%)
>       aarch64-linux: 25 (2.5%)
>   build rate: 128.06 builds per hour
>       x86_64-linux: 59.81 builds per hour
>       i686-linux: 55.88 builds per hour
>       powerpc64le-linux: 13.49 builds per hour
>       aarch64-linux: 42.47 builds per hour
> looking for 1 store items on https://bordeaux.guix.gnu.org...
> https://bordeaux.guix.gnu.org
>   100.0% substitutes available (1 out of 1)
>   0,5 MiB of nars (compressed)
>   4,7 MiB on disk (uncompressed)
>   (continuous integration information unavailable)
> --8<---------------cut here---------------end--------------->8---
> 
> Now, I get:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build julia-staticarrays --substitute-urls=https://ci.guix.gnu.org -n
> The following derivation would be built:
>   /gnu/store/cinn2zy26791mp4d3qv2gda635a5a2r3-julia-staticarrays-1.2.13.drv
> 
> $ guix build julia-staticarrays --substitute-urls=https://bordeaux.guix.gnu.org -n
> The following derivation would be built:
>   /gnu/store/cinn2zy26791mp4d3qv2gda635a5a2r3-julia-staticarrays-1.2.13.drv
> --8<---------------cut here---------------end--------------->8---
> 
> where the former is expected, not the latter.  Why is the package
> locally built when “guix weather” says the substitutes is available?
> 
> What am I doing wrong?

I added a patch to tbb on 4 July to add support for riscv64 (and maybe
other architectures) and that caused a rebuild of julia. I have since
rebuilt all the julia packages locally. I did find that sometimes while
building all the packages in one go some of them failed and needed to be
restarted. After 2 rounds all the packages (guix package -A ^julia-)
built without any issues on x86_64.

Checking ci.guix.gnu.org it looks like none have failed, but they
haven't all started to be built yet.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Julia packages on build farms?

[zimoun]

Hi,

On mer., 06 juil. 2022 at 14:33, Efraim Flashner <efraim@flashner.co.il> wrote:

> I added a patch to tbb on 4 July to add support for riscv64 (and maybe
> other architectures) and that caused a rebuild of julia. I have since
> rebuilt all the julia packages locally. I did find that sometimes while
> building all the packages in one go some of them failed and needed to be
> restarted. After 2 rounds all the packages (guix package -A ^julia-)
> built without any issues on x86_64.

Thanks for the explanations.

Instead of,

    guix build $(guix package -A ^julia-)

I use

   guix build -m julia-packages.scm

where julia-packages.scm reads

--8<---------------cut here---------------start------------->8---
(use-modules (guix build-system julia)
             (guix packages)
             (guix profiles)
             (gnu packages))


(manifest
 (map package->manifest-entry
      (fold-packages
       (lambda (package result)
         (if (eq? (package-build-system package) julia-build-system)
             (cons package result)
             result))
       '())))
--8<---------------cut here---------------end--------------->8---


Well, I do not know if it avoids the failure in the “one go” case.


Cheers,
simon

Re: Julia packages on build farms?

[Ludovic Courtès]

zimoun <zimon.toutoune@gmail.com> skribis:

> (use-modules (guix build-system julia)
>              (guix packages)
>              (guix profiles)
>              (gnu packages))
>
>
> (manifest
>  (map package->manifest-entry
>       (fold-packages
>        (lambda (package result)
>          (if (eq? (package-build-system package) julia-build-system)
>              (cons package result)
>              result))
>        '())))

With a commit from July 3rd, I get good coverage:

--8<---------------cut here---------------start------------->8---
$ guix weather -m /tmp/julia.scm
computing 296 package derivations for x86_64-linux...
looking for 311 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  99.7% substitutes available (310 out of 311)
  at least 300.7 MiB of nars (compressed)
  668.9 MiB on disk (uncompressed)
  0.026 seconds per request (0.9 seconds in total)
  38.6 requests per second

  0.0% (0 out of 1) of the missing items are queued
  at least 1,000 queued builds
      x86_64-linux: 381 (38.1%)
      i686-linux: 341 (34.1%)
      powerpc64le-linux: 257 (25.7%)
      aarch64-linux: 21 (2.1%)
  build rate: 43.80 builds per hour
      aarch64-linux: 11.74 builds per hour
      powerpc64le-linux: 9.46 builds per hour
      i686-linux: 9.55 builds per hour
      x86_64-linux: 13.16 builds per hour
looking for 311 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  98.4% substitutes available (306 out of 311)
  140.3 MiB of nars (compressed)
  665.7 MiB on disk (uncompressed)
  0.007 seconds per request (2.2 seconds in total)
  136.6 requests per second
  (continuous integration information unavailable)
$ guix describe
Generation 221  Jul 03 2022 23:52:07    (current)
  guix e069de4
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: e069de452a2c923868f5137421b4b6349c38d754
--8<---------------cut here---------------end--------------->8---

Ludo’.

Re: Julia packages on build farms?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2075 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> That’s said, I do not understand the result of “guix weather” and “guix
> build”.  Let pick an example.  Using Guix 06493e7, the package
> julia-staticarrays is unavailable on Berlin (ci.guix) and available on
> Bordeaux (bordeaux.guix).  So far, so good!
>
> $ guix weather julia-staticarrays
> computing 1 package derivations for x86_64-linux...
> looking for 1 store items on https://ci.guix.gnu.org...
> https://ci.guix.gnu.org
>   0.0% substitutes available (0 out of 1)
>   unknown substitute sizes
>   0,0 MiB on disk (uncompressed)
>
>   0.0% (0 out of 1) of the missing items are queued
>   at least 1 000 queued builds
>       x86_64-linux: 374 (37.4%)
>       i686-linux: 341 (34.1%)
>       powerpc64le-linux: 260 (26.0%)
>       aarch64-linux: 25 (2.5%)
>   build rate: 128.06 builds per hour
>       x86_64-linux: 59.81 builds per hour
>       i686-linux: 55.88 builds per hour
>       powerpc64le-linux: 13.49 builds per hour
>       aarch64-linux: 42.47 builds per hour
> looking for 1 store items on https://bordeaux.guix.gnu.org...
> https://bordeaux.guix.gnu.org
>   100.0% substitutes available (1 out of 1)
>   0,5 MiB of nars (compressed)
>   4,7 MiB on disk (uncompressed)
>   (continuous integration information unavailable)
>
>
> Now, I get:
>
> $ guix build julia-staticarrays --substitute-urls=https://ci.guix.gnu.org -n
> The following derivation would be built:
>   /gnu/store/cinn2zy26791mp4d3qv2gda635a5a2r3-julia-staticarrays-1.2.13.drv
>
> $ guix build julia-staticarrays --substitute-urls=https://bordeaux.guix.gnu.org -n
> The following derivation would be built:
>   /gnu/store/cinn2zy26791mp4d3qv2gda635a5a2r3-julia-staticarrays-1.2.13.drv
>
> where the former is expected, not the latter.  Why is the package
> locally built when “guix weather” says the substitutes is available?
>
> What am I doing wrong?

guix weather doesn't check substitute trust, so you've probably not got
the bordeaux.guix.gnu.org key in your ACL.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Julia packages on build farms?

[zimoun]

Hi,

On mer., 06 juil. 2022 at 12:41, Christopher Baines <mail@cbaines.net> wrote:

>> What am I doing wrong?
>
> guix weather doesn't check substitute trust, so you've probably not got
> the bordeaux.guix.gnu.org key in your ACL.

Indeed.  Thanks.

I thought it was done by default when answering yes to the question
“Allow substitutes” at install time on foreign distro.


Cheers,
simon

Re: Julia packages on build farms?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 528 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On mer., 06 juil. 2022 at 12:41, Christopher Baines <mail@cbaines.net> wrote:
>
>>> What am I doing wrong?
>>
>> guix weather doesn't check substitute trust, so you've probably not got
>> the bordeaux.guix.gnu.org key in your ACL.
>
> Indeed.  Thanks.
>
> I thought it was done by default when answering yes to the question
> “Allow substitutes” at install time on foreign distro.

Not yet, although I'm expecting this to change with the next release.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]