source tarballs potentially built for each derivation

source tarballs potentially built for each derivation

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 3206 bytes --]

Turning this conversation into a bug, original thread around here:

  https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427

On 2021-05-29, Vagrant Cascadian wrote:
> On 2021-05-01, Leo Famulari wrote:
>> On Sat, May 01, 2021 at 06:45:32PM -0700, Vagrant Cascadian wrote:
>>> Pragmatically speaking, on slower platforms this is a huge resource
>>> overhead. So much so that ci.guix.gnu.org *usually* times out when
>>> generating the linux-libre aarch64 tarballs:
>>> 
>>>   https://ci.guix.gnu.org/search?query=system%3Aaarch64-linux+linux-libre-arm64-generic
>>
>> Thanks for letting me know. I didn't know this was happening.
>>
>> The immediate solution is for me to make sure the tarballs have built
>> before committing the updates. I already do this for x86_64 and I can
>> start doing it for aarch64 too.
>
> This has definitely helped sometimes, thanks! I even saw a substitute of
> linux-libre for aarch64 earlier today! :)
>
> Still, I'm noticing another problem with the way way the tarballs are
> generated on ci.guix.gnu.org ...
>
> When it generates a tarball, all the various packages independently try
> to recreate the source tarball; so you have at least fours jobs
> ("linux-libre", "linux-libre-arm64-generic", "linux-libre-headers",
> "linux-libre-bpf") all concurrently trying to build the very same
> very-slow-to-build tarball on ci.guix.gnu.org. Sometimes one of them
> might succeed, but the others may not, and even though one of them
> succeeded, none of the failing ones retry...
>
> Not knowing exactly how ci.guix.gnu.org works, would it make sense to
> create a tarball package instead of the ... computed origin(?) tarball,
> so it could be better represented in the package dependency graph, and
> the various linux-libre-* packages can wait till it is available rather
> than all trying to recreate the same thing?
>
> That still requires the tarball generation to not time out in the first
> place, but maybe it would help with the resource limitations a bit to
> only build the source tarball once per architecture?

This seems to still be an issue for ci.guix.gnu.org, but the
linux-libre* substitutes for aarch64 seem to be available on
https://bordeaux.guix.gnu.org ...

$ guix weather linux-libre linux-libre-arm64-generic
computing 2 package derivations for aarch64-linux...
looking for 2 store items on https://ci.guix.gnu.org...
https://ci.guix.gnu.org
  0.0% substitutes available (0 out of 2)
  unknown substitute sizes
  0.0 MiB on disk (uncompressed)
  0.740 seconds per request (0.7 seconds in total)
  1.4 requests per second

  0.0% (0 out of 2) of the missing items are queued
  1 queued builds
      aarch64-linux: 1 (100.0%)
  build rate: .00 builds per hour
      x86_64-linux: 0.00 builds per hour
      aarch64-linux: 0.00 builds per hour
      i686-linux: 0.00 builds per hour
looking for 2 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org
  100.0% substitutes available (2 out of 2)
  83.9 MiB of nars (compressed)
  202.2 MiB on disk (uncompressed)
  (continuous integration information unavailable)


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: source tarballs potentially built for each derivation

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 295 bytes --]

On 2021-08-01, Vagrant Cascadian wrote:
> Turning this conversation into a bug, original thread around here:
>
>   https://lists.gnu.org/archive/html/guix-devel/2021-05/threads.html#00427

For reference:

  bug#49810: source tarballs potentially built for each derivation


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]