Re: Hundreds of failed build on master following git package update

Re: Hundreds of failed build on master following git package update

[Sarah Morgensen]

Hello,

brice@waegenei.re writes:

> Hello Guix,
>
> Today I pushed 2 commits¹ related to the package to master, mainly to update it to 2.33.0. But since then https://data.guix.gnu.org/ went offline and Cuirass' evaluation #4930² has 556 failed build. Newly failed builds have unhelpful logs, ending with "cannot build missing derivation [...]"³.
>
> Before pushing those commits, I build git and git-minimal successfully on x86_64 and assessed the total impacted package for this update to be inferior to 300. Did I do something wrong with the update commit? Or is it an unrelated issue?
>
> I'm currently locally building some packages which have failed on the CI to see if I can reproduce them and to obtain more verbose build log.
>
> ¹ https://git.savannah.gnu.org/cgit/guix.git/log/?id=ea04295256329511b3201feaefb17900b05053b0
> ² https://ci.guix.gnu.org/eval/4930?status=failed
> ³ https://ci.guix.gnu.org/build/396873/log/raw
>
> Cheers,
> - Brice

That sounds awfully familiar.  I wonder if this is related to the
following issue?

Mathieu Othacehe <othacehe@gnu.org> writes:

> Hello Sarah,
>
>> substitute: [Kupdating substitutes from 'http://141.80.167.131'...   0.0%
>> substitute: [Kupdating substitutes from 'http://141.80.167.131'... 100.0%
>> cannot build missing derivation ?/gnu/store/r1p932bjq21nv4g2qf8fydrphr6fwnw7-go-golang-org-x-lint-0.0.0-0.83fdc39.drv?
>
> Yes, I recently configured Cuirass to use the publish server behind
> ci.guix.gnu.org instead of its own server.
>
> This server uses caching and when a remote worker tries do build a
> derivation that is not available locally, it will also try to fetch the
> input derivations. If one of them is being baked and is above the cache
> bypass limit (50 MiB), the publish server will send a 404 error causing
> the build to stop altogether as you noticed.
>
> This is a clear limitation of this bypass/404 mechanism, that I intend
> to solve with: https://issues.guix.gnu.org/50040.
>
> Thanks,
>
> Mathieu

--
Sarah

Re: Hundreds of failed build on master following git package update

[Mathieu Othacehe]

Hello Brice & Sarah,

> That sounds awfully familiar.  I wonder if this is related to the
> following issue?

Yes, this is indeed completely related to the pointed issue. I'd like to
proceed with the patch soon so that Cuirass can recover a working
state. However, as this is a rather delicate change, it's probably
better to wait for some feedback.

Thanks,

Mathieu

Hundreds of failed build on master following git package update

[brice]

Hello Guix,

Today I pushed 2 commits¹ related to the package to master, mainly to update it to 2.33.0. But since then https://data.guix.gnu.org/ went offline and Cuirass' evaluation #4930² has 556 failed build. Newly failed builds have unhelpful logs, ending with "cannot build missing derivation [...]"³.

Before pushing those commits, I build git and git-minimal successfully on x86_64 and assessed the total impacted package for this update to be inferior to 300. Did I do something wrong with the update commit? Or is it an unrelated issue?

I'm currently locally building some packages which have failed on the CI to see if I can reproduce them and to obtain more verbose build log.

¹ https://git.savannah.gnu.org/cgit/guix.git/log/?id=ea04295256329511b3201feaefb17900b05053b0
² https://ci.guix.gnu.org/eval/4930?status=failed
³ https://ci.guix.gnu.org/build/396873/log/raw

Cheers,
- Brice

Re: Hundreds of failed build on master following git package update

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]


brice@waegenei.re writes:

> Today I pushed 2 commits¹ related to the package to master, mainly to
> update it to 2.33.0. But since then https://data.guix.gnu.org/ went
> offline and Cuirass' evaluation #4930² has 556 failed build. Newly
> failed builds have unhelpful logs, ending with "cannot build missing
> derivation [...]"³.
>
> Before pushing those commits, I build git and git-minimal successfully
> on x86_64 and assessed the total impacted package for this update to
> be inferior to 300. Did I do something wrong with the update commit?
> Or is it an unrelated issue?

Hey,

Thanks for noticing that data.guix.gnu.org was down. I've had a look,
and the machine that runs it ran out of disk space.

I'm not quite sure why it ran out of disk space, I suspect it was maybe
excessive PostgreSQL logs... anyway, I'll get things going again and
keep an eye on it.

Once data.guix.gnu.org has caught up it will be easier to see what the
impact of this change is. I don't have any reason to think it was wrong
to make these changes.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]