Re: guix weather -m etc/sources-manifest.scm and CI?

[Christopher Baines <mail@cbaines.net>]

[-- Attachment #1: Type: text/plain, Size: 3769 bytes --]


zimoun <zimon.toutoune@gmail.com> writes:

>> As for keeping build results, everything that's ever been built for
>> bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is
>> still around. In some ways, this is because deleting things is a bit
>> more difficult, as the files aren't in the store, you can't just do a
>> guix gc.
>>
>> However, I do want to try to never delete anything. That's going to
>> require a bit of work as the local storage on the bayfront machine will
>> be all used up at some point, but I do have the beginnings of a plan to
>> avoid this an keep building things.
>
> From my understanding, the project does not have a lot of resources;
> especially about storage.  And my question is: do we have a “coherent”
> plan between the machines behind Bordeaux and the ones behind Berlin?

Not that I'm following at least. The machines behind
bordeaux.guix.gnu.org were just ones that I could find to use when the
plan of dpeloying behind ci.guix.gnu.org was looking less and less
likely.

> I mean, IMHO, it does not seem a correct usage of project resource to
> back twice all from v1.0 to now (maybe from earlier to later).  And I
> also have in mind energy consumption which is becoming more and more a
> precious value.
>
> For instance, we could imagine the both build farms build and challenge
> the outputs.  Then, on Berlin the identical outputs are deleted after
> say 6 months and only live on Bordeaux.  Bordeaux being the backup
> time-machine.  That’s only an idea to open the discussion. :-) Not a
> concrete plan.

Some duplication of stored data is going to be necessary for redundancy
I think, for bordeaux.guix.gnu.org I was thinking of at least trying to
store all the substitute data in two different places.

As you've mentioned, building the same things multiple times is also
valuable for testing reproduciblity.

> Moreover, Disarchive is also something that should be added; probably to
> machines behind Bordeaux in the above plan.

I haven't been following Disarchive too closely. I think I remember
thinking it might be interesting for it to listen to data from
data.guix.gnu.org, to find out about new source tarballs (which isn't
really possible yet).

> From my point of view, what is really missing are the derivations for
> *all* the commits; for example module-import.drv, guix-33bc3fb2a.drv,
> guix-command.drv, guix-module-union.drv, guix-33bc3fb2a-modules.drv,
> guix-packages-modules.drv, guix-system-tests-modules.drv,
> guix-packages-base-modules.drv, etc.  And the backup time-machine should
> systematically build them for all the reachable commits.

If you're just talking about the derivations, data.guix.gnu.org does
store these derivations for the commits it processes (which are just the
states of the master branch, and that data is also missing for some
older commits that were processed a while ago).

If you're talking about substitutes for the outputs of these
derivations, the situation on bordeaux.guix.gnu.org isn't great for
x86_64-linux currently, since building guile@3.0.7 is blocking building
these things for recent revisions [1]. The underlying reason is a
problem with the testsuite I think, but it's hard to work around for
some other reasons as well. Once it can be built, all the blocked builds
will be attempted too.

1: https://data.guix.gnu.org/gnu/store/5z950sh1wpi59mfh53qdjdn8qnlcwp8g-guile-3.0.7.drv

I am interested in what can be done to support using guix time-machine
though, I think there's a range of things like storing substitutes,
testing things better (so they don't start failing to build in the
future), and helping users find the right commits to use.

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]