all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
From: zimoun <zimon.toutoune@gmail.com>
To: Christopher Baines <mail@cbaines.net>,
	Mathieu Othacehe <othacehe@gnu.org>
Cc: guix-devel@gnu.org
Subject: Re: guix weather -m etc/sources-manifest.scm and CI?
Date: Tue, 21 Sep 2021 10:32:55 +0200	[thread overview]
Message-ID: <86zgs6ba6g.fsf@gmail.com> (raw)
In-Reply-To: <87sfy3y0nn.fsf@cbaines.net>

Hi Chris,

On Fri, 17 Sep 2021 at 10:49, Christopher Baines <mail@cbaines.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

>> https://ci.guix.gnu.org
>>   74.6% substitutes available (12,556 out of 16,831)

[...]

>> https://bordeaux.guix.gnu.org
>>   99.8% substitutes available (16,804 out of 16,831)

>> The questions are:
>>
>> Why ci.guix.gnu.org contains only 75%?  And bordeaux almost everything?
>> (I guess the missing ones on bordeaux are corner cases as icecat, linux-libre).
>
> bordeaux.guix.gnu.org is hopefully only missing substitutes for things
> where there's actually an issue building them. It's possible not to
> guess though and instead ask guix weather what is missing.

[...]

> If you visit https://data.guix.gnu.org and add the store path to the
> URL, you can get links to the failed builds ([1] for example).
>
> 1:
> https://data.guix.gnu.org/gnu/store/42knh9b75m6kc30m8v247sswhdfqnn8i-xpp3-1.1.4_src.tgz

Cool!  Thanks for explaining.  Let try to systematically address this
failures about the source. :-)


>> Does it make sense to duplicate the storage of all these origins?
>>
>> Using extensively "guix time-machine", I note that a lot of
>> derivations are missing and thus they are built locally, which is
>> costly on poor machine.  Could we reduce the duplication and so save
>> some space in order to systematically keep these derivations?  It
>> would greatly ease the Guix experience for "guix time-machine" users.
>> :-)
>
> In terms of duplication, I still think that an argument can be made that
> bordeaux.guix.gnu.org is providing value, even if it's doing some of the
> same stuff as ci.guix.gnu.org.

My question with duplication is about “long term storage of all the
sources”.  Not about building twice (it is easy to find justification
via reproducibility checker for instance :-)).

Now Guix have 2 build farms so let consider it is a strength. :-) The
question is how to exploit such strength by sharing the workload and
have a better global scaling up (and not 2 independent scaling).  For
what my opinion is worth here. :-)

> As for keeping build results, everything that's ever been built for
> bordeaux.guix.gnu.org (that's only ~337739 things totalling ~1.4TiBs) is
> still around. In some ways, this is because deleting things is a bit
> more difficult, as the files aren't in the store, you can't just do a
> guix gc.
>
> However, I do want to try to never delete anything. That's going to
> require a bit of work as the local storage on the bayfront machine will
> be all used up at some point, but I do have the beginnings of a plan to
> avoid this an keep building things.

From my understanding, the project does not have a lot of resources;
especially about storage.  And my question is: do we have a “coherent”
plan between the machines behind Bordeaux and the ones behind Berlin?

I mean, IMHO, it does not seem a correct usage of project resource to
back twice all from v1.0 to now (maybe from earlier to later).  And I
also have in mind energy consumption which is becoming more and more a
precious value.

For instance, we could imagine the both build farms build and challenge
the outputs.  Then, on Berlin the identical outputs are deleted after
say 6 months and only live on Bordeaux.  Bordeaux being the backup
time-machine.  That’s only an idea to open the discussion. :-) Not a
concrete plan.

Moreover, Disarchive is also something that should be added; probably to
machines behind Bordeaux in the above plan.

From my point of view, what is really missing are the derivations for
*all* the commits; for example module-import.drv, guix-33bc3fb2a.drv,
guix-command.drv, guix-module-union.drv, guix-33bc3fb2a-modules.drv,
guix-packages-modules.drv, guix-system-tests-modules.drv,
guix-packages-base-modules.drv, etc.  And the backup time-machine should
systematically build them for all the reachable commits.


Cheers,
simon


  reply	other threads:[~2021-09-21  8:42 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-15 17:03 guix weather -m etc/sources-manifest.scm and CI? zimoun
2021-09-17  9:49 ` Christopher Baines
2021-09-21  8:32   ` zimoun [this message]
2021-09-21 20:36     ` Christopher Baines
2021-09-23 20:18 ` Ludovic Courtès
2021-09-23 20:36   ` zimoun
2021-09-28 12:28     ` Ludovic Courtès
2021-10-01  9:38       ` Mathieu Othacehe
2021-10-02 14:48         ` Ludovic Courtès
2021-10-02 17:08           ` Mathieu Othacehe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=86zgs6ba6g.fsf@gmail.com \
    --to=zimon.toutoune@gmail.com \
    --cc=guix-devel@gnu.org \
    --cc=mail@cbaines.net \
    --cc=othacehe@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.