bug#42162: gforge.inria.fr to be taken off-line in Dec. 2020

[zimoun <zimon.toutoune@gmail.com>]

Hi Ludo,

On Tue, 12 Oct 2021 at 11:24, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

> I sense a lot of impatience in your message :-), and I also see many
> questions.  It is up to us all to answer them, I’ll just reply
> selectively here.

Impatience?  Probably. :-)


> zimoun <zimon.toutoune@gmail.com> skribis:
>> On Thu, 7 Oct 2021 at 18:07, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
> [...]
>
>>> The second-best solution is to improve our tooling so we can actually
>>> keep source code in a more controlled way.  That’s what I had in mind
>>> with <https://ci.guix.gnu.org/jobset/source>.  We have storage space for
>>> that on berlin, but it’s not infinite.
>>
>> If Berlin has space, why so much derivations are missing when running
>> time-machine?
>
> That’s not related to the question at hand, but it would be worth
> investigating, first by trying to quantify that.

The question seems related. :-)  Because you are saying “we have storage
space for that on Berlin”…

> For the record, the ‘guix publish’ config on berlin is here:
>
>   https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/modules/sysadmin/services.scm#n485
>
> If I read that correctly, nars have a TTL of 180 days (this is the time
> a nar is retained after the last time it has been requested, so it’s a
> lower bound.)

…and the NARs are more or less removed after 180 days if no one asked
for them during these 180 days, IIUC.  This policy seems to keep under
control the size of the storage, I guess.  And I provide an annoying
example of such policy. :-)

Anyway, I agree it is not, for now, the core of the question at
hand. :-)


About quantifying, it is clearly not related to the question at
hand. ;-)

Just for the record, a back to envelope computations.  180 days before
today was April 15th (M-x calendar C-u 180 C-b).  It means 6996 commits
(35aaf1fe10 is my current last commit).

    git log --format="%cd" --after=2021-04-15 | wc -l
    6996

However, these commits are pushed by batch.  Roughly, it reads:

    git log --format="%cd" --after=2021-04-15 --date=unix \
        | awk 'NR == 1{old= $1; next}{print old - $1; old = $1}' \
        | sort -n | uniq -c | grep -e "0$" | head
          1 -1542620
       3388 0
         14 10
          6 20
          5 30
          2 40
          4 50
          1 60
          2 70
          2 80

(Take the ’awk’ with care, I am not sure of what I am doing. :-)  And,
it is rough because timezone etc.)

Other said 3388/6996= ~50% of commits are pushed at the same time, i.e.,
missed by both build farms using 2 different strategies to collect the
thing to build (fetch every 5 minutes or fetch from guix-commits).  It
is a quick back to envelope so keep that with some salt. :-)

On that number, after 180 days (6 months), it is hard to evaluate the
rate of the time-machine queries.  And from my experience (no number to
back), running time-machine on a commit older than this 180 days implies
to build derivations.  Or it is a lucky day. :-)

Drifting, right?  Let focus on the question at hand.   However, this
question of long-term policy asked at:

<https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00215.html>

appears to me worth. :-)

Cheers,
simon