'guix build' and garbage collection

'guix build' and garbage collection

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 746 bytes --]

Hi,

I saw this in guix/scripts.build.scm:

--8<---------------cut here---------------start------------->8---
(and (build-derivations store drv mode)
     (for-each show-derivation-outputs drv)
     (for-each (cut register-root store <> <>)
               (map (lambda (drv)
                      (map cdr
                           (derivation->output-paths drv)))
                    drv)
               roots))
--8<---------------cut here---------------end--------------->8---

Here, we build the derivations, and then we add indirect GC roots.  My
question is: what guarantee is there that the output paths will not be
garbage collected after we build the derivations but before we add the
indirect GC roots?

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: 'guix build' and garbage collection

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> I saw this in guix/scripts.build.scm:
>
> (and (build-derivations store drv mode)
>      (for-each show-derivation-outputs drv)
>      (for-each (cut register-root store <> <>)
>                (map (lambda (drv)
>                       (map cdr
>                            (derivation->output-paths drv)))
>                     drv)
>                roots))
>
> Here, we build the derivations, and then we add indirect GC roots.  My
> question is: what guarantee is there that the output paths will not be
> garbage collected after we build the derivations but before we add the
> indirect GC roots?

For the duration of the connection to the build daemon, build results
are registered as GC roots, so we’re fine.

You can see it in the ‘DerivationGoal’ constructor in libstore/build.cc:

    /* Prevent the .chroot directory from being
       garbage-collected. (See isActiveTempFile() in gc.cc.) */
    worker.store.addTempRoot(drvPath);

Likewise in ‘SubstitutionGoal::init’.

HTH,
Ludo’.

Re: 'guix build' and garbage collection

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1835 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> I saw this in guix/scripts.build.scm:
>>
>> (and (build-derivations store drv mode)
>>      (for-each show-derivation-outputs drv)
>>      (for-each (cut register-root store <> <>)
>>                (map (lambda (drv)
>>                       (map cdr
>>                            (derivation->output-paths drv)))
>>                     drv)
>>                roots))
>>
>> Here, we build the derivations, and then we add indirect GC roots.  My
>> question is: what guarantee is there that the output paths will not be
>> garbage collected after we build the derivations but before we add the
>> indirect GC roots?
>
> For the duration of the connection to the build daemon, build results
> are registered as GC roots, so we’re fine.
>
> You can see it in the ‘DerivationGoal’ constructor in libstore/build.cc:
>
>     /* Prevent the .chroot directory from being
>        garbage-collected. (See isActiveTempFile() in gc.cc.) */
>     worker.store.addTempRoot(drvPath);
>
> Likewise in ‘SubstitutionGoal::init’.
>
> HTH,
> Ludo’.

That's exactly what I was missing.  Thank you for the information.

My understanding is that Nix builds derivations in a temporary
directory.  My understanding is that eventually, we copy the results of
the build into the final, content-addressed store path.  Since we don't
know the final, content-addressed store path until after we calculate
the hash of the output, how do we prevent that final store path from
being garbage collected while the build process is running?  Do we use
addTempRoot to add the final, content-addressed store path as a
temporary root, too?  I looked in the code but couldn't figure this out.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: 'guix build' and garbage collection

[Ludovic Courtès]

Hi Chris,

Chris Marusich <cmmarusich@gmail.com> skribis:

> My understanding is that Nix builds derivations in a temporary
> directory.  My understanding is that eventually, we copy the results of
> the build into the final, content-addressed store path.  Since we don't
> know the final, content-addressed store path until after we calculate
> the hash of the output, how do we prevent that final store path from
> being garbage collected while the build process is running?  Do we use
> addTempRoot to add the final, content-addressed store path as a
> temporary root, too?  I looked in the code but couldn't figure this out.

Store items are not content-addressed, except for fixed-output
derivations and things added with ‘add-to-store’ and
‘add-text-to-store’.

Leaving these two special cases aside, store file names are computed as
a function of the build inputs of a derivation.  In Eelco Dolstra’s
thesis, both models are described: the “extensional model” (the one Nix
and Guix use) and the “intensional model” (content-addressability, which
has never been deployed, and which is challenging in many ways.)

HTH!

Ludo’.

Re: 'guix build' and garbage collection

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 2033 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi Chris,
>
> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> My understanding is that Nix builds derivations in a temporary
>> directory.  My understanding is that eventually, we copy the results of
>> the build into the final, content-addressed store path.  Since we don't
>> know the final, content-addressed store path until after we calculate
>> the hash of the output, how do we prevent that final store path from
>> being garbage collected while the build process is running?  Do we use
>> addTempRoot to add the final, content-addressed store path as a
>> temporary root, too?  I looked in the code but couldn't figure this out.
>
> Store items are not content-addressed, except for fixed-output
> derivations and things added with ‘add-to-store’ and
> ‘add-text-to-store’.

Oh!  This was my misunderstanding, then.  Thank you for clarifying it.

> Leaving these two special cases aside, store file names are computed as
> a function of the build inputs of a derivation.  In Eelco Dolstra’s
> thesis, both models are described: the “extensional model” (the one Nix
> and Guix use) and the “intensional model” (content-addressability, which
> has never been deployed, and which is challenging in many ways.)

I had incorrectly assumed that we (and Nix) were using the intensional
model.  When I read Eelco's thesis, it sounded like the intensional
model was being actively developed and used (in prototype form) in
2006. The intensional model presented in his thesis seemed very well
thought out, and it seemed to bring many benefits over the extensional
model, so I'm surprised that we're still using the extensional model
today.

Do you know why the intensional model hasn't been deployed in the 11
years since the thesis was published?  To learn more, I can think of a
few places to look (Nix email lists, other Nix-related research papers),
but if you have specific recommendations, it would be helpful!

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: 'guix build' and garbage collection

[Ludovic Courtès]

Hi!

Chris Marusich <cmmarusich@gmail.com> skribis:

> Do you know why the intensional model hasn't been deployed in the 11
> years since the thesis was published?  To learn more, I can think of a
> few places to look (Nix email lists, other Nix-related research papers),
> but if you have specific recommendations, it would be helpful!

The intensional model (content-address item) works well if and only if
packages are 100% bit-reproducible, which they are not (yet!).

Also I’m not sure how substitutes could work with this model since one
cannot refer to the build result until it’s available.

My guesses!

Ludo’.

Re: 'guix build' and garbage collection

[Chris Marusich]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

ludo@gnu.org (Ludovic Courtès) writes:

> Hi!
>
> Chris Marusich <cmmarusich@gmail.com> skribis:
>
>> Do you know why the intensional model hasn't been deployed in the 11
>> years since the thesis was published?  To learn more, I can think of a
>> few places to look (Nix email lists, other Nix-related research papers),
>> but if you have specific recommendations, it would be helpful!
>
> The intensional model (content-address item) works well if and only if
> packages are 100% bit-reproducible, which they are not (yet!).
>
> Also I’m not sure how substitutes could work with this model since one
> cannot refer to the build result until it’s available.

FYI, section 6.4.4 (p. 157) of Eelco's thesis describes how substitution
works in the intensional model.  When performing substitution for a
store path p, if a trusted substitute for p has been registered, then it
is tried, and the substitution is considered successful if and only if
the output's content-addressed store path equals p.

If I learn more about why Nix doesn't use the intensional model, I'll
let you know.

-- 
Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: 'guix build' and garbage collection

[Ludovic Courtès]

Chris Marusich <cmmarusich@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hi!
>>
>> Chris Marusich <cmmarusich@gmail.com> skribis:
>>
>>> Do you know why the intensional model hasn't been deployed in the 11
>>> years since the thesis was published?  To learn more, I can think of a
>>> few places to look (Nix email lists, other Nix-related research papers),
>>> but if you have specific recommendations, it would be helpful!
>>
>> The intensional model (content-address item) works well if and only if
>> packages are 100% bit-reproducible, which they are not (yet!).
>>
>> Also I’m not sure how substitutes could work with this model since one
>> cannot refer to the build result until it’s available.
>
> FYI, section 6.4.4 (p. 157) of Eelco's thesis describes how substitution
> works in the intensional model.  When performing substitution for a
> store path p, if a trusted substitute for p has been registered, then it
> is tried, and the substitution is considered successful if and only if
> the output's content-addressed store path equals p.

Oh indeed, thanks for the reminder!

Ludo’.