* 'guix build' and garbage collection @ 2017-04-02 1:06 Chris Marusich 2017-04-02 9:30 ` Ludovic Courtès 0 siblings, 1 reply; 8+ messages in thread From: Chris Marusich @ 2017-04-02 1:06 UTC (permalink / raw) To: guix-devel [-- Attachment #1: Type: text/plain, Size: 746 bytes --] Hi, I saw this in guix/scripts.build.scm: --8<---------------cut here---------------start------------->8--- (and (build-derivations store drv mode) (for-each show-derivation-outputs drv) (for-each (cut register-root store <> <>) (map (lambda (drv) (map cdr (derivation->output-paths drv))) drv) roots)) --8<---------------cut here---------------end--------------->8--- Here, we build the derivations, and then we add indirect GC roots. My question is: what guarantee is there that the output paths will not be garbage collected after we build the derivations but before we add the indirect GC roots? -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 'guix build' and garbage collection 2017-04-02 1:06 'guix build' and garbage collection Chris Marusich @ 2017-04-02 9:30 ` Ludovic Courtès 2017-04-03 6:40 ` Chris Marusich 0 siblings, 1 reply; 8+ messages in thread From: Ludovic Courtès @ 2017-04-02 9:30 UTC (permalink / raw) To: Chris Marusich; +Cc: guix-devel Hi Chris, Chris Marusich <cmmarusich@gmail.com> skribis: > I saw this in guix/scripts.build.scm: > > (and (build-derivations store drv mode) > (for-each show-derivation-outputs drv) > (for-each (cut register-root store <> <>) > (map (lambda (drv) > (map cdr > (derivation->output-paths drv))) > drv) > roots)) > > Here, we build the derivations, and then we add indirect GC roots. My > question is: what guarantee is there that the output paths will not be > garbage collected after we build the derivations but before we add the > indirect GC roots? For the duration of the connection to the build daemon, build results are registered as GC roots, so we’re fine. You can see it in the ‘DerivationGoal’ constructor in libstore/build.cc: /* Prevent the .chroot directory from being garbage-collected. (See isActiveTempFile() in gc.cc.) */ worker.store.addTempRoot(drvPath); Likewise in ‘SubstitutionGoal::init’. HTH, Ludo’. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 'guix build' and garbage collection 2017-04-02 9:30 ` Ludovic Courtès @ 2017-04-03 6:40 ` Chris Marusich 2017-04-03 8:53 ` Ludovic Courtès 0 siblings, 1 reply; 8+ messages in thread From: Chris Marusich @ 2017-04-03 6:40 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 1835 bytes --] ludo@gnu.org (Ludovic Courtès) writes: > Hi Chris, > > Chris Marusich <cmmarusich@gmail.com> skribis: > >> I saw this in guix/scripts.build.scm: >> >> (and (build-derivations store drv mode) >> (for-each show-derivation-outputs drv) >> (for-each (cut register-root store <> <>) >> (map (lambda (drv) >> (map cdr >> (derivation->output-paths drv))) >> drv) >> roots)) >> >> Here, we build the derivations, and then we add indirect GC roots. My >> question is: what guarantee is there that the output paths will not be >> garbage collected after we build the derivations but before we add the >> indirect GC roots? > > For the duration of the connection to the build daemon, build results > are registered as GC roots, so we’re fine. > > You can see it in the ‘DerivationGoal’ constructor in libstore/build.cc: > > /* Prevent the .chroot directory from being > garbage-collected. (See isActiveTempFile() in gc.cc.) */ > worker.store.addTempRoot(drvPath); > > Likewise in ‘SubstitutionGoal::init’. > > HTH, > Ludo’. That's exactly what I was missing. Thank you for the information. My understanding is that Nix builds derivations in a temporary directory. My understanding is that eventually, we copy the results of the build into the final, content-addressed store path. Since we don't know the final, content-addressed store path until after we calculate the hash of the output, how do we prevent that final store path from being garbage collected while the build process is running? Do we use addTempRoot to add the final, content-addressed store path as a temporary root, too? I looked in the code but couldn't figure this out. -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 'guix build' and garbage collection 2017-04-03 6:40 ` Chris Marusich @ 2017-04-03 8:53 ` Ludovic Courtès 2017-04-03 16:29 ` Chris Marusich 0 siblings, 1 reply; 8+ messages in thread From: Ludovic Courtès @ 2017-04-03 8:53 UTC (permalink / raw) To: Chris Marusich; +Cc: guix-devel Hi Chris, Chris Marusich <cmmarusich@gmail.com> skribis: > My understanding is that Nix builds derivations in a temporary > directory. My understanding is that eventually, we copy the results of > the build into the final, content-addressed store path. Since we don't > know the final, content-addressed store path until after we calculate > the hash of the output, how do we prevent that final store path from > being garbage collected while the build process is running? Do we use > addTempRoot to add the final, content-addressed store path as a > temporary root, too? I looked in the code but couldn't figure this out. Store items are not content-addressed, except for fixed-output derivations and things added with ‘add-to-store’ and ‘add-text-to-store’. Leaving these two special cases aside, store file names are computed as a function of the build inputs of a derivation. In Eelco Dolstra’s thesis, both models are described: the “extensional model” (the one Nix and Guix use) and the “intensional model” (content-addressability, which has never been deployed, and which is challenging in many ways.) HTH! Ludo’. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 'guix build' and garbage collection 2017-04-03 8:53 ` Ludovic Courtès @ 2017-04-03 16:29 ` Chris Marusich 2017-04-04 12:13 ` Ludovic Courtès 0 siblings, 1 reply; 8+ messages in thread From: Chris Marusich @ 2017-04-03 16:29 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 2033 bytes --] ludo@gnu.org (Ludovic Courtès) writes: > Hi Chris, > > Chris Marusich <cmmarusich@gmail.com> skribis: > >> My understanding is that Nix builds derivations in a temporary >> directory. My understanding is that eventually, we copy the results of >> the build into the final, content-addressed store path. Since we don't >> know the final, content-addressed store path until after we calculate >> the hash of the output, how do we prevent that final store path from >> being garbage collected while the build process is running? Do we use >> addTempRoot to add the final, content-addressed store path as a >> temporary root, too? I looked in the code but couldn't figure this out. > > Store items are not content-addressed, except for fixed-output > derivations and things added with ‘add-to-store’ and > ‘add-text-to-store’. Oh! This was my misunderstanding, then. Thank you for clarifying it. > Leaving these two special cases aside, store file names are computed as > a function of the build inputs of a derivation. In Eelco Dolstra’s > thesis, both models are described: the “extensional model” (the one Nix > and Guix use) and the “intensional model” (content-addressability, which > has never been deployed, and which is challenging in many ways.) I had incorrectly assumed that we (and Nix) were using the intensional model. When I read Eelco's thesis, it sounded like the intensional model was being actively developed and used (in prototype form) in 2006. The intensional model presented in his thesis seemed very well thought out, and it seemed to bring many benefits over the extensional model, so I'm surprised that we're still using the extensional model today. Do you know why the intensional model hasn't been deployed in the 11 years since the thesis was published? To learn more, I can think of a few places to look (Nix email lists, other Nix-related research papers), but if you have specific recommendations, it would be helpful! -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 'guix build' and garbage collection 2017-04-03 16:29 ` Chris Marusich @ 2017-04-04 12:13 ` Ludovic Courtès 2017-04-07 3:47 ` Chris Marusich 0 siblings, 1 reply; 8+ messages in thread From: Ludovic Courtès @ 2017-04-04 12:13 UTC (permalink / raw) To: Chris Marusich; +Cc: guix-devel Hi! Chris Marusich <cmmarusich@gmail.com> skribis: > Do you know why the intensional model hasn't been deployed in the 11 > years since the thesis was published? To learn more, I can think of a > few places to look (Nix email lists, other Nix-related research papers), > but if you have specific recommendations, it would be helpful! The intensional model (content-address item) works well if and only if packages are 100% bit-reproducible, which they are not (yet!). Also I’m not sure how substitutes could work with this model since one cannot refer to the build result until it’s available. My guesses! Ludo’. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 'guix build' and garbage collection 2017-04-04 12:13 ` Ludovic Courtès @ 2017-04-07 3:47 ` Chris Marusich 2017-04-07 19:53 ` Ludovic Courtès 0 siblings, 1 reply; 8+ messages in thread From: Chris Marusich @ 2017-04-07 3:47 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 1126 bytes --] ludo@gnu.org (Ludovic Courtès) writes: > Hi! > > Chris Marusich <cmmarusich@gmail.com> skribis: > >> Do you know why the intensional model hasn't been deployed in the 11 >> years since the thesis was published? To learn more, I can think of a >> few places to look (Nix email lists, other Nix-related research papers), >> but if you have specific recommendations, it would be helpful! > > The intensional model (content-address item) works well if and only if > packages are 100% bit-reproducible, which they are not (yet!). > > Also I’m not sure how substitutes could work with this model since one > cannot refer to the build result until it’s available. FYI, section 6.4.4 (p. 157) of Eelco's thesis describes how substitution works in the intensional model. When performing substitution for a store path p, if a trusted substitute for p has been registered, then it is tried, and the substitution is considered successful if and only if the output's content-addressed store path equals p. If I learn more about why Nix doesn't use the intensional model, I'll let you know. -- Chris [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 832 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 'guix build' and garbage collection 2017-04-07 3:47 ` Chris Marusich @ 2017-04-07 19:53 ` Ludovic Courtès 0 siblings, 0 replies; 8+ messages in thread From: Ludovic Courtès @ 2017-04-07 19:53 UTC (permalink / raw) To: Chris Marusich; +Cc: guix-devel Chris Marusich <cmmarusich@gmail.com> skribis: > ludo@gnu.org (Ludovic Courtès) writes: > >> Hi! >> >> Chris Marusich <cmmarusich@gmail.com> skribis: >> >>> Do you know why the intensional model hasn't been deployed in the 11 >>> years since the thesis was published? To learn more, I can think of a >>> few places to look (Nix email lists, other Nix-related research papers), >>> but if you have specific recommendations, it would be helpful! >> >> The intensional model (content-address item) works well if and only if >> packages are 100% bit-reproducible, which they are not (yet!). >> >> Also I’m not sure how substitutes could work with this model since one >> cannot refer to the build result until it’s available. > > FYI, section 6.4.4 (p. 157) of Eelco's thesis describes how substitution > works in the intensional model. When performing substitution for a > store path p, if a trusted substitute for p has been registered, then it > is tried, and the substitution is considered successful if and only if > the output's content-addressed store path equals p. Oh indeed, thanks for the reminder! Ludo’. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2017-04-07 19:53 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-04-02 1:06 'guix build' and garbage collection Chris Marusich 2017-04-02 9:30 ` Ludovic Courtès 2017-04-03 6:40 ` Chris Marusich 2017-04-03 8:53 ` Ludovic Courtès 2017-04-03 16:29 ` Chris Marusich 2017-04-04 12:13 ` Ludovic Courtès 2017-04-07 3:47 ` Chris Marusich 2017-04-07 19:53 ` Ludovic Courtès
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.