* bug#18747: Fixed-output derivations are not substituted @ 2014-10-16 15:26 Ludovic Courtès 2014-10-16 15:43 ` Ludovic Courtès 0 siblings, 1 reply; 7+ messages in thread From: Ludovic Courtès @ 2014-10-16 15:26 UTC (permalink / raw) To: 18747 With current ‘master’, fixed-output derivations are not substituted, even though a substitute is available: --8<---------------cut here---------------start------------->8--- $ guix build -S acct The following file will be downloaded: /gnu/store/0sszdqmiqmb4mzbgs8h2185fh7p8bfd2-acct-6.6.1.tar.gz @ build-started /gnu/store/b7c2lgbdnr5anzklxkzhjvqxl14n7cqp-acct-6.6.1.tar.gz.drv - x86_64-linux /var/log/guix/drvs/b7//c2lgbdnr5anzklxkzhjvqxl14n7cqp-acct-6.6.1.tar.gz.drv.bz2 starting download of `/gnu/store/0sszdqmiqmb4mzbgs8h2185fh7p8bfd2-acct-6.6.1.tar.gz' from `http://ftpmirror.gnu.org/acct/acct-6.6.1.tar.gz'... following redirection to `http://gnu.cardse.net/acct/acct-6.6.1.tar.gz'... http://gnu.cardse.net/.../acct-6.6.1.tar.gz 100.0% of 2802.4 KiB @ build-succeeded /gnu/store/b7c2lgbdnr5anzklxkzhjvqxl14n7cqp-acct-6.6.1.tar.gz.drv - /gnu/store/0sszdqmiqmb4mzbgs8h2185fh7p8bfd2-acct-6.6.1.tar.gz $ sudo cat /var/guix/substitute-binary/cache/0sszdqmiqmb4mzbgs8h2185fh7p8bfd2 (narinfo (version 1) (cache-uri "http://hydra.gnu.org") (date 1413472751) (value "StorePath:... $ guix build -S libwebsockets The following file will be downloaded: /gnu/store/1k4c1a6nckhm2hjpprvdrlkxx6yifclw-libwebsockets-1.3 @ build-started /gnu/store/8x0jgw0819l7aqi1k1f6i772skmwq1zq-libwebsockets-1.3.drv - x86_64-linux /var/log/guix/drvs/8x//0jgw0819l7aqi1k1f6i772skmwq1zq-libwebsockets-1.3.drv.bz2 Cloning into '/gnu/store/1k4c1a6nckhm2hjpprvdrlkxx6yifclw-libwebsockets-1.3'... C-c C-c $ sudo cat /var/guix/substitute-binary/cache/1k4c1a6nckhm2hjpprvdrlkxx6yifclw (narinfo (version 1) (cache-uri "http://hydra.gnu.org") (date 1413466633) (value "StorePath:... --8<---------------cut here---------------end--------------->8--- Here, “The following file will be downloaded” means that a substitute is found and will be used, as seen also in the cache. But then we find ourselves performing the derivation locally regardless. Ludo’. ^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#18747: Fixed-output derivations are not substituted 2014-10-16 15:26 bug#18747: Fixed-output derivations are not substituted Ludovic Courtès @ 2014-10-16 15:43 ` Ludovic Courtès 2014-10-16 15:46 ` Ludovic Courtès 0 siblings, 1 reply; 7+ messages in thread From: Ludovic Courtès @ 2014-10-16 15:43 UTC (permalink / raw) To: 18747 I just realized that this is fixed by just removing #:local-build? #t from (guix download). Initially #:local-build? (aka. ‘preferLocalBuild’) meant “avoid offloading this derivation”, but Nix commit 55586527 (June 2013) changed it to “avoid offloading *or substituting* this derivation”, which I somehow overlooked for all this time. I think we to distinguish between whether to offload and whether to substitute. Typically, we don’t want to offload downloads, because that’s inefficient. However, we may want to substitute downloads, because then if the upstream download URL has become invalid, we can still get the substitute from hydra.gnu.org. Until then, I think I’ll just comment out #:local-build? #t in (guix download) and friends. Thoughts? Thanks, Ludo’. ^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#18747: Fixed-output derivations are not substituted 2014-10-16 15:43 ` Ludovic Courtès @ 2014-10-16 15:46 ` Ludovic Courtès 2015-07-01 15:21 ` bug#18747: [PATCH] Distinguish between “offloadability” and “substitutability” Ludovic Courtès ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Ludovic Courtès @ 2014-10-16 15:46 UTC (permalink / raw) To: 18747 ludo@gnu.org (Ludovic Courtès) skribis: > Initially #:local-build? (aka. ‘preferLocalBuild’) meant “avoid > offloading this derivation”, but Nix commit 55586527 (June 2013) changed > it to “avoid offloading *or substituting* this derivation”, which I > somehow overlooked for all this time. The relevant code is in libstore/build.cc: --8<---------------cut here---------------start------------->8--- /* We are first going to try to create the invalid output paths through substitutes. If that doesn't work, we'll build them. */ if (settings.useSubstitutes && !willBuildLocally(drv)) foreach (PathSet::iterator, i, invalidOutputs) addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair)); --8<---------------cut here---------------end--------------->8--- Ludo’. ^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#18747: [PATCH] Distinguish between “offloadability” and “substitutability” 2014-10-16 15:46 ` Ludovic Courtès @ 2015-07-01 15:21 ` Ludovic Courtès [not found] ` <87d20boqcw.fsf_-_@gnu.org> 2015-07-02 22:33 ` bug#18747: Fixed-output derivations are not substituted Ludovic Courtès 2 siblings, 0 replies; 7+ messages in thread From: Ludovic Courtès @ 2015-07-01 15:21 UTC (permalink / raw) To: 18747, Eelco Dolstra; +Cc: nix-dev [-- Attachment #1: Type: text/plain, Size: 574 bytes --] Hi again, Nix commit 55586527 (June 2013) changed the semantics of ‘preferLocalBuild’ from “avoid offloading this derivation” to “avoid offloading *or substituting* this derivation” (see <http://bugs.gnu.org/18747>.) This patch introduces a new special key, ‘substitution’, to specify whether a derivation should be substituted. ‘preferLocalBuild’ is kept, but its initial semantics is restored. As a consequence, existing uses of ‘preferLocalBuild’ must be audited to check what exactly is intended. WDYT? Thanks, Ludo’. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: Type: text/x-patch, Size: 1636 bytes --] diff --git a/nix/libstore/misc.cc b/nix/libstore/misc.cc index 6ecf878..bdb7a6d 100644 --- a/nix/libstore/misc.cc +++ b/nix/libstore/misc.cc @@ -67,6 +67,15 @@ Path findOutput(const Derivation & drv, string id) } +/* Return true if we are allowed to substitute DRV. This is the case, unless + DRV specifies 'substitution = "0"' in its environment. */ +static bool substitutionAllowed(const Derivation & drv) +{ + auto env = drv.env; + auto i = env.find("substitution"); + return i == env.end() || i->second != "0"; +} + void queryMissing(StoreAPI & store, const PathSet & targets, PathSet & willBuild, PathSet & willSubstitute, PathSet & unknown, unsigned long long & downloadSize, unsigned long long & narSize) @@ -120,7 +129,7 @@ void queryMissing(StoreAPI & store, const PathSet & targets, if (invalid.empty()) continue; todoDrv.insert(*i); - if (settings.useSubstitutes && !willBuildLocally(drv)) + if (settings.useSubstitutes && substitutionAllowed(drv)) query.insert(invalid.begin(), invalid.end()); } @@ -144,7 +153,7 @@ void queryMissing(StoreAPI & store, const PathSet & targets, PathSet outputs; bool mustBuild = false; - if (settings.useSubstitutes && !willBuildLocally(drv)) { + if (settings.useSubstitutes && substitutionAllowed(drv)) { foreach (DerivationOutputs::iterator, j, drv.outputs) { if (!wantOutput(j->first, i2.second)) continue; if (!store.isValidPath(j->second.path)) { ^ permalink raw reply related [flat|nested] 7+ messages in thread
[parent not found: <87d20boqcw.fsf_-_@gnu.org>]
* bug#18747: [PATCH] Distinguish between “offloadability” and “substitutability” [not found] ` <87d20boqcw.fsf_-_@gnu.org> @ 2015-07-01 15:27 ` Eelco Dolstra [not found] ` <559406C6.7040209@logicblox.com> 1 sibling, 0 replies; 7+ messages in thread From: Eelco Dolstra @ 2015-07-01 15:27 UTC (permalink / raw) To: Ludovic Courtès, 18747; +Cc: nix-dev Hi, On 01/07/15 17:21, Ludovic Courtès wrote: > Nix commit 55586527 (June 2013) changed the semantics of > ‘preferLocalBuild’ from “avoid offloading this derivation” to “avoid > offloading *or substituting* this derivation” (see > <http://bugs.gnu.org/18747>.) > > This patch introduces a new special key, ‘substitution’, to specify > whether a derivation should be substituted. ‘preferLocalBuild’ is kept, > but its initial semantics is restored. Have you seen https://github.com/NixOS/nix/commit/b64988bb3585478676585a0f0aecbcf4e11d4432, which essentially does the same thing? -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <559406C6.7040209@logicblox.com>]
* bug#18747: [PATCH] Distinguish between “offloadability” and “substitutability” [not found] ` <559406C6.7040209@logicblox.com> @ 2015-07-01 21:14 ` Ludovic Courtès 0 siblings, 0 replies; 7+ messages in thread From: Ludovic Courtès @ 2015-07-01 21:14 UTC (permalink / raw) To: Eelco Dolstra; +Cc: 18747, nix-dev Eelco Dolstra <eelco.dolstra@logicblox.com> skribis: > On 01/07/15 17:21, Ludovic Courtès wrote: > >> Nix commit 55586527 (June 2013) changed the semantics of >> ‘preferLocalBuild’ from “avoid offloading this derivation” to “avoid >> offloading *or substituting* this derivation” (see >> <http://bugs.gnu.org/18747>.) >> >> This patch introduces a new special key, ‘substitution’, to specify >> whether a derivation should be substituted. ‘preferLocalBuild’ is kept, >> but its initial semantics is restored. > > Have you seen > https://github.com/NixOS/nix/commit/b64988bb3585478676585a0f0aecbcf4e11d4432, > which essentially does the same thing? I hadn’t, but that’s perfect–we’re on the same page. ;-) Thanks! Ludo’. ^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#18747: Fixed-output derivations are not substituted 2014-10-16 15:46 ` Ludovic Courtès 2015-07-01 15:21 ` bug#18747: [PATCH] Distinguish between “offloadability” and “substitutability” Ludovic Courtès [not found] ` <87d20boqcw.fsf_-_@gnu.org> @ 2015-07-02 22:33 ` Ludovic Courtès 2 siblings, 0 replies; 7+ messages in thread From: Ludovic Courtès @ 2015-07-02 22:33 UTC (permalink / raw) To: 18747-done Fixed in 4a6aeb6. Note that there are still a few cases, in guix/*download.scm, where we haven’t yet reintroduced #:local-build? #t. We’ll do that when daemons with the fix are more widely deployed. Ludo’. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-07-02 22:34 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-10-16 15:26 bug#18747: Fixed-output derivations are not substituted Ludovic Courtès 2014-10-16 15:43 ` Ludovic Courtès 2014-10-16 15:46 ` Ludovic Courtès 2015-07-01 15:21 ` bug#18747: [PATCH] Distinguish between “offloadability” and “substitutability” Ludovic Courtès [not found] ` <87d20boqcw.fsf_-_@gnu.org> 2015-07-01 15:27 ` Eelco Dolstra [not found] ` <559406C6.7040209@logicblox.com> 2015-07-01 21:14 ` Ludovic Courtès 2015-07-02 22:33 ` bug#18747: Fixed-output derivations are not substituted Ludovic Courtès
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.