According to the manual (6.3.6 Substitution Failure): > Even when a substitute for a derivation is available, sometimes the > substitution attempt will fail. Inconvenient, but expected. > This can happen for a variety of reasons: the substitute server > might be offline, the substitute may recently have been deleted, So, according to this paragraph, it is possible that simultaneously (1) a substitute is available (2) the substitution attempt fails (3) and the substitute server is offline but (1) and (3) contradict each other -- if the substitute server is offline, then from the client's perspective, the substitute might as well not exist, so then substitutes are unavailable, and likewise, if the substitute is deleted, it doesn't exist anymore, so it isn't available. This inconsistency is the specification of Guix substitution is the first bug I'd like to point out. (The paragraph also mentions ‘the connection might have been interrupted, etc.’, which I do not take any issue with.) This is not only a documentation issue, because it impacts --fallback: > When substitutes are enabled and a substitute for a derivation is > available, but the substitution attempt fails, Guix will attempt to > build the derivation locally depending on whether or not ‘--fallback’ > was given (*note common build option ‘--fallback’: fallback-option. I think its reasonable that if a substitute is unavailable (*), then Guix will built it locally. Often, this is already the case (e.g. think of all the times you locally updated a package and prepared a patch), yet this is apparently not always the case when the substitute server is offline or when the substitute previously existed but was deleted. This ‘Yet this is apparently not always the case when [...]’ is the second bug. (*) Unavailability caused by interruption is its own can of worms, which will be covered separately. Now, consider the following output of "guix home build [...]": > substitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerken... 100.0% Third bug: unless you're being informal and putting the NL in nl_NL, it's "bijwerken", not "bijwerke". Also, newlines are missing. Looking at po/guix/nl.po, the translation does, in fact, write "bijwerken" instead of "bijwerke" and it has the same format stuff at the end as the English translation, so there appears to be a bug in the code (instead of the translation). > 21,2 MB will be downloaded: Fourth bug: inconsistent units -- in some substitute output, Guix uses binary prefixes (see, e.g., #63784) and in other places it used metric prefixes. > /gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0 > /gnu/store/y1918l95a3wjij79456n9qdhlk8b8nja-python-qtconsole-5.3.0 > /gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0 > /gnu/store/c536wjwfxdx3z7s9bwwhj1vngpq50aq6-python-ipyparallel-bootstrap-8.2.1 > /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 > /gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0 > /gnu/store/ifk6bsddap0smgq7x6lgkw3174hy29gz-python-ipython-8.5.0 > /gnu/store/dsd9dnf54571qggry3g5qwihx2f3v23m-python-jupyter-client-7.2.2 > /gnu/store/55zn484d18xdy0iarwcfxc5lxzsrjszf-qtconnectivity-5.15.8 > /gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8 > /gnu/store/kzg742h88nd03yagry7ddhpk5plqhnlj-qtmultimedia-5.15.8 > /gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8 > /gnu/store/0dxzxs9yza70ypmw2hw07s9dqghmlda3-qtlocation-5.15.8 > /gnu/store/nb5wjfx56bsw0m4bvgjh2aas4g3zzq2w-qtwebchannel-5.15.8 > /gnu/store/9vpx7hb9dg9r9410sg2r2lmw6yx82ixq-qtsensors-5.15.8 > /gnu/store/m1nw1w8qhkmnj4vmn97bwy6dvhi1awli-qttools-5.15.8 > /gnu/store/9rmh4afafhc77q16gbf1080ffdc7cz1l-qtxmlpatterns-5.15.8 > /gnu/store/fwkxw2ylbx0bd5vx1rnw1zkk40mfx0yn-qtwebsockets-5.15.8 > /gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8 > /gnu/store/c9nq0sbciv9jcw02b61fp4xf82dnn53a-python-pyqt-5.15.9 > /gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0 > /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 vervangen... > /gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0 vervangen... > /gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0 vervangen... > /gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0 vervangen... > /gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0 vervangen... > /gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8 vervangen... > /gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8 vervangen... > /gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8 vervangen... (For context, these are the ‘substituting FOO’ messages) > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is (For context, this is the ‘server is somewhat slow’ message.) > retrying download of '/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0' with other substitute URLs... > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0: de server is een beetje traag > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > retrying download of '/gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0' with other substitute URLs... > retrying download of '/gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0' with other substitute URLs... > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > retrying download of '/gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8' with other substitute URLs... > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > retrying download of '/gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8' with other substitute URLs... > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > retrying download of '/gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0' with other substitute URLs... > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > retrying download of '/gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8' with other substitute URLs... > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > retrying download of '/gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0' with other substitute URLs... > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > guix substitute: fout: failed to find alternative substitute for '/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0' > vervanging van /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 mislukt > guix home: fout: some substitutes for the outputs of derivation `/gnu/store/nvynln5vrrd8983slpais1c4dqn397j4-python-jupyter-client-7.2.2.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source OK, substituting ipykernel-bootstrap failed. Let's look what happened to ipytkernel-bootstrap: > 21,2 MB will be downloaded: > [...] > /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 > [...] > [...] > /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 vervangen... > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > retrying download of '/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0' with other substitute URLs... > [...] > guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0: de server is een beetje traag > guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is > guix substitute: fout: failed to find alternative substitute for '/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0' > vervanging van /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 mislukt > guix home: fout: some substitutes for the outputs of derivation `/gnu/store/nvynln5vrrd8983slpais1c4dqn397j4-python-jupyter-client-7.2.2.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source I.e.: 1. Let's substitute python-pykernel-bootstrap-6.13.0 2. I'm downloading from ci.guix.gnu.org, but it's going a bit slow, patience! 3. For some reason I'm not disclosing, let's download from somewhere else instead. 4. Again, I'm downloading from ci.guix.gnu.org, but again it's going a bit slow. (Also, I lied, instead of downloading from somewhere else, I'm downloading from the same location!) 5. I couldn't find any substitute at another location. There are multiple bugs here: 5. The warning message is badly constructed: ‘during [...]: the server is a bit slow’ is technically English I suppose, but there exist much better ways to phrase this (e.g.: ‘Warning: the download of https://[...] is proceeding slowly.’) 6. When substituting from ci.guix.gnu.org failed, it then used the same location again as ‘other substitution location’ -- that message is a lie, and it also isn't going to accomplish anything (except, maybe, by brute force). 7. It doesn't give information on why it (tries to) switch to another substitution server: did the first server remove the substitute that Guix expected? Is the TLS certificate out-of-date? A time-out? A prematurely closed connection? A hash mismatch? Depending on the cause, the user should do nothing or report an issue to whoever manages the server. My best guess is that Guix decided that the old server is too slow, but it would be better if Guix actually said that, e.g. ‘warning: the download of https://[...] is proceeding too slowly; download aborted.’ 8. It looks like no server had a substitute available (or it was technically available, only very slowly, but for practical purposes that counts as unavailable), so Guix should have built it from source without having to add --fallback. About --fallback, I think that too much unexpected network behaviour is only handled by --fallback, and that more --fallback behaviour should be done unconditionally. I think that, by default, Guix should always build from source when a substitute is unavailable, except when the cause of the unavailability is a symptom of some kind of wrongness. The option --fallback would then also build from source even when the cause of unavailability is a symptom of wrongness. More concretely: * Being offline is inconvenient, but not wrong. * A substitute being deleted isn't wrong either. -- it might a symptom of TTL not being properly implemented, but it could also just be the server administrator going ‘OOps, I need more space, I'll need to delete some substitutes earlier than expected.’ (That's kind of a wrongness, but it's a wrongness that the client doesn't have to report.) (Improperly implemented TTL is wrong IMO, but it's not really problematic and Guix has bigger problems to handle.) * Slowness can be caused by peering problems, but that's not really anything that the client or server can do much about, so it doesn't count as wrongness. * I don't know about interrupted connections, because I don't know what it usually is a symptom of. (There can exist multiple causes, but I don't have a clue about their frequencies.) * Likewise, I don't know about timeouts. * Hash mismatches count as wrongness (likely symptom of FS corruption!) * TLS errors: wrongness, likely indicates misconfiguration or MITM attempt. Best regards, Maxime.