bug#63634: nar 404 leads to hard ‘guix substitute’ crash

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Ludovic Courtès]

Like this:

--8<---------------cut here---------------start------------->8---
 gst-plugins-bad-minimal-1.22.2  650KiB                 6.8MiB/s 00:00 ▕██████████████████▏ 100.0%
 guile-bytestructures-1.0.10  222KiB                    4.1MiB/s 00:00 ▕██████████████████▏ 100.0%
Backtrace:
In ice-9/boot-9.scm:
  1752:10 19 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          18 (apply-smob/0 #<thunk 7f6df0f172a0>)
In ice-9/boot-9.scm:
    724:2 17 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8 16 (_ #(#(#<directory (guile-user) 7f6df0f1cc80>)))
In guix/ui.scm:
   2300:7 15 (run-guix . _)
  2263:10 14 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1752:10 12 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
   854:15 11 (_)
    646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    463:7  8 (download-nar #<<narinfo> path: "/gnu/store/dg0rm1mp9q…> …)
In ice-9/boot-9.scm:
  1747:15  7 (with-exception-handler #<procedure 7f6df0fa71e0 at ic…> …)
  1685:16  6 (raise-exception _ #:continuable? _)
  1683:16  5 (raise-exception _ #:continuable? _)
  1685:16  4 (raise-exception _ #:continuable? _)
  1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1685:16  2 (raise-exception _ #:continuable? _)
  1683:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &http-get-error:
      uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1" query: #f fragment: #f>
      code: 404
      reason: "Not Found"
      headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 39 minute: 59 hour: 21 day: 21 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
  2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1: HTTP download failed: 404 (\"Not Found\")"
substitution of /gnu/store/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1 failed
guix system: error: corrupt input while restoring archive from #<closed: file 7f0e7113b3f0>
$ guix describe
Generation 263  May 21 2023 23:56:04    (current)
  guix c5bc698
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: c5bc698e8922d78ed85989985cc2ceb034de2f23
  guile 36fd2b4
    repository URL: https://git.savannah.gnu.org/git/guile.git
    branch: main
    commit: 36fd2b4920ae926c79b936c29e739e71a6dff2bc
  shepherd 954742c
    repository URL: https://git.savannah.gnu.org/git/shepherd.git
    branch: master
    commit: 954742cd0d066ac4be7de4a031d7e229fd445c5d
$ guix package -p /run/current-system/profile -I guix
guix      	1.4.0-6.dc5430c	out	/gnu/store/cgjddvw9zay626z8hyxl0zmn1354c24k-guix-1.4.0-6.dc5430c
guix-icons	0.1            	out	/gnu/store/arcpwjvhp3ws1i6g1sdnfxxrxc3zq4m9-guix-icons-0.1
--8<---------------cut here---------------end--------------->8---

This might be related to 8bd4126917f59f4af9a4323c3d5699201862dca2.  The
two ‘download-nar’ calls are guarded against ‘http-get-error?’ though.

Ludo’.

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> In guix/scripts/substitute.scm:
>    854:15 11 (_)
>     646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
> In ice-9/boot-9.scm:
>   1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
> In guix/scripts/substitute.scm:
>     463:7  8 (download-nar #<<narinfo> path: "/gnu/store/dg0rm1mp9q…> …)
> In ice-9/boot-9.scm:
>   1747:15  7 (with-exception-handler #<procedure 7f6df0fa71e0 at ic…> …)
>   1685:16  6 (raise-exception _ #:continuable? _)
>   1683:16  5 (raise-exception _ #:continuable? _)
>   1685:16  4 (raise-exception _ #:continuable? _)
>   1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
>   1685:16  2 (raise-exception _ #:continuable? _)
>   1683:16  1 (raise-exception _ #:continuable? _)
>   1685:16  0 (raise-exception _ #:continuable? _)
>
> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> ERROR:
>   1. &http-get-error:
>       uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1" query: #f fragment: #f>
>       code: 404
>       reason: "Not Found"
>       headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 39 minute: 59 hour: 21 day: 21 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))

Possibly duplicate of <https://issues.guix.gnu.org/55820>.

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Simon Tournier]

Hi,


On lun., 22 mai 2023 at 00:12, Ludovic Courtès <ludo@gnu.org> wrote:

> This might be related to 8bd4126917f59f4af9a4323c3d5699201862dca2.  The
> two ‘download-nar’ calls are guarded against ‘http-get-error?’ though.

Well, I am currently hitting something very similar.

--8<---------------cut here---------------start------------->8---
$ guix build rust-base64@0.13 -S
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0,0 MB will be downloaded:
  /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
substituting /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz...
Backtrace:
          16 (primitive-load "/gnu/store/jpcpva9gg7l337d0icn88pf76l4…")
In guix/ui.scm:
   2263:7 15 (run-guix . _)
  2226:10 14 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1752:10 12 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
   856:15 11 (_)
    650:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    467:7  8 (download-nar #<<narinfo> path: "/gnu/store/sx6sr6cs1x…> …)
In ice-9/boot-9.scm:
  1747:15  7 (with-exception-handler #<procedure 7f6a0ca624e0 at ic…> …)
  1685:16  6 (raise-exception _ #:continuable? _)
  1683:16  5 (raise-exception _ #:continuable? _)
  1685:16  4 (raise-exception _ #:continuable? _)
  1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1685:16  2 (raise-exception _ #:continuable? _)
  1683:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &http-get-error:
      uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz" query: #f fragment: #f>
      code: 404
      reason: "Not Found"
      headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 28 minute: 9 hour: 8 day: 22 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
  2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz: HTTP download failed: 404 (\"Not Found\")"
substitution of /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz failed
guix build: error: some substitutes for the outputs of derivation `/gnu/store/qcpzwblzpa2wprc4vwlm128bwckfj8rj-rust-base64-0.13.1.tar.xz.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
--8<---------------cut here---------------end--------------->8---

Then the option “--fallback” works.

>                                                                     The
> two ‘download-nar’ calls are guarded against ‘http-get-error?’ though.

Well, the fallback should happen transparently.

It rings this bell
<https://issues.guix.gnu.org/msgid/f7db538ea324545f2e297b6a6cdcd937f889ef63.1683293896.git.dev@jpoiret.xyz>.

Cheers,
simon

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Ludovic Courtès]

Simon Tournier <zimon.toutoune@gmail.com> skribis:

>   2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz: HTTP download failed: 404 (\"Not Found\")"

Look:

--8<---------------cut here---------------start------------->8---
$ wget -qO- https://bordeaux.guix.gnu.org/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x.narinfo
StorePath: /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
NarHash: sha256:0pynnlsiiq03h7ysfkzw53wkf6hfmplb63haz6bjqdc686rnf4q1
NarSize: 49128
References: 
System: x86_64-linux
Deriver: qcpzwblzpa2wprc4vwlm128bwckfj8rj-rust-base64-0.13.1.tar.xz.drv
Signature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjRUE2MTcwNUZENjNCOTU1NzcwNjIxN0EyQjczN0U5NzI2RjM4Rjk1NkQwRTRCQjI4MjVBRTA2MTdDQjcxMDQ1MyMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDNGRjU3ODU1RTNERDMxQkJFMUY2NDEyMkU0MzNBNzcwMEZGNkI1ODExRUQ0NTBCQzVEQTcwMkQyNDk3ODZEMCMpCiAgIChzICMwN0ZCQjREN0YwNTExQTlBNzZENzBFNzUxQTc2NTM1MzE0MzU1Mzk1RUUwNzgzODU3REM2NEYxRDIxMTEzQjMxIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==
URL: nar/lzip/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
Compression: lzip
FileSize: 49781
--8<---------------cut here---------------end--------------->8---

See? zstd’s gone!  This confirms my hypothesis.

BTW, ‘guix publish’ has provisions to not compress already-compressed
files, as in the case above.  We should really share code, Chris.

Ludo’.

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1802 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Simon Tournier <zimon.toutoune@gmail.com> skribis:
>
>> 2. &message:
> "https://bordeaux.guix.gnu.org/nar/zstd/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz:
> HTTP download failed: 404 (\"Not Found\")"
>
> Look:
>
> $ wget -qO- https://bordeaux.guix.gnu.org/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x.narinfo
> StorePath: /gnu/store/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
> NarHash: sha256:0pynnlsiiq03h7ysfkzw53wkf6hfmplb63haz6bjqdc686rnf4q1
> NarSize: 49128
> References: 
> System: x86_64-linux
> Deriver: qcpzwblzpa2wprc4vwlm128bwckfj8rj-rust-base64-0.13.1.tar.xz.drv
> Signature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjRUE2MTcwNUZENjNCOTU1NzcwNjIxN0EyQjczN0U5NzI2RjM4Rjk1NkQwRTRCQjI4MjVBRTA2MTdDQjcxMDQ1MyMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDNGRjU3ODU1RTNERDMxQkJFMUY2NDEyMkU0MzNBNzcwMEZGNkI1ODExRUQ0NTBCQzVEQTcwMkQyNDk3ODZEMCMpCiAgIChzICMwN0ZCQjREN0YwNTExQTlBNzZENzBFNzUxQTc2NTM1MzE0MzU1Mzk1RUUwNzgzODU3REM2NEYxRDIxMTEzQjMxIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==
> URL: nar/lzip/sx6sr6cs1x8sf5jhgb65rcr1yxk1q75x-rust-base64-0.13.1.tar.xz
> Compression: lzip
> FileSize: 49781
>
> See? zstd’s gone!  This confirms my hypothesis.
>
> BTW, ‘guix publish’ has provisions to not compress already-compressed
> files, as in the case above.  We should really share code, Chris.

I'm not sure if this is what you have in mind regarding code sharing,
but I'll use compressed-file? from (guix utils) in the nar-herder to
avoid generating cached compressions for these kind of files in the
future.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> BTW, ‘guix publish’ has provisions to not compress already-compressed
>> files, as in the case above.  We should really share code, Chris.
>
> I'm not sure if this is what you have in mind regarding code sharing,
> but I'll use compressed-file? from (guix utils) in the nar-herder to
> avoid generating cached compressions for these kind of files in the
> future.

Great, thank you.

Ideally we’d have a single ‘guix publish’ implementation, or at least
most of it would be a module shared by both ‘guix publish’ and
‘nar-herder’ because it’s tricky enough that we’d rather not have two of
them.  The (guix narinfo) module you introduced is a step in that
direction and as discussed earlier, we need to go further.

Ludo’.

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Ludovic Courtès]

I experienced it a second time and have more info:

--8<---------------cut here---------------start------------->8---
substituting /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0...
Backtrace:
In ice-9/boot-9.scm:
  1752:10 19 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
          18 (apply-smob/0 #<thunk 7f4f9bf172a0>)
In ice-9/boot-9.scm:
    724:2 17 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8 16 (_ #(#(#<directory (guile-user) 7f4f9bf1cc80>)))
In guix/ui.scm:
   2300:7 15 (run-guix . _)
  2263:10 14 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1752:10 13 (with-exception-handler _ _ #:unwind? _ # _)
  1752:10 12 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
   854:15 11 (_)
    646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
In ice-9/boot-9.scm:
  1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
    463:7  8 (download-nar #<<narinfo> path: "/gnu/store/apw1y9nf8r…> …)
In ice-9/boot-9.scm:
  1747:15  7 (with-exception-handler #<procedure 7f4f9846af00 at ic…> …)
  1685:16  6 (raise-exception _ #:continuable? _)
  1683:16  5 (raise-exception _ #:continuable? _)
  1685:16  4 (raise-exception _ #:continuable? _)
  1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
  1685:16  2 (raise-exception _ #:continuable? _)
  1683:16  1 (raise-exception _ #:continuable? _)
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &http-get-error:
      uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0" query: #f fragment: #f>
      code: 404
      reason: "Not Found"
      headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 31 minute: 59 hour: 8 day: 22 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
  2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0: HTTP download failed: 404 (\"Not Found\")"
substitution of /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 failed
guix build: error: corrupt input while restoring archive from #<closed: file 7f1693f9c9a0>
--8<---------------cut here---------------end--------------->8---

I have ‘--substitute-urls=https://ci.guix.gnu.org https://bordeaux.guix.gnu.org’.
Let’s see what’s available on each server:

--8<---------------cut here---------------start------------->8---
$ wget -O- https://ci.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
--2023-05-22 11:00:08--  https://ci.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-05-22 11:00:08 ERROR 404: Not Found.
$ wget -O- https://bordeaux.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
--2023-05-22 11:03:36--  https://bordeaux.guix.gnu.org/apw1y9nf8rqgxvjnlr1isbhpd502bcs5.narinfo
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1715 (1.7K) [text/plain]
Saving to: ‘STDOUT’

-                                     0%[                                                                   ]       0  --.-KB/s               StorePath: /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
NarHash: sha256:1am3v254jr9dpgd9r4dkjqxfgbp1xm0vl1hxl49cmmh97dg117z8
NarSize: 144790464
References: 8rz7yh6zdzp4b78f4n9wqj3hav2md4d4-isl-0.24 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 brzrf6qx5d83j1zvirc1xk14wrhyx4hf-binutils-cross-aarch64-linux-gnu-2.38 cinj0g8krh5s428jawsazawlfarp912k-gcc-cross-aarch64-linux-gnu-11.3.0-lib cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0 gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1 rbjq50q2ik8c1glkj5f0ksnwfz64g16g-mpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 rjgpaj54q1q9n3hcvpzyp29fgi2nx6ls-ld-wrapper-aarch64-linux-gnu-0 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 y4ndp9n5krkaiwgvw58c61n7h8ls9b0f-glibc-cross-aarch64-linux-gnu-2.35
System: x86_64-linux
Deriver: fqb09jygm70cmq03xmjm7al6gzajp1p5-gcc-cross-aarch64-linux-gnu-11.3.0.drv
Signature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNzA2MTI1QUFGMjcxQzM2MDM3RkY2MTIwMzI0NzI2NTVDMkU3OEY2NTA3OUY0MzgxMzY4NDhDQzA0OTVFRTIxRiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDg1RTA2ODE3RTFENTdCQzE1NTAxOUNGNjA0QTQxMTlFRjNBREVGQ0VFOTE4NjQzMDIzQTExQzk4RDM3REVBOCMpCiAgIChzICMwMUIwMkQyNjI0MDhCMDIzMUMxMUVDODI1Q0Q2MTkzNTU5NDRCQUNBMzg4NjEwRDIxOEIzNTVCNjM1NjFCQjk3IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==
URL: nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Compression: lzip
FileSize: 24767829
-                                   100%[==================================================================>]   1.67K  --.-KB/s    in 0s      

2023-05-22 11:03:36 (20.6 MB/s) - written to stdout [1715/1715]

$ wget -O/dev/null https://bordeaux.guix.gnu.org/nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
--2023-05-22 11:04:02--  https://bordeaux.guix.gnu.org/nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24767829 (24M) [text/plain]
Saving to: ‘/dev/null’

/dev/null                           100%[==================================================================>]  23.62M  44.8MB/s    in 0.5s    

2023-05-22 11:04:03 (44.8 MB/s) - ‘/dev/null’ saved [24767829/24767829]

$ wget -O/dev/null https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
--2023-05-22 11:04:28--  https://bordeaux.guix.gnu.org/nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0
Resolving bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)... 185.233.100.56, 2a0c:e300::58
Connecting to bordeaux.guix.gnu.org (bordeaux.guix.gnu.org)|185.233.100.56|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-05-22 11:04:28 ERROR 404: Not Found.
--8<---------------cut here---------------end--------------->8---

So it seems ‘guix substitute’ picked /nar/zstd, even though
bordeaux.guix is not advertising that.

Or is it?

--8<---------------cut here---------------start------------->8---
$ sudo cat /var/guix/substitute/cache/kzwjeblndsbkjzmjailrt4bnhguil7tqjmewzcyw22hgajbhfy3q/apw1y9nf8rqgxvjnlr1isbhpd502bcs5
(narinfo (version 2) (cache-uri "https://bordeaux.guix.gnu.org") (date 1683633757) (ttl 15552000) (value "StorePath: /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nNarHash: sha256:1am3v254jr9dpgd9r4dkjqxfgbp1xm0vl1hxl49cmmh97dg117z8\nNarSize: 144790464\nReferences: 8rz7yh6zdzp4b78f4n9wqj3hav2md4d4-isl-0.24 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 brzrf6qx5d83j1zvirc1xk14wrhyx4hf-binutils-cross-aarch64-linux-gnu-2.38 cinj0g8krh5s428jawsazawlfarp912k-gcc-cross-aarch64-linux-gnu-11.3.0-lib cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0 gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1 rbjq50q2ik8c1glkj5f0ksnwfz64g16g-mpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 rjgpaj54q1q9n3hcvpzyp29fgi2nx6ls-ld-wrapper-aarch64-linux-gnu-0 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 y4ndp9n5krkaiwgvw58c61n7h8ls9b0f-glibc-cross-aarch64-linux-gnu-2.35\nSystem: x86_64-linux\nDeriver: fqb09jygm70cmq03xmjm7al6gzajp1p5-gcc-cross-aarch64-linux-gnu-11.3.0.drv\nSignature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNzA2MTI1QUFGMjcxQzM2MDM3RkY2MTIwMzI0NzI2NTVDMkU3OEY2NTA3OUY0MzgxMzY4NDhDQzA0OTVFRTIxRiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDg1RTA2ODE3RTFENTdCQzE1NTAxOUNGNjA0QTQxMTlFRjNBREVGQ0VFOTE4NjQzMDIzQTExQzk4RDM3REVBOCMpCiAgIChzICMwMUIwMkQyNjI0MDhCMDIzMUMxMUVDODI1Q0Q2MTkzNTU5NDRCQUNBMzg4NjEwRDIxOEIzNTVCNjM1NjFCQjk3IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==\nURL: nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: lzip\nFileSize: 24767829\nURL: nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: zstd\nFileSize: 39822104"))ludo@ribbon ~/src/guix [env]$ 
--8<---------------cut here---------------end--------------->8---

So it seems that bordeaux.guix did advertise zstd at some point (and
that narinfo is still in cache) but no longer does.

Chris, can you confirm?

I’ll see how to get ‘guix substitute’ to handle that gracefully.

Ludo’.

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2636 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> So it seems ‘guix substitute’ picked /nar/zstd, even though
> bordeaux.guix is not advertising that.
>
> Or is it?
>
> $ sudo cat /var/guix/substitute/cache/kzwjeblndsbkjzmjailrt4bnhguil7tqjmewzcyw22hgajbhfy3q/apw1y9nf8rqgxvjnlr1isbhpd502bcs5
> (narinfo (version 2) (cache-uri "https://bordeaux.guix.gnu.org") (date 1683633757) (ttl 15552000) (value "StorePath: /gnu/store/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nNarHash: sha256:1am3v254jr9dpgd9r4dkjqxfgbp1xm0vl1hxl49cmmh97dg117z8\nNarSize: 144790464\nReferences: 8rz7yh6zdzp4b78f4n9wqj3hav2md4d4-isl-0.24 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0 brzrf6qx5d83j1zvirc1xk14wrhyx4hf-binutils-cross-aarch64-linux-gnu-2.38 cinj0g8krh5s428jawsazawlfarp912k-gcc-cross-aarch64-linux-gnu-11.3.0-lib cs3hw1wnxgijjzsd61whc8ar3qy9wjd6-mpfr-4.2.0 gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 ib2n2vzqpchc3bhh9i712w5sq9zapn8d-gmp-6.2.1 rbjq50q2ik8c1glkj5f0ksnwfz64g16g-mpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 rjgpaj54q1q9n3hcvpzyp29fgi2nx6ls-ld-wrapper-aarch64-linux-gnu-0 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 y4ndp9n5krkaiwgvw58c61n7h8ls9b0f-glibc-cross-aarch64-linux-gnu-2.35\nSystem: x86_64-linux\nDeriver: fqb09jygm70cmq03xmjm7al6gzajp1p5-gcc-cross-aarch64-linux-gnu-11.3.0.drv\nSignature: 1;bayfront;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNzA2MTI1QUFGMjcxQzM2MDM3RkY2MTIwMzI0NzI2NTVDMkU3OEY2NTA3OUY0MzgxMzY4NDhDQzA0OTVFRTIxRiMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDg1RTA2ODE3RTFENTdCQzE1NTAxOUNGNjA0QTQxMTlFRjNBREVGQ0VFOTE4NjQzMDIzQTExQzk4RDM3REVBOCMpCiAgIChzICMwMUIwMkQyNjI0MDhCMDIzMUMxMUVDODI1Q0Q2MTkzNTU5NDRCQUNBMzg4NjEwRDIxOEIzNTVCNjM1NjFCQjk3IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjN0Q2MDI5MDJEM0EyREJCODNGOEEwRkI5ODYwMkE3NTRDNTQ5M0IwQjc3OEM4RDFERDRFMEY0MURFMTRERTM0RiMpCiAgICkKICApCiApCg==\nURL: nar/lzip/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: lzip\nFileSize: 24767829\nURL: nar/zstd/apw1y9nf8rqgxvjnlr1isbhpd502bcs5-gcc-cross-aarch64-linux-gnu-11.3.0\nCompression: zstd\nFileSize: 39822104"))ludo@ribbon ~/src/guix [env]$
>
> So it seems that bordeaux.guix did advertise zstd at some point (and
> that narinfo is still in cache) but no longer does.
>
> Chris, can you confirm?

Yeah, this would have been a consequence of reducing the cache size [1].

1: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=ce8d3000fda6b80c0cf3e6f8204e0ee293024e6b

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Ludovic Courtès]

Christopher Baines <mail@cbaines.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> So it seems that bordeaux.guix did advertise zstd at some point (and
>> that narinfo is still in cache) but no longer does.
>>
>> Chris, can you confirm?
>
> Yeah, this would have been a consequence of reducing the cache size [1].
>
> 1: https://git.savannah.gnu.org/cgit/guix/maintenance.git/commit/?id=ce8d3000fda6b80c0cf3e6f8204e0ee293024e6b

As discussed on IRC, I think nar-herder should honor the TTL it
advertises for all the URLs of the narinfo (lzip, zstd, etc.) because
that’s how ‘guix substitute’ currently interprets the TTL.

Eventually it would be nice for ‘guix substitute’ to interpret the TTL
as applying to one of the URLs (not necessarily all of them), but this
kind of change takes months if not years to be widely deployed because
users rarely (if ever) update guix-daemon.

Ludo’.

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> So it seems that bordeaux.guix did advertise zstd at some point (and
> that narinfo is still in cache) but no longer does.

Here’s a fix:

  https://issues.guix.gnu.org/63646

Ludo’.

bug#63634: nar 404 leads to hard ‘guix substitute’ crash

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

>     646:2 10 (process-substitution _ _ _ #:cache-urls _ #:acl _ # _ # …)
> In ice-9/boot-9.scm:
>   1752:10  9 (with-exception-handler _ _ #:unwind? _ # _)
> In guix/scripts/substitute.scm:
>     463:7  8 (download-nar #<<narinfo> path: "/gnu/store/dg0rm1mp9q…> …)
> In ice-9/boot-9.scm:
>   1747:15  7 (with-exception-handler #<procedure 7f6df0fa71e0 at ic…> …)
>   1685:16  6 (raise-exception _ #:continuable? _)
>   1683:16  5 (raise-exception _ #:continuable? _)
>   1685:16  4 (raise-exception _ #:continuable? _)
>   1780:13  3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
>   1685:16  2 (raise-exception _ #:continuable? _)
>   1683:16  1 (raise-exception _ #:continuable? _)
>   1685:16  0 (raise-exception _ #:continuable? _)
>
> ice-9/boot-9.scm:1685:16: In procedure raise-exception:
> ERROR:
>   1. &http-get-error:
>       uri: #<<uri> scheme: https userinfo: #f host: "bordeaux.guix.gnu.org" port: #f path: "/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1" query: #f fragment: #f>
>       code: 404
>       reason: "Not Found"
>       headers: ((server . "nginx") (date . #<date nanosecond: 0 second: 39 minute: 59 hour: 21 day: 21 month: 5 year: 2023 zone-offset: 0>) (content-type text/plain (charset . "utf-8")) (content-length . 3) (connection keep-alive))
>   2. &message: "https://bordeaux.guix.gnu.org/nar/zstd/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1: HTTP download failed: 404 (\"Not Found\")"
> substitution of /gnu/store/dg0rm1mp9qvjhvlwra5mc8dz5pfky4hj-guile-colorized-0.1 failed

[...]

> This might be related to 8bd4126917f59f4af9a4323c3d5699201862dca2.  The
> two ‘download-nar’ calls are guarded against ‘http-get-error?’ though.

That the exception went through despite the ‘network-error?’ guard is
fixed by 3f59fd6d114548480c719d4b8f8509bdf3e8dcca.

The reason was that the ‘&http-get-error’ condition was getting wrapped
in a ‘kind-and-args’ exception upon rethrow, as illustrated here:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (catch #t
		       (lambda () (raise-exception (condition (&message (message "boo")))))
		       (lambda args args))
$5 = (%exception #<&message message: "boo">)
scheme@(guile-user)> (with-exception-handler (lambda (e) e)
		       (lambda () (apply throw $5))
		       #:unwind? #t)
$6 = #<&compound-exception components: (#<&error> #<&irritants irritants: (#<&message message: "boo">)> #<&exception-with-kind-and-args kind: %exception args: (#<&message message: "boo">)>)>
scheme@(guile-user)> (message-condition? $6)
$7 = #f
--8<---------------cut here---------------end--------------->8---

Ludo’.