Removing corrupted items from a mirror

Removing corrupted items from a mirror

[Leo Famulari]

While trying to build the trash-cli patch on top of the master branch
(commit 2a69d4df7d7), I got the following hash mismatch while
downloading a coreutils substitute from my local mirror. When I used
mirror.hydra.gnu.org instead, the correct data was downloaded.

My question is: How can I remove this corrupted file from the mirror? I
don't know how to figure the name of the file to remove.

$ ./pre-inst-env guix build trash-cli
substitute: warning: failed to install locale: Invalid argument
@ substituter-started /gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24 /gnu/store/hb25mzl9a2qjywzx41bfqgi9znc29h32-guix-0.10.0-0.e901/libexec/guix/substitute
warning: failed to install locale: Invalid argument

Found valid signature for /gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24
From https://my-mirror/nar/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24
Downloading vsrcn3...-coreutils-8.24 (13.8MiB installed)...
 coreutils-8.24                                                                                                                                                        3.8MiB/s 00:01 | 3.5MiB transferred
@ substituter-failed /gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24 0 hash mismatch in downloaded path `/gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24': expected a91641b020926f42d7dcbe6e9d82be3b5d39f401b7328ea6d0cd79f7893aabb0, got fc8cf24465d8a97fef7cc41dfccddca158b6258ad0e0929e002552d258dec7da
@ substituter-started /gnu/store/dgwa731j19lj41cds023sspcg9lqgh4j-module-import /gnu/store/hb25mzl9a2qjywzx41bfqgi9znc29h32-guix-0.10.0-0.e901/libexec/guix/substitute
killing process 28390
guix build: error: build failed: some substitutes for the outputs of derivation `/gnu/store/yk4nv2zz40f9k7hr6c9v04rn3pb5l00s-coreutils-8.24.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 

Re: Removing corrupted items from a mirror

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2426 bytes --]

On Sat, Jul 23, 2016 at 08:26:20PM -0400, Leo Famulari wrote:
> While trying to build the trash-cli patch on top of the master branch
> (commit 2a69d4df7d7), I got the following hash mismatch while
> downloading a coreutils substitute from my local mirror. When I used
> mirror.hydra.gnu.org instead, the correct data was downloaded.
> 
> My question is: How can I remove this corrupted file from the mirror? I
> don't know how to figure the name of the file to remove.
> 
> $ ./pre-inst-env guix build trash-cli
> substitute: warning: failed to install locale: Invalid argument
> @ substituter-started /gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24 /gnu/store/hb25mzl9a2qjywzx41bfqgi9znc29h32-guix-0.10.0-0.e901/libexec/guix/substitute
> warning: failed to install locale: Invalid argument
> 
> Found valid signature for /gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24
> From https://my-mirror/nar/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24
> Downloading vsrcn3...-coreutils-8.24 (13.8MiB installed)...
>  coreutils-8.24                                                                                                                                                        3.8MiB/s 00:01 | 3.5MiB transferred
> @ substituter-failed /gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24 0 hash mismatch in downloaded path `/gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24': expected a91641b020926f42d7dcbe6e9d82be3b5d39f401b7328ea6d0cd79f7893aabb0, got fc8cf24465d8a97fef7cc41dfccddca158b6258ad0e0929e002552d258dec7da
> @ substituter-started /gnu/store/dgwa731j19lj41cds023sspcg9lqgh4j-module-import /gnu/store/hb25mzl9a2qjywzx41bfqgi9znc29h32-guix-0.10.0-0.e901/libexec/guix/substitute
> killing process 28390
> guix build: error: build failed: some substitutes for the outputs of derivation `/gnu/store/yk4nv2zz40f9k7hr6c9v04rn3pb5l00s-coreutils-8.24.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
> 

I was trying to build something earlier, got bounced between two
mismatched hashes so neither would perminantly fail. Had to kill it and
build both with --no-subsitutes to continue

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Removing corrupted items from a mirror

[Ludovic Courtès]

Hi!

Leo Famulari <leo@famulari.name> skribis:

> My question is: How can I remove this corrupted file from the mirror? I
> don't know how to figure the name of the file to remove.

So first, note that there’s a narinfo cache in
/var/guix/substitute/cache.  First step is to rm -rf that.

Next, you can remove all the narinfo cache of your nginx instance.  If
that’s unreasonable, you can find the file corresponding to the faulty
narinfo by grepping it in the nginx cache.  So in your case:

> Found valid signature for /gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24
> From https://my-mirror/nar/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24

… you would:

  cd /var/cache/nginx/narinfo
  grep -r vsrcn35rjpgrrak1w31m7yhc12zz3hd1 .
  rm the-file-that-matches

Not very convenient!

I realize I’m answering a slightly different question, because your
problem is:

> @ substituter-failed /gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24 0 hash mismatch in downloaded path `/gnu/store/vsrcn35rjpgrrak1w31m7yhc12zz3hd1-coreutils-8.24': expected a91641b020926f42d7dcbe6e9d82be3b5d39f401b7328ea6d0cd79f7893aabb0, got fc8cf24465d8a97fef7cc41dfccddca158b6258ad0e0929e002552d258dec7da

That is, you got a binary whose content do not match what the narinfo
advertises.

That can happen with non-deterministic build processes, which I think is
the case with Coreutils.  Namely, we can have this scenario:

  1. nginx caches the narinfo for coreutils, which contains the hash of
     a first build;

  2. first build is removed from the machine that nginx mirrors;

  3. a second build is performed on that machine, leading different
     contents (different hash);

  4. Guix client requests coreutils binary, gets narinfo with old hash
     and contents with new hash, and errors out.

I’ve never seen this happen on mirror.hydra.gnu.org, but anyway, our
best protection is reproducible builds.

In the meantime, removing the old narinfo from nginx’s cache would help.

HTH!

Ludo’.