bug#68811: build hash inconsistency

bug#68811: build hash inconsistency

[Zacchaeus Scheffer]

Hi all,

tl;dr I run the following command on two aarch64-linux machines and get
two different hashes for the 'qutebrowser' package:

guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qutebrowser --dry-run

Both machines use only the main guix repository, and guix describe gives
the same output (except generation number and date, which is fine).
Coming from aarch64, building is incredibly expensive.  If the build
hash doesn't match, then (I believe) there is no hope that my machine
will find the packages on a substitute server.  To get around this
issue, I built my guix home once, guix copy'd the store items, and
manually added a symlink in
/var/guix/profiles/per-user/USER/guix-home-N-link to point to the
foreign guix home build.  I couldn't find this issue elsewhere in the
issues, but my "hashes don't match" problem is pretty vague.

Is this an expected problem?  Is this a novel problem?  Am I
misunderstanding guix time-machine (which seems like it should produce
an identical store item)?

-Zacchae

bug#68811: build hash inconsistency

[Zacchaeus Scheffer]

Some more context might be useful:

Device A (which successfully built qutebrowser over a couple days)
~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qutebrowser --dry-run
/gnu/store/i9ir7a26gv1ii98b4bzgvxp1sx0akind-qutebrowser-2.5.4

Device B (trying to avoid building qutebrowser)
~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qutebrowser --dry-run
The following derivations would be built:
  /gnu/store/6n9aq7l5x26xfgrbvws7gvscbzvq5gqn-qutebrowser-2.5.4.drv
  /gnu/store/05wj7wf7bdlkm1ar58kpakvp52drrz7p-qtwebengine-5.15.10.drv
  /gnu/store/7nmv55qqw62jwzrd650vm434s3wpi7i3-python-pyqtwebengine-5.15.9.drv

I have noticed in the past that sometimes a package gets built twice
with two hashes, so I went back and checked if device A had built the
6n9..5gqn qutebrowser store item reported by B in addition to the
i9i..ind one A was reporting.  There was no 6n9..5gqn qutebrowser build
on device A.  I don't believe I have never run guix gc on either device.

-Zacchae

bug#68811: build hash inconsistency

[Saku Laesvuori via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

> Some more context might be useful:
> 
> Device A (which successfully built qutebrowser over a couple days)
> ~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qutebrowser --dry-run
> /gnu/store/i9ir7a26gv1ii98b4bzgvxp1sx0akind-qutebrowser-2.5.4
> 
> Device B (trying to avoid building qutebrowser)
> ~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qutebrowser --dry-run
> The following derivations would be built:
>   /gnu/store/6n9aq7l5x26xfgrbvws7gvscbzvq5gqn-qutebrowser-2.5.4.drv
>   /gnu/store/05wj7wf7bdlkm1ar58kpakvp52drrz7p-qtwebengine-5.15.10.drv
>   /gnu/store/7nmv55qqw62jwzrd650vm434s3wpi7i3-python-pyqtwebengine-5.15.9.drv

Those hashes are not comparable: i9ir..nd (A) is the hash of the built
store item and 6n9aq..qn (B) is the hash of the derivation that builds
the store item.

> I have noticed in the past that sometimes a package gets built twice
> with two hashes, so I went back and checked if device A had built the
> 6n9..5gqn qutebrowser store item reported by B in addition to the
> i9i..ind one A was reporting.  There was no 6n9..5gqn qutebrowser build
> on device A.  I don't believe I have never run guix gc on either device.

But I do think it is weird if the derivation is not present on the
machine that build qutebrowser. Would you mind sending the output of
`guix time-machine ... -- build --derivations ...` from both machines?
If they return a different hash you could maybe also check the returned
`/gnu/store/...-qutebrowser-2.5.4.drv` files to see what is the
difference between them.

- Saku

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#68811: build hash inconsistency

[Zacchaeus Scheffer]

Saku Laesvuori <saku@laesvuori.fi> writes:

> Those hashes are not comparable: i9ir..nd (A) is the hash of the built
> store item and 6n9aq..qn (B) is the hash of the derivation that builds
> the store item.

Ah, rookie mistake :|

> But I do think it is weird if the derivation is not present on the
> machine that build qutebrowser. Would you mind sending the output of
> `guix time-machine ... -- build --derivations ...` from both machines?
> If they return a different hash you could maybe also check the returned
> `/gnu/store/...-qutebrowser-2.5.4.drv` files to see what is the
> difference between them.

The derivations returned by guix build are still different:

Device A:
~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qutebrowser --derivations --dry-run
/gnu/store/vyk5zfr2bjh4cilw4zk3d5xkjiar99h0-qutebrowser-2.5.4.drv

Device B:
~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qutebrowser --derivations --dry-run
The following derivations would be built:
  /gnu/store/6n9aq7l5x26xfgrbvws7gvscbzvq5gqn-qutebrowser-2.5.4.drv
  /gnu/store/05wj7wf7bdlkm1ar58kpakvp52drrz7p-qtwebengine-5.15.10.drv
  /gnu/store/7nmv55qqw62jwzrd650vm434s3wpi7i3-python-pyqtwebengine-5.15.9.drv

I wanted to build the .drv for qutebrowser as you suggested, but trying
to do so caused qtwebengine to actually begin to build (not an option).
I decided to try and build the .drv for qtwebengine on both devices and
got even more interesting differences (see below).  Main things I
noticed: qtwebengine-everywhere-src has different hashes but they are
both found on substitute servers, the qtwebengine .drv file also has
different hashes, oh no that's a lot of thing my device is trying to
build.

I get errors if I try to build for armhf-linux, so I don't think I
installed the wrong version of Guix.  Could it be a difference in
guix-daemon version?  I have Guix on foreign distro, but I modified the
guix-daemon service to point to /var/guix/../root/../guix daemon instead
of /usr/bin/guix-daemon.  Those root profile versions could be quite
different between the two devices (though in theory that shouldn't
matter).


-Zacchae


Device A:
~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qtwebengine --derivations --dry-run
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivation would be built:
  /gnu/store/gnrk76mlrv3ipm2k3lpmy1533mn9dqc3-qtwebengine-6.5.2.drv
470.1 MB would be downloaded:
  /gnu/store/yw7r6bzl0yya0s0h9glz06vy2j88f14d-python-soupsieve-2.2.1
  /gnu/store/i42j0d1797cw6jh3mnrdfgwz2mvsgnv7-clang-14.0.6
  /gnu/store/2ghh4x5j4hsanan9rpka80lxdh1kxnrq-lld-wrapper-0
  /gnu/store/9ylcnhf8rcgwd3jazghz72jcq67kb7hs-lld-14.0.6
  /gnu/store/zsa0xdc4pqjils8j8gn6s0cgcf22j48c-ld.lld-wrapper-0
  /gnu/store/bjd1gyvag1fjwbm4q8fdkx4ci348khwn-lld-as-ld-wrapper-14.0.6
  /gnu/store/b3j2dabjfk3z8vrcdn04dxdr4m8834hs-openh264-2.3.0
  /gnu/store/ymjvjq5cyi8cclsrkqmmih6jsmxidjwp-cmake-minimal-3.24.2
  /gnu/store/66anfjz8bv5a5rf9ii9fd66dzga9dlv6-jsoncpp-1.9.5
  /gnu/store/b2bkchln7na32bk5mfd1gl896ac2ng9d-clang-runtime-14.0.6
  /gnu/store/na1xngg46bwpxh9jl74hnsw5gp5q8mbp-python-beautifulsoup4-4.11.1
  /gnu/store/i5hhylp02w3qqf2xjyws9kbii93jjqcs-libxkbfile-1.1.0
  /gnu/store/cy356bl5cj9sd915pwzvx1l2njvvh80l-qtmultimedia-6.5.2
  /gnu/store/w3ja0fcg7x2g8pr0l5ggzpbjbs1sxzw2-python-html5lib-1.1
  /gnu/store/8c5al6nn6ihqm6vv71fszv0k3pm0zy4b-llvm-14.0.6
  /gnu/store/i74mgypyv2j7i6p28x5z5r1l7mn3q11d-python-chardet-5.1.0
  /gnu/store/x0j572gnp5az71lcxs58274z11g27rsg-qtwebchannel-6.5.2
  /gnu/store/yqppj5kr0fzph6wvbhxlq6v6ynpa5xsq-protobuf-3.21.9
  /gnu/store/0cwx6qvc0jl5amb6lcanninis5xwsm49-md4c-0.4.8
  /gnu/store/a51gd25m22p623c3mwpbsc0bah30r534-qtbase-6.5.2
  /gnu/store/xylrvhrdv72x62vi94f73m7wdq8361r6-qtlanguageserver-6.5.2
  /gnu/store/6m5andsmqhxk7jrk1f0fmsa5012jmiw9-qtdeclarative-6.5.2
  /gnu/store/qi88j5lvxg2cmc391frvvlz91m4jv3ij-qtwebengine-everywhere-src-6.5.2.tar.xz
  /gnu/store/733pxaaxhdqs95a7qp7ydh831a77lyzp-ninja-1.11.1
  /gnu/store/qjby2hz1d2154jx06755x8wxyfi5ycmd-python-webencodings-0.5.1
  /gnu/store/1ik19yadbhc959nbk42gb548139vypp3-node-18.19.0

Device B:
~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build --derivations qtwebengine --dry-run
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivations would be built:
  /gnu/store/qgk0k5312p9bfikhgi87644rdb9h71g5-qtwebengine-6.5.2.drv
  /gnu/store/3kr5g43593v5x42nhz2396hb8a4sp7k8-qtwebchannel-6.5.2.drv
  /gnu/store/1waf6s55grvdx686nn4fg5mfsxz0567g-qtwebsockets-6.5.2.drv
  /gnu/store/lffwivm3lgk8p5sgifa49wx3j29srlc7-qtdeclarative-6.5.2.drv
  /gnu/store/082dd9z9d8n5s09x4m6k4d1ckxx4yzj2-qtshadertools-6.5.2.drv
  /gnu/store/zi5iqbjqi7m4a6rd7d1ynvp6k4cklbh6-qtbase-6.5.2.drv
  /gnu/store/k8s6y1rmg7qz5k9vrbh34p0qp19zraqn-md4c-0.4.8.drv
  /gnu/store/nj9pps7kcr4y4xdv46l84n8vy07y63yy-qtbase-everywhere-src-6.5.2.tar.xz.drv
  /gnu/store/1x4swi9dmq7wvh58pdqn4qjyqvzf6z11-qtdeclarative-everywhere-src-6.5.2.tar.xz.drv
  /gnu/store/56q3r3nn2cm91r14pn3lwwwrmhx6i89p-qtimageformats-6.5.2.drv
  /gnu/store/bw3zc4phc52wiah8mwwmb5d4533c3rdi-qtimageformats-everywhere-src-6.5.2.tar.xz.drv
  /gnu/store/bdb9kkf7w2c41imdchx7b3f3sqv8i2dy-qtsvg-6.5.2.drv
  /gnu/store/y85bhd7s2jj5klrwvj5hnjj9gjiwry8q-qtlanguageserver-6.5.2.drv
  /gnu/store/3lfwgdq5qvwd6cywj0bq2japs7clv5qx-qtwebengine-everywhere-src-6.5.2.tar.xz.drv
  /gnu/store/a5vpw1yl0xc1r49r22wfzdj40s85pzab-lld-as-ld-wrapper-14.0.6.drv
  /gnu/store/l4j3wl0dvdw4f1xffk4vcax48anig5yp-lld-wrapper-0.drv
  /gnu/store/rigww18q26c102h93kj1yqklh8nr6fmw-lld-14.0.6.drv
  /gnu/store/akgv8z3df74x4438l7amhy461nsq0kws-llvm-14.0.6.drv
  /gnu/store/iqvgng8pcmkxn9c8z8552d5wkhg25hzg-llvm-project-14.0.6-checkout.drv
  /gnu/store/z1isl8dl5xhc9c6hqi1zhm4pj07lgiai-ld.lld-wrapper-0.drv
  /gnu/store/g1g308r6rs5frcx6qs22m8aivhjssj2c-clang-runtime-14.0.6.drv
  /gnu/store/ixhdpsvchd147ka9iizqi1gba8bgc8ws-qtmultimedia-6.5.2.drv
  /gnu/store/6qpjcxxzj1zz8hpwbd7h37ax9s7lghqy-qtmultimedia-everywhere-src-6.5.2.tar.xz.drv
  /gnu/store/pypzr5m6wqj867fjyp3166p9as71bn61-gst-plugins-good-1.22.2.drv
  /gnu/store/f9zys916nr1kz2brx4mp3p1wxaz8lp2i-libsoup-3.1.4.drv
  /gnu/store/syv3mj26j0ggvw46c8mhxv7qqrhsqn59-gst-libav-1.22.2.drv
  /gnu/store/k7ga8s1ds7dmwvpb031p8s5kahvm4nz8-clang-14.0.6.drv
  /gnu/store/rnj3bamx0wzdyfr9smi33cx1yw9gl62f-node-18.19.0.drv
  /gnu/store/h20fm9fy8xiybpw5xisrqq1w4k6jpv7z-llhttp-6.0.11.drv
  /gnu/store/91ws1b9ckzrj0hvbgx36ripgdf0hh1j7-node-llparse-7.1.0.drv
  /gnu/store/9za67qcddkchhl0m6xrpfh25hi4nr9pn-module-import-compiled.drv
  /gnu/store/s5h43lrgqfdh46s819arhxy64xdalxpv-node-llparse-7.1.0-checkout.drv
  /gnu/store/svw3w50if1mq5cv93zcwyw9hp5z5c8j2-node-debug-4.3.0.drv
  /gnu/store/8zlx7n5cnyy50yvwmj9pl0xkzgx7ip3k-node-ms-2.1.2.drv
  /gnu/store/z4l1iy97z9kpbl991vhk0yn7akya3pm4-node-llparse-frontend-3.0.0.drv
  /gnu/store/fp155qzisf9xvppnvxj51wazkilm6mpn-node-llparse-builder-1.5.2.drv
  /gnu/store/06rkh3k2qh776ipqxm8bzvi15y88y3zs-node-binary-search-1.3.6.drv
  /gnu/store/850nb2yf2z4lkdrfa3x21s5z040zxghm-node-llparse-builder-1.5.2-checkout.drv
  /gnu/store/iwzzpkigabb4lxgxkil439aabxbfrjpp-node-llparse-frontend-3.0.0-checkout.drv
  /gnu/store/i6a3c6k2g9dvhpzkjha352f866y9kpkv-llhttp-6.0.11-checkout.drv
  /gnu/store/qp1zjmb7wrnc5jdydlc47ijl25ypi0w1-node-semver-7.2.1.drv
  /gnu/store/j6hf21a1ckssx5g98bl10qgpwslj3ha7-node-v18.19.0.tar.xz.drv
422.4 MB would be downloaded:
  /gnu/store/qixglkg4928gixdiwrfb1raykab66j4r-ruby-3.1.4
  /gnu/store/1dv7a5a3l5ixpqmlb4jpys0jj8w2qchr-qtwebengine-everywhere-src-6.5.2.tar.xz
  /gnu/store/il59vx2siqx7bzqm0y3pahqk79w7p1s3-xorg-server-21.1.10
  /gnu/store/gagpiy55s58nrphxcyq1pkzffjywf8wy-xvfb-run-21.1.7-1
  /gnu/store/gi04nc9q1z9bm5kxy11ghc5yh5skg0m1-qtwebsockets-everywhere-src-6.5.2.tar.xz

bug#68811: build hash inconsistency

[Josselin Poiret via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 150 bytes --]

Hi Zacchaeus,

Can you try the same, but this time with the --no-grafts option?  That
could be a source of issues.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

bug#68811: build hash inconsistency

[Zacchaeus Scheffer]

Hi Josselin,

Alas, the problem persists ~.~

Device A:
~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qtwebengine --derivations --system=aarch64-linux --no-grafts --dry-run
/gnu/store/gnrk76mlrv3ipm2k3lpmy1533mn9dqc3-qtwebengine-6.5.2.drv

Device B:
~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qtwebengine --derivations --system=aarch64-linux --no-grafts --dry-run
/gnu/store/qpb6d1qqx357rkydk0xv6ail6b9dcqs6-qtwebengine-6.5.2.drv

The fact that Device A finds substitutes for nearly everything while
Device B doesn't makes me think that B is "wrong".  Device A is a 3GiB
RAM Librem 5 (made in China), and Device B is a 4GiB RAM Liberty Phone
(made in USA).  (Maybe US backdoor bots need to up their stealth game?)
You can find info on how I set up guix at
https://zacchae.us/guix-usd.org, specifically the section "Setting up
guix home"


-Zacchae


PS - I see your response on issues.guix.gnu.org, but didn't get an
email.  I thought submitting bugs automatically subscribed me?

bug#68811: build hash inconsistency

[Zacchaeus Scheffer]

Hi all,

I just tried the previous command on Device C, an x86_64-linux Guix System:

~ $ guix time-machine --commit=deeb7d1f53d7ddfa977b3eadd760312bbd0a2509 -- build qtwebengine --derivations --system=aarch64-linux --no-grafts --dry-run
<lots of non-error prints>
/gnu/store/gnrk76mlrv3ipm2k3lpmy1533mn9dqc3-qtwebengine-6.5.2.drv

As suspected, the hash matches that of device A.

I'm tempted to reflash Device B, wipe and reencrypt the SD (which has
/gnu/store), and reinstall Guix, but I also want to get to the bottom of
this.  So long as this thread doesn't get stale, I'm happy to try
whatever suggestions people have.

-Zacchae