bug#21803: Guitarix builds non-deterministically

bug#21803: Guitarix builds non-deterministically

[Ludovic Courtès]

Guitarix 0.33.0, as of Guix commit 3c3e697, builds
non-deterministically:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix challenge guitarix
updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
/gnu/store/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0 contents differ:
  local hash: 1rh7qxmylsbsaah59h7sclqqxcz0lwsixlc0krkzwhx8gfhlyam6
  http://hydra.gnu.org/nar/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0: 1cq4k1wdqibwraqk0wkjj6n5hgs9v9zcvwr2wfgxvgxnf5l1rfhf
$ wget -q -O - http://hydra.gnu.org/nar/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0 | bunzip2 | guix archive -x t
$ LC_ALL=C diff -r /gnu/store/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0 t
$ LC_ALL=C diff -r --no-dereference /gnu/store/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0 t
File /gnu/store/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0/lib/libgxw.so is a symbolic link while file t/lib/libgxw.so is a regular file
--8<---------------cut here---------------end--------------->8---

It appears to create libgxw.so either as a symlink or as a regular file
in a non-deterministic fashion.

It would be nice to see why this happens, and whether this affects all
Waf-based packages.

Ludo’.

bug#21803: Bug #21803 Hunting: status?

[zimoun]

Hi Ludo,

You reported this bug more than 4 years ago about Guitarix 0.33 and
now it is gone and replaced by Guitarix 0.38.

From my test with Guix d258d9c7d222e6b64531c14293f41bd8d62ea4f7,
"guix challenge guitarix" and "guix build --rounds=3" do not report
issues about reproducibility.

And from my knowledge, the waf-based packages are not affected.

Do you agree to close this bug since it is not relevant anymore?

Thanks in advance for any comments.


All the best,
simon

bug#21803: Bug #21803 Hunting: status?

[Ludovic Courtès]

Hello,

zimoun <zimon.toutoune@gmail.com> skribis:

> From my test with Guix d258d9c7d222e6b64531c14293f41bd8d62ea4f7,
> "guix challenge guitarix" and "guix build --rounds=3" do not report
> issues about reproducibility.
>
> And from my knowledge, the waf-based packages are not affected.
>
> Do you agree to close this bug since it is not relevant anymore?

If you’ve checked that a local build gives the same result several times
in a row (make sure it actually rebuilt things; “guix build --rounds=3
foo” does nothing if “foo” is already in the store), then you can
definitely close it!

Thanks,
Ludo’.

bug#21803: Bug #21803 Hunting: status?

[zimoun]

On Wed, 13 Nov 2019 at 14:42, Ludovic Courtès <ludo@gnu.org> wrote:

> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > From my test with Guix d258d9c7d222e6b64531c14293f41bd8d62ea4f7,
> > "guix challenge guitarix" and "guix build --rounds=3" do not report
> > issues about reproducibility.
> >
> > And from my knowledge, the waf-based packages are not affected.
> >
> > Do you agree to close this bug since it is not relevant anymore?
>
> If you’ve checked that a local build gives the same result several times
> in a row (make sure it actually rebuilt things; “guix build --rounds=3
> foo” does nothing if “foo” is already in the store), then you can
> definitely close it!

I think I did but how do you do that cleanly?



I am testing with "brute force" method: "guix gc -C" then proceed. I
notice unexpected behaviour; not sure it is the right place to report.


1. The first issue is that the two following commands do not populate
the same way.

(Because my machine is not very powerful, before building I populate
the store with the dependencies from substitutes.)

--8<---------------cut here---------------start------------->8---
$ guix environment guitarix -- echo Done
--8<---------------cut here---------------start------------->8---

--8<---------------cut here---------------start------------->8---
$ guix build `guix show guitarix | recsel -R dependencies`
--8<---------------cut here---------------start------------->8---

The main issue is about `gettext-minimal`.

(Below, I pinpoint with star (*) which is not common between the both
commands and with sharp (#) which appears twice in the same list.)


2. The second issue is the inconsistent outputs:

--8<---------------cut here---------------start------------->8---
$ guix build guitarix --no-substitutes --dry-run
The following derivations would be built:
   /gnu/store/ikdd9740fifdcqwmf170gmlrlkirwa8j-guitarix-0.38.1.drv
   /gnu/store/ismr6xqwsi165phwjx9kbcrmr9lsz61r-module-import.drv
   /gnu/store/zsigy6yfllikzxmnlii6ivxczlv8k5h3-guitarix2-0.38.1.tar.xz.drv
   /gnu/store/zya9lhiqqr24lbgkp510v1z9s4qdcqid-module-import-compiled.drv
--8<---------------cut here---------------start------------->8---

--8<---------------cut here---------------start------------->8---
$ guix build guitarix --no-substitutes
building /gnu/store/4fqc5dydkc4svkl1zjyz5ymnyycfakx8-module-import.drv...
successfully built /gnu/store/4fqc5dydkc4svkl1zjyz5ymnyycfakx8-module-import.drv
building /gnu/store/s659hxn6zh7havik6bghip7mslarxfcx-ghostscript-9.27.tar.xz.drv...

Starting download of
/gnu/store/gz3sh75g8rwvqqhmj7z1wbdrn36bwk1g-ghostscript-9.27.tar.xz
From https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs927/ghostscript-9.27.tar.xz...
following redirection to
`https://github-production-release-asset-2e65be.s3.amazonaws.com/50461376/70f10a80-56c2-11e9-8208-d05d335afc94?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191113%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191113T173022Z&X-Amz-Expires=300&X-Amz-Signature=6498073580142d43071eba62060a7379b0dae94f148fc191b1a8ec53f284107a&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dghostscript-9.27.tar.xz&response-content-type=application%2Foctet-stream'...
downloading from
https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs927/ghostscript-9.27.tar.xz...
 ghostscript-9.27.tar.xz  31.6MiB

4.9MiB/s 00:02 [######            ]  34.0%
^C
--8<---------------cut here---------------start------------->8---


Why `ghostscript` is downloaded and not reported; neither by the two
populating commands?
Do I miss the obvious?

So I add it to the store with "guix build ghostscript" then I run
again "guix build guitarix --no-substitutes" and again pieces are
missing:

--8<---------------cut here---------------start------------->8---
[...]
 downloading from
http://downloads.sourceforge.net/project/cunit/CUnit/2.1-3/CUnit-2.1-3.tar.bz2
[...]
 downloading from https://c-ares.haxx.se/download/c-ares-1.15.0.tar.gz
[...]
 downloading from http://www.digip.org/jansson/releases/jansson-2.12.tar.bz2
--8<---------------cut here---------------start------------->8---


Then other pieces are missing. I suppose it is related to BAG and
depth of the dependencies.


Therefore, how is it possible to check the reproducibility of a
package without compiling the World? For example, the test suite of
Git kills my desktop machine. :-)



Thank you in advance.
simon



--

--8<---------------cut here---------------start------------->8---
$ guix environment guitarix --dry-run -- echo Done
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations would be built:
   /gnu/store/9p90w999xdqyhv6k1x5wp0sq312l10ri-profile.drv
   /gnu/store/qxhk96270rgn61nv7znbmgj5d4pamzgw-config.scm.drv
116.4 MB would be downloaded:
*   /gnu/store/zjnz5gg04zfyzn5gi66lay8sgv6i4rj8-module-import-compiled
*   /gnu/store/05kyg8pg8zzbrn366imllhxavdcwqgsk-mkfontdir-1.0.7
*   /gnu/store/0dsc5kh6qfwshfaq01iqrfpdhlaici8n-libfontenc-1.1.4
*   /gnu/store/r1p07fn372rcxzvfzqwm44w16n8gcqfc-mkfontscale-1.2.1
*   /gnu/store/sclspjcznk50s894irfk2wzn6nfnxa2g-guile-gdbm-ffi-20120209.fa1d5b6
*   /gnu/store/s2hqjjp057l2k6ix3yaa7cc2dpwxpfm7-module-import
*   /gnu/store/pfqvay49nk9cja05sqw5cwm7fn3w4fn3-module-import-compiled
*   /gnu/store/zqyr73kc60f2dyj2ykl3ijk86kias0yc-module-import-compiled
*   /gnu/store/nsrym6zn3yb4390fazx1gq8rg0m8dhkd-llvm-8.0.0
*   /gnu/store/vf5lkc27z5rcbvaw6rkdmqisyg923m5r-mesa-19.1.4
*   /gnu/store/kjlnb30snlawpb4fb8qwhbzhcpbxlckk-libepoxy-1.5.3
*   /gnu/store/4ppvmpyir7qwmhzkfxnqlb27j85rjqyz-gtk+-3.24.10-bin
*   /gnu/store/wwwnp8025yb7k6qbv28hi1l6qy6j7d9z-module-import-compiled
*   /gnu/store/iaqmwj2290z5nnrk69bss6r0d9lpr8cs-python2-2.7.16
   /gnu/store/fzjnhsfgkqcsqjzan2dxkqgw6fl5kniy-lv2-1.16.0
   /gnu/store/p8yabbkcywr8a5pcy3dg6w0mjaw6ddlq-zita-resampler-1.3.0
*   /gnu/store/7zlxdamykwrd1vjp3kxv54qyyv7ya6jr-util-macros-1.19.2
   /gnu/store/ksjvb9m0ky0g7yv27v70l44h8nj6v1qy-gtkmm-2.24.5
   /gnu/store/pxwz526473fynfhwnljs0wb5vd2qcg4c-atkmm-2.28.0
   /gnu/store/l8nphg0idd8pfddyad8f92lx8d1hc053-python-wrapper-3.7.4
   /gnu/store/ms1q76ikx7f78y7i8crg42lyv88xfvfv-ghostscript-with-cups-9.27
   /gnu/store/xc98lvvcbaabn0v1md34hxbdn5ivg72g-cups-filters-1.25.1
   /gnu/store/lv9xckn9in3cbswaprv5j9xgsa8wghpr-cups-2.2.11
   /gnu/store/vb4g1m42k156gwiwq62hd1db92ndvnch-gtk+-2.24.32
*   /gnu/store/b824dq3bccq0bhjli3li0fzi11lg1bh3-xorgproto-2019.1
*   /gnu/store/arzn28zwj8bqv2qiid7ybx3aad49c3pd-libpthread-stubs-0.4
   /gnu/store/j1ldfckx81mm6fydlhzw03cv7hsr58ya-faust-0.9.90
   /gnu/store/h10dlkcf019047d8jmjkw2w0h96zad2d-gperf-3.1
   /gnu/store/zc4shgdr8pw9z5rcv9657p37wyha4nqa-fftwf-3.3.8
   /gnu/store/dgkwjigfbadm0jn2y9z6d29lq0i3wj97-zita-convolver-3.1.0
   /gnu/store/rm9q1vqhg333nwpywsbf017fj88dlgsf-eigen-3.3.5
   /gnu/store/3vgaq9ga221mjx21sll3lxqxdgclgws4-flac-1.3.3
   /gnu/store/y8wkxd3pmp29k1j0knmg9hnjnrfnm9iy-libsndfile-1.0.28
   /gnu/store/f5g4av3mwn7zr81yqr1gn9hpb5d2c4m4-boost-1.70.0
   /gnu/store/wrlkvgl0lz7b03gwqf27ql6pjkmj5v6r-bdb-6.2.32
   /gnu/store/q1mz5mid5y4y3z12g5ify10ci7h72dnq-jack-0.125.0
*   /gnu/store/5686hrxkbsm6ycf1ks9nja3mjjxjpl9a-desktop-file-utils-0.23
   /gnu/store/a9rsi05xscg0bq6q0rbhcv5586zvf3li-cairomm-1.12.2
   /gnu/store/s0djjrx5x8c12sa1p0wl10crvl7rzs0v-libsigc++-2.10.2
   /gnu/store/78w7y0lxar70j512iqw8x3nimzj10yga-python-3.7.4
*   /gnu/store/fpj5mspa7jmksixqpnzbvzs3q2vbqq31-glib-2.60.6-bin
   /gnu/store/b8pr2k0i2zd07zmb7kpffmcimqi337if-glib-2.60.6
   /gnu/store/94s92fpkk14y514qwf4f4hnc54qz0zfj-glibmm-2.60.0
   /gnu/store/dhnzrhs9vf40p5v09817rbcd3ks07slc-pangomm-2.42.0
*   /gnu/store/20ilkjz5kd98zdm6rsk6zdw5p9nh0hq7-perl-xml-parser-2.44
   /gnu/store/qw4p5qwd1f1kcwspm455njd4ny7v9gww-sratom-0.6.2
   /gnu/store/r0jrq3jvvxrbx7sbqhs168yxaan9311v-sord-0.16.2
   /gnu/store/77h391w9aynwwb5j83yakav5m6qi7bbg-lilv-0.24.4
   /gnu/store/vfcbn39fwng8d35gvic51f235fxvj7y1-intltool-0.51.0
*   /gnu/store/k3m2kz55qiklkyihnnvhxhv0ylsyyaqc-module-import-compiled
   /gnu/store/6zasp7vh5jww0naybhva026z1967scr7-ladspa-1.13
   /gnu/store/v557q2wd91sm5vj3lrwjzajafblklr6w-libxslt-1.1.33
   /gnu/store/k2m4q2av9hw73hw2jx6qrxqdyh855398-openssl-1.1.1c
   /gnu/store/f8aljw2qhv3d1br9czn8v5afbgfdrxkg-cyrus-sasl-2.1.27
   /gnu/store/bjxd9jzc560d6i3i35f5yy5mljk0ib6m-openldap-2.4.47
   /gnu/store/4m8dlhrzis07787xznx73ang35c3lly1-curl-7.65.3
   /gnu/store/dgv36cg3x3pi2v00arwlfcvq3p7id4h5-raptor2-2.0.15
   /gnu/store/x5k749kbwbmbd1sn0j6ybpcc9450fba2-lrdf-0.6.1
   /gnu/store/pvz6zmy4lwaicjk07999gbsaim0p4iai-serd-0.30.0
The following profile hooks would be built:
   /gnu/store/5jc3lsfm9mj8smp3a4g4aqkq05zrjsq5-xdg-desktop-database.drv
   /gnu/store/6csqwjlmnsmpcgcdrxjm841d54xf7pcl-xdg-mime-database.drv
   /gnu/store/6k5057zfsxqf3kn5iqf27zz91jl7zbmh-info-dir.drv
   /gnu/store/chzwnq3kl6gc36sh2krlbfvxfga13cq5-gtk-icon-themes.drv
   /gnu/store/i22j0197hy2byv23f884dd0j6y7r93hp-manual-database.drv
   /gnu/store/immlg222fg9wpnjb04j6czwp8gviw7as-gtk-im-modules.drv
   /gnu/store/j7m81fkl3qacc1cnzifqiy0a4hzb8i6s-fonts-dir.drv
   /gnu/store/j96f9a5m2d7wi8pdvj6d5jvldbzjc3k8-glib-schemas.drv
   /gnu/store/mf3abgwykhp243r8cbfjwl1w7x6d7dkm-ca-certificate-bundle.drv
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
$ guix build `guix show guitarix | recsel -R dependencies` --dry-run
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
76.1 MB would be downloaded:
   /gnu/store/f5g4av3mwn7zr81yqr1gn9hpb5d2c4m4-boost-1.70.0
*   /gnu/store/33f8qhxa69dmd43yqdx3wq1b2hqjddgb-curl-7.65.3-doc
#   /gnu/store/4m8dlhrzis07787xznx73ang35c3lly1-curl-7.65.3
   /gnu/store/rm9q1vqhg333nwpywsbf017fj88dlgsf-eigen-3.3.5
   /gnu/store/j1ldfckx81mm6fydlhzw03cv7hsr58ya-faust-0.9.90
*   /gnu/store/3pfj84hcpw0xfxh8briill4c5mnk51ha-gettext-minimal-0.20.1-doc
*   /gnu/store/ypwxvcnrsdn0snllv944ckylwx3p1m79-gettext-minimal-0.20.1
   /gnu/store/h10dlkcf019047d8jmjkw2w0h96zad2d-gperf-3.1
*   /gnu/store/vmwm54y790r3ipbyd1l8qzxhzw0byv7d-gtk+-2.24.32-doc
#   /gnu/store/vb4g1m42k156gwiwq62hd1db92ndvnch-gtk+-2.24.32
   /gnu/store/pxwz526473fynfhwnljs0wb5vd2qcg4c-atkmm-2.28.0
   /gnu/store/ms1q76ikx7f78y7i8crg42lyv88xfvfv-ghostscript-with-cups-9.27
   /gnu/store/xc98lvvcbaabn0v1md34hxbdn5ivg72g-cups-filters-1.25.1
   /gnu/store/lv9xckn9in3cbswaprv5j9xgsa8wghpr-cups-2.2.11
#   /gnu/store/vb4g1m42k156gwiwq62hd1db92ndvnch-gtk+-2.24.32
   /gnu/store/a9rsi05xscg0bq6q0rbhcv5586zvf3li-cairomm-1.12.2
   /gnu/store/s0djjrx5x8c12sa1p0wl10crvl7rzs0v-libsigc++-2.10.2
   /gnu/store/fpj5mspa7jmksixqpnzbvzs3q2vbqq31-glib-2.60.6-bin
   /gnu/store/b8pr2k0i2zd07zmb7kpffmcimqi337if-glib-2.60.6
   /gnu/store/94s92fpkk14y514qwf4f4hnc54qz0zfj-glibmm-2.60.0
   /gnu/store/dhnzrhs9vf40p5v09817rbcd3ks07slc-pangomm-2.42.0
   /gnu/store/ksjvb9m0ky0g7yv27v70l44h8nj6v1qy-gtkmm-2.24.5
   /gnu/store/vfcbn39fwng8d35gvic51f235fxvj7y1-intltool-0.51.0
   /gnu/store/wrlkvgl0lz7b03gwqf27ql6pjkmj5v6r-bdb-6.2.32
   /gnu/store/q1mz5mid5y4y3z12g5ify10ci7h72dnq-jack-0.125.0
   /gnu/store/6zasp7vh5jww0naybhva026z1967scr7-ladspa-1.13
   /gnu/store/3vgaq9ga221mjx21sll3lxqxdgclgws4-flac-1.3.3
   /gnu/store/y8wkxd3pmp29k1j0knmg9hnjnrfnm9iy-libsndfile-1.0.28
   /gnu/store/qw4p5qwd1f1kcwspm455njd4ny7v9gww-sratom-0.6.2
   /gnu/store/r0jrq3jvvxrbx7sbqhs168yxaan9311v-sord-0.16.2
   /gnu/store/pvz6zmy4lwaicjk07999gbsaim0p4iai-serd-0.30.0
   /gnu/store/77h391w9aynwwb5j83yakav5m6qi7bbg-lilv-0.24.4
   /gnu/store/v557q2wd91sm5vj3lrwjzajafblklr6w-libxslt-1.1.33
   /gnu/store/f8aljw2qhv3d1br9czn8v5afbgfdrxkg-cyrus-sasl-2.1.27
   /gnu/store/bjxd9jzc560d6i3i35f5yy5mljk0ib6m-openldap-2.4.47
#   /gnu/store/4m8dlhrzis07787xznx73ang35c3lly1-curl-7.65.3
   /gnu/store/dgv36cg3x3pi2v00arwlfcvq3p7id4h5-raptor2-2.0.15
   /gnu/store/x5k749kbwbmbd1sn0j6ybpcc9450fba2-lrdf-0.6.1
   /gnu/store/k2m4q2av9hw73hw2jx6qrxqdyh855398-openssl-1.1.1c
   /gnu/store/78w7y0lxar70j512iqw8x3nimzj10yga-python-3.7.4
   /gnu/store/l8nphg0idd8pfddyad8f92lx8d1hc053-python-wrapper-3.7.4
   /gnu/store/fzjnhsfgkqcsqjzan2dxkqgw6fl5kniy-lv2-1.16.0
   /gnu/store/zc4shgdr8pw9z5rcv9657p37wyha4nqa-fftwf-3.3.8
   /gnu/store/dgkwjigfbadm0jn2y9z6d29lq0i3wj97-zita-convolver-3.1.0
   /gnu/store/p8yabbkcywr8a5pcy3dg6w0mjaw6ddlq-zita-resampler-1.3.0
--8<---------------cut here---------------end--------------->8---

bug#21803: Bug #21803 Hunting: status?

[Ludovic Courtès]

Hello,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Wed, 13 Nov 2019 at 14:42, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> zimoun <zimon.toutoune@gmail.com> skribis:
>>
>> > From my test with Guix d258d9c7d222e6b64531c14293f41bd8d62ea4f7,
>> > "guix challenge guitarix" and "guix build --rounds=3" do not report
>> > issues about reproducibility.
>> >
>> > And from my knowledge, the waf-based packages are not affected.
>> >
>> > Do you agree to close this bug since it is not relevant anymore?
>>
>> If you’ve checked that a local build gives the same result several times
>> in a row (make sure it actually rebuilt things; “guix build --rounds=3
>> foo” does nothing if “foo” is already in the store), then you can
>> definitely close it!
>
> I think I did but how do you do that cleanly?
>
>
>
> I am testing with "brute force" method: "guix gc -C" then proceed. I
> notice unexpected behaviour; not sure it is the right place to report.

I would probably do “guix build guitarix” (get the substitute) and then
“guix build guitarix --check --no-grafts -K”, possibly several times*,
which builds guitarix alone from source.

(*) ‘--rounds’ is ignored when combined with ‘--check’, go figure…

> 1. The first issue is that the two following commands do not populate
> the same way.
>
> (Because my machine is not very powerful, before building I populate
> the store with the dependencies from substitutes.)
>
> $ guix environment guitarix -- echo Done

That should work.

> $ guix build `guix show guitarix | recsel -R dependencies`

That gives a different result because ‘dependencies’ does not show
implicit inputs.

> Why `ghostscript` is downloaded and not reported; neither by the two
> populating commands?

This is because of grafts, which lead to a poor UX as things are…

Perhaps there are genuine issues described in the rest of your message
but we’d need to isolate them first.  :-)

Thanks,
Ludo’.

bug#21803: Bug #21803 Hunting: status?

[zimoun]

Hi Ludo,

Thank you for the explanations.


On Thu, 14 Nov 2019 at 22:09, Ludovic Courtès <ludo@gnu.org> wrote:

> I would probably do “guix build guitarix” (get the substitute) and then
> “guix build guitarix --check --no-grafts -K”, possibly several times*,
> which builds guitarix alone from source.

I confirm what you reported:

--8<---------------cut here---------------start------------->8---
$ LC_ALL=C diff -r --no-dereference
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1-check
File /gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so
is a symbolic link while file
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1-check/lib/libgxw.so
is a regular file
--8<---------------cut here---------------end--------------->8---

As you said, it needs more investigation... :-)


> Perhaps there are genuine issues described in the rest of your message
> but we’d need to isolate them first.  :-)

Yes, first things first. :-)
And I will open discussion elsewhere because it is not related to this bug.


Thanks,
simon

bug#21803: other waf non reproducible: mpv and ardour

[zimoun]

Hi,

I do not know if it is related but these 2 packages using the waf
build system are not reproducible too.

 - mpv
 - ardour

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference
/gnu/store/gb935w89qyyi4ljyaz40brai2z9wg0fr-mpv-0.30.0
/gnu/store/gb935w89qyyi4ljyaz40brai2z9wg0fr-mpv-0.30.0-check
Binary files /gnu/store/gb935w89qyyi4ljyaz40brai2z9wg0fr-mpv-0.30.0/bin/mpv
and /gnu/store/gb935w89qyyi4ljyaz40brai2z9wg0fr-mpv-0.30.0-check/bin/mpv
differ
Binary files /gnu/store/gb935w89qyyi4ljyaz40brai2z9wg0fr-mpv-0.30.0/lib/libmpv.so.1.106.0
and /gnu/store/gb935w89qyyi4ljyaz40brai2z9wg0fr-mpv-0.30.0-check/lib/libmpv.so.1.106.0
differ
--8<---------------cut here---------------end--------------->8---


--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference
/gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12
/gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check
Binary files /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12/lib/ardour5/ardour-5.12.0
and /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check/lib/ardour5/ardour-5.12.0
differ
Binary files /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12/lib/ardour5/engines/libclearlooks.so
and /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check/lib/ardour5/engines/libclearlooks.so
differ
Binary files /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12/lib/ardour5/libardour.so.3.0.0
and /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check/lib/ardour5/libardour.so.3.0.0
differ
Binary files /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12/lib/ardour5/libevoral.so.0.0.0
and /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check/lib/ardour5/libevoral.so.0.0.0
differ
Binary files /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12/lib/ardour5/LV2/a-comp.lv2/a-comp.so
and /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check/lib/ardour5/LV2/a-comp.lv2/a-comp.so
differ
Binary files /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12/lib/ardour5/LV2/a-eq.lv2/a-eq.so
and /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check/lib/ardour5/LV2/a-eq.lv2/a-eq.so
differ
Binary files /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12/lib/ardour5/LV2/a-fluidsynth.lv2/a-fluidsynth.so
and /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check/lib/ardour5/LV2/a-fluidsynth.lv2/a-fluidsynth.so
differ
Binary files /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12/lib/ardour5/LV2/reasonablesynth.lv2/reasonablesynth.so
and /gnu/store/7l5isavp1db8q5mv3zq7bmk29l6ld9qp-ardour-5.12-check/lib/ardour5/LV2/reasonablesynth.lv2/reasonablesynth.so
differ
--8<---------------cut here---------------end--------------->8---


All the best,
simon

bug#21803: guitarix non-reproducible hard to reproduce

[zimoun]

Dear,

Resume:
The command "guix build guitarix" downloads from ci.guix.gnu.org.
Then "guix build guitarix --no-grafts -K --check" rebuilds locally.
The two differ of one symlink.

--8<---------------cut here---------------start------------->8---
diff -r --no-dereference
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1-check
File /gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so
is a symbolic link while file
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1-check/lib/libgxw.so
is a regular file
--8<---------------cut here---------------end--------------->8---


Inspect:

After more than 10 attempts to reproduce the behaviour where the plain
binary is replaced by the symlink, it never happens locally, so I
conclude that the bug is really hard to track. Kind of bad luck. :-)


My intuition is: the non-determinism comes from the WAF configuration
files (./waf or wscript or wafadmin/).

The process conditionally enters in functions such as "do_install" or
"symlink_as", so maybe something is hidden by Python module "os" or
similar. Moreover, note that wscript:l.990 "add_group()" is used after
"add_subdirs('libgxw/gxw')".


Basically, the output of "--check" is:

--8<---------------cut here---------------start------------->8---
[...]
* installing build/default/libgxw/gxw/libgxw.so as
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so.0.1
* symlink /gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so.0
(-> libgxw.so.0.1)
* symlink /gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so
(-> libgxw.so.0.1)
* installing build/default/libgxw/gxw/libgxw.so as
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so
* installing build/default/libgxwmm/gxwmm/libgxwmm.so as
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxwmm.so.0.1
[...]
--8<---------------cut here---------------end--------------->8---


Here the step (3rd *) "symlink
/gnu/store/...-guitarix-0.38.1/lib/libgxw.so (-> libgxw.so.0.1)" is
overridden by the next one "installing
build/default/libgxw/gxw/libgxw.so as
/gnu/store/...-guitarix-0.38.1/lib/libgxw.so".


Replaying with "guix environment guitarix" then from
"/tmp/guix-build-guitarix-0.38.1.drv-0", the classical "./waf
configure --prefix=install" and "./waf build" "./waf install". In this
case, one symlink is not done and the other one is overriden by the
next step.

--8<---------------cut here---------------start------------->8---
[...]
* installing build/default/libgxw/gxw/libgxw.so as
/tmp/guix-build-guitarix-0.38.1.drv-0/guitarix-0.38.1/install/lib/libgxw.so.0.1
* symlink /tmp/guix-build-guitarix-0.38.1.drv-0/guitarix-0.38.1/install/lib/libgxw.so
(-> libgxw.so.0.1)
* installing build/default/libgxw/gxw/libgxw.so as
/tmp/guix-build-guitarix-0.38.1.drv-0/guitarix-0.38.1/install/lib/libgxw.so
* installing build/default/libgxwmm/gxwmm/libgxwmm.so as
/tmp/guix-build-guitarix-0.38.1.drv-0/guitarix-0.38.1/install/lib/libgxwmm.so.0.1
[...]
--8<---------------cut here---------------end--------------->8---


Interestingly, the 3 last-last evaluations by Cuirass is of the kind "--check".

https://ci.guix.gnu.org/build/1818089/details
https://ci.guix.gnu.org/build/1871705/details
https://ci.guix.gnu.org/build/1903552/details

Therefore, if we were comparing on November 3rd, we would conclude
differently. ;-)


However, the last evaluation is creating the symlink without overriding it.

https://ci.guix.gnu.org/build/1920646/details

--8<---------------cut here---------------start------------->8---
[...]
* installing build/default/libgxw/gxw/libgxw.so as
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so.0.1
* symlink /gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so.0
(-> libgxw.so.0.1)
* symlink /gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxw.so
(-> libgxw.so.0.1)
* installing build/default/libgxwmm/gxwmm/libgxwmm.so as
/gnu/store/16g7l26rpwhza7fm5jm3lz1ycavky9yl-guitarix-0.38.1/lib/libgxwmm.so.0.1
[...]
--8<---------------cut here---------------end--------------->8---



Well, even this bug is annoying speaking about reproducibility --
because it it not the *exact* same installation -- it is mitigated by
the fact that the same binary is symlinked or not. I will revisit this
bug when WAF will fully use Python 3.


All the best,
simon

bug#21803: guitarix non-reproducible hard to reproduce

[Ludovic Courtès]

Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

> My intuition is: the non-determinism comes from the WAF configuration
> files (./waf or wscript or wafadmin/).
>
> The process conditionally enters in functions such as "do_install" or
> "symlink_as", so maybe something is hidden by Python module "os" or
> similar. Moreover, note that wscript:l.990 "add_group()" is used after
> "add_subdirs('libgxw/gxw')".

My intuition :-) is that waf traverses files using directly
opendir/readdir, which returns files in an order that’s file
system-dependent.  That, in turn, leads it to make .so a symlink or not
in a non-deterministic fashion.

So I would suggest looking for uses of ‘readdir’ (or anything equivalent
in Python).

It would also be worth checking what others involved in the Reproducible
Builds effort have done (Debian, openSuSE, etc.).

Thanks,
Ludo’.

bug#21803: guitarix non-reproducible hard to reproduce

[zimoun]

Hi Ludo,

On Sat, 16 Nov 2019 at 17:12, Ludovic Courtès <ludo@gnu.org> wrote:

> zimoun <zimon.toutoune@gmail.com> skribis:
>
> > My intuition is: the non-determinism comes from the WAF configuration
> > files (./waf or wscript or wafadmin/).
> >
> > The process conditionally enters in functions such as "do_install" or
> > "symlink_as", so maybe something is hidden by Python module "os" or
> > similar. Moreover, note that wscript:l.990 "add_group()" is used after
> > "add_subdirs('libgxw/gxw')".
>
> My intuition :-) is that waf traverses files using directly
> opendir/readdir, which returns files in an order that’s file
> system-dependent.  That, in turn, leads it to make .so a symlink or not
> in a non-deterministic fashion.

Yes, it should come from the function 'os.listdir' (readdir). The
documentation [1] says: "The list is in arbitrary order."

[1] https://docs.python.org/2/library/os.html#os.listdir


The function 'os.listdir' is called some times ;-)

--8<---------------cut here---------------end--------------->8---
$ egrep -r "os\.listdir\("
wafadmin3/py3kfixes.py:            for x in os.listdir(os.path.join(dir,y)):
wafadmin3/Scripting.py:    lst=os.listdir(cwd)
wafadmin3/Scripting.py:        if WSCRIPT_FILE in os.listdir(calldir):
wafadmin3/Scripting.py:        dirlst=os.listdir(cwd)
wafadmin3/Scripting.py:    names=os.listdir(src)
wafadmin3/Scripting.py:    lst=os.listdir('.')
wafadmin3/Tools/qt4.py:            lst=os.listdir('/usr/local/Trolltech/')
wafadmin3/Tools/javaw.py:        lst=os.listdir(path)
wafadmin3/Utils.py:        return os.listdir(s)
tools/check_rpc:    for f in os.listdir(basedir):
src/gx_head/builder/make:for f in os.listdir(__dir__):
wafadmin/py3kfixes.py:            for x in os.listdir(os.path.join(dir,y)):
wafadmin/Scripting.py:    lst=os.listdir(cwd)
wafadmin/Scripting.py:        if WSCRIPT_FILE in os.listdir(calldir):
wafadmin/Scripting.py:        dirlst=os.listdir(cwd)
wafadmin/Scripting.py:    names=os.listdir(src)
wafadmin/Scripting.py:    lst=os.listdir('.')
wafadmin/Tools/qt4.py:            lst=os.listdir('/usr/local/Trolltech/')
wafadmin/Tools/javaw.py:        lst=os.listdir(path)
wafadmin/Utils.py:        return os.listdir(s)
--8<---------------cut here---------------end--------------->8---

And do not talk about the function 'walk' which internally use 'listdir'. :-)


I have no idea how to address this issue.


> It would also be worth checking what others involved in the Reproducible
> Builds effort have done (Debian, openSuSE, etc.).

Lot of sun for Debian [2] ;-)
I mean from what I understand, they do not find any reproducibility
issue and they apply only this patch [3].

[2] https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/guitarix.html
[3] https://sources.debian.org/patches/guitarix/0.36.1-1/02-libdl.patch/


Do not know about openSuSE.


Cheers,
simon

bug#21803: guitarix non-reproducible hard to reproduce

[Ludovic Courtès]

Hello,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sat, 16 Nov 2019 at 17:12, Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> My intuition :-) is that waf traverses files using directly
>> opendir/readdir, which returns files in an order that’s file
>> system-dependent.  That, in turn, leads it to make .so a symlink or not
>> in a non-deterministic fashion.
>
> Yes, it should come from the function 'os.listdir' (readdir). The
> documentation [1] says: "The list is in arbitrary order."
>
> [1] https://docs.python.org/2/library/os.html#os.listdir
>
>
> The function 'os.listdir' is called some times ;-)

Hmm.

BTW, did you try comparing both build logs, in particular the lines
corresponding to the creation of the offending .so file?  That might
help narrow the search space.

Also, we can play with disorderfs to perhaps reproduce the problem
locally and be in more favorable debugging conditions.

>> It would also be worth checking what others involved in the Reproducible
>> Builds effort have done (Debian, openSuSE, etc.).
>
> Lot of sun for Debian [2] ;-)
> I mean from what I understand, they do not find any reproducibility
> issue and they apply only this patch [3].

Heh, OK.

Thanks,
Ludo’.

bug#21803: Guitarix builds non-deterministically

[zimoun]

Dear,

On Sun, 01 Nov 2015 at 11:30, ludo@gnu.org (Ludovic Courtès) wrote:
> Guitarix 0.33.0, as of Guix commit 3c3e697, builds
> non-deterministically:
>
> $ ./pre-inst-env guix challenge guitarix
> updating list of substitutes from 'http://hydra.gnu.org'... 100.0%
> /gnu/store/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0 contents differ:
>   local hash: 1rh7qxmylsbsaah59h7sclqqxcz0lwsixlc0krkzwhx8gfhlyam6
>   http://hydra.gnu.org/nar/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0: 1cq4k1wdqibwraqk0wkjj6n5hgs9v9zcvwr2wfgxvgxnf5l1rfhf
> $ wget -q -O - http://hydra.gnu.org/nar/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0 | bunzip2 | guix archive -x t
> $ LC_ALL=C diff -r /gnu/store/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0 t
> $ LC_ALL=C diff -r --no-dereference /gnu/store/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0 t
> File /gnu/store/6ksnwcqn92z2nf6qw5js7njnfqlifgpb-guitarix-0.33.0/lib/libgxw.so is a symbolic link while file t/lib/libgxw.so is a regular file
>
> It appears to create libgxw.so either as a symlink or as a regular file
> in a non-deterministic fashion.

Guitarix has been updated to 0.41.0 by commit
bf592ef506e1db0340dc11faa0514fe80793e6d6.

I have tried "guix build guitarix --no-grafts --check -K" at least 10
times without noticing an unreproducible behaviour.

Could someone confirm this and then close this almost 5 years old bug?


> It would be nice to see why this happens, and whether this affects all
> Waf-based packages.

I propose to open another bug report to track the waf-based packages issues.


All the best,
simon

bug#21803: Guitarix builds non-deterministically

[Maxim Cournoyer]

Hello,

[...]

> Guitarix has been updated to 0.41.0 by commit
> bf592ef506e1db0340dc11faa0514fe80793e6d6.
>
> I have tried "guix build guitarix --no-grafts --check -K" at least 10
> times without noticing an unreproducible behaviour.
>
> Could someone confirm this and then close this almost 5 years old bug?

Confirmed!

>> It would be nice to see why this happens, and whether this affects all
>> Waf-based packages.
>
> I propose to open another bug report to track the waf-based packages issues.

This sounds reasonable, given this is 5 years old and perhaps not
relevant anymore.

Closing, thank you!

Maxim