From: Ricardo Wurmus <rekado@elephly.net>
To: Konrad Hinsen <konrad.hinsen@fastmail.net>
Cc: Guix Devel <guix-devel@gnu.org>, zimoun <zimon.toutoune@gmail.com>
Subject: Re: Investigating a reproducibility failure
Date: Thu, 03 Feb 2022 12:41:58 +0100 [thread overview]
Message-ID: <87a6f8415g.fsf@elephly.net> (raw)
In-Reply-To: <m17dacb988.fsf@fastmail.net>
Hi Konrad,
>> CPU detection is a bottomless can of worms.
>
> That sounds very credible. But what can we do about this?
>
> There is obviously a trade-off between reproducibility and performance
> here. Can we support both, in a way that users can understand and manage?
So far our default approach has been to use the lowest common set of CPU
instructions, which generally leads to poorly performing code. Some
packages are smarter and provide different code paths for different
CPUs. The resulting binary is built the same, but at runtime different
parts of the code run dependent on the features the CPU reports.
The case of OpenBLAS is an anomaly in that this mechanism seems to
produce different binaries dependent on where it is built. When I first
encountered this problem I guessed that perhaps it can only build these
different code paths up to the feature set of the CPU on the build
machine, so if you’re building with an older CPU your binary will lack
components that would be used on newer CPUs. This is just a guess,
though.
Your problem is that the OpenBLAS build system doesn’t recognize your
modern CPU. Ideally, it wouldn’t need to know anything about the
build-time CPU to build all the different code paths for different CPU
features. The only way around this — retroactively — is to pretend to
have an older CPU, e.g. by using qemu.
In the long term it would be great if we could patch OpenBLAS to not
attempt to detect CPU features at build time. I’m not sure this will
work if it does indeed use the currently available CPU features to
determine “how far up” to build modules in support of certain CPU
features / instruction sets.
> There is of course the issue that we can never be sure if a build will
> be reproducible in the future. But we can at least take care of the
> cases where the packager is aware of non-reproducibility issues, and
> make them transparent and manageable.
The new “--tune” feature is supposed to take care of cases like this.
We would still patch the code so that by default you’d get a package
that is reproducible (= you get the same exact binary no matter when or
where you build it) but that many not have optimal performance. With
“--tune” you could opt to replace that generic build with one that uses
features of your current CPU, using grafts to swap the generic library
for the more performant library.
--
Ricardo
next prev parent reply other threads:[~2022-02-03 11:53 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-02-02 20:35 Investigating a reproducibility failure zimoun
2022-02-02 23:43 ` zimoun
2022-02-03 9:16 ` Konrad Hinsen
2022-02-03 11:41 ` Ricardo Wurmus [this message]
2022-02-03 17:05 ` Konrad Hinsen
2022-02-03 12:07 ` zimoun
2022-02-05 14:12 ` Ludovic Courtès
2022-02-15 14:10 ` Bengt Richter
2022-02-16 12:03 ` zimoun
2022-02-16 13:04 ` Konrad Hinsen
2022-02-17 11:21 ` zimoun
2022-02-17 16:55 ` Konrad Hinsen
-- strict thread matches above, loose matches on Subject: below --
2022-02-01 14:05 Konrad Hinsen
2022-02-01 14:30 ` Konrad Hinsen
2022-02-02 23:19 ` Ricardo Wurmus
2022-02-02 23:36 ` Ricardo Wurmus
2022-02-05 14:05 ` Ludovic Courtès
2022-02-08 5:57 ` Konrad Hinsen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87a6f8415g.fsf@elephly.net \
--to=rekado@elephly.net \
--cc=guix-devel@gnu.org \
--cc=konrad.hinsen@fastmail.net \
--cc=zimon.toutoune@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).