Re: Deterministic Library Calls when Building

[ludo@gnu.org (Ludovic Courtès)]

Hello!

Karl Semich <fuzzytew@gmail.com> skribis:

> I'm pretty new at looking under the hood of linux, but I can imagine these
> approaches at least:
> - preload system library wrappers around key nondeterministic functions

Often, preloaded libraries are a bit fragile, in part because they need
to intercept a wide range of libc system call wrappers and to maintain
state.

An example is libfaketime, which people have tried to use in the context
of Debian’s reproducible builds effort.  In this particular case,
there’s the additional problem that tools that heavily rely on
timestamps, such as Make, break in unexpected ways in such environments.

> - replace /dev/*random with fakes (could be named pipes, dummy devices fed
> by modules, or just flat files!)

This one sounds like it could easily be implemented.

I think /dev/*random cannot be a flat file, because applications such as
test suites expect some sort of randomness, but it could be seeded with
a constant value.

In practice I don’t think we’ve ever identified a case where
/dev/*random was having a visible effect on build results.

> - replace system libraries with fullblown libraries with nondeterministic
> calls rewritten (could merge changes upstream, provide a flag)
> - create a kernel module which alters the behavior of the running kernel to
> be more deterministic
> - change the kernel itself to have a "deterministic mode" (could merge
> upstream)

In practice, the non-determinism issues we stumble upon are often either
“trivial” (e.g., a timestamp is stored somewhere), or super tricky
(e.g., result depends on thread/process scheduling.)

Because of that, I think that the changes that are easy to implement
would in fact be of little help, and that the changes that would be the
most helpful (e.g., deterministic scheduling) would take a lot of
effort and/or make the build environment much more complex.

But I may well be too pessimistic, and I think it’s good to investigate
how we could improve things!

Thanks,
Ludo’.