Re: Proposal: Build timers

[zimoun <zimon.toutoune@gmail.com>]

Hi,

On Tue, 23 Nov 2021 at 14:39, Jacob Hrbek <kreyren@rixotstudio.cz> wrote:

>> This approximation would not even be accurate enough for the same
>> machine.  For instance, the test suite of the julia package runs
>> mainly sequential using one thread... 
>
> I am aware of this scenario and I adapted the equasion for it, but I
> recognize that this exponentially increases the inaccuracy with more
> threads and I don't believe that there is a mathematical way with the
> provided values to handle that scenario so we would have to adjust the
> calculation for those packages.

What I am trying to explain is that the model cannot work to be
predictable enough with what I consider a meaningful accuracy.
Obviously, relaxing the precision, it is easy to infer a rule of thumb;
a simple cross-multiplication fits the job. ;-)

The “pokémon-battle” model is a simple linear model
(cross-multiplication); using Jacob’s “notation”:

 - HP: time to build on machine A
 - DPS = nthread * cpufreq : “power” of machine

Then it is expected to evaluate ’a’ and ’b’ on average such that:

  HP = a * DPS + b

based on some experiments.  Last, on machine B, knowing both nthread'
and cpufreq' for that machine B, you are expecting to evaluate HP' for
that machine B applying the formula:

  HP' = a * nthread' * cpufreq' + b

Jacob, do I correctly understand the model?


In any case, that’s what LFS is doing, instead HP is named SBU.  And
instead DPS, they use a reference package.  And this normalization is
better, IMHO.  Other said, for one specific package considered as
reference, they compute HP1 (resp. HP2) for machine A (resp. B), then
for machine A, they know HP for another package and they deduce,

  HP' = HP2/HP1 * HP

All this is trivial. :-) The key is the accuracy, i.e., the error
between the prediction HP' and the real time.  Here, the issue is that
HP1 and HP2 capture for one specific package the overall time; which
depends on hidden parameters as nthread, cpufreq, IO, and other
parameters from hardware.  But that a strong assumption when considering
these hidden parameters (evaluated for one specific package) are equally
the same for any other package.

It is a strong assumption because the hidden parameters depends on
hardware specifications (nthread, cpufreq, etc.) *and* how the package
itself exploits them.

Therefore, the difference between the prediction and the real time is
highly variable, and thus personally I am not convince the effort is
worth; for local build.  That’s another story. ;-)

LSF is well-aware of the issue and it is documented [1,2].

The root of the issue is the model based on a strong assumption; both
(model and assumption) do not fit how the reality concrete works, IMHO.

One straightforward way — requiring some work though – for improving the
accuracy is to use statistical regressions.  We cannot do really better
to capture the hardware specification – noticing that the machine stress
(what the machine is currently doing when the build happens) introduces
a variability hard to estimate beforehand.  However, it is possible to
do better when dealing with packages.  Other said, exploit the data from
the build farms.


Well, I stop here because it rings a bell: model could be discussed at
length if it is never applied to concrete numbers. :-)

Let keep it pragmatic! :-)

Using the simple LFS model and SBU, what would be the typical error?

For instance, I propose that we collectively send the timings of
packages: bash, gmsh, julia, emacs, vim; or any other 5 packages for
x86_64 architecture.  Then we can compare typical errors between
prediction and real, i.e., evaluate “accuracy“ for SBU and then decide
if it is acceptable or not. :-)


Cheers,
simon


1: <https://www.linuxfromscratch.org/lfs/view/stable/chapter04/aboutsbus.html>
2: <https://www.linuxfromscratch.org/~bdubbs/about.html>