bug#53533: [DISCUSSION] Quality of services in reproducible build environment Guix

bug#53533: [DISCUSSION] Quality of services in reproducible build environment Guix

[Sharlatan Hellseher]

Hi Guix team!

The current QA for the accepted changes in Gujx is far away from
trustible. For example, some
changes in package update may cause a faileur of the whole chain of
packages depending on it. It
would be nice to have some soft policy of changes, check list or some
procedure to have a "stable"
branch which may guarantee all packages build successfully and pass of
all enabled tests.

I find current [[id:60941898-0ed4-4188-b473-d2dcda158d61][CI/CD]]
(https://ci.guix.gnu.org/) is missleading in case of providing some
visibility
of all successful builds for the current pushed changes (on master
branch). I would like to conclude
from the CI is which commit broke how many packages. Other open
question - if I've sticked to a
specific "stable" branch how I may be sure that another ~guix pull~
will not break my packages in
case of un-pinned version?

Some missing features of CI
- Timing - current view has missing a clear representation of build date-time
- Overall slats for the current commit to the specific branch - and
how many package are become
  broken after update of package X.
- Sort of "blocking on merge" of a commit which causes some issues (do
not merge broken packages
  into stable branch)
- Some documentation for all UI features (green dots, red dots, grey dots etc.
  https://ci.guix.gnu.org/eval/54326/dashboard)

Some missing practice of packaging:
- Some essential message of the reason why tests were disabled and any
sort of suggestions on how to
  make them enabled. Contact upstream if required.
- Before sending patch make sure (at least for the localhost
architecture) it's built, linted and in
  case of bumping version - all dependent chain still can be built.

Related commits and issues which broke other packages in ~master~ branch:
- https://issues.guix.gnu.org/53230
- fix 6445f412b993ec7b52dc4c81e99f49af38b3a967 RawTherapee stopped
building with wrong configure key,
  the previous update to 5.8 was never tested before been merged.

  #+begin_src sh
git checkout master
git pull
git log -n1 --pretty="%h %s %cd - %cn"
  #+end_src
  : 4235c6ee92 gnu: QGIS: Build without QtWebKit. Tue Jan 25 15:10:19
2022 -0500 - Leo Famulari

  Excess changes which could be prevented with just local attempt to build
  #+begin_src sh
git log --grep="Fix build" --pretty="%h %s %cd - %cn" | wc -l
  #+end_src
  : 1025


-- 
… наш разум - превосходная объяснительная машина которая способна
найти смысл почти в чем угодно, истолковать любой феномен, но
совершенно не в состоянии принять мысль о непредсказуемости.

bug#53533: [DISCUSSION] Quality of services in reproducible build environment Guix

[Leo Famulari]

On Tue, Jan 25, 2022 at 08:45:22PM +0000, Sharlatan Hellseher wrote:
> Related commits and issues which broke other packages in ~master~ branch:
> - https://issues.guix.gnu.org/53230
> - fix 6445f412b993ec7b52dc4c81e99f49af38b3a967 RawTherapee stopped
> building with wrong configure key,
>   the previous update to 5.8 was never tested before been merged.

The previous update of the Rawtherapee package happened in February
2020, almost two years ago. It was definitely tested at that time. Since
then, some other changes in the distro broke it.

>   #+begin_src sh
> git checkout master
> git pull
> git log -n1 --pretty="%h %s %cd - %cn"
>   #+end_src
>   : 4235c6ee92 gnu: QGIS: Build without QtWebKit. Tue Jan 25 15:10:19
> 2022 -0500 - Leo Famulari
> 
>   Excess changes which could be prevented with just local attempt to build
>   #+begin_src sh
> git log --grep="Fix build" --pretty="%h %s %cd - %cn" | wc -l
>   #+end_src
>   : 1025

Can you say what this shell snippet means?

bug#53533: [DISCUSSION] Quality of services in reproducible build environment Guix

[Leo Famulari]

On Tue, Jan 25, 2022 at 08:45:22PM +0000, Sharlatan Hellseher wrote:
> The current QA for the accepted changes in Gujx is far away from
> trustible. For example, some
> changes in package update may cause a faileur of the whole chain of
> packages depending on it. It
> would be nice to have some soft policy of changes, check list or some
> procedure to have a "stable"
> branch which may guarantee all packages build successfully and pass of
> all enabled tests.

People are definitely supposed to check that their changes don't break
things before they push them, but it doesn't always happen.

It's not easy to make sure that all packages build successfully. I've
never seen 100% of packages build successfully since I joined Guix in
2015. It's a nice goal but it requires some work... a lot more work.
Everyone is invited to help. Probably, the first step is to remove
several hundred packages that don't build; it might even be a couple
thousand.

> I would like to conclude from the CI is which commit broke how many
> packages.

Agreed, this is a very important missing feature. Also, we need the
capability to compare commits in terms of CI results.

Our CI software is called Cuirass:

https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git

We need more people to help develop Cuirass!

> Some missing practice of packaging:
> - Some essential message of the reason why tests were disabled and any
> sort of suggestions on how to
>   make them enabled. Contact upstream if required.

Everyone is supposed to do this when writing packages. It's a failure of
the code review process if that is not happening.

> - Before sending patch make sure (at least for the localhost
> architecture) it's built, linted and in
>   case of bumping version - all dependent chain still can be built.

This is supposed to be done for patches that go to the master branch.
That is, patches that affect <300 packages. Other patches that affect
more than 300 packages are batched on development branches such as
'core-updates' [0], and then we need a lot of Guix contributors to help
get all the packages building again. Maybe we should remove broken
packages more casually, like I suggested above.

>   Excess changes which could be prevented with just local attempt to build
>   #+begin_src sh
> git log --grep="Fix build" --pretty="%h %s %cd - %cn" | wc -l
>   #+end_src
>   : 1025

We have 92225 commits total:

------
$ git log --oneline | wc -l
92225
------

So, about 1.1% of them are "Fix build" commits.

However, not all of these commits were made on the master branch. Many
of them are on development branches such as 'core-updates' and
'staging', and so users never experienced the problems that were fixed.

That doesn't mean that Guix QA is perfect, but rather that your
measurement is inaccurate and misleading.

[0] To learn more about development branches like core-updates, see
item 8:
https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html

bug#53533: [DISCUSSION] Quality of services in reproducible build environment Guix

[zimoun]

Hi,

On Wed, 26 Jan 2022 at 03:53, Leo Famulari <leo@famulari.name> wrote:

> It's not easy to make sure that all packages build successfully. I've
> never seen 100% of packages build successfully since I joined Guix in
> 2015. It's a nice goal but it requires some work... a lot more work.

I think this goal is impossible by design of the current workflow.

Do not take me wrong, I am saying that some clean is not necessary.
My aim with this email is to point that the annoyance of broken
packages comes from our current workflow and our scaling up, yeah! :-)


> Everyone is invited to help. Probably, the first step is to remove
> several hundred packages that don't build; it might even be a couple
> thousand.

We agreed to do that in 2019 with a reminder on June, 20th 2020:

<https://yhetil.org/guix/86y2oisj41.fsf@gmail.com/>

and nothing happened.  Because the issue is the current modelling:
master/staging/core-updates.  Continuing with that workflow, it is not
realistic to purge broken packages.  Here, we have to make a choice:

 a) current workflow with broken packages
 b) adopt another workflow

The pro for a) is that it is already working enough, we have the
habits, etc.  And the cons of a) are some packages break sometimes.
My rough guess is, these days, ~5% of packages are broken per
revision.  And my guess is that this percentage of broken package
stays the same as the number of packages increases; other said, more
packages often means more users, so this ~5% means more "visibility"
to hit a broken package.

For b), the switch comes with a cost and I am not convinced all people
are agreeing if this cost would be counter-balanced.  More details
about such discussion are for instance in this thread:

<https://yhetil.org/guix/86mtv29erk.fsf@gmail.com/>

and the archive guix-devel contains many similar discussions.  Nothing new. ;-)


> > I would like to conclude from the CI is which commit broke how many
> > packages.
>
> Agreed, this is a very important missing feature. Also, we need the
> capability to compare commits in terms of CI results.

The idea was to have this feature provided by the Data Service.
However, AFAIK, many pieces are since then broken between
ci.guix.gnu.org and data.guix.gnu.org.

For instance, using this:

<https://data.guix.gnu.org/repository/1/branch/master/package/rawtherapee/output-history>

you can detect which commit breaks the package.  Well, 2 important notes:

 1. All is Scheduled because data.guix.gnu.org is fetching data from
the other CI managed by the Build Coordinator, not from the CI managed
by Cuirass.
 2. The commits listed there are an approximation.  Basically, there
are based on the mailing list guix-commits and a complete batch pushed
as once is considered as a change, and more than often, this batch
contains irrelevant change, see details there:

<https://yhetil.org/guix/863617oe1h.fsf@gmail.com/>


> > Some missing practice of packaging:
> > - Some essential message of the reason why tests were disabled and any
> > sort of suggestions on how to
> >   make them enabled. Contact upstream if required.
>
> Everyone is supposed to do this when writing packages. It's a failure of
> the code review process if that is not happening.

I would not say it is a failure of the code review process.  Instead,
I would say it is a failure of the current workflow.

On one hand, the project cannot complain that not enough people are
doing reviewing (see this thread [1]), when on the other hand,
potential reviewer would have to build many more than just the change
itself, especially when many potential reviewers do not have enough
power at hand (see this thread [2]).

1: <https://yhetil.org/guix/87mtn56mzg.fsf_-_@inria.fr/>
2: <https://yhetil.org/guix/878rynh0yq.fsf@systemreboot.net/#t>


Note that since I joined the project (around 2018), we are discussing
about more automation using patchwork and other similar tools.  For
instance, Chris started this initiative:

<https://patches.guix-patches.cbaines.net/project/guix-patches/list/>

which would really help to improve the CI process.  However, we, as a
project, are not putting enough love in such initiative.  Well, the
hidden point barely discussed is twofold: necessity of a powerful
infrastructure for building more when the resource are somehow
splitted and strong integration with current CI.  Both require many
more resources: manpower and money.  That's why, I think the current
workflow will make more visible broken packages.


> That doesn't mean that Guix QA is perfect, but rather that your
> measurement is inaccurate and misleading.

Yes, this 1.1% does not make sense.  Firstly, counting the commits
containing "Fix build" completely underestimates the number of real
fixes, because some fixes contains instead "Fix tests", some fixes are
update, etc.  Secondly, the important number is the number of broken
packages at one instant t (revision) and for that, the Git history is
useless.  Basically, the important number is the ratio between the
green vs red bullets [3].  Other said, compare the coverage, say,
using "guix weather" for past revisions.  I guess, ~5% of packages are
broken, on average.

3: <https://ci.guix.gnu.org/eval/63833/dashboard>


My 2 cents.

Cheers,
simon

PS: The discussion usually happens on guix-devel and not in the bug
tracker.  Other said, since it does not appear to me an actionable bug
report, I would like to close it.

bug#53533: [DISCUSSION] Quality of services in reproducible build environment Guix

[Sharlatan Hellseher]

[-- Attachment #1: Type: text/plain, Size: 80 bytes --]


Hi,

I think I can close this now as all questions were covered.

Thanks,
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]