Re: Cuirass web writeup

Re: Cuirass web writeup

[Gábor Boskovits]

[-- Attachment #1.1: Type: text/plain, Size: 1440 bytes --]

Gábor Boskovits <boskovits@gmail.com> ezt írta (időpont: 2018. júl. 30., H,
8:04):

> Clément Lassieur <clement@lassieur.org> ezt írta (időpont: 2018. júl.
> 30., H, 1:29):
>
>> Gábor Boskovits <boskovits@gmail.com> writes:
>>
>> > I did not want to send this on the mailing list, as evaluations are not
>> > public yet. I've collected important communication regarding the
>> project. I
>> > also added some additional ideas. If you can please fill in the missing
>> > details. Also how should we share this document? Civodul recommended
>> git.
>>
>> So, I pushed the web interface.  There are probably lots of missing
>> features, I didn't check the list.  But at least we have an existing
>> basis on which we can add new things.
>>
>> Maybe, Gábor, your writeup can be edited accordingly :-)
>>
>> Thanks, I'll take a look.
>

Here is the new, extended version of my writeup. I marked the items already
completed, and at the end I collected the missing features.
What I really would like to get implemented is filtering and shorting, as
it would make this more easily extensible. WDYT?
It also seems to me, that we actually don't have build farm status
information right now. How could we do this? Should we add something like
pending jobs and currently building jobs? Do we have this at the database
level?
How should we incorporate architecture information?



>
>
>> Clément
>>
>

[-- Attachment #1.2: Type: text/html, Size: 2566 bytes --]

[-- Attachment #2: cuirass-web.org --]
[-- Type: application/octet-stream, Size: 5932 bytes --]

* TODO Features

** DONE Add the list of all evaluations of a specification
*** DONE displaying numbers of successful and failed builds for each evaluation
(like this list https://hydra.nixos.org/jobset/gnu/guile-2-0/evals).

** TODO Add the table with status information of each job-evaluation pair
like https://hydra.nixos.org/jobset/gnu/guile-2-0/#tabs-jobs.

Am I right that in terms of Cuirass database derivations correspond to jobs? So,
we need to display "derivations" table with derivations as rows and evaluations
as columns and fill the cells using information from the "builds" table? Also,
it is not clear to me how to order evaluations by their creation time like it's
done in Hydra.

** TODO Add page displaying information about a build
similar to https://hydra.nixos.org/build/74870692.

Am I right that here we should display the information stored in hydra-like
build description (defined in http.scm) alongside with links to the build log
and outputs?

** TODO Add additional information about previous builds
(latest successful, first broken, etc) on this build page. For this feature,
we need to extend database requests functionality.

** DONE Add job information page containing the list of job's latest builds
like https://hydra.nixos.org/job/gnu/guile-2-0/build.x86_64-linux.

* What we want to know?

** TODO is the build farm building something?
** TODO if so, what is it building?
** TODO how many more things does it need to build in this round (=“evaluation”)?
** DONE has any build failed?
** TODO if so, how did the builds fail?
** TODO what build failure is blocking a certain package from being built?
** TODO is the percentage of failing builds decreasing over time as we modify the repository?

* Danny's list

Generally, the continuous integration process should enable developers
to get feedback about the effects of their changes.

(This is not for the web interface project)
** TODO This means that as soon as a commit is made, usually an evaluation of the build source on the continuous integration server starts.

(Sometimes there are exceptions to this (for example in order to not overload
the build servers) - but generally it's true)

For a new evaluation, as a developer I'd like to know:

** TODO Are now more packages broken than before?  Which ones?
** TODO Are now more packages working than before?  Which ones?
** TODO Do some packages work on more architectures than before? Fewer?
** TODO Is the build server still building my change?
** TODO Or is it done and I can trust that the information I see is now complete?
** TODO If not, what is it building now or later?

"before" means "with the previous evaluation" or "with some specific past
evaluation" or "in another branch".

I think this would be the most basic functionality.

More advanced functionalities would include automatic tracking on the reason
of the failure:

** TODO If it's dependency failure, specifically mark this package
so I know I don't have to fix this package
** TODO I have to fix a package this one depends on (which one?).
** TODO What kind of failure is it?
** TODO What's the latest non-noise error message in the build log?
** TODO Display suggestions on what to do about it.

* DONE Clément's changes

** DONE Adding support for multiple inputs.

Currently Cuirass only supports one input per specification,
which has to be the Guix git repository.
But one might want inputs for:
  - a repository containing the 'proc' that returns the jobs,
  - a repository containing extra packages (GUIX_PACKAGE_PATH).
Those inputs would be fetched at INTERVAL as well, and any change would
trigger an evaluation.  This is a follow-up of
https://lists.gnu.org/archive/html/guix-patches/2018-06/msg00311.html.

** DONE Removing the notion of 'project'.
Cuirass really has no project, and what is called a 'project' is in fact a
'specification' (which Hydra calls 'jobset').  And what is called a 'jobset'
by Cuirass is in fact the branch of the Guix input.  So the patch associates
to the name 'jobset' what it really is: a specification.  A consequence is
that the 'jobset' filtering of the API now filters by specification name, and
it is not possible anymore to filter by branch.  (But it was useless IMHO.)

* DONE Ricardo - Clément communication

> I’m supportive of all of the proposed changes.  My only concern is about
> potential conflicts with Tatiana’s work on the web interface.  Do you
> think we could merge Tatiana’s patches first?

I think we should merge whichever patchset is ready first.  I will
surely help with the conflicts resolution, but it shouldn't be too
difficult, because we didn't work on the same parts.  It's a matter of
renaming a few database fields, that's all.  (There is no git conflict.)

* What's next?
** DONE Checking what features are implented.
** DONE Collecting what's missing.
** TODO Feature selection for last round.
** DONE Planning merge, resolving conflicts, if any.
** TODO Planning testing and deployment.

* What is missing?
** TODO filtering and shorting
This would make a lot of missing points low hanging fruit.
** TODO list of failed builds
** TODO detailed build info page
*** TODO reasons of build failure
**** TODO if dependency failure, which dependency is it?
** TODO information on previous builds
** TODO build farm status (what is it doing, what it still needs to do)
** TODO incremental information to the evaluation page
*** TODO Absolute difference of successful builds
**** TODO list of new successful builds
*** TODO Absoulte difference of failing builds
**** TODO list of new failing builds
*** TODO Add relative success rate and change of relative success rate
** TODO Add support to view builds based on architectures

* Questions

** what does the gray backgrounded number show on the evaluations page?
** do you think that we need to document the interface?
** do you think that we need to document the URL parameters?