unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
From: ludo@gnu.org (Ludovic Courtès)
To: Mathieu Lirzin <mthl@gnu.org>
Cc: guix-devel <guix-devel@gnu.org>
Subject: Re: [GSoC] Continuous integration tool à la Hydra.
Date: Mon, 25 Jul 2016 23:36:16 +0200	[thread overview]
Message-ID: <87lh0pe6y7.fsf@gnu.org> (raw)
In-Reply-To: <87vazueezx.fsf@gnu.org> (Mathieu Lirzin's message of "Mon, 25 Jul 2016 02:30:10 +0200")

Hello!

Mathieu Lirzin <mthl@gnu.org> skribis:

> Since my second update I have first fixed a major bug.  When building
> different branches of Guix and evaluating package derivations the
> results were always the same.  The issue was that the evaluations were
> happening in the same Guile process which does not play well with module
> changes.  To fix that I have used a separate process + pipe to get the
> evaluation results.

Sounds good.  Using a separate process makes sure what’s evaluated
doesn’t erroneously ends up using modules already loaded by the
evaluator, and vice versa.

> In the last update, I have introduced usage of SRFI-9 records for
> specifications, jobs, and builds.  While they are nice to organize data,
> they have major drawbacks:
>
> - not flexible when you want to informally create a container.
> - require serialization when passing them throught pipes.
>
> For those reasons I have switched to good ol' alists, which are flexible,
> persistant, directly readable and don't require messing with load paths.
> The downside is of course that there is no compile time checks when
> manipulating data.

OK.

> As stated in my last update.  I have been working on storing data in a
> database.  For that I have decided to use Guile-sqlite3.  The principal
> efforts have consist of using an external schema file and design it.
>
> I have come up with this, but this will likely evolve in the future:
>
> CREATE TABLE Specifications (
>   id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
>   repo_name     TEXT NOT NULL,
>   url           TEXT NOT NULL,
>   load_path     TEXT NOT NULL,
>   file          TEXT NOT NULL,
>   proc          TEXT NOT NULL,
>   arguments     TEXT NOT NULL,
>   -- The following columns are optional.
>   branch        TEXT,
>   tag           TEXT,
>   revision      TEXT
> );
>
> CREATE TABLE Evaluations (
>   derivation    TEXT NOT NULL PRIMARY KEY,
>   job_name      TEXT NOT NULL,
>   specification INTEGER NOT NULL,
>   FOREIGN KEY (specification) REFERENCES Specifications (id)
> );

An evaluation leads to several derivations (for Guix, roughly one
derivation per package and per system type), but the table above seems
to suggest that each evaluation is mapped to only one derivation?

> CREATE TABLE Builds (
>   id              INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
>   derivation      TEXT NOT NULL,
>   log             TEXT NOT NULL,
>   output          TEXT,         -- NULL if build failed
>   FOREIGN KEY (derivation) REFERENCES Evaluations (derivation)
> );
>
> The next step will be to continue improving the database communication,
> and start looking how to implement the HTTP API.

Cool!  It seems that the database already has or is about to have the
info we typically look for at
<https://hydra.gnu.org/jobset/gnu/core-updates>: which evaluations were
performed, what commit (specification) they correspond to, which
derivations they correspond to and whether they
succeeded/failed/aborted.

If the HTTP API exposes this info, possibly using the same API as Hydra
(see guix-hydra-jobset.el), that will cover an important part of our
daily needs.

> For those willing to follow my work, a Git repository is available here:
>
>   https://notabug.org/mthl/cuirass

… and the README has instructions on how to run it.  If anyone has spare
CPU cycles, run Cuirass, ‘guix publish’, and share.  :-)

I’m happy with the progress that’s been made, thank you!

Ludo’.

  reply	other threads:[~2016-07-25 21:36 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-05-29 20:10 [GSoC] Continuous integration tool à la Hydra Mathieu Lirzin
2016-05-30 21:34 ` Ludovic Courtès
2016-05-30 21:54 ` Ludovic Courtès
2016-06-20 22:44 ` Mathieu Lirzin
2016-06-24 12:42   ` Ludovic Courtès
2016-07-25  0:30 ` Mathieu Lirzin
2016-07-25 21:36   ` Ludovic Courtès [this message]
2016-07-27 14:28     ` Mathieu Lirzin
2016-07-28 12:38       ` Ludovic Courtès
2016-07-29 11:20     ` Florian Paul Schmidt
2016-07-29 19:26       ` Mathieu Lirzin
2016-07-30 22:49         ` Ludovic Courtès
2016-07-31  2:05           ` 'guix environment' as a build tool. (was: [GSoC] Continuous integration tool à la Hydra.) Mathieu Lirzin
2016-07-31  2:20             ` Thompson, David
2016-07-31  4:17               ` 'guix environment' as a build tool Mathieu Lirzin
2016-07-31 13:55               ` Ludovic Courtès
2016-07-31 14:07                 ` Thompson, David
2016-07-31 20:09                   ` Ludovic Courtès
2016-07-31 11:13             ` Ludovic Courtès
2016-07-31  7:09         ` [GSoC] Continuous integration tool à la Hydra Florian Paul Schmidt
2016-07-31 12:03           ` Mathieu Lirzin
2016-08-01 18:55             ` Florian Paul Schmidt
  -- strict thread matches above, loose matches on Subject: below --
2016-08-06 11:05 David Craven
2016-08-06 16:19 ` Mark H Weaver
2016-08-06 16:23   ` David Craven
2016-08-06 16:58     ` David Craven
2016-08-06 23:45 ` Mathieu Lirzin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87lh0pe6y7.fsf@gnu.org \
    --to=ludo@gnu.org \
    --cc=guix-devel@gnu.org \
    --cc=mthl@gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).