[GSoC] Continuous integration tool à la Hydra.

[GSoC] Continuous integration tool à la Hydra.

[Mathieu Lirzin]

Hello Guix!

Here is a first update on my GSoC project after the first week.

Hydra (https://nixos.org/hydra/) is a Nix-based continuous build system
which is used by Guix to compile packages on different platforms and to
distribute packages substitutes.  The aim of this project is to replace
Hydra with a more integrated software written in Guile.

Since it would be cumbersome to speak about something without giving it
a name, I have decided to call this software “Cuirass”.

While being intended as a general purpose continuous integration tool,
the first iteration of Cuirass is focusing only on building Guix
packages.  The logic of the program is simple since 'cuirass' sits in a
loop:

 - Fetching new changes from Guix master branch
 - Compiling Guix code
 - Evaluating packages derivations
 - Building those derivations

The Guix repository is kept in a directory set by 'CUIRASS_CACHEDIR'
environment variable to avoid cloning it every time.  Even if the
algorithm is trivial, the implementation has required some thought in
order to allow using guile code fetched from the Guix git repository.
Hopefully the solution was already in 'hydra-eval-guile-jobs' script
from Hydra, so I have used the same technique.

Currently the build phase is failing with the following error:

--8<---------------cut here---------------start------------->8---
Starting download of /gnu/store/y17blrd1x80nqik4ss4djbm92a5i5ivl-zsh-5.1.1.tar.gz
From http://www.zsh.org/pub/zsh-5.1.1.tar.gz...
ERROR: In procedure connect*: Network is unreachable

Starting download of /gnu/store/y17blrd1x80nqik4ss4djbm92a5i5ivl-zsh-5.1.1.tar.gz
From http://www.zsh.org/pub/old/zsh-5.1.1.tar.gz...
ERROR: In procedure connect*: Network is unreachable

Starting download of /gnu/store/y17blrd1x80nqik4ss4djbm92a5i5ivl-zsh-5.1.1.tar.gz
From http://tarballs.nixos.org/sha256/11shllzhq53fg8ngy3bgbmpf09fn2czifg7hsb41nxi3410mpvcl...
ERROR: download failed "http://tarballs.nixos.org/sha256/11shllzhq53fg8ngy3bgbmpf09fn2czifg7hsb41nxi3410mpvcl" 404 "Not Found"
failed to download "/gnu/store/y17blrd1x80nqik4ss4djbm92a5i5ivl-zsh-5.1.1.tar.gz" from ("http://www.zsh.org/pub/zsh-5.1.1.tar.gz" "http://www.zsh.org/pub/old/zsh-5.1.1.tar.gz")
builder for `/gnu/store/srsm569hhqiz56abp90dksknn6mqb03j-zsh-5.1.1.tar.gz.drv' failed to produce output path `/gnu/store/y17blrd1x80nqik4ss4djbm92a5i5ivl-zsh-5.1.1.tar.gz'
@ build-failed /gnu/store/srsm569hhqiz56abp90dksknn6mqb03j-zsh-5.1.1.tar.gz.drv - 1 builder for `/gnu/store/srsm569hhqiz56abp90dksknn6mqb03j-zsh-5.1.1.tar.gz.drv' failed to produce output path `/gnu/store/y17blrd1x80nqik4ss4djbm92a5i5ivl-zsh-5.1.1.tar.gz'
cannot build derivation `/gnu/store/ri86p3f5w84r6pxyz54icyb0r00lmqaj-zsh-5.1.1.drv': 1 dependencies couldn't be built
Backtrace:
In ice-9/boot-9.scm:
 157: 9 [catch #t #<catch-closure 1ffeba0> ...]
In unknown file:
   ?: 8 [apply-smob/1 #<catch-closure 1ffeba0>]
In ice-9/boot-9.scm:
  63: 7 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 6 [eval # #]
In ice-9/boot-9.scm:
  63: 5 [call-with-prompt break486 ...]
  63: 4 [call-with-prompt continue487 ...]
In ice-9/eval.scm:
 432: 3 [eval # #]
In ice-9/r4rs.scm:
  90: 2 [dynamic-wind #<procedure 4c92620 at ice-9/boot-9.scm:1133:2 _> ...]
In ice-9/boot-9.scm:
 702: 1 [map #<procedure 3b489120 at ice-9/eval.scm:416:20 (a)> (# # # # ...)]
In guix/store.scm:
 684: 0 [build # # 0]

guix/store.scm:684:15: In procedure build:
guix/store.scm:684:15: Throw to key `srfi-34' with args `(#<condition &nix-protocol-error [message: "build of `/gnu/store/ri86p3f5w84r6pxyz54icyb0r00lmqaj-zsh-5.1.1.drv' failed" status: 1] 3b50d5a0>)'.
--8<---------------cut here---------------end--------------->8---

which is probably related to the fact that https is not handled.

The next step will be to fix this issue, to provide a better abstraction
for the 'build jobs', and to find a way to make some test scripts using
dummy jobs.

For those willing to see more, a Git repository is available here:

  https://notabug.org/mthl/cuirass

Everyone is of course welcome to provide any feedback.

Thanks.

-- 
Mathieu Lirzin

Re: [GSoC] Continuous integration tool à la Hydra.

[Ludovic Courtès]

Hello!

Mathieu Lirzin <mthl@gnu.org> skribis:

> Since it would be cumbersome to speak about something without giving it
> a name, I have decided to call this software “Cuirass”.

Interesting.  :-)

> While being intended as a general purpose continuous integration tool,
> the first iteration of Cuirass is focusing only on building Guix
> packages.  The logic of the program is simple since 'cuirass' sits in a
> loop:
>
>  - Fetching new changes from Guix master branch
>  - Compiling Guix code
>  - Evaluating packages derivations
>  - Building those derivations
>
> The Guix repository is kept in a directory set by 'CUIRASS_CACHEDIR'
> environment variable to avoid cloning it every time.  Even if the
> algorithm is trivial, the implementation has required some thought in
> order to allow using guile code fetched from the Guix git repository.
> Hopefully the solution was already in 'hydra-eval-guile-jobs' script
> from Hydra, so I have used the same technique.

So far, so good!

I think eventually ‘compile’ should disappear.  It should be up to the
project’s ‘evaluate.scm’ file (for Guix, it’s
‘build-aux/hydra/gnu-system.scm’) to compute whatever needs to be
computed to produce a list of derivations to build.

> Currently the build phase is failing with the following error:
>
> Starting download of /gnu/store/y17blrd1x80nqik4ss4djbm92a5i5ivl-zsh-5.1.1.tar.gz
> From http://www.zsh.org/pub/zsh-5.1.1.tar.gz...
> ERROR: In procedure connect*: Network is unreachable
>
> Starting download of /gnu/store/y17blrd1x80nqik4ss4djbm92a5i5ivl-zsh-5.1.1.tar.gz
> From http://www.zsh.org/pub/old/zsh-5.1.1.tar.gz...
> ERROR: In procedure connect*: Network is unreachable

Probably just a transient error on zsh.org.  The 1st URL is 404, and the
second one works for me.  I don’t think this has anything to do with
https.

Anyway, it’s good to see that the basics are in place!  :-)

Thanks for the update,
Ludo’.

Re: [GSoC] Continuous integration tool à la Hydra.

[Ludovic Courtès]

Mathieu Lirzin <mthl@gnu.org> skribis:

> For those willing to see more, a Git repository is available here:
>
>   https://notabug.org/mthl/cuirass

BTW, I can confirm that it “builds stuff” here when I let it go.

An improvement would be to redirect or filter
(current-build-output-port).  For instance, we could let only the “@”
lines go through (build started, build failed, etc.)

Ludo’.

Re: [GSoC] Continuous integration tool à la Hydra.

[Mathieu Lirzin]

Hello Guix!

Here is a second update on my GSoC project after the first month.

As a reminder, Hydra (https://nixos.org/hydra/) is a Nix-based
continuous build system which is used by Guix to compile packages on
different platforms and to distribute packages substitutes.  The aim of
this project is to replace Hydra with a more integrated software written
in Guile, named “Cuirass”.

After providing a first basic evaluation loop described in my previous
mail, I have started thinking about the architecture of the global job
evaluation/compilation process in order to identify what type of data
structures would make sense.  It was not easy for me to reason about the
different layers of current Hydra usage in Guix, so it took me 2 weeks
to analyse it and provide a first decomposition of logic steps.  For now
this decomposition is:

  job-spec > job > build-result

where:
  
  - 'job-spec' defines all the information required by Cuirass to get
    the actual job definitions.  These information contains the
    repository type and url, the file and procedure name which yields a
    list of job, and the list of arguments passed to that procedure.

  - 'job' contains the derivation file name which describe all the
    dependencies.

  - 'build-result' contains the output obtained when realizing/building
    the derivation from a job + some logs.

'job-spec' and 'job' are already implemented in Cuirass. However
'build-result' will require a database to be useful.  Since I have no
experience with databases at all, this last week has been dedicated to
learn more about them, play with SQL queries, and use Guile-dbi and
Guile-sqlite3 bindings.  My plan is to use Sqlite first and eventually
switch to Postgresql later when concurrent writers would be critical.

The next step is to apply my newly acquired knowledge by allowing
'build-result' entries to be added to the database.  I will be AFK for 5
days so I will start working on that, next Sunday.

For those willing to follow my work, a Git repository is available here:

  https://notabug.org/mthl/cuirass

Everyone is of course welcome to provide any feedback.

Thanks.

-- 
Mathieu Lirzin

Re: [GSoC] Continuous integration tool à la Hydra.

[Ludovic Courtès]

Hello Mathieu!

Mathieu Lirzin <mthl@gnu.org> skribis:

> After providing a first basic evaluation loop described in my previous
> mail, I have started thinking about the architecture of the global job
> evaluation/compilation process in order to identify what type of data
> structures would make sense.  It was not easy for me to reason about the
> different layers of current Hydra usage in Guix, so it took me 2 weeks
> to analyse it and provide a first decomposition of logic steps.  For now
> this decomposition is:
>
>   job-spec > job > build-result
>
> where:
>   
>   - 'job-spec' defines all the information required by Cuirass to get
>     the actual job definitions.  These information contains the
>     repository type and url, the file and procedure name which yields a
>     list of job, and the list of arguments passed to that procedure.
>
>   - 'job' contains the derivation file name which describe all the
>     dependencies.
>
>   - 'build-result' contains the output obtained when realizing/building
>     the derivation from a job + some logs.

Sounds good to me.

> 'job-spec' and 'job' are already implemented in Cuirass. However
> 'build-result' will require a database to be useful.  Since I have no
> experience with databases at all, this last week has been dedicated to
> learn more about them, play with SQL queries, and use Guile-dbi and
> Guile-sqlite3 bindings.  My plan is to use Sqlite first and eventually
> switch to Postgresql later when concurrent writers would be critical.

When and if.  :-)

I’m saying “if” because while Hydra uses the database for inter-process
communication, I think we may as well use a message passing toolkit such
as ZeroMQ rather than abuse the database.  But that’s for later!

Looks like a good plan anyway!

Thank you,
Ludo’.

Re: [GSoC] Continuous integration tool à la Hydra.

[Mathieu Lirzin]

Hello Guix!

Here is a third update on my GSoC project after the second month.

As a reminder, Hydra (https://nixos.org/hydra/) is a Nix-based
continuous build system which is used by Guix to compile packages on
different platforms and to distribute packages substitutes.  The aim of
this project is to replace Hydra with a more integrated software written
in Guile, named “Cuirass”.

Since my second update I have first fixed a major bug.  When building
different branches of Guix and evaluating package derivations the
results were always the same.  The issue was that the evaluations were
happening in the same Guile process which does not play well with module
changes.  To fix that I have used a separate process + pipe to get the
evaluation results.

In the last update, I have introduced usage of SRFI-9 records for
specifications, jobs, and builds.  While they are nice to organize data,
they have major drawbacks:

- not flexible when you want to informally create a container.
- require serialization when passing them throught pipes.

For those reasons I have switched to good ol' alists, which are flexible,
persistant, directly readable and don't require messing with load paths.
The downside is of course that there is no compile time checks when
manipulating data.

As stated in my last update.  I have been working on storing data in a
database.  For that I have decided to use Guile-sqlite3.  The principal
efforts have consist of using an external schema file and design it.

I have come up with this, but this will likely evolve in the future:

--8<---------------cut here---------------start------------->8---
CREATE TABLE Specifications (
  id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
  repo_name     TEXT NOT NULL,
  url           TEXT NOT NULL,
  load_path     TEXT NOT NULL,
  file          TEXT NOT NULL,
  proc          TEXT NOT NULL,
  arguments     TEXT NOT NULL,
  -- The following columns are optional.
  branch        TEXT,
  tag           TEXT,
  revision      TEXT
);

CREATE TABLE Evaluations (
  derivation    TEXT NOT NULL PRIMARY KEY,
  job_name      TEXT NOT NULL,
  specification INTEGER NOT NULL,
  FOREIGN KEY (specification) REFERENCES Specifications (id)
);

CREATE TABLE Builds (
  id              INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
  derivation      TEXT NOT NULL,
  log             TEXT NOT NULL,
  output          TEXT,         -- NULL if build failed
  FOREIGN KEY (derivation) REFERENCES Evaluations (derivation)
);
--8<---------------cut here---------------end--------------->8---

The next step will be to continue improving the database communication,
and start looking how to implement the HTTP API.

For those willing to follow my work, a Git repository is available here:

  https://notabug.org/mthl/cuirass

Everyone is of course welcome to provide any feedback.

Thanks.

-- 
Mathieu Lirzin

Re: [GSoC] Continuous integration tool à la Hydra.

[Ludovic Courtès]

Hello!

Mathieu Lirzin <mthl@gnu.org> skribis:

> Since my second update I have first fixed a major bug.  When building
> different branches of Guix and evaluating package derivations the
> results were always the same.  The issue was that the evaluations were
> happening in the same Guile process which does not play well with module
> changes.  To fix that I have used a separate process + pipe to get the
> evaluation results.

Sounds good.  Using a separate process makes sure what’s evaluated
doesn’t erroneously ends up using modules already loaded by the
evaluator, and vice versa.

> In the last update, I have introduced usage of SRFI-9 records for
> specifications, jobs, and builds.  While they are nice to organize data,
> they have major drawbacks:
>
> - not flexible when you want to informally create a container.
> - require serialization when passing them throught pipes.
>
> For those reasons I have switched to good ol' alists, which are flexible,
> persistant, directly readable and don't require messing with load paths.
> The downside is of course that there is no compile time checks when
> manipulating data.

OK.

> As stated in my last update.  I have been working on storing data in a
> database.  For that I have decided to use Guile-sqlite3.  The principal
> efforts have consist of using an external schema file and design it.
>
> I have come up with this, but this will likely evolve in the future:
>
> CREATE TABLE Specifications (
>   id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
>   repo_name     TEXT NOT NULL,
>   url           TEXT NOT NULL,
>   load_path     TEXT NOT NULL,
>   file          TEXT NOT NULL,
>   proc          TEXT NOT NULL,
>   arguments     TEXT NOT NULL,
>   -- The following columns are optional.
>   branch        TEXT,
>   tag           TEXT,
>   revision      TEXT
> );
>
> CREATE TABLE Evaluations (
>   derivation    TEXT NOT NULL PRIMARY KEY,
>   job_name      TEXT NOT NULL,
>   specification INTEGER NOT NULL,
>   FOREIGN KEY (specification) REFERENCES Specifications (id)
> );

An evaluation leads to several derivations (for Guix, roughly one
derivation per package and per system type), but the table above seems
to suggest that each evaluation is mapped to only one derivation?

> CREATE TABLE Builds (
>   id              INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
>   derivation      TEXT NOT NULL,
>   log             TEXT NOT NULL,
>   output          TEXT,         -- NULL if build failed
>   FOREIGN KEY (derivation) REFERENCES Evaluations (derivation)
> );
>
> The next step will be to continue improving the database communication,
> and start looking how to implement the HTTP API.

Cool!  It seems that the database already has or is about to have the
info we typically look for at
<https://hydra.gnu.org/jobset/gnu/core-updates>: which evaluations were
performed, what commit (specification) they correspond to, which
derivations they correspond to and whether they
succeeded/failed/aborted.

If the HTTP API exposes this info, possibly using the same API as Hydra
(see guix-hydra-jobset.el), that will cover an important part of our
daily needs.

> For those willing to follow my work, a Git repository is available here:
>
>   https://notabug.org/mthl/cuirass

… and the README has instructions on how to run it.  If anyone has spare
CPU cycles, run Cuirass, ‘guix publish’, and share.  :-)

I’m happy with the progress that’s been made, thank you!

Ludo’.

Re: [GSoC] Continuous integration tool à la Hydra.

[Mathieu Lirzin]

Hello,

ludo@gnu.org (Ludovic Courtès) writes:

> Mathieu Lirzin <mthl@gnu.org> skribis:
>
[...]
>> CREATE TABLE Specifications (
>>   id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
>>   repo_name     TEXT NOT NULL,
>>   url           TEXT NOT NULL,
>>   load_path     TEXT NOT NULL,
>>   file          TEXT NOT NULL,
>>   proc          TEXT NOT NULL,
>>   arguments     TEXT NOT NULL,
>>   -- The following columns are optional.
>>   branch        TEXT,
>>   tag           TEXT,
>>   revision      TEXT
>> );
>>
>> CREATE TABLE Evaluations (
>>   derivation    TEXT NOT NULL PRIMARY KEY,
>>   job_name      TEXT NOT NULL,
>>   specification INTEGER NOT NULL,
>>   FOREIGN KEY (specification) REFERENCES Specifications (id)
>> );
>
> An evaluation leads to several derivations (for Guix, roughly one
> derivation per package and per system type), but the table above seems
> to suggest that each evaluation is mapped to only one derivation?

In my "confused" mind each derivation was considered an individual
evaluation.  But as you pointed out this makes more sense to decouple
those.  Fixed in commit d493a58823aed8c556bf795d02207e57718b96c9

Thanks,

-- 
Mathieu Lirzin

Re: [GSoC] Continuous integration tool à la Hydra.

[Ludovic Courtès]

Mathieu Lirzin <mthl@gnu.org> skribis:

> Hello,
>
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mathieu Lirzin <mthl@gnu.org> skribis:
>>
> [...]
>>> CREATE TABLE Specifications (
>>>   id            INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
>>>   repo_name     TEXT NOT NULL,
>>>   url           TEXT NOT NULL,
>>>   load_path     TEXT NOT NULL,
>>>   file          TEXT NOT NULL,
>>>   proc          TEXT NOT NULL,
>>>   arguments     TEXT NOT NULL,
>>>   -- The following columns are optional.
>>>   branch        TEXT,
>>>   tag           TEXT,
>>>   revision      TEXT
>>> );
>>>
>>> CREATE TABLE Evaluations (
>>>   derivation    TEXT NOT NULL PRIMARY KEY,
>>>   job_name      TEXT NOT NULL,
>>>   specification INTEGER NOT NULL,
>>>   FOREIGN KEY (specification) REFERENCES Specifications (id)
>>> );
>>
>> An evaluation leads to several derivations (for Guix, roughly one
>> derivation per package and per system type), but the table above seems
>> to suggest that each evaluation is mapped to only one derivation?
>
> In my "confused" mind each derivation was considered an individual
> evaluation.  But as you pointed out this makes more sense to decouple
> those.  Fixed in commit d493a58823aed8c556bf795d02207e57718b96c9

Perfect, thanks!

Ludo’.

Re: [GSoC] Continuous integration tool à la Hydra.

[Florian Paul Schmidt]

[-- Attachment #1.1: Type: text/plain, Size: 2039 bytes --]

On 07/25/2016 11:36 PM, Ludovic Courtès wrote:

>> For those willing to follow my work, a Git repository is available here:
>>
>>   https://notabug.org/mthl/cuirass
> 
> … and the README has instructions on how to run it.  If anyone has spare
> CPU cycles, run Cuirass, ‘guix publish’, and share.  :-)

Sorry, if this mail comes around twice. Thunderbird crashed :(

Sadly the instructions to build it are a bit unclear. I tried:

fps@guix ~/cuirass$ guix environment --ad-hoc gcc gcc:lib automake guix
guile
guix environment: warning: ambiguous package specification `guile'
guix environment: warning: choosing guile-2.0.11 from
gnu/packages/guile.scm:131:2
fps@guix ~/cuirass [env]$
GUILE_LOAD_PATH=$GUILE_LOAD_PATH:/home/fps/guix/ ./configure




checking for a BSD-compatible install...
/run/current-system/profile/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p...
/run/current-system/profile/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... no
checking whether make supports nested variables... no
checking whether make supports nested variables... (cached) no
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for pkg-config... no
checking for GUILE... no
configure: error: in `/home/fps/cuirass':
configure: error: The pkg-config script could not be found or is too
old.  Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.

Alternatively, you may set the environment variables GUILE_CFLAGS
and GUILE_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

To get pkg-config, see <http://pkg-config.freedesktop.org/>.
See `config.log' for more details

How about including a guix package definition then we can easily build
it assuming "we" know how to do out-of-guix-tree package building :)

Regards,
Flo


-- 
https://fps.io


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [GSoC] Continuous integration tool à la Hydra.

[Mathieu Lirzin]

Hello,

Florian Paul Schmidt <mista.tapas@gmx.net> writes:

> On 07/25/2016 11:36 PM, Ludovic Courtès wrote:
>
>>> For those willing to follow my work, a Git repository is available here:
>>>
>>>   https://notabug.org/mthl/cuirass
>> 
>> … and the README has instructions on how to run it.  If anyone has spare
>> CPU cycles, run Cuirass, ‘guix publish’, and share.  :-)
>
> Sorry, if this mail comes around twice. Thunderbird crashed :(
>
> Sadly the instructions to build it are a bit unclear. I tried:
>
> fps@guix ~/cuirass$ guix environment --ad-hoc gcc gcc:lib automake guix
> guile
> guix environment: warning: ambiguous package specification `guile'
> guix environment: warning: choosing guile-2.0.11 from
> gnu/packages/guile.scm:131:2
> fps@guix ~/cuirass [env]$
> GUILE_LOAD_PATH=$GUILE_LOAD_PATH:/home/fps/guix/ ./configure
>
>
>
>
> checking for a BSD-compatible install...
> /run/current-system/profile/bin/install -c
> checking whether build environment is sane... yes
> checking for a thread-safe mkdir -p...
> /run/current-system/profile/bin/mkdir -p
> checking for gawk... gawk
> checking whether make sets $(MAKE)... no
> checking whether make supports nested variables... no
> checking whether make supports nested variables... (cached) no
> checking build system type... x86_64-unknown-linux-gnu
> checking host system type... x86_64-unknown-linux-gnu
> checking for pkg-config... no
> checking for GUILE... no
> configure: error: in `/home/fps/cuirass':
> configure: error: The pkg-config script could not be found or is too
> old.  Make sure it
> is in your PATH or set the PKG_CONFIG environment variable to the full
> path to pkg-config.
>
> Alternatively, you may set the environment variables GUILE_CFLAGS
> and GUILE_LIBS to avoid the need to call pkg-config.
> See the pkg-config man page for more details.
>
> To get pkg-config, see <http://pkg-config.freedesktop.org/>.
> See `config.log' for more details

I have tested successfully with the following command on a foreign
system:

  guix environment --ad-hoc automake pkg-config guile guix libgcrypt sqlite guile-sqlite3

Tell me if it works for you.

> How about including a guix package definition then we can easily build
> it assuming "we" know how to do out-of-guix-tree package building :)

It would indeed be nice to provide an easy way for Guix users to install
Cuirass.  IMHO package definitions meant as a development build tool is
confusing and should be avoided.  Nonetheless, I think it is useful to
document the previous 'guix environment ...' command in the README.

Thank you for your feedback.

-- 
Mathieu Lirzin

Re: [GSoC] Continuous integration tool à la Hydra.

[Ludovic Courtès]

Mathieu Lirzin <mthl@gnu.org> skribis:

> I have tested successfully with the following command on a foreign
> system:
>
>   guix environment --ad-hoc automake pkg-config guile guix libgcrypt sqlite guile-sqlite3
>
> Tell me if it works for you.
>
>> How about including a guix package definition then we can easily build
>> it assuming "we" know how to do out-of-guix-tree package building :)
>
> It would indeed be nice to provide an easy way for Guix users to install
> Cuirass.  IMHO package definitions meant as a development build tool is
> confusing and should be avoided.  Nonetheless, I think it is useful to
> document the previous 'guix environment ...' command in the README.

What about providing a ‘guix.scm’ file that people could pass to ‘guix
environment -l’ (instead of typing the long command above), and to ‘guix
package -f’ (info "(guix) Invoking guix package")?

Ludo’.

'guix environment' as a build tool. (was: [GSoC] Continuous integration tool à la Hydra.)

[Mathieu Lirzin]

ludo@gnu.org (Ludovic Courtès) writes:

> Mathieu Lirzin <mthl@gnu.org> skribis:
>
>> I have tested successfully with the following command on a foreign
>> system:
>>
>>   guix environment --ad-hoc automake pkg-config guile guix libgcrypt sqlite guile-sqlite3
>>
>> Tell me if it works for you.
>>
>>> How about including a guix package definition then we can easily build
>>> it assuming "we" know how to do out-of-guix-tree package building :)
>>
>> It would indeed be nice to provide an easy way for Guix users to install
>> Cuirass.  IMHO package definitions meant as a development build tool is
>> confusing and should be avoided.  Nonetheless, I think it is useful to
>> document the previous 'guix environment ...' command in the README.
>
> What about providing a ‘guix.scm’ file that people could pass to ‘guix
> environment -l’ (instead of typing the long command above), and to ‘guix
> package -f’ (info "(guix) Invoking guix package")?

'guix environment -l' uses a package definition.  To me this abstraction
doesn't fit well in a development context:

 - the origin hash doesn't make sense.
 - packages already included in Guix have redundant description and synopsis.
 - package definitions rely on Guix internals.

In fact what 'guix.scm' contains feels more like a "debian" directory or
a "PACKAGE.spec" file on steroid because of the "guix environment -l"
feature which derives from it but doesn't appear as first class.

An idea that I like better and is less invasive, would be to complement
bootstrap scripts with:

  ./bootstrap --with-guix

This command would:

- generate a guix-env script that wraps 'guix environment ...' if it
  doesn't exist.
- Invoke ./guix-env
- Invoke autoreconf -vfi

if the user wants to enter this environment Later it will have to invoke
'./guix-env'.

Some interesting things could be done beyond this, for example by using
per repository profiles that would save development environments.  This
would allow developpers to easily use different setups.

WDYT?

-- 
Mathieu Lirzin

Re: 'guix environment' as a build tool. (was: [GSoC] Continuous integration tool à la Hydra.)

[Thompson, David]

On Sat, Jul 30, 2016 at 10:05 PM, Mathieu Lirzin <mthl@gnu.org> wrote:
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mathieu Lirzin <mthl@gnu.org> skribis:
>>
>>> I have tested successfully with the following command on a foreign
>>> system:
>>>
>>>   guix environment --ad-hoc automake pkg-config guile guix libgcrypt sqlite guile-sqlite3
>>>
>>> Tell me if it works for you.
>>>
>>>> How about including a guix package definition then we can easily build
>>>> it assuming "we" know how to do out-of-guix-tree package building :)
>>>
>>> It would indeed be nice to provide an easy way for Guix users to install
>>> Cuirass.  IMHO package definitions meant as a development build tool is
>>> confusing and should be avoided.  Nonetheless, I think it is useful to
>>> document the previous 'guix environment ...' command in the README.
>>
>> What about providing a ‘guix.scm’ file that people could pass to ‘guix
>> environment -l’ (instead of typing the long command above), and to ‘guix
>> package -f’ (info "(guix) Invoking guix package")?
>
> 'guix environment -l' uses a package definition.  To me this abstraction
> doesn't fit well in a development context:

It *does* fit well.  This use-case is why I wrote 'guix environment'
in the first place.

>  - the origin hash doesn't make sense.

You don't need one.  Use local-file for the source field.

>  - packages already included in Guix have redundant description and synopsis.

I don't understand what this means.

>  - package definitions rely on Guix internals.

The package API rarely changes in practice.

> In fact what 'guix.scm' contains feels more like a "debian" directory or
> a "PACKAGE.spec" file on steroid because of the "guix environment -l"
> feature which derives from it but doesn't appear as first class.
>
> An idea that I like better and is less invasive, would be to complement
> bootstrap scripts with:
>
>   ./bootstrap --with-guix
>
> This command would:
>
> - generate a guix-env script that wraps 'guix environment ...' if it
>   doesn't exist.
> - Invoke ./guix-env
> - Invoke autoreconf -vfi
>
> if the user wants to enter this environment Later it will have to invoke
> './guix-env'.

This just makes things more inconvenient and limits potential utility.
You would lose the ability to tweak the dependency graph with custom
package recipes.  I do this frequently in my projects that use
bleeding edge features of other software that may not even have an
official release yet.  Also, with a package file, Guix users can
install the development snapshot with 'guix package -f' or simply
build it with 'guix build -f'.

> Some interesting things could be done beyond this, for example by using
> per repository profiles that would save development environments.  This
> would allow developpers to easily use different setups.

'guix environment' already serves this purpose.  I do want to extend
'guix environment' with a --save flag that will register the profile
it generates as a GC root so that it can be saved for future sessions
so that the environment doesn't need to be rebuilt every time the user
upgrades Guix.

Hopefully this clears things up.

- Dave

Re: 'guix environment' as a build tool.

[Mathieu Lirzin]

"Thompson, David" <dthompson2@worcester.edu> writes:

> On Sat, Jul 30, 2016 at 10:05 PM, Mathieu Lirzin <mthl@gnu.org> wrote:
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> Mathieu Lirzin <mthl@gnu.org> skribis:
>>>
>>>> I have tested successfully with the following command on a foreign
>>>> system:
>>>>
>>>>   guix environment --ad-hoc automake pkg-config guile guix libgcrypt sqlite guile-sqlite3
>>>>
>>>> Tell me if it works for you.
>>>>
>>>>> How about including a guix package definition then we can easily build
>>>>> it assuming "we" know how to do out-of-guix-tree package building :)
>>>>
>>>> It would indeed be nice to provide an easy way for Guix users to install
>>>> Cuirass.  IMHO package definitions meant as a development build tool is
>>>> confusing and should be avoided.  Nonetheless, I think it is useful to
>>>> document the previous 'guix environment ...' command in the README.
>>>
>>> What about providing a ‘guix.scm’ file that people could pass to ‘guix
>>> environment -l’ (instead of typing the long command above), and to ‘guix
>>> package -f’ (info "(guix) Invoking guix package")?
>>
>> 'guix environment -l' uses a package definition.  To me this abstraction
>> doesn't fit well in a development context:
>
> It *does* fit well.  This use-case is why I wrote 'guix environment'
> in the first place.

Obviously I disagree.

>>  - the origin hash doesn't make sense.
>
> You don't need one.  Use local-file for the source field.

I would be happy to, however I vaguely remember trying this without
success for Mcron.  Do you have an example to provide?

>>  - packages already included in Guix have redundant description and synopsis.
>
> I don't understand what this means.
>
>>  - package definitions rely on Guix internals.
>
> The package API rarely changes in practice.

This is relative, IME packages moving across modules is not an
exception.

>> In fact what 'guix.scm' contains feels more like a "debian" directory or
>> a "PACKAGE.spec" file on steroid because of the "guix environment -l"
>> feature which derives from it but doesn't appear as first class.
>>
>> An idea that I like better and is less invasive, would be to complement
>> bootstrap scripts with:
>>
>>   ./bootstrap --with-guix
>>
>> This command would:
>>
>> - generate a guix-env script that wraps 'guix environment ...' if it
>>   doesn't exist.
>> - Invoke ./guix-env
>> - Invoke autoreconf -vfi
>>
>> if the user wants to enter this environment Later it will have to invoke
>> './guix-env'.
>
> This just makes things more inconvenient and limits potential utility.
> You would lose the ability to tweak the dependency graph with custom
> package recipes.
>  I do this frequently in my projects that use bleeding edge features
> of other software that may not even have an official release yet.

Since this script is supposed to be wrapper around 'guix environment' I
don't understand how it could limit anything 'guix environment' could
do?

> Also, with a package file, Guix users can install the development
> snapshot with 'guix package -f' or simply build it with 'guix build
> -f'.

I am not sure what you mean by "development snapshot". If you an
arbitrary timely chosen commit like what is done for Haunt:

  (origin
    (method git-fetch)
    (uri (git-reference
         (url "git://dthompson.us/haunt.git")
         (commit "f0a7c2b14a201448432d3564d851ee0686d5b1b1")))
         (sha256
          (base32
           "1dnzsw18blhr8admw48zbl3ilz3iiqmb149i37y820h0imqfli0v"))))

This is not useful for me.  In most case I want 'guix build -f' to build
the current commit from the local repository.  If 'local-file' can work
then this is fine, but I have never achieved the expected result.

>> Some interesting things could be done beyond this, for example by using
>> per repository profiles that would save development environments.  This
>> would allow developpers to easily use different setups.
>
> 'guix environment' already serves this purpose.
>  I do want to extend 'guix environment' with a --save flag that will
> register the profile it generates as a GC root so that it can be saved
> for future sessions so that the environment doesn't need to be rebuilt
> every time the user upgrades Guix.

That would be nice.

> Hopefully this clears things up

Maybe I am misinterpreting, but this read a bit patronizing.  No need to
say that if this is the case I don't appreciate much.

-- 
Mathieu Lirzin

Re: 'guix environment' as a build tool.

[Ludovic Courtès]

Hello!

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Sat, Jul 30, 2016 at 10:05 PM, Mathieu Lirzin <mthl@gnu.org> wrote:
>> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>>> What about providing a ‘guix.scm’ file that people could pass to ‘guix
>>> environment -l’ (instead of typing the long command above), and to ‘guix
>>> package -f’ (info "(guix) Invoking guix package")?
>>
>> 'guix environment -l' uses a package definition.  To me this abstraction
>> doesn't fit well in a development context:
>
> It *does* fit well.  This use-case is why I wrote 'guix environment'
> in the first place.

[...]

>> if the user wants to enter this environment Later it will have to invoke
>> './guix-env'.
>
> This just makes things more inconvenient and limits potential utility.

That sounds harsh.

I don’t have a better answer for Mathieu other than ‘guix environment
-l’, and I think it does the job well.

But I also think that Mathieu’s concerns must not be dismissed.  For
instance, it’s true that some of the metadata in ‘package’ forms looks
irrelevant for the purposes of setting up a build environment—no big
deal, but still it doesn’t “feel” completely right.

Conversely, useful metadata is missing: for instance, I’d like to add
something that would allow me to specify the equivalent of ‘--network
--expose=$HOME/.gdbinit’ in development environments I use.

Perhaps the solution is to introduce a new way to declare development
environments?  It would be similar to ‘package’, but without ‘synopsis’,
‘description’, and a couple other things; it could have additional
fields to describe container setups and such likes; it would compile
down to a bag, just like packages.

What do you think?

Thanks,
Ludo’.

Re: 'guix environment' as a build tool.

[Thompson, David]

On Sun, Jul 31, 2016 at 9:55 AM, Ludovic Courtès <ludo@gnu.org> wrote:
> Hello!
>
> "Thompson, David" <dthompson2@worcester.edu> skribis:
>
>> On Sat, Jul 30, 2016 at 10:05 PM, Mathieu Lirzin <mthl@gnu.org> wrote:
>>> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>>>> What about providing a ‘guix.scm’ file that people could pass to ‘guix
>>>> environment -l’ (instead of typing the long command above), and to ‘guix
>>>> package -f’ (info "(guix) Invoking guix package")?
>>>
>>> 'guix environment -l' uses a package definition.  To me this abstraction
>>> doesn't fit well in a development context:
>>
>> It *does* fit well.  This use-case is why I wrote 'guix environment'
>> in the first place.
>
> [...]
>
>>> if the user wants to enter this environment Later it will have to invoke
>>> './guix-env'.
>>
>> This just makes things more inconvenient and limits potential utility.
>
> That sounds harsh.

I'm sorry.

> I don’t have a better answer for Mathieu other than ‘guix environment
> -l’, and I think it does the job well.
>
> But I also think that Mathieu’s concerns must not be dismissed.  For
> instance, it’s true that some of the metadata in ‘package’ forms looks
> irrelevant for the purposes of setting up a build environment—no big
> deal, but still it doesn’t “feel” completely right.

My intention was to define it just like a regular package so that
users can do whatever they want with it: build it, install it, or make
a development environment using it.

> Conversely, useful metadata is missing: for instance, I’d like to add
> something that would allow me to specify the equivalent of ‘--network
> --expose=$HOME/.gdbinit’ in development environments I use.
>
> Perhaps the solution is to introduce a new way to declare development
> environments?  It would be similar to ‘package’, but without ‘synopsis’,
> ‘description’, and a couple other things; it could have additional
> fields to describe container setups and such likes; it would compile
> down to a bag, just like packages.
>
> What do you think?

Hmm, that sounds like a good idea.  Maybe I'll try to write a
prototype sometime.  The downside of this method is that one could no
longer use the same expression as input to 'guix build -f' or 'guix
package -f'.

- Dave

Re: 'guix environment' as a build tool.

[Ludovic Courtès]

Hi!

"Thompson, David" <dthompson2@worcester.edu> skribis:

> On Sun, Jul 31, 2016 at 9:55 AM, Ludovic Courtès <ludo@gnu.org> wrote:
>> Conversely, useful metadata is missing: for instance, I’d like to add
>> something that would allow me to specify the equivalent of ‘--network
>> --expose=$HOME/.gdbinit’ in development environments I use.
>>
>> Perhaps the solution is to introduce a new way to declare development
>> environments?  It would be similar to ‘package’, but without ‘synopsis’,
>> ‘description’, and a couple other things; it could have additional
>> fields to describe container setups and such likes; it would compile
>> down to a bag, just like packages.
>>
>> What do you think?
>
> Hmm, that sounds like a good idea.  Maybe I'll try to write a
> prototype sometime.  The downside of this method is that one could no
> longer use the same expression as input to 'guix build -f' or 'guix
> package -f'.

Good point.  That would be a drawback of the approach, but probably we
can support both styles.

Thanks,
Ludo’.

Re: 'guix environment' as a build tool.

[Ludovic Courtès]

Hello,

Mathieu Lirzin <mthl@gnu.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mathieu Lirzin <mthl@gnu.org> skribis:
>>
>>> I have tested successfully with the following command on a foreign
>>> system:
>>>
>>>   guix environment --ad-hoc automake pkg-config guile guix libgcrypt sqlite guile-sqlite3
>>>
>>> Tell me if it works for you.
>>>
>>>> How about including a guix package definition then we can easily build
>>>> it assuming "we" know how to do out-of-guix-tree package building :)
>>>
>>> It would indeed be nice to provide an easy way for Guix users to install
>>> Cuirass.  IMHO package definitions meant as a development build tool is
>>> confusing and should be avoided.  Nonetheless, I think it is useful to
>>> document the previous 'guix environment ...' command in the README.
>>
>> What about providing a ‘guix.scm’ file that people could pass to ‘guix
>> environment -l’ (instead of typing the long command above), and to ‘guix
>> package -f’ (info "(guix) Invoking guix package")?
>
> 'guix environment -l' uses a package definition.  To me this abstraction
> doesn't fit well in a development context:
>
>  - the origin hash doesn't make sense.

Not a problem with ‘local-file’, as David wrote.

>  - packages already included in Guix have redundant description and synopsis.

Yeah, though for such packages, a guix.scm is typically less useful
since most of the time ‘guix environment PACKAGE’ is enough.

>  - package definitions rely on Guix internals.

I sympathize both with this and with what David wrote here.

I can sympathize with the idea that conceptually a package definition is
not quite the same thing as a development environment definition, in
practice ‘guix environment -l’ remains much better than the long command
line above.  :-)

> An idea that I like better and is less invasive, would be to complement
> bootstrap scripts with:
>
>   ./bootstrap --with-guix
>
> This command would:
>
> - generate a guix-env script that wraps 'guix environment ...' if it
>   doesn't exist.
> - Invoke ./guix-env
> - Invoke autoreconf -vfi
>
> if the user wants to enter this environment Later it will have to invoke
> './guix-env'.

I’m not convinced by generated scripts.

Now, we’re at a stage where everyone is welcome to explore their own
way.  Some may prefer a ‘guix.scm’ file, while others would prefer a
script that runs ‘guix environment --ad-hoc’.  With more experience,
maybe we can come up with a solution that better caters to everyone’s
needs.

Thanks,
Ludo’.

Re: [GSoC] Continuous integration tool à la Hydra.

[Florian Paul Schmidt]

[-- Attachment #1.1: Type: text/plain, Size: 4965 bytes --]

Hi!

On 07/29/2016 09:26 PM, Mathieu Lirzin wrote:
>> To get pkg-config, see <http://pkg-config.freedesktop.org/>.
>> See `config.log' for more details
> 
> I have tested successfully with the following command on a foreign
> system:
> 
>   guix environment --ad-hoc automake pkg-config guile guix libgcrypt sqlite guile-sqlite3
> 
> Tell me if it works for you.

Yes, that makes configure run fine. Thanks

This is what I get when I run it:

fps@guix ~/cuirass [env]$ ./pre-inst-env cuirass
--specifications=tests/hello-singleton.scm --database=test.db
Cloning into 'guix'...
remote: Counting objects: 101761, done.
remote: Compressing objects: 100% (28866/28866), done.
remote: Total 101761 (delta 74214), reused 99101 (delta 72173)
Receiving objects: 100% (101761/101761), 35.86 MiB | 6.27 MiB/s, done.
Resolving deltas: 100% (74214/74214), done.
Checking connectivity... done.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
+ exec autoreconf -vfi
./bootstrap: line 5: exec: autoreconf: not found
In execvp of ./configure: No such file or directory
In execvp of make: No such file or directory
;;; note: source file /home/fps/.cache/cuirass/guix/./guix/derivations.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/derivations.go
;;; note: source file /home/fps/.cache/cuirass/guix/./guix/store.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/store.go
;;; note: source file /home/fps/.cache/cuirass/guix/./guix/utils.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/utils.go
;;; note: source file /home/fps/.cache/cuirass/guix/./guix/combinators.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/combinators.go
;;; note: source file /home/fps/.cache/cuirass/guix/./guix/build/utils.scm

[.... more here ....]

;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/unrtf.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/unrtf.go
;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/uucp.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/uucp.go
;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/vtk.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/vtk.go
;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/wdiff.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/wdiff.go
;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/wine.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/wine.go
;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/xfce.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/xfce.go
;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/xnee.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/xnee.go
;;; note: source file
/home/fps/.cache/cuirass/guix/./gnu/packages/yubico.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/yubico.go
;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/zsh.scm
;;;       newer than compiled
/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/zsh.go
evaluate 'hello-2.10.x86_64-linux': 32.563 seconds
building /gnu/store/cw502hifb3q32h2x0gl1afgzmbx9295y-hello-2.10.drv...
/gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
^C
fps@guix ~/cuirass [env]$

And then:

fps@guix ~/cuirass [env]$  ./pre-inst-env cuirass --database=test.db
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
[....]

Is that the expected output/behaviour?

Regards,
Flo


-- 
https://fps.io


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [GSoC] Continuous integration tool à la Hydra.

[Mathieu Lirzin]

Hi,

Florian Paul Schmidt <mista.tapas@gmx.net> writes:

> On 07/29/2016 09:26 PM, Mathieu Lirzin wrote:
>>> To get pkg-config, see <http://pkg-config.freedesktop.org/>.
>>> See `config.log' for more details
>> 
>> I have tested successfully with the following command on a foreign
>> system:
>> 
>>   guix environment --ad-hoc automake pkg-config guile guix libgcrypt sqlite guile-sqlite3
>> 
>> Tell me if it works for you.
>
> Yes, that makes configure run fine. Thanks

Cool. :)

> This is what I get when I run it:
>
> fps@guix ~/cuirass [env]$ ./pre-inst-env cuirass
> --specifications=tests/hello-singleton.scm --database=test.db
> Cloning into 'guix'...
> remote: Counting objects: 101761, done.
> remote: Compressing objects: 100% (28866/28866), done.
> remote: Total 101761 (delta 74214), reused 99101 (delta 72173)
> Receiving objects: 100% (101761/101761), 35.86 MiB | 6.27 MiB/s, done.
> Resolving deltas: 100% (74214/74214), done.
> Checking connectivity... done.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> + exec autoreconf -vfi
> ./bootstrap: line 5: exec: autoreconf: not found
> In execvp of ./configure: No such file or directory
> In execvp of make: No such file or directory
> ;;; note: source file /home/fps/.cache/cuirass/guix/./guix/derivations.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/derivations.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./guix/store.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/store.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./guix/utils.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/utils.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./guix/combinators.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/combinators.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./guix/build/utils.scm
>
> [.... more here ....]
>
> ;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/unrtf.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/unrtf.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/uucp.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/uucp.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/vtk.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/vtk.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/wdiff.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/wdiff.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/wine.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/wine.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/xfce.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/xfce.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/xnee.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/xnee.go
> ;;; note: source file
> /home/fps/.cache/cuirass/guix/./gnu/packages/yubico.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/yubico.go
> ;;; note: source file /home/fps/.cache/cuirass/guix/./gnu/packages/zsh.scm
> ;;;       newer than compiled
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/zsh.go
> evaluate 'hello-2.10.x86_64-linux': 32.563 seconds
> building /gnu/store/cw502hifb3q32h2x0gl1afgzmbx9295y-hello-2.10.drv...
> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> ^C
> fps@guix ~/cuirass [env]$
>
> And then:
>
> fps@guix ~/cuirass [env]$  ./pre-inst-env cuirass --database=test.db
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> [....]
>
> Is that the expected output/behaviour?

Yes The output is not really explicit nor clean and should be filtered.
However this is working like intended.  To explain What is happening:

 > HEAD is now at XXX....

each time this line is displayed the repository is polled and if a
commit happened since, the evaluation/building process restarts which is
what the first lines was displaying.

> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
> [...]
> /gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/zsh.go
> evaluate 'hello-2.10.x86_64-linux': 32.563 seconds
> building /gnu/store/cw502hifb3q32h2x0gl1afgzmbx9295y-hello-2.10.drv...
> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10

If you have any suggestion to improve Cuirass, feel free to share.
Thank you very much for going this far.  :)

-- 
Mathieu Lirzin

Re: [GSoC] Continuous integration tool à la Hydra.

[Florian Paul Schmidt]

[-- Attachment #1: Type: text/plain, Size: 6529 bytes --]

Hi again. Sorry I'm on vacation in Sweden so the formatting of this mail suffered ;) The one thing that had me worried was the errors right after the git pull..

Regards,
Flo

On July 31, 2016 2:03:59 PM GMT+02:00, Mathieu Lirzin <mthl@gnu.org> wrote:
>Hi,
>
>Florian Paul Schmidt <mista.tapas@gmx.net> writes:
>
>> On 07/29/2016 09:26 PM, Mathieu Lirzin wrote:
>>>> To get pkg-config, see <http://pkg-config.freedesktop.org/>.
>>>> See `config.log' for more details
>>> 
>>> I have tested successfully with the following command on a foreign
>>> system:
>>> 
>>>   guix environment --ad-hoc automake pkg-config guile guix libgcrypt
>sqlite guile-sqlite3
>>> 
>>> Tell me if it works for you.
>>
>> Yes, that makes configure run fine. Thanks
>
>Cool. :)
>
>> This is what I get when I run it:
>>
>> fps@guix ~/cuirass [env]$ ./pre-inst-env cuirass
>> --specifications=tests/hello-singleton.scm --database=test.db
>> Cloning into 'guix'...
>> remote: Counting objects: 101761, done.
>> remote: Compressing objects: 100% (28866/28866), done.
>> remote: Total 101761 (delta 74214), reused 99101 (delta 72173)
>> Receiving objects: 100% (101761/101761), 35.86 MiB | 6.27 MiB/s,
>done.
>> Resolving deltas: 100% (74214/74214), done.
>> Checking connectivity... done.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> + exec autoreconf -vfi
>> ./bootstrap: line 5: exec: autoreconf: not found
>> In execvp of ./configure: No such file or directory
>> In execvp of make: No such file or directory
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./guix/derivations.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/derivations.go
>> ;;; note: source file /home/fps/.cache/cuirass/guix/./guix/store.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/store.go
>> ;;; note: source file /home/fps/.cache/cuirass/guix/./guix/utils.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/utils.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./guix/combinators.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/guix/combinators.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./guix/build/utils.scm
>>
>> [.... more here ....]
>>
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./gnu/packages/unrtf.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/unrtf.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./gnu/packages/uucp.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/uucp.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./gnu/packages/vtk.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/vtk.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./gnu/packages/wdiff.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/wdiff.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./gnu/packages/wine.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/wine.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./gnu/packages/xfce.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/xfce.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./gnu/packages/xnee.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/xnee.go
>> ;;; note: source file
>> /home/fps/.cache/cuirass/guix/./gnu/packages/yubico.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/yubico.go
>> ;;; note: source file
>/home/fps/.cache/cuirass/guix/./gnu/packages/zsh.scm
>> ;;;       newer than compiled
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/zsh.go
>> evaluate 'hello-2.10.x86_64-linux': 32.563 seconds
>> building
>/gnu/store/cw502hifb3q32h2x0gl1afgzmbx9295y-hello-2.10.drv...
>> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> ^C
>> fps@guix ~/cuirass [env]$
>>
>> And then:
>>
>> fps@guix ~/cuirass [env]$  ./pre-inst-env cuirass --database=test.db
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> [....]
>>
>> Is that the expected output/behaviour?
>
>Yes The output is not really explicit nor clean and should be filtered.
>However this is working like intended.  To explain What is happening:
>
> > HEAD is now at XXX....
>
>each time this line is displayed the repository is polled and if a
>commit happened since, the evaluation/building process restarts which
>is
>what the first lines was displaying.
>
>> HEAD is now at 2fa66f4 gnu: mutt: Update to 1.6.2.
>> [...]
>>
>/gnu/store/044sj7g4gxmy9garq6ii5aips4jn6hr6-profile/share/guile/site/2.0/gnu/packages/zsh.go
>> evaluate 'hello-2.10.x86_64-linux': 32.563 seconds
>> building
>/gnu/store/cw502hifb3q32h2x0gl1afgzmbx9295y-hello-2.10.drv...
>> /gnu/store/zby49aqfbd9w9br4l52mvb3y6f9vfv22-hello-2.10
>
>If you have any suggestion to improve Cuirass, feel free to share.
>Thank you very much for going this far.  :)
>
>-- 
>Mathieu Lirzin

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[-- Attachment #2: Type: text/html, Size: 7816 bytes --]

Re: [GSoC] Continuous integration tool à la Hydra.

[David Craven]

Hi Mathieu,

Are you already working on a package/service for guix?

I'd like to deploy it to my server to avoid silly mistakes
like in my last two pushes to master...

Does it rebuild from commit individually or only HEAD?
This would be nice to check for rebase mistakes where
HEAD builds but a commit doesn't.

Does it already support building packages for all guix
supported systems?

```
(define (local-file file)
;; In the common case jobs will be defined relative to the repository.
;; However for testing purpose use local gnu-system.scm instead.
(string-append (dirname (current-filename)) "/" file))
(define hello-master
`((#:name . "guix")
(#:url . "git://git.savannah.gnu.org/guix.git")
(#:load-path . ".")
(#:file . ,(local-file "gnu-system.scm"))
(#:proc . hydra-jobs)
(#:arguments (subset . "hello"))
(#:branch . "master")))
(list hello-master)
```

Can I also define custom jobs like running guix --rounds=2 and
guix lint?

Thank you,
David

Re: [GSoC] Continuous integration tool à la Hydra.

[Mark H Weaver]

David Craven <david@craven.ch> writes:

> Hi Mathieu,
>
> Are you already working on a package/service for guix?
>
> I'd like to deploy it to my server to avoid silly mistakes
> like in my last two pushes to master...
>
> Does it rebuild from commit individually or only HEAD?
> This would be nice to check for rebase mistakes where
> HEAD builds but a commit doesn't.

We don't have enough build farm capacity to make it practical to check
every commit.  It might make sense to include an option for this, but
most of the time we'd need to disable it.

      Mark

Re: [GSoC] Continuous integration tool à la Hydra.

[David Craven]

> We don't have enough build farm capacity to make it practical to check
> every commit.  It might make sense to include an option for this, but
> most of the time we'd need to disable it.

I understand that, but developers should have a simple way to check
their own work. Hopefully commits made to master are good commits
and hydra only has to rebuild from HEAD.

Re: [GSoC] Continuous integration tool à la Hydra.

[David Craven]

> I understand that, but developers should have a simple way to check
> their own work. Hopefully commits made to master are good commits
> and hydra only has to rebuild from HEAD.

It would also be nice if developers could easily hook up their servers to
the guix build farm, if I do run my own continuous integration system,
my server would be idling most of the time anyway, so it would be
nice to use that time.

Re: [GSoC] Continuous integration tool à la Hydra.

[Mathieu Lirzin]

Hello David,

David Craven <david@craven.ch> writes:

> Are you already working on a package/service for guix?

This would indeed be the goal, however I am not there yet.

> I'd like to deploy it to my server to avoid silly mistakes
> like in my last two pushes to master...
>
> Does it rebuild from commit individually or only HEAD?
> This would be nice to check for rebase mistakes where
> HEAD builds but a commit doesn't.

That would make sense as an option, however the current model is
simplistic and only evaluates the new HEAD commit.

> Does it already support building packages for all guix
> supported systems?

For now, you have to modify the 'gnu-system.scm' file to your needs to
achieve that.  basically you need to define another "subset" case in
'gnu-system.scm' and adjust the '#:arguments' value in the job
specification.

> ```
> (define (local-file file)
> ;; In the common case jobs will be defined relative to the repository.
> ;; However for testing purpose use local gnu-system.scm instead.
> (string-append (dirname (current-filename)) "/" file))
> (define hello-master
> `((#:name . "guix")
> (#:url . "git://git.savannah.gnu.org/guix.git")
> (#:load-path . ".")
> (#:file . ,(local-file "gnu-system.scm"))
> (#:proc . hydra-jobs)
> (#:arguments (subset . "hello"))
> (#:branch . "master")))
> (list hello-master)
> ```
>
> Can I also define custom jobs like running guix --rounds=2 and
> guix lint?

Likewise.  However this would require more work.  :)

I guess the future approach will be to define both the scheme code
running on the client side and on the build (remote) side in the job
specification.  I mean something similar to what Guix is doing for its
package definitions.  The details are not defined yet.

Thanks for your questions.

-- 
Mathieu Lirzin