cuirass evaluate

cuirass evaluate

[Mathieu Othacehe]

Hi,

Now that Cuirass uses (guix git), I'm trying to robustify specification
evaluation. Currently Cuirass calls a binary called "evaluate" to get a
job list from a specification.

I don't like the idea of this extra "evaluate" script because :

* We have to give evaluate almost all arguments given to Cuirass
  (load-path, package-path, cachedir, spec, database).
* The script is made available for the user but it's very unclear how to
  call it. The help says : "Usage evaluate FILE" which is wrong (5
  arguments are expected).

So my question is, is this a strong requirement to have a separate
script to parse specification, or can I put this stuff back in main
Cuirass program ?

Thanks,

Mathieu

Re: cuirass evaluate

[Ludovic Courtès]

Hi Mathieu,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> Now that Cuirass uses (guix git), I'm trying to robustify specification
> evaluation. Currently Cuirass calls a binary called "evaluate" to get a
> job list from a specification.
>
> I don't like the idea of this extra "evaluate" script because :
>
> * We have to give evaluate almost all arguments given to Cuirass
>   (load-path, package-path, cachedir, spec, database).
> * The script is made available for the user but it's very unclear how to
>   call it. The help says : "Usage evaluate FILE" which is wrong (5
>   arguments are expected).
>
> So my question is, is this a strong requirement to have a separate
> script to parse specification, or can I put this stuff back in main
> Cuirass program ?

It’s a requirement because the evaluation process has side effects on
the Guile that runs it; for instance, it loads tons of modules in it.
Also, the evaluation process may need to load modules that have the same
name as currently-loaded modules, but different content—and Guile
supports only one module with a given name.

That said, this program should be moved to $libexecdir/cuirass/VERSION.

HTH,
Ludo’.

Re: cuirass evaluate

[Mathieu Othacehe]

Hi Ludo,

> It’s a requirement because the evaluation process has side effects on
> the Guile that runs it; for instance, it loads tons of modules in it.

Ok thanks for the explanation.

I just sent a patch to add minimal error checking on "evaluate" errors.

Mathieu

Re: cuirass evaluate

[Mathieu Othacehe]

Hi Ludo,

A cuirass question :

In src/cuirass/base.scm, you wrote the following comment :

--8<---------------cut here---------------start------------->8---
  ;; Register the results in the database.
  ;; XXX: The 'build-derivations' call is blocking so we end updating the
  ;; database potentially long after things have been built.
--8<---------------cut here---------------end--------------->8---

How would it be possible to be notified when a build succeeds, to update
the database asap ?

Thanks,

Mathieu

Re: cuirass evaluate

[Ludovic Courtès]

Hi,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> A cuirass question :
>
> In src/cuirass/base.scm, you wrote the following comment :
>
>   ;; Register the results in the database.
>   ;; XXX: The 'build-derivations' call is blocking so we end updating the
>   ;; database potentially long after things have been built.
>
> How would it be possible to be notified when a build succeeds, to update
> the database asap ?

Unfortunately, the current daemon protocol makes it hard to get such
notifications, unless you parse its output (the “@ build-succeeded”
lines) like I did in ‘wip-ui’.

Perhaps we’ll have to do this parsing anyway, or just change the
protocol altogether and have a “channel” mechanism like SSH to multiplex
the number of communication channels on one connection.  reepca, if you
read this, that’s something to keep in mind.  :-)

Alternately, Cuirass could also periodically call ‘valid-path?’ for all
the outputs of the pending builds (in a separate thread; note that the
daemon connection cannot be shared among threads.)

HTH,
Ludo’.

Re: cuirass evaluate

[Mathieu Othacehe]

Hi Ludo,

> Unfortunately, the current daemon protocol makes it hard to get such
> notifications, unless you parse its output (the “@ build-succeeded”
> lines) like I did in ‘wip-ui’.
>
> Perhaps we’ll have to do this parsing anyway, or just change the
> protocol altogether and have a “channel” mechanism like SSH to multiplex
> the number of communication channels on one connection.  reepca, if you
> read this, that’s something to keep in mind.  :-)
>
> Alternately, Cuirass could also periodically call ‘valid-path?’ for all
> the outputs of the pending builds (in a separate thread; note that the
> daemon connection cannot be shared among threads.)

Ok, the solution you implemented in wip-ui seems viable.

For now I'm implementing basic web API in cuirass :

* api/build/:build
* build/:build/log/raw
* api/latestbuilds

This allows emacs-guix to query cuirass as per hydra. The "queue" api
will be implementable when we will support build start/stop/failure
detection.

Thanks,

Mathieu

Re: cuirass evaluate

[Ludovic Courtès]

Hello,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> Unfortunately, the current daemon protocol makes it hard to get such
>> notifications, unless you parse its output (the “@ build-succeeded”
>> lines) like I did in ‘wip-ui’.
>>
>> Perhaps we’ll have to do this parsing anyway, or just change the
>> protocol altogether and have a “channel” mechanism like SSH to multiplex
>> the number of communication channels on one connection.  reepca, if you
>> read this, that’s something to keep in mind.  :-)
>>
>> Alternately, Cuirass could also periodically call ‘valid-path?’ for all
>> the outputs of the pending builds (in a separate thread; note that the
>> daemon connection cannot be shared among threads.)
>
> Ok, the solution you implemented in wip-ui seems viable.
>
> For now I'm implementing basic web API in cuirass :
>
> * api/build/:build
> * build/:build/log/raw
> * api/latestbuilds
>
> This allows emacs-guix to query cuirass as per hydra. The "queue" api
> will be implementable when we will support build start/stop/failure
> detection.

Sounds good.  Let us know if you stumble upon other issues.

Thanks,
Ludo.’