[bug#27876] [PATCH] cuirass: add Hydra compatible HTTP API.

[ludo@gnu.org (Ludovic Courtès)]

Hey!

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> * bin/evaluate.in (fill-job): New procedure.
> (main): Use it to fill informations (nix-name, system) that will later be
> added to database.
> * doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP API.
> (Database)[Derivation]: Add system and nix_name fields.
> (Database)[Builds]: Add id, status, timestamp, starttime and stoptime
> fields. Remove output field.
> (Database)[Outputs]: New table describing the build outputs.
> * src/cuirass/base.scm (build-packages): Add new fields to build object before
> adding it to database.
> * src/cuirass/database.scm (db-get-build, db-get-builds): New procedures to get
> a build by id from database and a list of builds using filter parameters
> respectively.
> * src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rename it
>   object->json-string.
> (object->json-scm): Move it utils.scm.
> (handle-*-request): New helpers procedures.
> (request-parameters): New procedure to parse a request query.
> (url-handler): Add new API's.
> * src/cuirass/utils.scm (object->json-scm, object->json-string): Exported
> procedures moved from http.scm.
> * src/schema.sql (Outputs) : New table.
> (Derivations): Add system and nix_name columns.
> (Builds): Remove output column and add id, status, timestamp, starttime and
> stoptime columns.
> ---
>
> Hi,
>
> Here's a first draft adding partial support for Hydra API in Cuirass.
> It can be tested using curl or, better, with Emacs Guix.
>
> The following elisp will change hydra url to a local running Cuirass server.
>
> (setq guix-hydra-url "http://127.0.0.1:8080/")
>
> Then, it should be possible to use M-x guix-hydra-latest-builds.

Woow!

> The commands guix-hydra-jobsets and guix-hydra-queued-builds won't function
> because the associated API's are not implemented yet.
>
> There's a problem with /build/:build-id/log/raw API because it is trying to fork
> while multiple threads are running (because of decompressed-port function). It seems
> to work but a warning message is printed.

I think libbz2 and libz have the same API, so we could probably adapt
(guix zlib) so that it supports libbz2 as well, which would solve this
problem.

That said, there’s only one place where we work, which is where we spawn
the ‘evaluate’ command.  It may be that spawning it with ‘open-pipe’
from (ice-9 popen) would sidestep the problem because ‘open-pipe’ is
specifically written to permit this.

Some comments follow.

> +@section API description.
> +@cindex description, json
> +
> +@subsection Build informations.

“Information” is always singular (uncountable).  Also, no trailing
period in section names.

> +@example
> +0 -> succeded
            ^
“succeeded”

> +@subsection Build raw log output.
> +
> +It is possible to ask Cuirass for the raw build output log with the API
> +@code{"/build/:build-id/log/raw"} where @code{build-id} is the
                 ^                         ^^^^^^
Nitpick: we should probably write @var{build-id} (and remove the colon).

> +unique id associated to the build in database.
> +
> +The output is a raw text, for example :
> +
> +@example
> +$ curl http://localhost:8080/build/2/log/raw

Initially I was thinking about having something similar in ‘guix
publish’, but that would not include the “/build” part anyway, so it’s
good to have this in Cuirass.

> +(define (db-get-builds db filters)
> +  "Retrieve all builds in database DB which are matched by given FILTERS.
> +FILTERS is an assoc list which possible keys are 'project | 'jobset | 'job |
> +'system | 'nr."

Perhaps the database part of this change could have been a separate
commit (first commit: store the relevant info in the DB and provide
procedures to access it; second commit: implement the HTTP API to access
the DB.)

> +    (("build" build)
> +     (let ((hydra-build (handle-build-request db build)))
> +       (if hydra-build
> +           (respond-json (object->json-string hydra-build))
> +           (respond-build-not-found build))))
> +    (("build" build "log" "raw")
> +     (let ((log-response (handle-log-request db build)))
> +       (if log-response
> +           (respond-text log-response)
> +           (respond-build-not-found build))))
> +    (("api" "latestbuilds")
> +     (let* ((params (request-parameters request))
> +            ;; 'nr parameter is mandatory to limit query size.
> +            (valid-params? (assq-ref params 'nr)))
> +       (if valid-params?
> +           (respond-json (object->json-string
> +                          (handle-builds-request db params)))
> +           (respond-json-with-error 500 "Parameter not defined!"))))

Nice.  :-)

I think it would be nice to have a couple of tests for the HTTP API.
There are helpers in tests/publish.scm and (guix tests http) that could
probably be borrowed here.  WDYT?

Anyway, awesome work!  It’s a really important gap that you’re filling
here.

In the future it would be nice to have an API to add jobsets, trigger an
evaluation, things like that.  That’s probably more difficult though
because we’ll need an authentication mechanism.

Also we should consider Fiberizing the whole thing eventually, so that
Cuirass can actually perform all its activities concurrently.

Thanks!

Ludo’.