From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40520) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dcC8z-0007qC-Iz for guix-patches@gnu.org; Mon, 31 Jul 2017 10:58:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dcC8w-0007te-EW for guix-patches@gnu.org; Mon, 31 Jul 2017 10:58:05 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:32808) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dcC8w-0007tL-A8 for guix-patches@gnu.org; Mon, 31 Jul 2017 10:58:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dcC8v-00043P-W5 for guix-patches@gnu.org; Mon, 31 Jul 2017 10:58:02 -0400 Subject: [bug#27876] [PATCH] cuirass: add Hydra compatible HTTP API. Resent-Message-ID: From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) References: <20170730100759.17734-1-m.othacehe@gmail.com> Date: Mon, 31 Jul 2017 16:57:08 +0200 In-Reply-To: <20170730100759.17734-1-m.othacehe@gmail.com> (Mathieu Othacehe's message of "Sun, 30 Jul 2017 12:07:59 +0200") Message-ID: <87ini84ptn.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: Mathieu Othacehe Cc: 27876@debbugs.gnu.org Hey! Mathieu Othacehe skribis: > * bin/evaluate.in (fill-job): New procedure. > (main): Use it to fill informations (nix-name, system) that will later be > added to database. > * doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP A= PI. > (Database)[Derivation]: Add system and nix_name fields. > (Database)[Builds]: Add id, status, timestamp, starttime and stoptime > fields. Remove output field. > (Database)[Outputs]: New table describing the build outputs. > * src/cuirass/base.scm (build-packages): Add new fields to build object b= efore > adding it to database. > * src/cuirass/database.scm (db-get-build, db-get-builds): New procedures = to get > a build by id from database and a list of builds using filter parameters > respectively. > * src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rena= me it > object->json-string. > (object->json-scm): Move it utils.scm. > (handle-*-request): New helpers procedures. > (request-parameters): New procedure to parse a request query. > (url-handler): Add new API's. > * src/cuirass/utils.scm (object->json-scm, object->json-string): Exported > procedures moved from http.scm. > * src/schema.sql (Outputs) : New table. > (Derivations): Add system and nix_name columns. > (Builds): Remove output column and add id, status, timestamp, starttime a= nd > stoptime columns. > --- > > Hi, > > Here's a first draft adding partial support for Hydra API in Cuirass. > It can be tested using curl or, better, with Emacs Guix. > > The following elisp will change hydra url to a local running Cuirass serv= er. > > (setq guix-hydra-url "http://127.0.0.1:8080/") > > Then, it should be possible to use M-x guix-hydra-latest-builds. Woow! > The commands guix-hydra-jobsets and guix-hydra-queued-builds won't functi= on > because the associated API's are not implemented yet. > > There's a problem with /build/:build-id/log/raw API because it is trying = to fork > while multiple threads are running (because of decompressed-port function= ). It seems > to work but a warning message is printed. I think libbz2 and libz have the same API, so we could probably adapt (guix zlib) so that it supports libbz2 as well, which would solve this problem. That said, there=E2=80=99s only one place where we work, which is where we = spawn the =E2=80=98evaluate=E2=80=99 command. It may be that spawning it with = =E2=80=98open-pipe=E2=80=99 from (ice-9 popen) would sidestep the problem because =E2=80=98open-pipe=E2= =80=99 is specifically written to permit this. Some comments follow. > +@section API description. > +@cindex description, json > + > +@subsection Build informations. =E2=80=9CInformation=E2=80=9D is always singular (uncountable). Also, no t= railing period in section names. > +@example > +0 -> succeded ^ =E2=80=9Csucceeded=E2=80=9D > +@subsection Build raw log output. > + > +It is possible to ask Cuirass for the raw build output log with the API > +@code{"/build/:build-id/log/raw"} where @code{build-id} is the ^ ^^^^^^ Nitpick: we should probably write @var{build-id} (and remove the colon). > +unique id associated to the build in database. > + > +The output is a raw text, for example : > + > +@example > +$ curl http://localhost:8080/build/2/log/raw Initially I was thinking about having something similar in =E2=80=98guix publish=E2=80=99, but that would not include the =E2=80=9C/build=E2=80=9D p= art anyway, so it=E2=80=99s good to have this in Cuirass. > +(define (db-get-builds db filters) > + "Retrieve all builds in database DB which are matched by given FILTERS. > +FILTERS is an assoc list which possible keys are 'project | 'jobset | 'j= ob | > +'system | 'nr." Perhaps the database part of this change could have been a separate commit (first commit: store the relevant info in the DB and provide procedures to access it; second commit: implement the HTTP API to access the DB.) > + (("build" build) > + (let ((hydra-build (handle-build-request db build))) > + (if hydra-build > + (respond-json (object->json-string hydra-build)) > + (respond-build-not-found build)))) > + (("build" build "log" "raw") > + (let ((log-response (handle-log-request db build))) > + (if log-response > + (respond-text log-response) > + (respond-build-not-found build)))) > + (("api" "latestbuilds") > + (let* ((params (request-parameters request)) > + ;; 'nr parameter is mandatory to limit query size. > + (valid-params? (assq-ref params 'nr))) > + (if valid-params? > + (respond-json (object->json-string > + (handle-builds-request db params))) > + (respond-json-with-error 500 "Parameter not defined!")))) Nice. :-) I think it would be nice to have a couple of tests for the HTTP API. There are helpers in tests/publish.scm and (guix tests http) that could probably be borrowed here. WDYT? Anyway, awesome work! It=E2=80=99s a really important gap that you=E2=80= =99re filling here. In the future it would be nice to have an API to add jobsets, trigger an evaluation, things like that. That=E2=80=99s probably more difficult though because we=E2=80=99ll need an authentication mechanism. Also we should consider Fiberizing the whole thing eventually, so that Cuirass can actually perform all its activities concurrently. Thanks! Ludo=E2=80=99.