From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36889) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dcdD4-0003kw-5u for guix-patches@gnu.org; Tue, 01 Aug 2017 15:52:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dcdD1-0008LA-1o for guix-patches@gnu.org; Tue, 01 Aug 2017 15:52:06 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:34630) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dcdD0-0008L3-Tg for guix-patches@gnu.org; Tue, 01 Aug 2017 15:52:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dcdD0-0001J0-NT for guix-patches@gnu.org; Tue, 01 Aug 2017 15:52:02 -0400 Subject: [bug#27876] [PATCH v2 2/3] cuirass: add Hydra compatible HTTP API. Resent-Message-ID: From: Mathieu Othacehe Date: Tue, 1 Aug 2017 21:51:23 +0200 Message-Id: <20170801195124.7030-2-m.othacehe@gmail.com> In-Reply-To: <20170801195124.7030-1-m.othacehe@gmail.com> References: <20170801195124.7030-1-m.othacehe@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+kyle=kyleam.com@gnu.org Sender: "Guix-patches" To: 27876@debbugs.gnu.org * doc/cuirass.texi (Sections)[Web API]: New section describing the HTTP API. * src/cuirass/http.scm (spec->json-string): Move it to utils.scm and rename it object->json-string. (object->json-scm): Move it utils.scm. (handle-*-request): New helpers procedures. (request-parameters): New procedure to parse a request query. (url-handler): Add new API's. * src/cuirass/utils.scm (object->json-scm, object->json-string): Exported procedures moved from http.scm. --- doc/cuirass.texi | 191 +++++++++++++++++++++++++++++++++++++++++++++++ src/cuirass/database.scm | 29 +++---- src/cuirass/http.scm | 150 ++++++++++++++++++++++++++++++------- src/cuirass/utils.scm | 22 +++++- 4 files changed, 350 insertions(+), 42 deletions(-) diff --git a/doc/cuirass.texi b/doc/cuirass.texi index 443b53c..bac2e54 100644 --- a/doc/cuirass.texi +++ b/doc/cuirass.texi @@ -57,6 +57,7 @@ Tutorial sections: Reference sections: * Invocation:: How to run Cuirass. * Database:: About the database schema. +* Web API:: Description of the Web API. * Contributing:: Your help needed! * GNU Free Documentation License:: The license of this manual. @@ -380,8 +381,198 @@ This text field holds the name of the output. This text field holds the path of the output. @end table + +@c ********************************************************************* +@node Web API +@chapter Web API +@cindex web api + +Cuirass web API is derived from Hydra one, see @url{https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml, Hydra API description}. + +For now only a subset of this API is implemented. + +@section API description +@cindex description, json + +@subsection Build information + +It is possible to query Cuirass web server for build informations. The +dedicated API is "/build/@var{build-id}" where @var{build-id} is the +unique id associated to the build in database. + +For instance, querying a local Cuirass web server can be done with +@code{curl} and @code{jq} to format the JSON response : + +@example +$ curl -s "http://localhost:8080/build/2" | jq + +@{ + "id": 2, + "project": "guix", + "jobset": "master", + "job": "acpica-20150410-job", + "timestamp": 1501347493, + "starttime": 1501347493, + "stoptime": 1501347493, + "buildoutputs": @{ + "out": @{ + "path": "/gnu/store/6g3njhfzqpdm335s7qhvmwvs5l7gcbq1-acpica-20150410" + @} + @}, + "system": "x86_64-linux", + "nixname": "acpica-20150410", + "buildstatus": 0, + "busy": 0, + "priority": 0, + "finished": 1, + "buildproducts": null, + "releasename": null, + "buildinputs_builds": null +@} +@end example + +If requested @var{build-id} is not known, the HTTP code 404 is +answered with a JSON error message. For example : + +@example +$ curl -s "http://localhost:8080/build/fff" + +@{"error" : "Build with ID fff doesn't exist."@} +@end example + +The nominal output is a JSON object whose fields are described +hereafter. + +@table @code +@item id +The unique build id. + +@item project +The associated specification name, as a string. + +@item jobset +The associated specification branch, as a string. + +@item job +The associated job-name, as a string. + +@item timestamp +Timestamp taken at build creation time. + +@item starttime +Timestamp taken at build start time. + +@item stoptime +Timestamp taken at build stop time. + +@item buildoutputs +Build outputs as a JSON object. The keys names are referring to the +eventual output names. The associated value is another JSON object which +only key is @code{path}. @code{path} value is the output directory in +store as a string. + +@item system +System name of the build, as a string. + +@item nixname +Derivation name, as a string. + +@item buildstatus +Build status, as an integer. Possible values are : + +@example +0 -> succeeded +1 -> failed +2 -> failed dependency +3 -> failed other +4 -> cancelled +@end example + +@item busy +Whether the build is pending, as an integer (not implemented yet). + +@item priority +Build priority, as an integer (not implemented yet). + +@item finished +Build finished, as an integer (not implemented yet : always 1). + +@item buildproducts +Build products in store as a JSON object (not implemented yet). + +@item releasename +Unknown, not implemented yet. + +@item buildinputs_builds +Inputs used for the build, as a JSON object (not implemented yet). + +@end table + +@subsection Build raw log output + +It is possible to ask Cuirass for the raw build output log with the API +"/build/@var{build-id}/log/raw" where @var{build-id} is the +unique id associated to the build in database. + +The output is a raw text, for example : + +@example +$ curl http://localhost:8080/build/2/log/raw + +starting phase `set-SOURCE-DATE-EPOCH' +phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds +starting phase `set-paths' +... +@end example + +If requested @var{build-id} is not known, the HTTP code 404 is +answered with a JSON error message. For example : + +@example +$ curl -s "http://localhost:8080/build/fff/log/raw" + +@{"error" : "Build with ID fff doesn't exist."@} +@end example + +@subsection Latest builds + +The list of latest builds can be obtained with the API +"/api/latestbuilds". The output is a JSON array of +builds. Builds are represented as in "/build/@var{build-id} API. + +This request accepts a mandatory parameter and multiple optional ones. + +@table @code +@item nr +Limit query result to nr elements. This parameter is @emph{mandatory}. + +@item project +Filter query result to builds with the given @code{project}. + +@item jobset +Filter query result to builds with the given @code{jobset}. + +@item job +Filter query result to builds with the given @code{job} name. + +@item system +Filter query result to builds with the given @code{system}. + @end table +For example, to ask for the ten last builds : + +@example +$ curl "http://localhost:8080/api/latestbuilds?nr=10" +@end example + +or the five last builds which project is ``guix'' and jobset ``master' : + +@example +$ curl "http://localhost:8080/api/latestbuilds?nr=5&project=guix&jobset=master" +@end example + +If no builds matching given parameters are found and empty JSON array is returned. @c ********************************************************************* @node Contributing diff --git a/src/cuirass/database.scm b/src/cuirass/database.scm index 5f60fac..6b84a48 100644 --- a/src/cuirass/database.scm +++ b/src/cuirass/database.scm @@ -226,7 +226,7 @@ INSERT INTO Outputs (build, name, path) VALUES ('~A', '~A', '~A');" outputs)))))) (define db-build-request "\ -SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime, Builds.log, Builds.status,\ +SELECT Builds.id, Builds.timestamp, Builds.starttime, Builds.stoptime, Builds.log, Builds.status, Builds.derivation,\ Derivations.job_name, Derivations.system, Derivations.nix_name,\ Specifications.repo_name, Specifications.branch \ FROM Builds \ @@ -236,20 +236,21 @@ INNER JOIN Specifications ON Evaluations.specification = Specifications.repo_nam (define (db-format-build db build) (match build - (#(id timestamp starttime stoptime log status job-name system + (#(id timestamp starttime stoptime log status derivation job-name system nix-name repo-name branch) - `((#:id . ,id) - (#:timestamp . ,timestamp) - (#:starttime . ,starttime) - (#:stoptime . ,stoptime) - (#:log . ,log) - (#:status . ,status) - (#:job-name . ,job-name) - (#:system . ,system) - (#:nix-name . ,nix-name) - (#:repo-name . ,repo-name) - (#:outputs . ,(db-get-outputs db id)) - (#:branch . ,branch))))) + `((#:id . ,id) + (#:timestamp . ,timestamp) + (#:starttime . ,starttime) + (#:stoptime . ,stoptime) + (#:log . ,log) + (#:status . ,status) + (#:derivation . ,derivation) + (#:job-name . ,job-name) + (#:system . ,system) + (#:nix-name . ,nix-name) + (#:repo-name . ,repo-name) + (#:outputs . ,(db-get-outputs db id)) + (#:branch . ,branch))))) (define (db-get-build db id) "Retrieve a build in database DB which corresponds to ID." diff --git a/src/cuirass/http.scm b/src/cuirass/http.scm index 33cd37b..23c3ad7 100644 --- a/src/cuirass/http.scm +++ b/src/cuirass/http.scm @@ -1,5 +1,6 @@ ;;;; http.scm -- HTTP API ;;; Copyright © 2016 Mathieu Lirzin +;;; Copyright © 2017 Mathieu Othacehe ;;; ;;; This file is part of Cuirass. ;;; @@ -19,52 +20,147 @@ (define-module (cuirass http) #:use-module (cuirass database) #:use-module (cuirass utils) - #:use-module (ice-9 hash-table) + #:use-module (guix config) + #:use-module (guix build utils) + #:use-module (guix utils) #:use-module (ice-9 match) + #:use-module (ice-9 popen) #:use-module (json) #:use-module (web request) #:use-module (web response) #:use-module (web server) #:use-module (web uri) - #:export (spec->json-string - run-cuirass-server)) + #:export (run-cuirass-server)) -;;; -;;; JSON format. -;;; +(define (build->hydra-build build) + "Convert BUILD to an assoc list matching hydra API format." + `((#:id . ,(assq-ref build #:id)) + (#:project . ,(assq-ref build #:repo-name)) + (#:jobset . ,(assq-ref build #:branch)) + (#:job . ,(assq-ref build #:job-name)) + (#:timestamp . ,(assq-ref build #:timestamp)) + (#:starttime . ,(assq-ref build #:starttime)) + (#:stoptime . ,(assq-ref build #:stoptime)) + (#:buildoutputs . ,(assq-ref build #:outputs)) + (#:system . ,(assq-ref build #:system)) + (#:nixname . ,(assq-ref build #:nix-name)) + (#:buildstatus . ,(assq-ref build #:status)) + + ;; TODO: Fill the fields above with correct values. + (#:busy . 0) + (#:priority . 0) + (#:finished . 1) + (#:buildproducts . #nil) + (#:releasename . #nil) + (#:buildinputs_builds . #nil))) + +(define (handle-build-request db build-id) + "Retrieve build identified by BUILD-ID in DB and convert it to hydra + format. Return #f is not build was found." + (let ((build (db-get-build db build-id))) + (and=> build build->hydra-build))) -(define (object->json-scm obj) - "Prepare OBJ for JSON usage." - (cond ((string? obj) obj) - ((number? obj) obj) - ((boolean? obj) obj) - ((null? obj) obj) - ((symbol? obj) (symbol->string obj)) - ((keyword? obj) (object->json-scm (keyword->symbol obj))) - ((alist? obj) (alist->hash-table (map object->json-scm obj))) - ((pair? obj) (cons (object->json-scm (car obj)) - (object->json-scm (cdr obj)))) - (else (object->string obj)))) - -(define* (spec->json-string spec #:key pretty) - "Return SPEC as a JSON object." - (scm->json-string (object->json-scm spec) #:pretty pretty)) +(define (handle-builds-request db filters) + "Retrieve all builds matched by FILTERS in DB and convert them to hydra + format." + (let ((builds (db-get-builds db filters))) + (map build->hydra-build builds))) + +(define (handle-log-request db build) + "Retrieve the log file of BUILD. Return a lambda which PORT argument is an + input port from which the content of the log file can be read or #f if the + log file is not readable." + (let* ((log (assq-ref build #:log)) + (access (and (string? log) + (access? log R_OK)))) + (and access + (lambda (out-port) + (let ((in-pipe-port + (open-input-pipe + (format #f "~a -dc ~a" %bzip2 log)))) + (dump-port in-pipe-port out-port) + (close-pipe in-pipe-port)))))) + +(define (request-parameters request) + "Parse the REQUEST query parameters and return them under the form + '((parameter value) ...)." + (let* ((uri (request-uri request)) + (query (uri-query uri))) + (and query + (map (lambda (param) + (match (string-split param #\=) + ((key param) + (list (string->symbol key) param)))) + (string-split query #\&))))) ;;; ;;; Web server. ;;; +;;; The api is derived from the hydra one. It is partially described here : +;;; +;;; https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml +;;; (define (request-path-components request) (split-and-decode-uri-path (uri-path (request-uri request)))) (define (url-handler request body db) + (define* (respond response #:key body (db db)) (values response body db)) + + (define-syntax-rule (respond-json body ...) + (respond '((content-type . (application/json))) + #:body body ...)) + + (define-syntax-rule (respond-text body ...) + (respond '((content-type . (text/plain))) + #:body body ...)) + + (define-syntax-rule (respond-json-with-error error-code message) + (respond + (build-response #:headers '((content-type . (application/json))) + #:code error-code) + #:body + (object->json-string + `((error . ,message))))) + + (define (respond-build-not-found build-id) + (respond-json-with-error + 404 + (format #f "Build with ID ~a doesn't exist." build-id))) + + (define (respond-build-log-not-found build) + (let ((drv (assq-ref build #:derivation))) + (respond-json-with-error + 404 + (format #f "The build log of derivation ~a is not available." drv)))) + (match (request-path-components request) (((or "jobsets" "specifications") . rest) - (respond '((content-type . (application/json))) - #:body (spec->json-string (car (db-get-specifications db))))) + (respond-json (object->json-string (car (db-get-specifications db))))) + (("build" build-id) + (let ((hydra-build (handle-build-request db build-id))) + (if hydra-build + (respond-json (object->json-string hydra-build)) + (respond-build-not-found build-id)))) + (("build" build-id "log" "raw") + (let ((build (db-get-build db build-id))) + (if build + (let ((log-response (handle-log-request db build))) + (if log-response + (respond-text log-response) + (respond-build-log-not-found build))) + (respond-build-not-found build-id)))) + (("api" "latestbuilds") + (let* ((params (request-parameters request)) + ;; 'nr parameter is mandatory to limit query size. + (valid-params? (assq-ref params 'nr))) + (if valid-params? + (respond-json (object->json-string + (handle-builds-request db params))) + (respond-json-with-error 500 "Parameter not defined!")))) (_ (respond (build-response #:code 404) #:body (string-append "Resource not found: " @@ -73,6 +169,6 @@ (define* (run-cuirass-server db #:key (port 8080)) (format (current-error-port) "listening on port ~A~%" port) (run-server url-handler - 'http ;server implementation - `(#:port ,port) ;implementation parameters - db)) ;state + 'http + `(#:port ,port) + db)) diff --git a/src/cuirass/utils.scm b/src/cuirass/utils.scm index d966543..a932674 100644 --- a/src/cuirass/utils.scm +++ b/src/cuirass/utils.scm @@ -21,9 +21,29 @@ (define-module (cuirass utils) #:use-module (ice-9 match) #:use-module (srfi srfi-1) - #:export (alist?)) + #:use-module (json) + #:export (alist? + object->json-scm + object->json-string)) (define (alist? obj) "Return #t if OBJ is an alist." (and (list? obj) (every pair? obj))) + +(define (object->json-scm obj) + "Prepare OBJ for JSON usage." + (cond ((string? obj) obj) + ((number? obj) obj) + ((boolean? obj) obj) + ((null? obj) obj) + ((symbol? obj) (symbol->string obj)) + ((keyword? obj) (object->json-scm (keyword->symbol obj))) + ((alist? obj) (map object->json-scm obj)) + ((pair? obj) (cons (object->json-scm (car obj)) + (object->json-scm (cdr obj)))) + (else (object->string obj)))) + +(define* (object->json-string object #:key pretty) + "Return OBJECT as a JSON object." + (scm->json-string (object->json-scm object) #:pretty pretty)) -- 2.13.2