From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 6TTUJvQYgGDiJwEAgWs5BA (envelope-from ) for ; Wed, 21 Apr 2021 14:22:12 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id MNo4IfQYgGAOBQAAbx9fmQ (envelope-from ) for ; Wed, 21 Apr 2021 12:22:12 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 288DF1AA19 for ; Wed, 21 Apr 2021 14:22:12 +0200 (CEST) Received: from localhost ([::1]:41408 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZBrv-0002Xq-3M for larch@yhetil.org; Wed, 21 Apr 2021 08:22:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48628) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZBrm-0002X3-Ng for guix-patches@gnu.org; Wed, 21 Apr 2021 08:22:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:45576) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lZBrm-00079q-GI for guix-patches@gnu.org; Wed, 21 Apr 2021 08:22:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lZBrm-0005a5-A5 for guix-patches@gnu.org; Wed, 21 Apr 2021 08:22:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47929] [PATCH 1/5] ci: Add manifest support to channel-with-substitutes-available. References: <20210421121610.2045-1-othacehe@gnu.org> In-Reply-To: <20210421121610.2045-1-othacehe@gnu.org> Resent-From: Mathieu Othacehe Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 21 Apr 2021 12:22:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47929 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 47929@debbugs.gnu.org Cc: Mathieu Othacehe Received: via spool by 47929-submit@debbugs.gnu.org id=B47929.161900768921373 (code B ref 47929); Wed, 21 Apr 2021 12:22:02 +0000 Received: (at 47929) by debbugs.gnu.org; 21 Apr 2021 12:21:29 +0000 Received: from localhost ([127.0.0.1]:57111 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZBrE-0005YX-VN for submit@debbugs.gnu.org; Wed, 21 Apr 2021 08:21:29 -0400 Received: from eggs.gnu.org ([209.51.188.92]:35874) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lZBrD-0005YB-0x for 47929@debbugs.gnu.org; Wed, 21 Apr 2021 08:21:27 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:54763) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZBr7-0006lD-SI for 47929@debbugs.gnu.org; Wed, 21 Apr 2021 08:21:21 -0400 Received: from [2a01:e0a:19b:d9a0:9576:8bbf:4795:82ee] (port=38154 helo=localhost.localdomain) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lZBr6-0003pa-P3; Wed, 21 Apr 2021 08:21:21 -0400 From: Mathieu Othacehe Date: Wed, 21 Apr 2021 14:21:04 +0200 Message-Id: <20210421122108.2344-1-othacehe@gnu.org> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619007732; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=JmcUez/hGOWQ3I/n6QyA81UJ051JBP61QDHcESm3i2E=; b=btDadyFSftT0Q7ij5oCdlWB6HYkwPfnagoTl/uwJ03Ro8kjhXHZSREmlt0FXDDByPqwCDJ Akfbi/sskEjT2ZqCQ6vMqtKv1GyWlKYCAb4BuzayRZeeo9WT068RiZdziyC8+V4vLbGZ9u d2bQoLo5blUpCSN2O6U1gFker1hPa7hgIIhEjmWoGEE49UFUoJR4Npm+pCdy6zBVfYbTMN 2U2cyKPjkOmgcPO02nOTfscHChw081ZP/V0swRFThFqXp2ptPCqq33Ar3fO6hY/T827MSE NbLnc5vzREUxllKeYMZouFsLY/vjr5i+VExCJZE6s0JOSCsswVdSKSQUvCaJ1A== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1619007732; a=rsa-sha256; cv=none; b=HjXmgB1vutuJCgabPOqwb1MGcXiscXvp+uSo3VoYcrQCc8TtRbJl0aPvH5gLID4pSzSRSM fT32EeMWtMUCii7hn+DD0CrWWb45HPUVXRdl7qNxxQ7m/iZdJ1Tb+ySB1Erow4AmsKH2MH FUt3veAqqnNCvEOqea4cchI+a1sRbo2/ngLuQX6uzByRC9rY7YXirQOs9y85vf+CzHsa7f wA3oUpiz9GGmFRq+fJL9RjNpXbIcdVmAWB+Av+YktrxwfsFB6zpTOS4qvces08tHnJf9U/ 17b3Ro0rocBMDHPaaBmQyV324BhUeDeBZne9lccucAhFkOV8CkxZeLcElZwLkw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: 2.06 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 288DF1AA19 X-Spam-Score: 2.06 X-Migadu-Scanner: scn0.migadu.com X-TUID: NVoRYOiIdTQk * guix/ci.scm (%default-guix-specification, %default-package-specification): New variables. (, ): New records. (job, job-history, sort-history-by-coverage, channel-commit, package->job-name, manifest->jobs): New procedures. (find-latest-commit-with-substitutes): Rename it into ... (latest-checkouts-with-substitutes): ... this new procedure. (channel-with-substitutes-available): Add an optional manifest argument and honor it. * doc/guix.texi (Channels with Substitutes): Update it. --- doc/guix.texi | 31 ++++++-- guix/ci.scm | 205 ++++++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 207 insertions(+), 29 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index b9019d5550..c39bbdb3d5 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -5201,11 +5201,32 @@ server at @url{https://ci.guix.gnu.org}. "https://ci.guix.gnu.org")) @end lisp -Note that this does not mean that all the packages that you will -install after running @command{guix pull} will have available -substitutes. It only ensures that @command{guix pull} will not try to -compile package definitions. This is particularly useful when using -machines with limited resources. +It is also possible to ask @command{guix pull} to use the latest commit +with the maximal number of available substitutes for a given manifest +this way: + +@lisp +(use-modules (guix ci)) + +(list (channel-with-substitutes-available + %default-guix-channel + "https://ci.guix.gnu.org" + "/path/to/manifest)) +@end lisp + +or this way: + +@lisp +(use-modules (guix ci)) + +(list (channel-with-substitutes-available + %default-guix-channel + "https://ci.guix.gnu.org" + (specifications->manifest + '("git" "emacs-minimal")))) +@end lisp + +This is particularly useful when using machines with limited resources. @node Creating a Channel @section Creating a Channel diff --git a/guix/ci.scm b/guix/ci.scm index c70e5bb9e6..780e90ef32 100644 --- a/guix/ci.scm +++ b/guix/ci.scm @@ -18,10 +18,16 @@ ;;; along with GNU Guix. If not, see . (define-module (guix ci) + #:use-module (gnu packages) + #:use-module (guix channels) #:use-module (guix http-client) + #:use-module (guix packages) + #:use-module (guix profiles) + #:use-module (guix ui) #:use-module (guix utils) #:use-module (json) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) #:use-module (ice-9 match) #:use-module (guix i18n) #:use-module (guix diagnostics) @@ -58,6 +64,7 @@ latest-evaluations evaluations-for-commit + manifest->jobs channel-with-substitutes-available)) ;;; Commentary: @@ -67,6 +74,14 @@ ;;; ;;; Code: +;; The name of the CI specification building the 'guix-modular' package. +(define %default-guix-specification + (make-parameter "guix")) + +;; The default name of the CI specification building all the packages. +(define %default-package-specification + (make-parameter "master")) + (define-json-mapping make-build-product build-product? json->build-product @@ -109,6 +124,24 @@ (map json->checkout (vector->list checkouts))))) +(define-json-mapping make-job job? + json->job + (name job-name) ;string + (build job-build) ;integer + (status job-status)) ;integer + +(define-json-mapping make-history history? + json->history + (evaluation history-evaluation) ;integer + (checkouts history-checkouts "checkouts" ;* + (lambda (checkouts) + (map json->checkout + (vector->list checkouts)))) + (jobs history-jobs "jobs" + (lambda (jobs) + (map json->job + (vector->list jobs))))) + (define %query-limit ;; Max number of builds requested in queries. 1000) @@ -172,34 +205,158 @@ as one of their inputs." (evaluation-checkouts evaluation))) (latest-evaluations url limit))) -(define (find-latest-commit-with-substitutes url) - "Return the latest commit with available substitutes for the Guix package -definitions at URL. Return false if no commit were found." - (let* ((job-name (string-append "guix." (%current-system))) - (build (match (latest-builds url 1 - #:job job-name - #:status 0) ;success - ((build) build) - (_ #f))) - (evaluation (and build - (evaluation url (build-evaluation build)))) - (commit (and evaluation - (match (evaluation-checkouts evaluation) - ((checkout) - (checkout-commit checkout)))))) - commit)) - -(define (channel-with-substitutes-available chan url) +(define* (job url name #:key evaluation) + "Return the job which name is NAME for the given EVALUATION, from the CI +server at URL." + (map json->job + (vector->list + (json->scm + (http-fetch + (format #f "~a/api/jobs?evaluation=~a&names=~a" + url evaluation name)))))) + +(define* (jobs-history url jobs + #:key + (specification "master") + (limit 20)) + "Return the job history for the SPECIFICATION jobs which names are part of +the JOBS list, from the CI server at URL. Limit the history to the latest +LIMIT evaluations. " + (let ((names (string-join jobs ","))) + (map json->history + (vector->list + (json->scm + (http-fetch + (format #f "~a/api/jobs/history?spec=~a&names=~a&nr=~a" + url specification names (number->string limit)))))))) + +(define (sort-history-by-coverage history) + "Sort and return the given evaluation HISTORY list by descending successful +jobs count. This means that the first element of the list will be the +evaluation with the higher successful jobs count." + (let ((coverage + (map (cut fold + (lambda (status prev) + (if (eq? status 0) ;successful + (1+ prev) + prev)) + 0 <>) + (map (compose + (cut map job-status <>) history-jobs) + history)))) + (map (match-lambda + ((cov . hist) hist)) + (sort (map cons coverage history) + (match-lambda* + (((c1 . h1) (c2 . h2)) + (> c1 c2))))))) + +(define (channel-commit checkouts channel) + "Return the CHANNEL commit from CHECKOUTS." + (any (lambda (checkout) + (and (string=? (checkout-channel checkout) channel) + (checkout-commit checkout))) + checkouts)) + +(define (package->job-name package) + "Return the CI job name for the given PACKAGE name." + (string-append package "." (%current-system))) + +(define (manifest->jobs manifest) + "Return the list of job names that are part of the given MANIFEST." + (define (load-manifest file) + (let ((user-module (make-user-module '((guix profiles) (gnu))))) + (load* file user-module))) + + (let* ((manifest (cond + ((string? manifest) + (load-manifest manifest)) + ((manifest? manifest) + manifest) + (else #f))) + (packages (delete-duplicates + (map manifest-entry-item + (manifest-transitive-entries manifest)) + eq?))) + (map (lambda (package) + (package->job-name (package-name package))) + packages))) + +(define* (latest-checkouts-with-substitutes url jobs) + "Return a list of latest checkouts, sorted by descending substitutes +coverage of the given JOBS list on the CI server at URL. Only evaluations for +which the Guix package is built are considered. + +If JOBS is false, return a list of latest checkouts for which the Guix package +is built. Return false if no checkouts were found." + (define guix-history + (filter (lambda (hist) + (let ((jobs (history-jobs hist))) + (match jobs + ((job) + (eq? (job-status job) 0)) + (else #f)))) + (jobs-history url (list (package->job-name "guix")) + #:specification + (%default-guix-specification)))) + + (define (guix-commit checkouts) + (let ((name (symbol->string + (channel-name %default-guix-channel)))) + (channel-commit checkouts name))) + + (define (guix-package-available? hist) + (any (lambda (guix-hist) + (string=? (guix-commit + (history-checkouts hist)) + (guix-commit + (history-checkouts guix-hist))) + hist) + guix-history)) + + (define (first-checkout checkouts) + (match checkouts + ((checkouts _ ...) + checkouts) + (() #f))) + + (if jobs + (let* ((jobs-history + (sort-history-by-coverage + (jobs-history url jobs + #:specification + (%default-package-specification)))) + (checkouts + (map history-checkouts + (filter-map guix-package-available? + jobs-history)))) + (first-checkout checkouts)) + (first-checkout + (map history-checkouts guix-history)))) + +(define* (channel-with-substitutes-available chan url + #:optional manifest) "Return a channel inheriting from CHAN but which commit field is set to the latest commit with available substitutes for the Guix package definitions at -URL. The current system is taken into account. +URL. If the MANIFEST argument is passed, return the latest commit with the +maximal substitutes coverage of MANIFEST. MANIFEST can be an absolute path as +a string, or a record. The current system is taken into account. If no commit with available substitutes were found, the commit field is set to false and a warning message is printed." - (let ((commit (find-latest-commit-with-substitutes url))) - (unless commit + (let* ((jobs (and manifest + (manifest->jobs manifest))) + (checkouts + (latest-checkouts-with-substitutes url jobs))) + (unless checkouts (warning (G_ "could not find available substitutes at ~a~%") url)) - (channel - (inherit chan) - (commit commit)))) + (let* ((name (channel-name chan)) + (name-str (if (symbol? name) + (symbol->string name) + name)) + (commit (and checkouts + (channel-commit checkouts name-str)))) + (channel + (inherit chan) + (commit commit))))) -- 2.31.1