From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id cE7EBP+8Z2PR2gAAbAwnHQ (envelope-from ) for ; Sun, 06 Nov 2022 14:56:15 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id +GOaBP+8Z2NBIgEAauVa8A (envelope-from ) for ; Sun, 06 Nov 2022 14:56:15 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8F40D23F95 for ; Sun, 6 Nov 2022 14:56:14 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1org85-0006K0-1k; Sun, 06 Nov 2022 08:56:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1org83-0006JQ-1d for guix-patches@gnu.org; Sun, 06 Nov 2022 08:56:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1org82-0003iY-PY for guix-patches@gnu.org; Sun, 06 Nov 2022 08:56:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1org82-0004lP-KN for guix-patches@gnu.org; Sun, 06 Nov 2022 08:56:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#59078] [PATCH] lint: Split the derivation lint checker by system. Resent-From: Christopher Baines Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 06 Nov 2022 13:56:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 59078 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 59078@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.166774294618285 (code B ref -1); Sun, 06 Nov 2022 13:56:02 +0000 Received: (at submit) by debbugs.gnu.org; 6 Nov 2022 13:55:46 +0000 Received: from localhost ([127.0.0.1]:58966 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1org7l-0004kr-3O for submit@debbugs.gnu.org; Sun, 06 Nov 2022 08:55:45 -0500 Received: from lists.gnu.org ([209.51.188.17]:46312) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1org7i-0004kh-0G for submit@debbugs.gnu.org; Sun, 06 Nov 2022 08:55:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1org7h-000693-Qk for guix-patches@gnu.org; Sun, 06 Nov 2022 08:55:41 -0500 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1org7f-0003fv-BX for guix-patches@gnu.org; Sun, 06 Nov 2022 08:55:41 -0500 Received: from localhost (host-82-184-251-30.business.telecomitalia.it [82.184.251.30]) by mira.cbaines.net (Postfix) with ESMTPSA id EDA3827BBE9 for ; Sun, 6 Nov 2022 13:55:35 +0000 (GMT) Received: from localhost (localhost [local]) by localhost (OpenSMTPD) with ESMTPA id 61b65099 for ; Sun, 6 Nov 2022 13:55:32 +0000 (UTC) From: Christopher Baines Date: Sun, 6 Nov 2022 14:55:32 +0100 Message-Id: <20221106135532.5724-1-mail@cbaines.net> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Guix-patches" Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1667742974; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=sLMcJrTl70t15EYJLjNmvz34PXYxwCH4QmA/LQJoX3Y=; b=cqT2ShnsY7HtWhh7a65LA9deNJpUdZItdnaNapXYGu95SFtRbYo43SBWQEFD9HPxaRMon+ Mv8gR/Gh0WA8uWnZ6Cv1+awswT7Q7aPteDC92PoCIvOT+Q2Dk0OfJ3WALcaAvv4kSVYkuc F5Xx17iwtT5qEhqAacf5b3EZ1H4h8fJNDciHfy3XFlfqJF4ryUbZEAK3qB93Ah1lZJ14HQ 3u2Bb5aBrRj2jRlU/NfEjr72MZTC5oyrEPMJa6UihwOv3CYrEi6leDuovUUDQi7YZS6+Bm gyoPCICLXWkJgXzMbBkB5Ud8G/l8Oe6OjxQMECxv0PnhuSDEcdP8e4cC3N1JvQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1667742974; a=rsa-sha256; cv=none; b=Q+UhDi3eCRqnrKZb/v43a3V+GUghqW2XoguwhlL/VlLyKGMKWiLgUj9v6cjE7D/S2kEIIq PVjSNHQiYvSXmYK9WO78BEC22FIxg2cWPC6s3UlqSTDgE+qlcwwyribWGRBG8Bd9iNIW5W 3LU5UZMD3skByiKcDhmLH1UqLZzQ5+Eu1GOvSgJuHu7pvVdEU7k6SgiwH/s4P/boIJpoBB iwwKlL3T+EGGwFFiKEN5AC6j79Hpgp3IBBHK/brbSkj0n9wAFoefzhsUwxIqkHzd4RHp6g MDSY5zNQY4wKYsnPNmTTDmGFynq5kjAUqgvCQ09l1B13SzXP89Y/AEjcfS7C/g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.10 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 8F40D23F95 X-Spam-Score: -1.10 X-Migadu-Scanner: scn0.migadu.com X-TUID: DYZyCTJnqwRr Currently, if you attempt to run the derivation checker on all packages, the Guile process will run out of memory. I think a contributing factor to this is that the checker forces an inefficient order when you want to generate derivations for all the supported systems of each package, constantly switching system then package. This problem also impacts the Guix Data Service, since it tries to run the derivation checker for all packages. The changes in this commit to split the derivation lint checker in to several, one for each system, means that you can now treat each system separately, which should be better for caching purposes. If it's desirable to keep some notion of checking all supported systems for a single package, I think lint checker groups could be added, so that you could ask for the "derivation" checker, and this would run all the derivation checkers. * guix/lint.scm (check-derivation): Adapt to make-check-derivation-for-system. (%derivation-checkers): New variable. (%local-checkers): Include all %derivation-checkers. * doc/guix.texi (Invoking guix lint): Update. --- doc/guix.texi | 4 +- guix/lint.scm | 139 +++++++++++++++++++++++++++++--------------------- 2 files changed, 83 insertions(+), 60 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 7806b21a0f..8d4989a60c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -14422,9 +14422,9 @@ Parse the @code{source} URL to determine if a tarball from GitHub is autogenerated or if it is a release tarball. Unfortunately GitHub's autogenerated tarballs are sometimes regenerated. -@item derivation +@item derivation/SYSTEM Check that the derivation of the given packages can be successfully -computed for all the supported systems (@pxref{Derivations}). +computed for the specified system (@pxref{Derivations}). @item profile-collisions Check whether installing the given packages in a profile would lead to diff --git a/guix/lint.scm b/guix/lint.scm index 8e3976171f..f692856f42 100644 --- a/guix/lint.scm +++ b/guix/lint.scm @@ -52,6 +52,7 @@ (define-module (guix lint) #:use-module (guix memoization) #:use-module (guix profiles) #:use-module (guix monads) + #:use-module (guix platform) #:use-module (guix scripts) #:use-module ((guix ui) #:select (texi->plain-text fill-paragraph)) #:use-module (guix gnu-maintenance) @@ -98,7 +99,6 @@ (define-module (guix lint) check-patch-file-names check-patch-headers check-synopsis-style - check-derivation check-home-page check-name check-source @@ -116,6 +116,8 @@ (define-module (guix lint) check-haskell-stackage check-tests-true + make-check-derivation-for-system + lint-warning lint-warning? lint-warning-package @@ -1369,56 +1371,6 @@ (define (check-phases-deltas deltas) (append-map check-phases-delta deltas)) (find-phase-deltas package check-phases-deltas)) -(define* (check-derivation package #:key store) - "Emit a warning if we fail to compile PACKAGE to a derivation." - (define (try store system) - (guard (c ((store-protocol-error? c) - (make-warning package - (G_ "failed to create ~a derivation: ~a") - (list system - (store-protocol-error-message c)))) - ((exception-with-kind-and-args? c) - (make-warning package - (G_ "failed to create ~a derivation: ~s") - (list system - (cons (exception-kind c) - (exception-args c))))) - ((message-condition? c) - (make-warning package - (G_ "failed to create ~a derivation: ~a") - (list system - (condition-message c)))) - ((formatted-message? c) - (let ((str (apply format #f - (formatted-message-string c) - (formatted-message-arguments c)))) - (make-warning package - (G_ "failed to create ~a derivation: ~a") - (list system str)))) - (else - (make-warning package - (G_ "failed to create ~a derivation: ~a") - (list system c)))) - (parameterize ((%graft? #f)) - (package-derivation store package system #:graft? #f) - - ;; If there's a replacement, make sure we can compute its - ;; derivation. - (match (package-replacement package) - (#f #t) - (replacement - (package-derivation store replacement system - #:graft? #f)))))) - - (define (check-with-store store) - (filter lint-warning? - (map (cut try store <>) (package-supported-systems package)))) - - ;; For backwards compatability, don't rely on store being set - (or (and=> store check-with-store) - (with-store store - (check-with-store store)))) - (define* (check-profile-collisions package #:key store) "Check for collisions that would occur when installing PACKAGE as a result of the propagated inputs it pulls in." @@ -1843,13 +1795,88 @@ (define (check-formatting package) (G_ "source file not found")))))))) '()))) +(define (make-check-derivation-for-system system) + (define (try package proc) + (guard (c ((store-protocol-error? c) + (make-warning package + (G_ "failed to create ~a derivation: ~a") + (list system + (store-protocol-error-message c)))) + ((exception-with-kind-and-args? c) + (make-warning package + (G_ "failed to create ~a derivation: ~s") + (list system + (cons (exception-kind c) + (exception-args c))))) + ((message-condition? c) + (make-warning package + (G_ "failed to create ~a derivation: ~a") + (list system + (condition-message c)))) + ((formatted-message? c) + (let ((str (apply format #f + (formatted-message-string c) + (formatted-message-arguments c)))) + (make-warning package + (G_ "failed to create ~a derivation: ~a") + (list system str)))) + (else + (make-warning package + (G_ "failed to create ~a derivation: ~a") + (list system c)))) + (proc))) + + + + (lambda* (package #:key store) + "Emit a warning if we fail to compile PACKAGE to a derivation." + + (define (check-with-store store) + (if (member system (package-supported-systems package)) + (filter + lint-warning? + (map (cut try package <>) + (list + (lambda () + (parameterize ((%graft? #f)) + (package-derivation store package system #:graft? #f))) + (lambda () + ;; If there's a replacement, make sure we can compute its + ;; derivation. + (match (package-replacement package) + (#f #t) + (replacement + (parameterize ((%graft? #f)) + (package-derivation store replacement system + #:graft? #f)))))))) + '())) + + ;; For backwards compatability, don't rely on store being set + (or (and=> store check-with-store) + (with-store store + (check-with-store store))))) + ;;; ;;; List of checkers. ;;; +(define %derivation-checkers + (map (lambda (system) + (lint-checker + (name (string->symbol + (simple-format #f "derivation/~A" system))) + (description + (simple-format + #f + "Report failure to compile a package to a derivation for ~A" + system)) + (check (make-check-derivation-for-system system)) + (requires-store? #t))) + (systems))) + (define %local-checkers - (list + (cons* (lint-checker (name 'name) (description "Validate package names") @@ -1901,11 +1928,6 @@ (define %local-checkers (name 'source-unstable-tarball) (description "Check for autogenerated tarballs") (check check-source-unstable-tarball)) - (lint-checker - (name 'derivation) - (description "Report failure to compile a package to a derivation") - (check check-derivation) - (requires-store? #t)) (lint-checker (name 'profile-collisions) (description "Report collisions that would occur due to propagated inputs") @@ -1922,7 +1944,8 @@ (define %local-checkers (lint-checker (name 'formatting) (description "Look for formatting issues in the source") - (check check-formatting)))) + (check check-formatting)) + %derivation-checkers)) (define %network-dependent-checkers (list -- 2.37.3