From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: Package input loop detection Date: Mon, 05 Feb 2018 16:42:22 +0000 Message-ID: <87eflzfkwh.fsf@cbaines.net> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:53602) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eijqi-0004NX-47 for guix-devel@gnu.org; Mon, 05 Feb 2018 11:42:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eijqe-0003CS-41 for guix-devel@gnu.org; Mon, 05 Feb 2018 11:42:32 -0500 Received: from mira.cbaines.net ([2a01:7e00::f03c:91ff:fe69:8da9]:60406) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eijqd-00037r-NW for guix-devel@gnu.org; Mon, 05 Feb 2018 11:42:28 -0500 Received: from localhost (cpc102582-walt20-2-0-cust14.13-2.cable.virginm.net [86.27.34.15]) by mira.cbaines.net (Postfix) with ESMTPSA id 2B5A013D064 for ; Mon, 5 Feb 2018 16:42:26 +0000 (GMT) Received: from phact (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id c10ff4df for ; Mon, 5 Feb 2018 16:42:25 +0000 (UTC) List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: guix-devel@gnu.org --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain I've had this issue for a while now, while adding some packages, I'll create a loop in the package graph, which causes Guix to just loop infinitely when trying to generate derivations. I've included a very rough patch which detects and informs the user what has happened, this is an example of what this looks like (with a version of the wip-rails branch I've broken): $ guix build ruby-rails error: input loop detected, error generating a derivation for # This shouldn't happen with Guix packages, please consider reporting a bug. Report bugs to: bug-guix@gnu.org. GNU Guix home page: General help using GNU software: If any of the packages below are not included in Guix, it could be that one of them is causing the loop. The packages are listed in reverse order, so the first package listed is a input to the second package for example, and the start and end of the detected loop is highlighted with an arrow (--->). ---> # # # # ---> # # # # # # # I'm not particularly fond of the implementation, because the package-derivation function is called from expand-input called from bag->derivation, the information about the part of the graph that has been traversed is passed through each function. The seen-package-list argument could be removed, but the ordering information is really useful when printing out the error message. I think it should be still possible to generate this after finding the issue by searching through the graph of packages, which would allow removing this one argument. One other thought I had is that this could be extracted to something in guix lint, which would at least allow finding these problems, without touching the core derivation related code. What do people think? Thanks, Chris --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-WIP-Add-package-input-loop-detection.patch Content-Transfer-Encoding: quoted-printable From=20ac8434025cc16fee2bc84345313305fc9146ca20 Mon Sep 17 00:00:00 2001 From: Christopher Baines Date: Mon, 5 Feb 2018 16:28:42 +0000 Subject: [PATCH] WIP: Add package input loop detection. =2D-- guix/packages.scm | 60 +++++++++++++++++++++++++++++++++++++++++++++------= ---- 1 file changed, 50 insertions(+), 10 deletions(-) diff --git a/guix/packages.scm b/guix/packages.scm index 7d884aa36..c34191183 100644 =2D-- a/guix/packages.scm +++ b/guix/packages.scm @@ -21,6 +21,7 @@ ;;; along with GNU Guix. If not, see . =20 (define-module (guix packages) + #:use-module (guix ui) #:use-module (guix utils) #:use-module (guix records) #:use-module (guix store) @@ -860,7 +861,8 @@ Return the cached result when available." ((_ package system body ...) (cached (=3D> %derivation-cache) package system body ...)))) =20 =2D(define* (expand-input store package input system #:optional cross-syste= m) +(define* (expand-input store package input system #:optional cross-system + #:key seen-packages seen-package-list) "Expand INPUT, an input tuple, such that it contains only references to derivation paths or store paths. PACKAGE is only used to provide contextu= al information in exceptions." @@ -873,7 +875,9 @@ information in exceptions." (if cross-system (cut package-cross-derivation store <> cross-system system #:graft? #f) =2D (cut package-derivation store <> system #:graft? #f))) + (cut package-derivation store <> system #:graft? #f + #:seen-packages seen-packages + #:seen-package-list seen-package-list))) =20 (match input (((? string? name) (? package? package)) @@ -1077,7 +1081,8 @@ TARGET." (bag-grafts store bag))) =20 (define* (bag->derivation store bag =2D #:optional context) + #:optional context + #:key seen-packages seen-package-list) "Return the derivation to build BAG for SYSTEM. Optionally, CONTEXT can= be a package object describing the context in which the call occurs, for impr= oved error reporting." @@ -1085,7 +1090,9 @@ error reporting." (bag->cross-derivation store bag) (let* ((system (bag-system bag)) (inputs (bag-transitive-inputs bag)) =2D (input-drvs (map (cut expand-input store context <> system) + (input-drvs (map (cut expand-input store context <> system + #:seen-packages seen-packages + #:seen-package-list seen-package-list) inputs)) (paths (delete-duplicates (append-map (match-lambda @@ -1102,20 +1109,27 @@ error reporting." (bag-arguments bag))))) =20 (define* (bag->cross-derivation store bag =2D #:optional context) + #:optional context + #:key seen-packages seen-package-list) "Return the derivation to build BAG, which is actually a cross build. Optionally, CONTEXT can be a package object denoting the context of the ca= ll. This is an internal procedure." (let* ((system (bag-system bag)) (target (bag-target bag)) (host (bag-transitive-host-inputs bag)) =2D (host-drvs (map (cut expand-input store context <> system tar= get) + (host-drvs (map (cut expand-input store context <> system target + #:seen-packages seen-packages + #:seen-package-list seen-package-list) host)) (target* (bag-transitive-target-inputs bag)) =2D (target-drvs (map (cut expand-input store context <> system) + (target-drvs (map (cut expand-input store context <> system + #:seen-packages seen-packages + #:seen-package-list seen-package-list) target*)) (build (bag-transitive-build-inputs bag)) =2D (build-drvs (map (cut expand-input store context <> system) + (build-drvs (map (cut expand-input store context <> system + #:seen-packages seen-packages + #:seen-package-list seen-package-list) build)) (all (append build target* host)) (paths (delete-duplicates @@ -1144,15 +1158,41 @@ This is an internal procedure." =20 (define* (package-derivation store package #:optional (system (%current-system)) =2D #:key (graft? (%graft?))) + #:key (graft? (%graft?)) + (seen-packages (setq)) + (seen-package-list '())) "Return the object of PACKAGE for SYSTEM." =20 + (if (set-contains? seen-packages package) + (begin + (simple-format #t "\nerror: input loop detected, error generating = a derivation for ~A\n" + (last seen-package-list)) + (display " +This shouldn't happen with Guix packages, please consider reporting a bug.= \n") + (show-bug-report-information) + (display " +If any of the packages below are not included in Guix, it could be that on= e of +them is causing the loop. The packages are listed in reverse order, so the +first package listed is a input to the second package for example, and the +start and end of the detected loop is highlighted with an arrow (--->).\n\= n") + (for-each (lambda (seen-package) + (if (eq? package seen-package) + (display " --->")) + (simple-format #t "\t~A\n" seen-package)) + (cons package + seen-package-list)) + (exit 1))) + ;; Compute the derivation and cache the result. Caching is important ;; because some derivations, such as the implicit inputs of the GNU build ;; system, will be queried many, many times in a row. (cached package (cons system graft?) (let* ((bag (package->bag package system #f #:graft? graft?)) =2D (drv (bag->derivation store bag package))) + (drv (bag->derivation store bag package + #:seen-packages + (set-insert package seen-packages) + #:seen-package-list + (cons package seen-package-list)))) (if graft? (match (bag-grafts store bag) (() =2D-=20 2.16.1 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAlp4iW5fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XdCxBAApCZox89IFZDo5xNlN5lWWOc1c6eEBG2JoA3UYSWMcyZq5qM++vmh3B2l VYg7BHA34ddhrtlz1x9rbbv3Y183rtgvIkysmyn9zAPL1vOaOwsc5N131Lg7tEIY q3tOz5wY83AEvYArect0dHSMwl1pionKwl7atAiLjxJbzniryl3ZzeE7Cs/2DLfH p6qXy+p7CqQsImXTnBad28MyCErJDma5YyaKs7HyhZ1oSw2wCwUIoPZmAvVISeay meAX6hm6In2jgxyss2s5RF7T/2EAN4GVaAQQWL3VPBARag1oXoEk6FCf373ts5oo 7RMdDJLNcm+uVVzrHI1Znq8sEoyq3Xqw2OrOZYBTbNXoFw/Iivls2b9qxE/Zyghr HDbzfPWFETQqFLb2W/S2/r8jHZYXJzZLPfz+WH67/dzKAFXthXutwSrR7ag5Ggq2 p4x+M+k/ee3vNEZ2aePaPOgVi6d5cMl3VN2ThCzXtfN1yTKhKC60twKQkYiOVrVC B3pcJSZM/5vblfUxigB3hvxLw2FliqdBr15HOKAmWhEncehEqvgodlKkHOt+37/S kUdSUnDDibhp032q7QAMT25I94iHlTzVFBDfKhGbDIJdBHC0s3RyKXaO3xaG+XL5 6p0ZfAw2FbeRAEAaJPz7F6B57fujoi3jzv8pIyGUBP3rqjftwJc= =0CWI -----END PGP SIGNATURE----- --==-=-=--