From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 0ApPEk0cD2CUYgAA0tVLHw (envelope-from ) for ; Mon, 25 Jan 2021 19:30:21 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id CFMUDk0cD2DkHwAAbx9fmQ (envelope-from ) for ; Mon, 25 Jan 2021 19:30:21 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 598FE9402C2 for ; Mon, 25 Jan 2021 19:30:20 +0000 (UTC) Received: from localhost ([::1]:51498 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l47Z5-0000aA-5x for larch@yhetil.org; Mon, 25 Jan 2021 14:30:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44246) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l47Yp-0000ZM-D2 for guix-patches@gnu.org; Mon, 25 Jan 2021 14:30:03 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:56692) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1l47Yo-0002cO-7x for guix-patches@gnu.org; Mon, 25 Jan 2021 14:30:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1l47Yo-00073D-3S for guix-patches@gnu.org; Mon, 25 Jan 2021 14:30:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#45712] [PATCHES] Improve Python package quality Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 25 Jan 2021 19:30:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 45712 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Received: via spool by 45712-submit@debbugs.gnu.org id=B45712.161160296527034 (code B ref 45712); Mon, 25 Jan 2021 19:30:02 +0000 Received: (at 45712) by debbugs.gnu.org; 25 Jan 2021 19:29:25 +0000 Received: from localhost ([127.0.0.1]:40005 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l47Y4-00071n-Ve for submit@debbugs.gnu.org; Mon, 25 Jan 2021 14:29:25 -0500 Received: from mail-qt1-f169.google.com ([209.85.160.169]:40401) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l47Y3-00071b-0g for 45712@debbugs.gnu.org; Mon, 25 Jan 2021 14:29:16 -0500 Received: by mail-qt1-f169.google.com with SMTP id z22so10536658qto.7 for <45712@debbugs.gnu.org>; Mon, 25 Jan 2021 11:29:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=2Daplu2Uu3IM/Zc2Twl5kSxNxyJrj7HMj8oOGH01tfY=; b=TNQXpRi4r+uBCU2CZjrX1IiR2Ve1aiAVcow+qFCSd9PJM8Kgllhki5rQKwEPejTkzC LwTVddfnV/Jadb/rvySFcwhdpNJ8OMf8vLinkOVJg6Rnq3ammqFVAPET/s7a56wF8cyb r7XDbXMSDWwbVL/b5jhdzjYdwHtIvHvkNcS/hZQhtuAM6ZessOouBF1tXdmtIVqvTNa4 sMq5M67LgDkIYYT96HFwILdRo4SCLsHKZMIXWvXufaN1J7cp3pDrSvilVlVDadaCNsGZ DJQS6BrANZ6ff9+uEWQHcnJMQq56HfOE7LRyHmgU4nXz3Rff7o9B59CXNvPxVAPzXHWt mY+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=2Daplu2Uu3IM/Zc2Twl5kSxNxyJrj7HMj8oOGH01tfY=; b=RfbkRQ/SG7mwg2ma5SrXkbgMpgUMpzeWdNKihdOvZstrVcMVrlgaSF+jMku1yp7BFA lsKBTkuvI/bStlhrwF9W0V1ivq8OUUKHuTFb5aWXJcpgHayQLv6ctGmIeANgYA4L+F3s VWC1SyXP+XoYUrFLSvbhfANieq3YSj1Kiid6gwlm1eDeDWGWDsqQTidTKhBqFDevSb11 tDTf34j0TSzfvYyGKbMMw9qnC7zLnGlRVm285YN9J1KNCSPg0/5OxZexkSeBgllaGP79 peVAm1/XzenXX+ynJ3TeYgebT60Vr3ugAwxIqZ/WXt8tPbLdlJUdwzeJ2hYrgfaweRuI qVcQ== X-Gm-Message-State: AOAM531mniJESINkosItzepuQZjYFaTkj/1foUZPrVwek+mSrLNAaI+d 1kBGDWUyYrF44K5lUtPeEO9j83jL78tniw== X-Google-Smtp-Source: ABdhPJw5u0MZ9UuccX+KmpxmC5Q5UGTJ+5VEK5VfQ36FxSjBKkKAjaOpS+oe+M1XonROqBt9erSkoA== X-Received: by 2002:ac8:7c89:: with SMTP id y9mr2066798qtv.14.1611602949191; Mon, 25 Jan 2021 11:29:09 -0800 (PST) Received: from hurd (dsl-10-132-34.b2b2c.ca. [72.10.132.34]) by smtp.gmail.com with ESMTPSA id w9sm13172947qka.76.2021.01.25.11.29.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:29:08 -0800 (PST) From: Maxim Cournoyer References: Date: Mon, 25 Jan 2021 14:29:06 -0500 In-Reply-To: (Lars-Dominik Braun's message of "Tue, 12 Jan 2021 10:37:07 +0100") Message-ID: <87czxs3jel.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Hartmut Goebel , 45712@debbugs.gnu.org Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -1.25 Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gmail.com header.s=20161025 header.b=TNQXpRi4; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 598FE9402C2 X-Spam-Score: -1.25 X-Migadu-Scanner: scn0.migadu.com X-TUID: zRhbse2mTo+S --=-=-= Content-Type: text/plain Hi Lars-Dominik, Lars-Dominik Braun writes: > Adds a new phase validating usalibity of installed Python packages. > > * guix/build/python-build-system.scm (validate-script): Add script. > (validate-loadable): New phase. > (%standard-phases): Use it. > * tests/builders.scm (make-python-dummy): Add test package generator. > (check-build-{success,failure}): Add build helper functions. > (python-dummy-*): Add test packages. > ("python-build-system: &"): Add tests. Attached is a small rework of your original patch. I've made the Python script standalone, which should make it easier to maintain. I've also refactored the tests somewhat and added your copyright information. Is this OK with you? Thanks! Maxim --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0001-build-system-python-Add-a-sanity-check-phase.patch Content-Transfer-Encoding: quoted-printable >From 2df41c3fb476822efac1aa8dac8368e91a0e360a Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 3 Jan 2021 10:30:29 +0100 Subject: [PATCH] build-system/python: Add a sanity check phase. Add a new phase validating the usability of installed Python packages. * gnu/packages/aux-files/python/sanity-check.py: New file. * Makefile.am (AUX_FILES): Register it. * guix/build-system/python.scm (sanity-check.py): New variable. (lower): Add the script as an implicit input. * guix/build/python-build-system.scm: Remove trailing #t. (sanity-check): New phase. (%standard-phases): Use it. * tests/builders.scm (test-build-package): New syntax. ("python-build-system: dummy-ok") ("python-build-system: dummy-fail-requirements") ("python-build-system: dummy-fail-import") ("python-build-system: dummy-fail-console-script"): Add tests. --- Makefile.am | 1 + gnu/packages/aux-files/python/sanity-check.py | 85 ++++++++++++++ guix/build-system/python.scm | 8 ++ guix/build/python-build-system.scm | 26 +++-- tests/builders.scm | 106 +++++++++++++++++- 5 files changed, 212 insertions(+), 14 deletions(-) create mode 100644 gnu/packages/aux-files/python/sanity-check.py diff --git a/Makefile.am b/Makefile.am index dc5cf9babc..dddae69ff1 100644 --- a/Makefile.am +++ b/Makefile.am @@ -380,6 +380,7 @@ AUX_FILES =3D \ gnu/packages/aux-files/linux-libre/4.4-i686.conf \ gnu/packages/aux-files/linux-libre/4.4-x86_64.conf \ gnu/packages/aux-files/pack-audit.c \ + gnu/packages/aux-files/python/sanity-check.py \ gnu/packages/aux-files/run-in-namespace.c =20 # Templates, examples. diff --git a/gnu/packages/aux-files/python/sanity-check.py b/gnu/packages/a= ux-files/python/sanity-check.py new file mode 100644 index 0000000000..1ba912a931 --- /dev/null +++ b/gnu/packages/aux-files/python/sanity-check.py @@ -0,0 +1,85 @@ +# GNU Guix --- Functional package management for GNU +# Copyright =C2=A9 2021 Lars-Dominik Braun +# +# This file is part of GNU Guix. +# +# GNU Guix is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3 of the License, or (at +# your option) any later version. +# +# GNU Guix is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Guix. If not, see . + +from __future__ import print_function # Python 2 support. +import importlib +import pkg_resources +import sys +import traceback + +try: + from importlib.machinery import PathFinder +except ImportError: + PathFinder =3D None + +ret =3D 0 + +# Only check site-packages installed by this package, but not dependencies +# (which pkg_resources.working_set would include). Path supplied via argv. +ws =3D pkg_resources.find_distributions(sys.argv[1]) + +for dist in ws: + print('validating', repr(dist.project_name), dist.location) + try: + print('...checking requirements', end=3D': ') + req =3D str(dist.as_requirement()) + # dist.activate() is not enough to actually check requirements, we + # have to .require() it. + pkg_resources.require(req) + print('OK') + except Exception as e: + print('ERROR:', req, e) + ret =3D 1 + continue + + # Try to load entry points of console scripts too, making sure they + # work. They should be removed if they don't. Other groups may not be + # safe, as they can depend on optional packages. + for group, v in dist.get_entry_map().items(): + if group not in {'console_scripts', }: + continue + for name, ep in v.items(): + try: + print('...trying to load endpoint', group, name, end=3D': = ') + ep.load() + print('OK') + except Exception: + print('ERROR:') + traceback.print_exc(file=3Dsys.stdout) + ret =3D 1 + continue + + # And finally try to load top level modules. This should not have any + # side-effects. + for name in dist.get_metadata_lines('top_level.txt'): + # Only available on Python 3. + if PathFinder and PathFinder.find_spec(name) is None: + # Ignore unavailable modules. Cannot use ModuleNotFoundError, + # because it is raised by failed imports too. + continue + try: + print('...trying to load module', name, end=3D': ') + importlib.import_module(name) + print('OK') + except Exception: + print('ERROR:') + traceback.print_exc(file=3Dsys.stdout) + ret =3D 1 + continue + +sys.exit(ret) diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index e39c06528e..2bb6fa87ca 100644 --- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2013, 2014, 2015, 2016, 2017 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2013 Andreas Enge ;;; Copyright =C2=A9 2013 Nikita Karetnikov +;;; Copyright =C2=A9 2021 Lars-Dominik Braun ;;; ;;; This file is part of GNU Guix. ;;; @@ -19,6 +20,8 @@ ;;; along with GNU Guix. If not, see . =20 (define-module (guix build-system python) + #:use-module ((gnu packages) #:select (search-auxiliary-file)) + #:use-module (guix gexp) #:use-module (guix store) #:use-module (guix utils) #:use-module (guix memoization) @@ -70,6 +73,10 @@ extension, such as '.tar.gz'." (let ((python (resolve-interface '(gnu packages python)))) (module-ref python 'python-2))) =20 +(define sanity-check.py + ;; The script used to validate the installation of a Python package. + (search-auxiliary-file "python/sanity-check.py")) + (define* (package-with-explicit-python python old-prefix new-prefix #:key variant-property) "Return a procedure of one argument, P. The procedure creates a package= with @@ -156,6 +163,7 @@ pre-defined variants." ;; Keep the standard inputs of 'gnu-build-system'. ,@(standard-packages))) (build-inputs `(("python" ,python) + ("sanity-check.py" ,(local-file sanity-check.py)) ,@native-inputs)) (outputs outputs) (build python-build) diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-s= ystem.scm index 7fef0b2278..1f11dd2b0a 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -9,6 +9,7 @@ ;;; Copyright =C2=A9 2019, 2020, 2021 Maxim Cournoyer ;;; Copyright =C2=A9 2020 Jakub K=C4=85dzio=C5=82ka ;;; Copyright =C2=A9 2020 Efraim Flashner +;;; Copyright =C2=A9 2021 Lars-Dominik Braun ;;; ;;; This file is part of GNU Guix. ;;; @@ -134,6 +135,15 @@ (apply invoke "python" "./setup.py" command params))) (error "no setup.py found"))) =20 +(define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) + "Ensure packages depending on this package via setuptools work properly, +their advertised endpoints work and their top level modules are importable +without errors." + (let ((sanity-check.py (assoc-ref inputs "sanity-check.py"))) + ;; Make sure the working directory is empty (i.e. no Python modules in= it) + (with-directory-excursion "/tmp" + (invoke "python" sanity-check.py (site-packages inputs outputs))))) + (define* (build #:key use-setuptools? #:allow-other-keys) "Build a given Python package." (call-setuppy "build" '() use-setuptools?) @@ -225,8 +235,7 @@ useful when running checks after installing the package= ." ;; '--invalidation-mode' option, do not generate any. (unless <3.7? (invoke "python" "-m" "compileall" "--invalidation-mode=3Dunchecked-= hash" - out)) - #t)) + out)))) =20 (define* (wrap #:key inputs outputs #:allow-other-keys) (let ((pythonpath (guix-pythonpath inputs))) @@ -262,8 +271,7 @@ installed with setuptools." (easy-install-pth (string-append site-packages "/easy-install.pth= ")) (new-pth (string-append site-packages "/" name ".pth"))) (when (file-exists? easy-install-pth) - (rename-file easy-install-pth new-pth)) - #t)) + (rename-file easy-install-pth new-pth)))) =20 (define* (ensure-no-mtimes-pre-1980 #:rest _) "Ensure that there are no mtimes before 1980-01-02 in the source tree." @@ -275,8 +283,7 @@ installed with setuptools." (ftw "." (lambda (file stat flag) (unless (<=3D early-1980 (stat:mtime stat)) (utime file early-1980 early-1980)) - #t)) - #t)) + #t)))) =20 (define* (enable-bytecode-determinism #:rest _) "Improve determinism of pyc files." @@ -284,8 +291,7 @@ installed with setuptools." (setenv "PYTHONHASHSEED" "0") ;; Prevent Python from creating .pyc files when loading modules (such as ;; when running a test suite). - (setenv "PYTHONDONTWRITEBYTECODE" "1") - #t) + (setenv "PYTHONDONTWRITEBYTECODE" "1")) =20 (define* (ensure-no-cythonized-files #:rest _) "Check the source code for @code{.c} files which may have been pre-gener= ated @@ -296,8 +302,7 @@ by Cython." (string-append (string-drop-right file 3) "c"))) (when (file-exists? generated-file) (format #t "Possible Cythonized file found: ~a~%" generated-file= )))) - (find-files "." "\\.pyx$")) - #t) + (find-files "." "\\.pyx$"))) =20 (define %standard-phases ;; The build phase only builds C extensions and copies the Python source= s, @@ -319,6 +324,7 @@ by Cython." (add-after 'install 'wrap wrap) (add-before 'check 'add-install-to-pythonpath add-install-to-pythonpat= h) (add-before 'check 'add-install-to-path add-install-to-path) + (add-after 'check 'sanity-check sanity-check) (add-before 'strip 'rename-pth-file rename-pth-file))) =20 (define* (python-build #:key inputs (phases %standard-phases) diff --git a/tests/builders.scm b/tests/builders.scm index fdcf38ded3..c5528b2593 100644 --- a/tests/builders.scm +++ b/tests/builders.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2012, 2013, 2014, 2015, 2019 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2021 Lars-Dominik Braun ;;; ;;; This file is part of GNU Guix. ;;; @@ -17,19 +18,19 @@ ;;; along with GNU Guix. If not, see . =20 =20 -(define-module (test-builders) +(define-module (tests builders) #:use-module (guix download) #:use-module (guix build-system) #:use-module (guix build-system gnu) + #:use-module (guix build-system python) #:use-module (guix store) + #:use-module (guix monads) #:use-module (guix utils) #:use-module (guix base32) #:use-module (guix derivations) #:use-module (gcrypt hash) #:use-module (guix tests) - #:use-module ((guix packages) - #:select (package? - package-derivation package-native-search-paths)) + #:use-module (guix packages) #:use-module (gnu packages bootstrap) #:use-module (ice-9 match) #:use-module (srfi srfi-1) @@ -78,4 +79,101 @@ (test-assert "gnu-build-system" (build-system? gnu-build-system)) =20 + +;;; +;;; Test the sanity-check phase of the Python build system. +;;; + +(define-syntax-rule (test-build-package name expect-failure? package) + "Return a test named NAME, building PACKAGE in the external store." + (with-external-store store + (unless store (test-skip 1)) + (let ((build (lambda (p) + (build-derivations + store (list (package-derivation store p)))))) + (if expect-failure? + (test-error name + (store-protocol-error?) + (build package)) + (test-assert name (build package)))))) + +(test-build-package "python-build-system: dummy-ok" #f + (dummy-package "python-dummy-ok" + (build-system python-build-system) + (arguments + `(#:phases + (modify-phases %standard-phases + (replace 'unpack + (lambda _ + (mkdir-p "dummy") + (invoke "touch" "dummy/__init__.py") + (with-output-to-file "setup.py" + (lambda _ + (display "\ +from setuptools import setup +setup(name=3D'dummy-ok', + version=3D'0', + packages=3D['dummy'])")))))))))) + +(test-build-package "python-build-system: dummy-fail-requirements" #t + (dummy-package "python-dummy-fail-requirements" + (build-system python-build-system) + (arguments + `(#:tests? #f + #:phases + (modify-phases %standard-phases + (replace 'unpack + (lambda _ + (mkdir-p "dummy") + (invoke "touch" "dummy/__init__.py") + (with-output-to-file "setup.py" + (lambda _ + (display "\ +from setuptools import setup +setup(name=3D'dummy-fail-requirements', + version=3D'0', + packages=3D['dummy'], + install_requires=3D['nonexistent'])")))))))))) + +(test-build-package "python-build-system: dummy-fail-import" #t + (dummy-package "python-dummy-fail-import" + (build-system python-build-system) + (arguments + `(#:tests? #f + #:phases + (modify-phases %standard-phases + (replace 'unpack + (lambda _ + (mkdir-p "dummy") + (with-output-to-file "dummy/__init__.py" + (lambda _ + (display "import nonexistent"))) + (with-output-to-file "setup.py" + (lambda _ + (display "\ +from setuptools import setup +setup(name=3D'dummy-fail-import', + version=3D'0', + packages=3D['dummy'])")))))))))) + +(test-build-package "python-build-system: dummy-fail-console-script" #f + (dummy-package "python-dummy-fail-console-script" + (build-system python-build-system) + (arguments + `(#:tests? #f + #:phases + (modify-phases %standard-phases + (replace 'unpack + (lambda _ + (mkdir-p "dummy") + (invoke "touch" "dummy/__init__.py") + (with-output-to-file "setup.py" + (lambda _ + (display "\ +from setuptools import setup +setup(name=3D'dummy-fail-console-script', + version=3D'0', + packages=3D['dummy'], + entry_points=3D{'console_scripts': ['broken =3D dummy:nonexistent']}= )")))))))))) + (test-end "builders") --=20 2.30.0 --=-=-=--