From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id +FJ0MQ2W8V9qAwAA0tVLHw (envelope-from ) for ; Sun, 03 Jan 2021 10:01:49 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id iDYvLQ2W8V9RAQAA1q6Kng (envelope-from ) for ; Sun, 03 Jan 2021 10:01:49 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5FE6F9401C0 for ; Sun, 3 Jan 2021 10:01:49 +0000 (UTC) Received: from localhost ([::1]:59302 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kw0Co-00013u-LP for larch@yhetil.org; Sun, 03 Jan 2021 05:01:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:38472) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kw0CH-00013i-2f for guix-devel@gnu.org; Sun, 03 Jan 2021 05:01:13 -0500 Received: from mout-p-102.mailbox.org ([2001:67c:2050::465:102]:55998) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1kw0CC-0005fB-7H for guix-devel@gnu.org; Sun, 03 Jan 2021 05:01:11 -0500 Received: from smtp1.mailbox.org (smtp1.mailbox.org [80.241.60.240]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4D7vR10XY0zQlY3 for ; Sun, 3 Jan 2021 11:01:01 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1609668059; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=U+7u65x+H0UBWLegVbnJ2/B4ZbbHvBuKa8nVwidRsQQ=; b=OJEWTlrVHqqACywdMVSd9KgrTQ7N4XDnwN7maoqOy8lcx1LomaCBGLx2q1feRrrzCntS7M 9Pu1mzTEqhOb3VyQ5IuKKK7DHGt4i2mdnkkOBiWt10kbdFAblTCf5uxQDA5Y2C2v4+6ejQ ME55oNqCLw3qRpzJ25xhiWKyh/J7NBxKTNfmejq3nbcoX5JLtHIVnG7LNiAClli2q8CiGK IK4eTmjEUqpwM6xyzZAAYuK/vajP6qnMUZGOj/rM5XRh6WvaQwWI+Y9OtvKpcx2mQMiCQg 5h2YlF9F9UbjHM4j1os1L1L/iPlWzMCTWfZ54vPapR3IJ4Ravl8cyYUe5d6xaQ== Received: from smtp1.mailbox.org ([80.241.60.240]) by spamfilter04.heinlein-hosting.de (spamfilter04.heinlein-hosting.de [80.241.56.122]) (amavisd-new, port 10030) with ESMTP id DWhVj6KUmiWW for ; Sun, 3 Jan 2021 11:00:57 +0100 (CET) Date: Sun, 3 Jan 2021 11:00:56 +0100 From: Lars-Dominik Braun To: guix-devel@gnu.org Subject: [RFC] Improve Python package quality Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="uOaizfWA8odHYD67" Content-Disposition: inline Content-Transfer-Encoding: 8bit X-MBO-SPAM-Probability: X-Rspamd-Score: -4.52 / 15.00 / 15.00 X-Rspamd-Queue-Id: 1B56D1825 X-Rspamd-UID: 66b720 Received-SPF: pass client-ip=2001:67c:2050::465:102; envelope-from=lars@6xq.net; helo=mout-p-102.mailbox.org X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -3.03 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=6xq.net header.s=MBO0001 header.b=OJEWTlrV; dmarc=pass (policy=none) header.from=6xq.net; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 5FE6F9401C0 X-Spam-Score: -3.03 X-Migadu-Scanner: scn1.migadu.com X-TUID: X2h2QfpYvR5f --uOaizfWA8odHYD67 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Hi, I’d like to propose adding an additional phase to python-build-system to improve Guix’ Python package quality. Python is an interpreted language without any compile-time checks. Any errors are only visible at run-time, including missing or wrong imports. Additionally most Python packages use an additional packaging layer through setuptools, which also declares mandatory and optional dependencies. This can result in buildable, but broken packages. An example of a botched upgrade is fixed in commit 22e06297b1982f75aaadddba616b1052e506e4a0. The package python-httpx was upgraded, but that new version declared a dependency to a newer python-httpcore via setuptools, which wasn’t available and thus packages depending on python-httpx were broken. My proposal adds some build-time checks to guarentee three properties: 1) Depending on the package (distribution in Python language) via setuptools works. This ensures dependencies are complete and have the correct version as determined by setup.{py,cfg}. 2) Console entry points can be loaded. This ensures scripts installed to bin/ actually work. 3) Top-level modules can be imported. This ensures the package is actually usable and has no undeclared dependencies. The attached patch implements all three. I’ve been rebuilding a few Python packages using the patch and uncovered some issues already. I’m sure it’s not perfect yet and thus I’m open to improvements. If this idea is well received I’m willing to rebuild and fix *all* Python packages broken by this new phase. I’m also aware this change needs to go to core-updates. Cheers, Lars --uOaizfWA8odHYD67 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0001-WIP-build-system-python-Validate-installed-package.patch" Content-Transfer-Encoding: 8bit >From 919fd78b1aff6c277baca396ef962a5dcd4e23ae Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 3 Jan 2021 10:30:29 +0100 Subject: [PATCH] [WIP] build-system/python: Validate installed package * guix/build/python-build-system.scm (validate-loadable): New phase. (%standard-phases): Use it. --- guix/build/python-build-system.scm | 51 ++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm index 09bd8465c8..edb772a7a4 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -148,6 +148,56 @@ (format #t "test suite not run~%")) #t) +(define* (validate-loadable #:key tests? inputs outputs #:allow-other-keys) + "Ensure packages depending on this package via setuptools work properly, +their advertised endpoints work and their top level modules are importable +without errors." + (let ((script (string-join +'( +;; Python 2 support. +"from __future__ import print_function" +"import pkg_resources, sys, importlib" +;; Only check site-packages installed by this package, but not dependencies +;; (which pkg_resources.working_set would include). Path supplied via argv. +"ws = pkg_resources.find_distributions (sys.argv[1])" +"for dist in ws:" +" print ('validating', repr (dist.project_name), dist.location)" +" try:" +" req = str (dist.as_requirement ())" +;; dist.activate() is not enough to actually check requirements, we have to +;; .require() it. +" pkg_resources.require (req)" +" except Exception as e:" +" print (req, e)" +" sys.exit (1)" +;; Try to load entry points of console scripts too, making sure they work. They +;; should be removed if they don’t. Other groups may not be safe, as they can +;; depend on optional packages. +" for group, v in dist.get_entry_map ().items ():" +" if group not in {'console_scripts', }:" +" continue" +" for name, ep in v.items ():" +" print ('...trying to load endpoint', group, name)" +" ep.load ()" +;; And finally try to load top level modules. This should not have any +;; side-effects. +" for name in dist.get_metadata_lines ('top_level.txt'):" +" print ('...trying to load module', name)" +" try:" +" importlib.import_module (name)" +;; Ignore non-existent modules, we only want to know if the existing ones work. +" except ModuleNotFoundError:" +" print ('......WARNING: module', name, 'not found, continuing')" +" continue") +"\n"))) + (add-installed-pythonpath inputs outputs) + ;; Make sure the working directory is empty (i.e. no Python modules in it) + (with-directory-excursion "/tmp" + ;; XXX: Cloak command run. Long and unreadable if it fails, provide an + ;; explanation instead. + (invoke "python" "-c" script (site-packages inputs outputs)))) + #t) + (define (python-version python) (let* ((version (last (string-split python #\-))) (components (string-split version #\.)) @@ -267,6 +317,7 @@ installed with setuptools." (replace 'install install) (add-after 'install 'check check) (add-after 'install 'wrap wrap) + (add-after 'check 'validate-loadable validate-loadable) (add-before 'strip 'rename-pth-file rename-pth-file))) (define* (python-build #:key inputs (phases %standard-phases) -- 2.26.2 --uOaizfWA8odHYD67--