From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id sAk6AsFBZmRgdQAASxT56A (envelope-from ) for ; Thu, 18 May 2023 17:18:25 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id iF8RAsFBZmRrhAEA9RJhRA (envelope-from ) for ; Thu, 18 May 2023 17:18:25 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A7AFE243DD for ; Thu, 18 May 2023 17:18:24 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pzfNq-0003n6-4l; Thu, 18 May 2023 11:17:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pzfNN-0003KQ-Tw for guix-patches@gnu.org; Thu, 18 May 2023 11:17:11 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pzfNK-0001yh-9h for guix-patches@gnu.org; Thu, 18 May 2023 11:17:07 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pzfNF-0005TT-Up; Thu, 18 May 2023 11:17:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#63571] [PATCH 01/14] tests: pypi: Factorize tarball and wheel file creation. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: lars@6xq.net, jgart@dismail.de, guix-patches@gnu.org Resent-Date: Thu, 18 May 2023 15:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 63571 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 63571@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Lars-Dominik Braun , jgart X-Debbugs-Original-Xcc: Lars-Dominik Braun , jgart Received: via spool by 63571-submit@debbugs.gnu.org id=B63571.168442300620917 (code B ref 63571); Thu, 18 May 2023 15:17:01 +0000 Received: (at 63571) by debbugs.gnu.org; 18 May 2023 15:16:46 +0000 Received: from localhost ([127.0.0.1]:54100 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pzfMy-0005R1-Dv for submit@debbugs.gnu.org; Thu, 18 May 2023 11:16:45 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36032) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pzfMt-0005QH-Tn for 63571@debbugs.gnu.org; Thu, 18 May 2023 11:16:42 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pzfMn-0001u5-VO; Thu, 18 May 2023 11:16:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:References:In-Reply-To:Date:Subject:To: From; bh=TV5eqNoYHxzAbEd8qPVouhgZPISTlzx4ipu5kBGrYoE=; b=oaGYiyIF5YRX2hjvK/Iw qhVyP3yYPcRdFZf1kl8DgLDmnbcPWwHj4rjJ/ZYGVd50Am9+DUM+EjIubo8KIUF5pPRZ5RtTEaoHi kOnOHOGMC35DkBi5kWd8h+S3q5zogQDWADTKDOz76x0xJuYFN+Dvd0GbxWF/wZl8nwAAeyjazuqhT 6yf4/G4WbtHKtHxkvyZC2eMRRkdTbtXxsJ6kJLg6YWxuMP7xLQVWjvrTaO3+1YfulUuUorT8QlMhr Bezly7q7zsvQ0QP4BtkGNrp4jayYWFcU6sY6H37ZBmfaLZ2kQU/0V0p7HQdpUxKQPKtn8qBaidDon kugvQ8DobPfTxQ==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pzfMn-0007G6-Ib; Thu, 18 May 2023 11:16:33 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 18 May 2023 17:16:09 +0200 Message-Id: X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: guix-patches-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=1; s=key1; d=yhetil.org; t=1684423104; a=rsa-sha256; cv=none; b=gAXfsWQ6JAw+0bpw4nNBIVZy7sEgR/JM0fTyJ1EmC9+tAZT5UcUAufURJiAbOgudT7H1si 5tTVSzeeyPxqDC+APN5ywU3DYQHE4lg0N1/TvV8UvBvc1bWnMnA4D6/UbTVoJSlVdwYzI8 xEeHuLrsxN2c0HwJcaWOepFEsTs9bLveohgj8OUMRXjeAaKdAQRu5P3ATQcb4XaCXzdTbx Gj9n403Y+jR/G9csdvxCuy6Y38MbHHES4VabQkNVHKhMIvGNhDA1+Qy8CMF0G1guWGd/VR tuFYHqqwpY4MuIYD21zWcI0DPXb78/8vwGMJHxuXg+0jg0DkURLCUcPtC7xB5A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gnu.org header.s=fencepost-gnu-org header.b=oaGYiyIF; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1684423104; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=JIy56dlo3rbhTxUoxrdOd0pBIJGzwC/+57tesMdUERQ=; b=hDrGYHMd00iB3m5TPhk0a1J4Ec3q7fFCoXQCCX/n/FGY2+e5pq0ZYZLJFy2dnzDMa170a3 79IkipP/n6+TEy3RcwuA43JmssgQrC+SX84qctAUNEh6KTFpRJbHAdn7N/iAR9394deG5z JfygnCXXnamro8zF6+3P8VyS2t7D79q6IsNw2eSZg4Qira1vMwwGhq2TMpafzDcnXpir55 4i6eu1qmFNTyIRYn8nYejmqs7NlyZi+dxDmNJNHko20T6vJxxiS7615q4abtzsnrpyJNOw jDbwXjFz+QJ6d3IRnfyy2/g8XNOMa7nZm45uT8V8mfWW0F+aw4SNnsUKn6wptA== X-Migadu-Spam-Score: -2.41 X-Spam-Score: -2.41 X-Migadu-Queue-Id: A7AFE243DD X-Migadu-Scanner: scn0.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=fail ("body hash did not verify") header.d=gnu.org header.s=fencepost-gnu-org header.b=oaGYiyIF; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-patches-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-patches-bounces+larch=yhetil.org@gnu.org" X-TUID: No1Xq+O/g7HM * tests/pypi.scm (sample-directory): New variable. (pypi-tarball, wheel-file): New procedures. ("pypi->guix-package, no wheel") ("pypi->guix-package, wheels") ("pypi->guix-package, no usable requirement file.") ("pypi->guix-package, package name contains \"-\" followed by digits"): Use them. --- tests/pypi.scm | 126 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 82 insertions(+), 44 deletions(-) diff --git a/tests/pypi.scm b/tests/pypi.scm index 1ddcc542ff..1c85e6a16f 100644 --- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -28,8 +28,12 @@ (define-module (test-pypi) #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) - #:use-module ((guix build utils) #:select (delete-file-recursively which mkdir-p)) + #:use-module ((guix build utils) + #:select (delete-file-recursively + which mkdir-p + with-directory-excursion)) #:use-module ((guix diagnostics) #:select (guix-warning-port)) + #:use-module ((guix build syscalls) #:select (mkdtemp!)) #:use-module (json) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) @@ -131,6 +135,58 @@ (define test-metadata-with-extras-jedi "\ Requires-Dist: pytest (>=3.1.0); extra == 'testing' ") +(define sample-directory + ;; Directory containing tarballs and .whl files for this test. + (let ((template (string-append (or (getenv "TMPDIR") "/tmp") + "/guix-pypi-test-XXXXXX"))) + (mkdtemp! template))) + +(define (pypi-tarball name specs) + "Return a PyPI tarball called NAME suffixed with '.tar.gz' and containing +the files specified in SPECS. Return its file name." + (let ((directory (in-vicinity sample-directory name)) + (tarball (in-vicinity sample-directory (string-append name ".tar.gz")))) + (false-if-exception (delete-file tarball)) + (mkdir-p directory) + (for-each (match-lambda + ((file content) + (mkdir-p (in-vicinity directory (dirname file))) + (call-with-output-file (in-vicinity directory file) + (lambda (port) + (display content port))))) + specs) + (parameterize ((current-output-port (%make-void-port "w0"))) + (system* "tar" "-C" sample-directory "-czvf" tarball + (basename directory))) + (delete-file-recursively directory) + tarball)) + +(define (wheel-file name specs) + "Return a Wheel file called NAME suffixed with '.whl' and containing the +files specified by SPECS. Return its file name." + (let* ((directory (in-vicinity sample-directory + (string-append name ".dist-info"))) + (zip-file (in-vicinity sample-directory + (string-append name ".zip"))) + (whl-file (in-vicinity sample-directory + (string-append name ".whl")))) + (false-if-exception (delete-file whl-file)) + (mkdir-p directory) + (for-each (match-lambda + ((file content) + (mkdir-p (in-vicinity directory (dirname file))) + (call-with-output-file (in-vicinity directory file) + (lambda (port) + (display content port))))) + specs) + ;; zip always adds a "zip" extension to the file it creates, + ;; so we need to rename it. + (with-directory-excursion (dirname directory) + (system* "zip" "-qr" zip-file (basename directory))) + (rename-file zip-file whl-file) + (delete-file-recursively directory) + whl-file)) + (test-begin "pypi") @@ -224,17 +280,13 @@ (define test-metadata-with-extras-jedi "\ (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" - (begin - ;; Unusual requires.txt location should still be found. - (mkdir-p "foo-1.0.0/src/bizarre.egg-info") - (with-output-to-file "foo-1.0.0/src/bizarre.egg-info/requires.txt" - (lambda () - (display test-requires.txt))) - (parameterize ((current-output-port (%make-void-port "rw+"))) - (system* "tar" "czvf" file-name "foo-1.0.0/")) - (delete-file-recursively "foo-1.0.0") + ;; Unusual requires.txt location should still be found. + (let ((tarball (pypi-tarball "foo-1.0.0" + `(("src/bizarre.egg-info/requires.txt" + ,test-requires.txt))))) + (copy-file tarball file-name) (set! test-source-hash - (call-with-input-file file-name port-sha256)))) + (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch @@ -279,28 +331,18 @@ (define test-metadata-with-extras-jedi "\ (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" - (begin - (mkdir-p "foo-1.0.0/foo.egg-info/") - (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" - (lambda () - (display "wrong data to make sure we're testing wheels "))) - (parameterize ((current-output-port (%make-void-port "rw+"))) - (system* "tar" "czvf" file-name "foo-1.0.0/")) - (delete-file-recursively "foo-1.0.0") + (let ((tarball (pypi-tarball + "foo-1.0.0" + '(("foo-1.0.0/foo.egg-info/requires.txt" + "wrong data \ +to make sure we're testing wheels"))))) + (copy-file tarball file-name) (set! test-source-hash (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" - (begin - (mkdir "foo-1.0.0.dist-info") - (with-output-to-file "foo-1.0.0.dist-info/METADATA" - (lambda () - (display test-metadata))) - (let ((zip-file (string-append file-name ".zip"))) - ;; zip always adds a "zip" extension to the file it creates, - ;; so we need to rename it. - (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/METADATA") - (rename-file zip-file file-name)) - (delete-file-recursively "foo-1.0.0.dist-info"))) + (let ((wheel (wheel-file "foo-1.0.0" + `(("METADATA" ,test-metadata))))) + (copy-file wheel file-name))) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -342,12 +384,11 @@ (define test-metadata-with-extras-jedi "\ (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" - (mkdir-p "foo-1.0.0/foo.egg-info/") - (parameterize ((current-output-port (%make-void-port "rw+"))) - (system* "tar" "czvf" file-name "foo-1.0.0/")) - (delete-file-recursively "foo-1.0.0") - (set! test-source-hash - (call-with-input-file file-name port-sha256))) + (let ((tarball (pypi-tarball "foo-1.0.0" + '(("foo.egg-info/.empty" ""))))) + (copy-file tarball file-name) + (set! test-source-hash + (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch @@ -388,15 +429,11 @@ (define test-metadata-with-extras-jedi "\ (lambda (url file-name) (match url ("https://example.com/foo-99-1.0.0.tar.gz" - (begin + (let ((tarball (pypi-tarball "foo-99-1.0.0" + `(("src/bizarre.egg-info/requires.txt" + ,test-requires.txt))))) ;; Unusual requires.txt location should still be found. - (mkdir-p "foo-99-1.0.0/src/bizarre.egg-info") - (with-output-to-file "foo-99-1.0.0/src/bizarre.egg-info/requires.txt" - (lambda () - (display test-requires.txt))) - (parameterize ((current-output-port (%make-void-port "rw+"))) - (system* "tar" "czvf" file-name "foo-99-1.0.0/")) - (delete-file-recursively "foo-99-1.0.0") + (copy-file tarball file-name) (set! test-source-hash (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-99-1.0.0-py2.py3-none-any.whl" #f) @@ -434,3 +471,4 @@ (define test-metadata-with-extras-jedi "\ (pk 'fail x #f)))))) (test-end "pypi") +(delete-file-recursively sample-directory) -- 2.40.1