bug#74445: Python respect --cores build argument

unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed

* bug#74445: Python respect --cores build argument
@ 2024-11-20 15:10 Greg Hogan
  2024-11-20 18:47 ` Sharlatan Hellseher
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: Greg Hogan @ 2024-11-20 15:10 UTC (permalink / raw)
  To: 74445

#74436 fixed two instances of "--numprocesses" "auto" but a simple
grep shows 18 additional cases (one of which is commented out).




^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#74445: Python respect --cores build argument
  2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
@ 2024-11-20 18:47 ` Sharlatan Hellseher
  2024-11-20 20:54 ` bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages Sharlatan Hellseher
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-20 18:47 UTC (permalink / raw)
  To: 74445; +Cc: Greg Hogan

[-- Attachment #1: Type: text/plain, Size: 147 bytes --]

Hi,

There a lot in astronomy module I guess ^.^

Would you like to send a patch adjusting them?

They could be "-n" "auto" as well.

Thanks,
Oleg

[-- Attachment #2: Type: text/html, Size: 395 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages.
  2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
  2024-11-20 18:47 ` Sharlatan Hellseher
@ 2024-11-20 20:54 ` Sharlatan Hellseher
  2024-11-20 20:54   ` bug#74445: [PATCH 1/1] " Sharlatan Hellseher
  2024-11-23 14:54 ` bug#74445: [python-team] Respect --cores build argument Sharlatan Hellseher
  2024-11-23 16:00 ` Sharlatan Hellseher
  3 siblings, 1 reply; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-20 20:54 UTC (permalink / raw)
  To: 74445
  Cc: Sharlatan Hellseher, Andreas Enge, Eric Bavier,
	Lars-Dominik Braun, Marius Bakke, Munyoki Kilyungi,
	Sharlatan Hellseher, Tanguy Le Carrour, jgart

Hi Greg,

Thanks for highlighting that, I was too much keen to make tests faster.

This patch adjust all "-n" or "--numprocesses" from "auto" to
"parallel-job-count" parameter.
This quite save to be pushed to master directly:

--8<---------------cut here---------------start------------->8---
Building the following 77 packages would ensure 180 dependent packages are
rebuilt: python-yt-astro-analysis@1.1.3 variant-tools@3.1.2 osc@0.172.0
h-client@0.0a0-1.e6c78b1 python-urlgrabber@4.1.0 xapers@0.8.2 ganeti@3.0.2
awscli@1.36.0 instrain@1.5.4 python-osmnx@1.9.3 conda@22.9.0
python-awscrt@0.23.0 awscli@2.20.0 python-spacy@3.5.3 snakemake@6.15.5
python-scm-sr-ht@0.22.23 awslogs@0.14.0 python-tibanna@5.4.3
python-hicexplorer@3.7.4 python-vaex-core@4.13.0 python-kanon@0.6.6
python-zodipy@1.1.0 python-coolest@0.1.9 python-sncosmo@2.11.1
python-halotools@0.9.1 python-healpy@1.16.6 scamp@2.10.0
python-asdf-compression@0.0.1-0.57cc7e7 python-asdf-zarr@0.0.4
python-regularizepsf@0.4.0 python-statmorph@0.5.7 python-pyxsim@4.4.2
python-viresclient@0.12.0 python-naima@0.10.0 python-aplpy@2.1.0
python-astroalign@2.6.0 ginga-qt5@5.1.0 python-poliastro@0.17.0
python-astroplan@0.10.1 python-bulkvis@2.0.0-2.00a82a9 python-cooltools@0.7.0
fanc@0-1.354401e python-fanc@0.9.25 python-pytorch-geometric@2.4.0 cura@4.13.1
python-sunkit-image@0.5.1 python-radiospectra@0.6.0 python-webbpsf@1.4.0
python-acstools@3.7.2 python-jwst-reffiles@1.0.1 python-sbpy@0.5.0
python-glue-qt@0.3.1 python-glue-astronomy@0.10.0 python-pyrodigal@3.3.0
python-verde@1.8.0 python-scikit-allel@1.3.5 python-pandera@0.18.0
python-modin@0.15.1 python-cesium@0.12.1 synapse@1.29.0 python-sparse@0.14.0
python-demuxem@0.1.7 python-cellbender@0.2.2 python-scikit-bio@0.6.0
python-ikarus@0.0.2 python-muon@0.1.6 python-liana-py@1.1.0
r-doubletcollection@1.1.0-1.c0d62f1 python-episcanpy@0.4.0
python-metacells@0.9.4 python-celltypist@1.6.2 python-hotspotsc@1.1.1
python-multivelo@0.1.2 python-baltica@1.1.2 scregseg@0.1.3
python-magic-impute@1.2.1 python-multiscale-spatial-image@1.0.0
--8<---------------cut here---------------end--------------->8---

Thanks,
Oleg

Sharlatan Hellseher (1):
  gnu: Respect --cores build argument in some python packages.

 gnu/packages/astronomy.scm  | 31 ++++++++++++++++---------------
 gnu/packages/check.scm      |  2 +-
 gnu/packages/databases.scm  |  2 +-
 gnu/packages/python-web.scm |  6 +++---
 gnu/packages/python-xyz.scm |  6 +++---
 5 files changed, 24 insertions(+), 23 deletions(-)


base-commit: da8afaa65fe30ae59e1bedbbb231490ad01c013c
-- 
2.46.0





^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#74445: [PATCH 1/1] gnu: Respect --cores build argument in some python packages.
  2024-11-20 20:54 ` bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages Sharlatan Hellseher
@ 2024-11-20 20:54   ` Sharlatan Hellseher
  0 siblings, 0 replies; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-20 20:54 UTC (permalink / raw)
  To: 74445
  Cc: Sharlatan Hellseher, Andreas Enge, Eric Bavier,
	Lars-Dominik Braun, Marius Bakke, Munyoki Kilyungi,
	Sharlatan Hellseher, Tanguy Le Carrour, jgart

Reported by Greg Hogan <code@greghogan.com> in #74445.

* gnu/packages/astronomy.scm (python-asdf-astropy, python-astropy,
python-photutils, python-poppy, python-regions, python-reproject,
python-sunpy, python-spectral-cube, python-stdatamodels, python-pysiaf,
python-sbpy, python-asdf-coordinates-schemas, python-roman-datamodels,
python-webbpsf, python-yt):
[arguments]<test-flags>: Adjust "-n" or "--numprocess" to respect
"--cores" build argument.

* gnu/packages/check.scm (python-crosshair): Likewise.

* gnu/packages/databases.scm (python-fastparquet, python-pycurl, awscli,
python-s3transfer): Likewise.

* gnu/packages/python-xyz.scm (python-glymur, python-zarr, python-dask):
Likewise.

Change-Id: Ifbc6435e4ad22b0ae822b485bccca41eaa165cc5
---
 gnu/packages/astronomy.scm  | 31 ++++++++++++++++---------------
 gnu/packages/check.scm      |  2 +-
 gnu/packages/databases.scm  |  2 +-
 gnu/packages/python-web.scm |  6 +++---
 gnu/packages/python-xyz.scm |  6 +++---
 5 files changed, 24 insertions(+), 23 deletions(-)

diff --git a/gnu/packages/astronomy.scm b/gnu/packages/astronomy.scm
index 6cc5142370..296ac44dbf 100644
--- a/gnu/packages/astronomy.scm
+++ b/gnu/packages/astronomy.scm
@@ -1653,7 +1653,7 @@ (define-public python-asdf-astropy
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases #~(modify-phases %standard-phases
                    (add-before 'check 'set-home-env
                      (lambda _ (setenv "HOME" "/tmp"))))))
@@ -3034,7 +3034,7 @@ (define-public python-astropy
      (list
       #:test-flags
       #~(list "--pyargs" "astropy"
-              "--numprocesses" "auto"
+              "--numprocesses" (number->string (parallel-job-count))
               "-k" (string-append
                     ;; Skip tests that need remote data.
                     "not remote_data"
@@ -3924,7 +3924,7 @@ (define-public python-photutils
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
          (add-after 'unpack 'relax-requirements
@@ -4085,7 +4085,7 @@ (define-public python-poppy
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (propagated-inputs
      ;; XXX: With python-synphot (marked as optional) package added to the list
      ;; it tries to download from remote host during tests and fails. Overall
@@ -4191,7 +4191,7 @@ (define-public python-regions
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           ;; setup.py was removed in 84c80a280431adda00641cda5264c7de18b43b2f
@@ -4295,7 +4295,7 @@ (define-public python-reproject
       #:test-flags
       #~(list "--arraydiff"
               "--arraydiff-default-format=fits"
-              "--numprocesses" "auto"
+              "--numprocesses" (number->string (parallel-job-count))
               "--pyargs" "reproject")
       #:phases
       #~(modify-phases %standard-phases
@@ -4399,7 +4399,7 @@ (define-public python-sunpy
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Requries SpicePy not packed in Guix yet.
               "--ignore=sunpy/coordinates/tests/test_spice.py")
       #:phases
@@ -4589,7 +4589,7 @@ (define-public python-spectral-cube
       ;; See <https://github.com/radio-astro-tools/radio-beam/issues/129>.
       #:tests? #f
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (propagated-inputs
      (list python-astropy
            ;; XXX: Currently failing in upstream as it's optional silent
@@ -4981,7 +4981,7 @@ (define-public python-stdatamodels
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Disable tests requiring access to CRDS servers to download
               ;; ~500MiB of data.
               "-k" "not test_crds_selectors_vs_datamodel")
@@ -5450,7 +5450,7 @@ (define-public python-pysiaf
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Disable 2 failing tests, see
               ;; <https://github.com/spacetelescope/pysiaf/issues/338>
               "-k" (string-append "not test_write_jwst_siaf_xlsx"
@@ -5543,7 +5543,8 @@ (define-public python-sbpy
     (build-system pyproject-build-system)
     (arguments
      (list
-      #:test-flags #~(list "--numprocesses" "auto")
+      #:test-flags
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-before 'check 'set-home-env
@@ -5835,7 +5836,7 @@ (define python-asdf-coordinates-schemas
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (native-inputs
      (list python-pytest
            python-pytest-xdist
@@ -6137,7 +6138,7 @@ (define-public python-roman-datamodels
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-after 'unpack 'set-env
@@ -6295,7 +6296,7 @@ (define-public python-webbpsf
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               "-k" (string-append
                     ;; Test requiring network access
                     "not test_monthly_trending_plot_auto_opdtable"
@@ -6383,7 +6384,7 @@ (define-public python-yt
      (list
       #:build-backend "setuptools.build_meta"
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
          (add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 6593eadf6a..ef1c08f8f3 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -2744,7 +2744,7 @@ (define-public python-crosshair
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; check_examples_test.py contains failing tests that
               ;; show what happens if a counterexample is found.
               "--ignore=crosshair/examples/check_examples_test.py"
diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 5f762eb5be..142f9090c5 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -5203,7 +5203,7 @@ (define-public python-fastparquet
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index c030deeb12..cf3c15857c 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -2026,7 +2026,7 @@ (define-public python-pycurl
     (build-system pyproject-build-system)
     (arguments
      '(#:test-flags
-       (list "-n" "auto"
+       (list "--n" (number->string (parallel-job-count))
              "-k" (string-append
                    ;; Disable hanginging tests
                    "not test_multi_socket_select"
@@ -4235,7 +4235,7 @@ (define-public awscli
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Tests require networking.
               "--ignore" "tests/integration"
               ;; It strugles to set PYTHONPATH.
@@ -4909,7 +4909,7 @@ (define-public python-s3transfer
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Tests require networking.
               "--ignore" "tests/integration")))
     (native-inputs
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index ceb122e08d..3872b511a1 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -2296,7 +2296,7 @@ (define-public python-glymur
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Failing test due to inability of ctypes.util.find_library()
               ;; to determine library path, which is patched above.
               "--ignore=tests/test_config.py")
@@ -28320,7 +28320,7 @@ (define-public python-zarr
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; This tests are flaky.  The pass several times on my laptop
               ;; but occasionally fail.  They fail pretty reliably on the
               ;; build farm.
@@ -28845,7 +28845,7 @@ (define-public python-dask
      (list
       ;; Avoid coverage
       #:test-flags
-      #~(list "-n" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               "-m" "not gpu and not slow and not network"
               ;; These all fail with different hashes.  Doesn't seem
               ;; problematic.
-- 
2.46.0





^ permalink raw reply related	[flat|nested] 6+ messages in thread

* bug#74445: [python-team] Respect --cores build argument.
  2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
  2024-11-20 18:47 ` Sharlatan Hellseher
  2024-11-20 20:54 ` bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages Sharlatan Hellseher
@ 2024-11-23 14:54 ` Sharlatan Hellseher
  2024-11-23 16:00 ` Sharlatan Hellseher
  3 siblings, 0 replies; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-23 14:54 UTC (permalink / raw)
  To: 74445-done

[-- Attachment #1: Type: text/plain, Size: 85 bytes --]


Hi,

Pushed as 2d1a7ec92d91b8346d257e5352b65fdce35070a5 to master.

--
Thanks,
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* bug#74445: [python-team] Respect --cores build argument.
  2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
                   ` (2 preceding siblings ...)
  2024-11-23 14:54 ` bug#74445: [python-team] Respect --cores build argument Sharlatan Hellseher
@ 2024-11-23 16:00 ` Sharlatan Hellseher
  3 siblings, 0 replies; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-23 16:00 UTC (permalink / raw)
  To: 74445

[-- Attachment #1: Type: text/plain, Size: 81 bytes --]


Correct commit: origin/master d3b010c38c449ea17bf7beba48de39acec26ae04

--
Oleg

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2024-11-23 16:02 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
2024-11-20 18:47 ` Sharlatan Hellseher
2024-11-20 20:54 ` bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages Sharlatan Hellseher
2024-11-20 20:54   ` bug#74445: [PATCH 1/1] " Sharlatan Hellseher
2024-11-23 14:54 ` bug#74445: [python-team] Respect --cores build argument Sharlatan Hellseher
2024-11-23 16:00 ` Sharlatan Hellseher

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).