* bug#74445: Python respect --cores build argument
@ 2024-11-20 15:10 Greg Hogan
2024-11-20 18:47 ` Sharlatan Hellseher
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Greg Hogan @ 2024-11-20 15:10 UTC (permalink / raw)
To: 74445
#74436 fixed two instances of "--numprocesses" "auto" but a simple
grep shows 18 additional cases (one of which is commented out).
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#74445: Python respect --cores build argument
2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
@ 2024-11-20 18:47 ` Sharlatan Hellseher
2024-11-20 20:54 ` bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages Sharlatan Hellseher
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-20 18:47 UTC (permalink / raw)
To: 74445; +Cc: Greg Hogan
[-- Attachment #1: Type: text/plain, Size: 147 bytes --]
Hi,
There a lot in astronomy module I guess ^.^
Would you like to send a patch adjusting them?
They could be "-n" "auto" as well.
Thanks,
Oleg
[-- Attachment #2: Type: text/html, Size: 395 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages.
2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
2024-11-20 18:47 ` Sharlatan Hellseher
@ 2024-11-20 20:54 ` Sharlatan Hellseher
2024-11-20 20:54 ` bug#74445: [PATCH 1/1] " Sharlatan Hellseher
2024-11-23 14:54 ` bug#74445: [python-team] Respect --cores build argument Sharlatan Hellseher
2024-11-23 16:00 ` Sharlatan Hellseher
3 siblings, 1 reply; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-20 20:54 UTC (permalink / raw)
To: 74445
Cc: Sharlatan Hellseher, Andreas Enge, Eric Bavier,
Lars-Dominik Braun, Marius Bakke, Munyoki Kilyungi,
Sharlatan Hellseher, Tanguy Le Carrour, jgart
Hi Greg,
Thanks for highlighting that, I was too much keen to make tests faster.
This patch adjust all "-n" or "--numprocesses" from "auto" to
"parallel-job-count" parameter.
This quite save to be pushed to master directly:
--8<---------------cut here---------------start------------->8---
Building the following 77 packages would ensure 180 dependent packages are
rebuilt: python-yt-astro-analysis@1.1.3 variant-tools@3.1.2 osc@0.172.0
h-client@0.0a0-1.e6c78b1 python-urlgrabber@4.1.0 xapers@0.8.2 ganeti@3.0.2
awscli@1.36.0 instrain@1.5.4 python-osmnx@1.9.3 conda@22.9.0
python-awscrt@0.23.0 awscli@2.20.0 python-spacy@3.5.3 snakemake@6.15.5
python-scm-sr-ht@0.22.23 awslogs@0.14.0 python-tibanna@5.4.3
python-hicexplorer@3.7.4 python-vaex-core@4.13.0 python-kanon@0.6.6
python-zodipy@1.1.0 python-coolest@0.1.9 python-sncosmo@2.11.1
python-halotools@0.9.1 python-healpy@1.16.6 scamp@2.10.0
python-asdf-compression@0.0.1-0.57cc7e7 python-asdf-zarr@0.0.4
python-regularizepsf@0.4.0 python-statmorph@0.5.7 python-pyxsim@4.4.2
python-viresclient@0.12.0 python-naima@0.10.0 python-aplpy@2.1.0
python-astroalign@2.6.0 ginga-qt5@5.1.0 python-poliastro@0.17.0
python-astroplan@0.10.1 python-bulkvis@2.0.0-2.00a82a9 python-cooltools@0.7.0
fanc@0-1.354401e python-fanc@0.9.25 python-pytorch-geometric@2.4.0 cura@4.13.1
python-sunkit-image@0.5.1 python-radiospectra@0.6.0 python-webbpsf@1.4.0
python-acstools@3.7.2 python-jwst-reffiles@1.0.1 python-sbpy@0.5.0
python-glue-qt@0.3.1 python-glue-astronomy@0.10.0 python-pyrodigal@3.3.0
python-verde@1.8.0 python-scikit-allel@1.3.5 python-pandera@0.18.0
python-modin@0.15.1 python-cesium@0.12.1 synapse@1.29.0 python-sparse@0.14.0
python-demuxem@0.1.7 python-cellbender@0.2.2 python-scikit-bio@0.6.0
python-ikarus@0.0.2 python-muon@0.1.6 python-liana-py@1.1.0
r-doubletcollection@1.1.0-1.c0d62f1 python-episcanpy@0.4.0
python-metacells@0.9.4 python-celltypist@1.6.2 python-hotspotsc@1.1.1
python-multivelo@0.1.2 python-baltica@1.1.2 scregseg@0.1.3
python-magic-impute@1.2.1 python-multiscale-spatial-image@1.0.0
--8<---------------cut here---------------end--------------->8---
Thanks,
Oleg
Sharlatan Hellseher (1):
gnu: Respect --cores build argument in some python packages.
gnu/packages/astronomy.scm | 31 ++++++++++++++++---------------
gnu/packages/check.scm | 2 +-
gnu/packages/databases.scm | 2 +-
gnu/packages/python-web.scm | 6 +++---
gnu/packages/python-xyz.scm | 6 +++---
5 files changed, 24 insertions(+), 23 deletions(-)
base-commit: da8afaa65fe30ae59e1bedbbb231490ad01c013c
--
2.46.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#74445: [PATCH 1/1] gnu: Respect --cores build argument in some python packages.
2024-11-20 20:54 ` bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages Sharlatan Hellseher
@ 2024-11-20 20:54 ` Sharlatan Hellseher
0 siblings, 0 replies; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-20 20:54 UTC (permalink / raw)
To: 74445
Cc: Sharlatan Hellseher, Andreas Enge, Eric Bavier,
Lars-Dominik Braun, Marius Bakke, Munyoki Kilyungi,
Sharlatan Hellseher, Tanguy Le Carrour, jgart
Reported by Greg Hogan <code@greghogan.com> in #74445.
* gnu/packages/astronomy.scm (python-asdf-astropy, python-astropy,
python-photutils, python-poppy, python-regions, python-reproject,
python-sunpy, python-spectral-cube, python-stdatamodels, python-pysiaf,
python-sbpy, python-asdf-coordinates-schemas, python-roman-datamodels,
python-webbpsf, python-yt):
[arguments]<test-flags>: Adjust "-n" or "--numprocess" to respect
"--cores" build argument.
* gnu/packages/check.scm (python-crosshair): Likewise.
* gnu/packages/databases.scm (python-fastparquet, python-pycurl, awscli,
python-s3transfer): Likewise.
* gnu/packages/python-xyz.scm (python-glymur, python-zarr, python-dask):
Likewise.
Change-Id: Ifbc6435e4ad22b0ae822b485bccca41eaa165cc5
---
gnu/packages/astronomy.scm | 31 ++++++++++++++++---------------
gnu/packages/check.scm | 2 +-
gnu/packages/databases.scm | 2 +-
gnu/packages/python-web.scm | 6 +++---
gnu/packages/python-xyz.scm | 6 +++---
5 files changed, 24 insertions(+), 23 deletions(-)
diff --git a/gnu/packages/astronomy.scm b/gnu/packages/astronomy.scm
index 6cc5142370..296ac44dbf 100644
--- a/gnu/packages/astronomy.scm
+++ b/gnu/packages/astronomy.scm
@@ -1653,7 +1653,7 @@ (define-public python-asdf-astropy
(arguments
(list
#:test-flags
- #~(list "-n" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases #~(modify-phases %standard-phases
(add-before 'check 'set-home-env
(lambda _ (setenv "HOME" "/tmp"))))))
@@ -3034,7 +3034,7 @@ (define-public python-astropy
(list
#:test-flags
#~(list "--pyargs" "astropy"
- "--numprocesses" "auto"
+ "--numprocesses" (number->string (parallel-job-count))
"-k" (string-append
;; Skip tests that need remote data.
"not remote_data"
@@ -3924,7 +3924,7 @@ (define-public python-photutils
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'relax-requirements
@@ -4085,7 +4085,7 @@ (define-public python-poppy
(arguments
(list
#:test-flags
- #~(list "-n" "auto")))
+ #~(list "--numprocesses" (number->string (parallel-job-count)))))
(propagated-inputs
;; XXX: With python-synphot (marked as optional) package added to the list
;; it tries to download from remote host during tests and fails. Overall
@@ -4191,7 +4191,7 @@ (define-public python-regions
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
;; setup.py was removed in 84c80a280431adda00641cda5264c7de18b43b2f
@@ -4295,7 +4295,7 @@ (define-public python-reproject
#:test-flags
#~(list "--arraydiff"
"--arraydiff-default-format=fits"
- "--numprocesses" "auto"
+ "--numprocesses" (number->string (parallel-job-count))
"--pyargs" "reproject")
#:phases
#~(modify-phases %standard-phases
@@ -4399,7 +4399,7 @@ (define-public python-sunpy
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Requries SpicePy not packed in Guix yet.
"--ignore=sunpy/coordinates/tests/test_spice.py")
#:phases
@@ -4589,7 +4589,7 @@ (define-public python-spectral-cube
;; See <https://github.com/radio-astro-tools/radio-beam/issues/129>.
#:tests? #f
#:test-flags
- #~(list "-n" "auto")))
+ #~(list "--numprocesses" (number->string (parallel-job-count)))))
(propagated-inputs
(list python-astropy
;; XXX: Currently failing in upstream as it's optional silent
@@ -4981,7 +4981,7 @@ (define-public python-stdatamodels
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Disable tests requiring access to CRDS servers to download
;; ~500MiB of data.
"-k" "not test_crds_selectors_vs_datamodel")
@@ -5450,7 +5450,7 @@ (define-public python-pysiaf
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Disable 2 failing tests, see
;; <https://github.com/spacetelescope/pysiaf/issues/338>
"-k" (string-append "not test_write_jwst_siaf_xlsx"
@@ -5543,7 +5543,8 @@ (define-public python-sbpy
(build-system pyproject-build-system)
(arguments
(list
- #:test-flags #~(list "--numprocesses" "auto")
+ #:test-flags
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-before 'check 'set-home-env
@@ -5835,7 +5836,7 @@ (define python-asdf-coordinates-schemas
(arguments
(list
#:test-flags
- #~(list "-n" "auto")))
+ #~(list "--numprocesses" (number->string (parallel-job-count)))))
(native-inputs
(list python-pytest
python-pytest-xdist
@@ -6137,7 +6138,7 @@ (define-public python-roman-datamodels
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'set-env
@@ -6295,7 +6296,7 @@ (define-public python-webbpsf
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
"-k" (string-append
;; Test requiring network access
"not test_monthly_trending_plot_auto_opdtable"
@@ -6383,7 +6384,7 @@ (define-public python-yt
(list
#:build-backend "setuptools.build_meta"
#:test-flags
- #~(list "--numprocesses" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 6593eadf6a..ef1c08f8f3 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -2744,7 +2744,7 @@ (define-public python-crosshair
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; check_examples_test.py contains failing tests that
;; show what happens if a counterexample is found.
"--ignore=crosshair/examples/check_examples_test.py"
diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 5f762eb5be..142f9090c5 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -5203,7 +5203,7 @@ (define-public python-fastparquet
(arguments
(list
#:test-flags
- #~(list "-n" "auto")
+ #~(list "--numprocesses" (number->string (parallel-job-count)))
#:phases
#~(modify-phases %standard-phases
(add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index c030deeb12..cf3c15857c 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -2026,7 +2026,7 @@ (define-public python-pycurl
(build-system pyproject-build-system)
(arguments
'(#:test-flags
- (list "-n" "auto"
+ (list "--n" (number->string (parallel-job-count))
"-k" (string-append
;; Disable hanginging tests
"not test_multi_socket_select"
@@ -4235,7 +4235,7 @@ (define-public awscli
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Tests require networking.
"--ignore" "tests/integration"
;; It strugles to set PYTHONPATH.
@@ -4909,7 +4909,7 @@ (define-public python-s3transfer
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Tests require networking.
"--ignore" "tests/integration")))
(native-inputs
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index ceb122e08d..3872b511a1 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -2296,7 +2296,7 @@ (define-public python-glymur
(arguments
(list
#:test-flags
- #~(list "--numprocesses" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; Failing test due to inability of ctypes.util.find_library()
;; to determine library path, which is patched above.
"--ignore=tests/test_config.py")
@@ -28320,7 +28320,7 @@ (define-public python-zarr
(arguments
(list
#:test-flags
- #~(list "-n" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
;; This tests are flaky. The pass several times on my laptop
;; but occasionally fail. They fail pretty reliably on the
;; build farm.
@@ -28845,7 +28845,7 @@ (define-public python-dask
(list
;; Avoid coverage
#:test-flags
- #~(list "-n" "auto"
+ #~(list "--numprocesses" (number->string (parallel-job-count))
"-m" "not gpu and not slow and not network"
;; These all fail with different hashes. Doesn't seem
;; problematic.
--
2.46.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* bug#74445: [python-team] Respect --cores build argument.
2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
2024-11-20 18:47 ` Sharlatan Hellseher
2024-11-20 20:54 ` bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages Sharlatan Hellseher
@ 2024-11-23 14:54 ` Sharlatan Hellseher
2024-11-23 16:00 ` Sharlatan Hellseher
3 siblings, 0 replies; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-23 14:54 UTC (permalink / raw)
To: 74445-done
[-- Attachment #1: Type: text/plain, Size: 85 bytes --]
Hi,
Pushed as 2d1a7ec92d91b8346d257e5352b65fdce35070a5 to master.
--
Thanks,
Oleg
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* bug#74445: [python-team] Respect --cores build argument.
2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
` (2 preceding siblings ...)
2024-11-23 14:54 ` bug#74445: [python-team] Respect --cores build argument Sharlatan Hellseher
@ 2024-11-23 16:00 ` Sharlatan Hellseher
3 siblings, 0 replies; 6+ messages in thread
From: Sharlatan Hellseher @ 2024-11-23 16:00 UTC (permalink / raw)
To: 74445
[-- Attachment #1: Type: text/plain, Size: 81 bytes --]
Correct commit: origin/master d3b010c38c449ea17bf7beba48de39acec26ae04
--
Oleg
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-11-23 16:02 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-11-20 15:10 bug#74445: Python respect --cores build argument Greg Hogan
2024-11-20 18:47 ` Sharlatan Hellseher
2024-11-20 20:54 ` bug#74445: [PATCH 0/1] gnu: Respect --cores build argument in some python packages Sharlatan Hellseher
2024-11-20 20:54 ` bug#74445: [PATCH 1/1] " Sharlatan Hellseher
2024-11-23 14:54 ` bug#74445: [python-team] Respect --cores build argument Sharlatan Hellseher
2024-11-23 16:00 ` Sharlatan Hellseher
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).