bug#52001: numpy CPU dispatch probably prevents builds of python-numpy from being reproducible

bug#52001: numpy CPU dispatch probably prevents builds of python-numpy from being reproducible

[Ricardo Wurmus]

Numpy accelerates operations with native code.  Early in the build 
it probes the CPU / compiler for supported instruction extensions 
and then proceeds to build extensions for different features.  I’m 
guessing that this won’t work when building on older CPUs, so the 
results of two builds on different CPUs may differ.

It would be helpful if someone could confirm this by building 
numpy on an older CPU and then compare the build result with that 
provided by ci.guix.gnu.org.

-- 
Ricardo

bug#52001: numpy CPU dispatch probably prevents builds of python-numpy from being reproducible

[Maxime Devos]

Ricardo Wurmus schreef op za 20-11-2021 om 15:36 [+0000]:
> Numpy accelerates operations with native code.  Early in the build 
> it probes the CPU / compiler for supported instruction extensions 
> and then proceeds to build extensions for different features.  I’m 
> guessing that this won’t work when building on older CPUs, so the 
> results of two builds on different CPUs may differ.
> 
> It would be helpful if someone could confirm this by building 
> numpy on an older CPU and then compare the build result with that 
> provided by ci.guix.gnu.org.
> 

I built it locally, and have the same result as ci.guix.gnu.org.
However, the bordeaux substitute is different:

$ LANGUAGE=en guix challenge python-numpy
/gnu/store/k311v0843hgq5frcbscylwimqssl1h54-python-numpy-1.17.3
contents differ:
  local hash: 07var0wn8fywxchldz5pjqpnlavrlbc8s81aqwsqyi0i7qlh6ka7
 
https://ci.guix.gnu.org/nar/lzip/k311v0843hgq5frcbscylwimqssl1h54-python-numpy-1.17.3
: 07var0wn8fywxchldz5pjqpnlavrlbc8s81aqwsqyi0i7qlh6ka7
 
https://bordeaux.guix.gnu.org/nar/lzip/k311v0843hgq5frcbscylwimqssl1h54-python-numpy-1.17.3
: 0cbl3q19bshb6ddz8xkcrjzkcmillsqii4z852ybzixyp7rg40qa
 bordeaux.guix.gnu.org  3.9MiB                                 6.7MiB/s
00:01 [##################] 100.0%
1 store items were analyzed:
  - 0 (0.0%) were identical
  - 1 (100.0%) differed
  - 0 (0.0%) were inconclusive

Diffoscope output:

 bordeaux.guix.gnu.org  3.9MiB                                 4.2MiB/s 00:01 [##################] 100.0%
[...]
│ │ │ │ │ │ │ │   --- /tmp/guix-directory.6ivbVt/lib/python3.8/site-packages/numpy/distutils/fcompiler/__pycache__/vast.cpython-38.pyc
│ │ │ │ │ │ │ ├── +++ /gnu/store/k311v0843hgq5frcbscylwimqssl1h54-python-numpy-1.17.3/lib/python3.8/site-packages/numpy/distutils/fcompiler/__pycache__/vast.cpython-38.pyc
│ │ │ │ │ │ │ │ @@ -106,15 +106,15 @@
│ │ │ │ │ │ │ │  00000690: 0104 0106 0104 0104 0106 0104 f906 0904  ................
│ │ │ │ │ │ │ │  000006a0: 0104 0208 0308 06da 085f 5f6d 6169 6e5f  .........__main_
│ │ │ │ │ │ │ │  000006b0: 5f29 01da 036c 6f67 e902 0000 0029 01da  _)...log.....)..
│ │ │ │ │ │ │ │  000006c0: 1463 7573 746f 6d69 7a65 645f 6663 6f6d  .customized_fcom
│ │ │ │ │ │ │ │  000006d0: 7069 6c65 7272 0700 0000 2901 da08 636f  pilerr....)...co
│ │ │ │ │ │ │ │  000006e0: 6d70 696c 6572 2911 da0a 5f5f 6675 7475  mpiler)...__futu
│ │ │ │ │ │ │ │  000006f0: 7265 5f5f 7202 0000 0072 0300 0000 7204  re__r....r....r.
│ │ │ │ │ │ │ │ -00000700: 0000 0072 1900 0000 da1d 6e75 6d70 792e  ...r......numpy.
│ │ │ │ │ │ │ │ +00000700: 0000 0072 1900 0000 5a1d 6e75 6d70 792e  ...r....Z.numpy.
│ │ │ │ │ │ │ │  00000710: 6469 7374 7574 696c 732e 6663 6f6d 7069  distutils.fcompi
│ │ │ │ │ │ │ │  00000720: 6c65 722e 676e 7572 0500 0000 da09 636f  ler.gnur......co
│ │ │ │ │ │ │ │  00000730: 6d70 696c 6572 7372 0600 0000 7227 0000  mpilersr....r'..
│ │ │ │ │ │ │ │  00000740: 00da 0964 6973 7475 7469 6c73 7233 0000  ...distutilsr3..
│ │ │ │ │ │ │ │  00000750: 00da 0d73 6574 5f76 6572 626f 7369 7479  ...set_verbosity
│ │ │ │ │ │ │ │  00000760: da0f 6e75 6d70 792e 6469 7374 7574 696c  ..numpy.distutil
│ │ │ │ │ │ │ │  00000770: 7372 3500 0000 da05 7072 696e 7472 2000  sr5.....printr .

1 bit is different.

Greetings,
Maxime.

bug#52001: numpy CPU dispatch probably prevents builds of python-numpy from being reproducible

[Ludovic Courtès]

Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> Numpy accelerates operations with native code.  Early in the build it
> probes the CPU / compiler for supported instruction extensions and
> then proceeds to build extensions for different features.  I’m
> guessing that this won’t work when building on older CPUs, so the 
> results of two builds on different CPUs may differ.

I don’t see anything in the build log that suggests it might be building
for a specific ISA extension (on x86_64).

On the contrary, it seems to do the right thing by dispatching to the
right optimized implementation at run time, in numpy/core/src/umath.
Apparently this directory contains templates for home-made function
multi-versioning, with variants and dispatch code generated by
numpy/core/code_generators/generate_umath.py.

Besides, I see the same .pyc difference as Maxime, but I don’t think it
relates to ISA extensions.

Ludo’.