bug#22010: Python 2.7.10 not deterministic

bug#22010: Python 2.7.10 not deterministic

[Ludovic Courtès]

As of v0.9.0-238-gd7572b4 (currently ‘tk-update’), Python 2.7.10 is not
reproducible:

--8<---------------cut here---------------start------------->8---
$ diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10 ,py
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/libpython2.7.so.1.0 and ,py/lib/libpython2.7.so.1.0 differ
diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/config/Makefile ,py/lib/python2.7/config/Makefile
31,32c31,32
< abs_srcdir=	/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
< abs_builddir=	/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
---
> abs_srcdir=	/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
> abs_builddir=	/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
146c146
< RUNSHARED=       LD_LIBRARY_PATH=/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
---
> RUNSHARED=       LD_LIBRARY_PATH=/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/getopt.pyc and ,py/lib/python2.7/getopt.pyc differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/getopt.pyo and ,py/lib/python2.7/getopt.pyo differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/lib2to3/Grammar2.7.10.final.0.pickle and ,py/lib/python2.7/lib2to3/Grammar2.7.10.final.0.pickle differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/lib-dynload/array.so and ,py/lib/python2.7/lib-dynload/array.so differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/lib-dynload/gdbm.so and ,py/lib/python2.7/lib-dynload/gdbm.so differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/lib-dynload/pyexpat.so and ,py/lib/python2.7/lib-dynload/pyexpat.so differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/py_compile.pyc and ,py/lib/python2.7/py_compile.pyc differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/py_compile.pyo and ,py/lib/python2.7/py_compile.pyo differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/struct.pyc and ,py/lib/python2.7/struct.pyc differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/struct.pyo and ,py/lib/python2.7/struct.pyo differ
diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/_sysconfigdata.py ,py/lib/python2.7/_sysconfigdata.py
458c458
<  'RUNSHARED': 'LD_LIBRARY_PATH=/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
---
>  'RUNSHARED': 'LD_LIBRARY_PATH=/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
502c502
<  'TESTPYTHON': 'LD_LIBRARY_PATH=/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10 ./python -Wd -3 -E -tt',
---
>  'TESTPYTHON': 'LD_LIBRARY_PATH=/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10 ./python -Wd -3 -E -tt',
526,527c526,527
<  'abs_builddir': '/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
<  'abs_srcdir': '/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
---
>  'abs_builddir': '/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
>  'abs_srcdir': '/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10',
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/_sysconfigdata.pyc and ,py/lib/python2.7/_sysconfigdata.pyc differ
Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/_sysconfigdata.pyo and ,py/lib/python2.7/_sysconfigdata.pyo differ
--8<---------------cut here---------------end--------------->8---

The pyc/pyo differ by one 32-bit value starting on the 5th byte of the
file.

Ludo’.

bug#22010: Python 2.7.10 not deterministic

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> As of v0.9.0-238-gd7572b4 (currently ‘tk-update’), Python 2.7.10 is not
> reproducible:
>
> $ diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10 ,py
> Binary files /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/libpython2.7.so.1.0 and ,py/lib/libpython2.7.so.1.0 differ
> diff -r --no-dereference /gnu/store/q405y94pbnx028mygranaww5pnbqhnq1-python-2.7.10/lib/python2.7/config/Makefile ,py/lib/python2.7/config/Makefile
> 31,32c31,32
> < abs_srcdir=	/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
> < abs_builddir=	/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
> ---
>> abs_srcdir=	/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10
>> abs_builddir=	/gnu/tmp/nix-build-python-2.7.10.drv-0/Python-2.7.10

This problem is sidestepped with commit cb96010, where the daemon always
use /tmp/nix-build*.drv-0 inside the build environment.

It will disappear when we have upgraded the daemon on the build
machines.

Ludo’.

bug#22010: bytecode timestamps

[Leo Famulari]

Those differing bytes are the timestamps of the .py sources files that
correspond to the compiled .pyc / .pyo files. So, the
python-2.7-source-date-epoch.patch is working for software compiled by
the patched compiler, but not on python-2.7 itself.

It's related to <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22533>.

bug#22010: bytecode timestamps

[Ricardo Wurmus]

Leo Famulari <leo@famulari.name> writes:

> Those differing bytes are the timestamps of the .py sources files that
> correspond to the compiled .pyc / .pyo files. So, the
> python-2.7-source-date-epoch.patch is working for software compiled by
> the patched compiler, but not on python-2.7 itself.
>
> It's related to <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22533>.

It looks easy to fix this:

--8<---------------cut here---------------start------------->8---
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 1c4ea720f..34c01bd50 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -276,6 +276,38 @@ data types.")
                         (search-patches "python2-CVE-2018-14647.patch"
                                         "python2-CVE-2018-1000802.patch")))))))
 
+(define-public python-2-rekado
+  (package
+    (inherit python-2)
+    (name "python2-rekado")
+    (arguments
+     `(#:tests? #f ; XXX
+       ,@(substitute-keyword-arguments (package-arguments python-2)
+           ((#:phases phases)
+            `(modify-phases ,phases
+               (add-after 'remove-tests 'rebuild-bytecode
+                 (lambda* (#:key outputs #:allow-other-keys)
+                   (let ((out (assoc-ref outputs "out")))
+                     ;; Disable hash randomization to ensure the generated .pycs
+                     ;; are reproducible.
+                     (setenv "PYTHONHASHSEED" "0")
+                     (for-each
+                      (lambda (opt)
+                        (format #t "Compiling with optimization level: ~a\n"
+                                (if (null? opt) "none" (car opt)))
+                        (for-each (lambda (file)
+                                    (apply invoke
+                                           `(,(string-append out "/bin/python")
+                                             ,@opt
+                                             "-m" "compileall"
+                                             "-f" ; force rebuild
+                                             ;; Don't build lib2to3, because it contains Python 3 code.
+                                             "-x" "lib2to3/.*"
+                                             ,file)))
+                                  (find-files out "\\.py$")))
+                      (list '() '("-O") '("-OO")))
+                     #t))))))))))
+
 (define-public python2-called-python
   ;; Both 2.x and 3.x used to be called "python".  In commit
   ;; a7714d42de2c3082f3609d1e63c83d703fb39cf9 (March 2018), we renamed
   the
--8<---------------cut here---------------end--------------->8---

This Python variant builds reproducibly (I disabled the tests for speed
and did this in a separate package to avoid grafting shenanigans and
rebuilds).  The phase is virtually the same as the phase of the same
name in the Python 3 package; only difference is the use of
“/bin/python” instead of “/bin/python3”.

I wonder if we can simplify this with (setenv "PYTHONHASHSEED" "0")
before any pyc files are built, but I expect that to be problematic.

~~ Ricardo

bug#22010: bytecode timestamps

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> I wonder if we can simplify this with (setenv "PYTHONHASHSEED" "0")
> before any pyc files are built, but I expect that to be problematic.

Doing this has no discernable effect.

I’ll push the change to the core-updates branch.

-- 
Ricardo

bug#22010: bytecode timestamps

[Ricardo Wurmus]

Ricardo Wurmus <rekado@elephly.net> writes:

> I’ll push the change to the core-updates branch.

This is now on core-updates.  Python 2 now builds reproducibly.

--
Ricardo