From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?G=C3=A1bor?= Boskovits Subject: bug#22533: Python bytecode reproducibility Date: Thu, 8 Mar 2018 11:39:52 +0100 Message-ID: References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a114c9d88c18df00566e44b12" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:33455) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1etsxz-0004xP-4e for bug-guix@gnu.org; Thu, 08 Mar 2018 05:40:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1etsxv-00059t-3O for bug-guix@gnu.org; Thu, 08 Mar 2018 05:40:07 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:42327) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1etsxu-00059n-Tx for bug-guix@gnu.org; Thu, 08 Mar 2018 05:40:03 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1etsxu-0005Nf-ND for bug-guix@gnu.org; Thu, 08 Mar 2018 05:40:02 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <871sgz1wg0.fsf@elephly.net> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Ricardo Wurmus Cc: 22533@debbugs.gnu.org --001a114c9d88c18df00566e44b12 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-04 20:18 GMT+01:00 Ricardo Wurmus : > I have applied this patch locally: > > > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 5f701701a..0d1ecc3c6 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -359,8 +359,42 @@ data types.") > "Lib/ctypes/test/test_win32.py" ; fails on > aarch64 > "Lib/test/test_fcntl.py")) ; fails on > aarch64 > #t)))) > - (arguments (substitute-keyword-arguments (package-arguments python-2= ) > - ((#:tests? _) #t))) > + (arguments > + (substitute-keyword-arguments (package-arguments python-2) > + ((#:tests? _) #t) > + ((#:phases phases) > + `(modify-phases ,phases > + (add-after 'unpack 'patch-timestamp-for-pyc-files > + (lambda _ > + ;; We set DETERMINISTIC_BUILD to only override the mtime > when > + ;; building with Guix, lest we break auto-compilation in > + ;; environments. > + (setenv "DETERMINISTIC_BUILD" "1") > + (substitute* "Lib/py_compile.py" > + (("source_stats\\['mtime'\\]") > + "(1 if 'DETERMINISTIC_BUILD' in os.environ else > source_stats['mtime'])")) > + > + ;; Use deterministic hashes for strings, bytes, and > datetime > + ;; objects. > + (setenv "PYTHONHASHSEED" "0") > + > + ;; Reset mtime when validating bytecode header. > + (substitute* "Lib/importlib/_bootstrap_external.py" > + (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)"= ) > + "source_mtime =3D 1")) > + #t)) > + (add-after 'unpack 'disable-timestamp-tests > + (lambda _ > + (substitute* "Lib/test/test_importlib/ > source/test_file_loader.py" > + (("test_bad_marshal") > + "disable_test_bad_marshal") > + (("test_no_marshal") > + "disable_test_no_marshal") > + (("test_non_code_marshal") > + "disable_test_non_code_marshal")) > + #t)) > + (add-before 'check 'allow-non-deterministic-compilation > + (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t)))))) > (native-search-paths > (list (search-path-specification > (variable "PYTHONPATH") > > > It allows me to build python-six and python-sip reproducibly. It does > not fix problems with Python 2, and I haven=E2=80=99t yet tested if it ca= uses > any new problems. > > It=E2=80=99s a little worrying that I had to disable three more tests tha= t I > think shouldn=E2=80=99t have failed. > > Ok, I've checked the test issue again. If we change the _bootstrap_external.py substitution to: "source_mtime =3D 1 if 'DETERMINISTIC_BUILD' in _os.environ else int(source_stats['mtime'])" the test do not fail any more. WDYT? > What do you think? > > -- > Ricardo > > GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC > https://elephly.net > > --001a114c9d88c18df00566e44b12 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-04 20:18 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:=
I have applied this patch locally:


diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 5f701701a..0d1ecc3c6 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -359,8 +359,42 @@ data types.")
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"Lib/ctypes/test/test_win32.py" ; fails on aarch64
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"Lib/test/test_fcntl.py")) = ; fails on aarch64
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))))=
-=C2=A0 =C2=A0 (arguments (substitute-keyword-arguments (package-arguments = python-2)
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((#:tests? _= ) #t)))
+=C2=A0 =C2=A0 (arguments
+=C2=A0 =C2=A0 =C2=A0(substitute-keyword-arguments (package-arguments pytho= n-2)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0((#:tests? _) #t)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0((#:phases phases)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 `(modify-phases ,phases
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-after 'unpack 'patch= -timestamp-for-pyc-files
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; We set DETERMINI= STIC_BUILD to only override the mtime when
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; building with Gu= ix, lest we break auto-compilation in
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; environments. +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setenv "DETER= MINISTIC_BUILD" "1")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* "= Lib/py_compile.py"
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("sour= ce_stats\\['mtime'\\]")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "(1 if= 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime&#= 39;])"))
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; Use deterministi= c hashes for strings, bytes, and datetime
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; objects.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setenv "PYTHO= NHASHSEED" "0")
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; Reset mtime when= validating bytecode header.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* "= Lib/importlib/_bootstrap_external.py"
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("sour= ce_mtime =3D int\\(source_stats\\['mtime'\\]\\)")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "sourc= e_mtime =3D 1"))
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-after 'unpack 'disab= le-timestamp-tests
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* "= Lib/test/test_importlib/source/test_file_loader.py"
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("test= _bad_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "disab= le_test_bad_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("test= _no_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "disab= le_test_no_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("test= _non_code_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "disab= le_test_non_code_marshal"))
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-before 'check 'allow= -non-deterministic-compilation
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _ (unsetenv "= DETERMINISTIC_BUILD") #t))))))
=C2=A0 =C2=A0 =C2=A0(native-search-paths
=C2=A0 =C2=A0 =C2=A0 (list (search-path-specification
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(variable "PYTHONPATH&= quot;)


It allows me to build python-six and python-sip reproducibly.=C2=A0 It does=
not fix problems with Python 2, and I haven=E2=80=99t yet tested if it caus= es
any new problems.

It=E2=80=99s a little worrying that I had to disable three more tests that = I
think shouldn=E2=80=99t have failed.


Ok, I've checked the test issue ag= ain. If we change the _bootstrap_external.py
substitution to:
"source_mtime =3D 1 if 'DETERMINISTIC_BUILD' in _os.env= iron else int(source_stats['mtime'])"
the test do no= t fail any more. WDYT?

=C2=A0
What do you think?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net


--001a114c9d88c18df00566e44b12--