From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: bug#22533: Python bytecode reproducibility Date: Mon, 05 Mar 2018 01:02:15 +0100 Message-ID: <87y3j7z8xk.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:49966) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1esdar-00050P-Uw for bug-guix@gnu.org; Sun, 04 Mar 2018 19:03:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1esdao-0001DT-If for bug-guix@gnu.org; Sun, 04 Mar 2018 19:03:05 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:36963) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1esdao-0001Ct-Db for bug-guix@gnu.org; Sun, 04 Mar 2018 19:03:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1esdan-0002my-Uq for bug-guix@gnu.org; Sun, 04 Mar 2018 19:03:01 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-reply-to: <871sgz1wg0.fsf@elephly.net> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: =?UTF-8?Q?G=C3=A1bor?= Boskovits Cc: 22533@debbugs.gnu.org Ricardo Wurmus writes: > I have applied this patch locally: > > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 5f701701a..0d1ecc3c6 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -359,8 +359,42 @@ data types.") > "Lib/ctypes/test/test_win32.py" ; fails on= aarch64 > "Lib/test/test_fcntl.py")) ; fails on aarc= h64 > #t)))) > - (arguments (substitute-keyword-arguments (package-arguments python-2= ) > - ((#:tests? _) #t))) > + (arguments > + (substitute-keyword-arguments (package-arguments python-2) > + ((#:tests? _) #t) > + ((#:phases phases) > + `(modify-phases ,phases > + (add-after 'unpack 'patch-timestamp-for-pyc-files > + (lambda _ > + ;; We set DETERMINISTIC_BUILD to only override the mtime = when > + ;; building with Guix, lest we break auto-compilation in > + ;; environments. > + (setenv "DETERMINISTIC_BUILD" "1") > + (substitute* "Lib/py_compile.py" > + (("source_stats\\['mtime'\\]") > + "(1 if 'DETERMINISTIC_BUILD' in os.environ else source= _stats['mtime'])")) > + > + ;; Use deterministic hashes for strings, bytes, and datet= ime > + ;; objects. > + (setenv "PYTHONHASHSEED" "0") > + > + ;; Reset mtime when validating bytecode header. > + (substitute* "Lib/importlib/_bootstrap_external.py" > + (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)"= ) > + "source_mtime =3D 1")) > + #t)) > + (add-after 'unpack 'disable-timestamp-tests > + (lambda _ > + (substitute* "Lib/test/test_importlib/source/test_file_lo= ader.py" > + (("test_bad_marshal") > + "disable_test_bad_marshal") > + (("test_no_marshal") > + "disable_test_no_marshal") > + (("test_non_code_marshal") > + "disable_test_non_code_marshal")) > + #t)) > + (add-before 'check 'allow-non-deterministic-compilation > + (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t)))))) > (native-search-paths > (list (search-path-specification > (variable "PYTHONPATH") > > It allows me to build python-six and python-sip reproducibly. It does > not fix problems with Python 2, and I haven=E2=80=99t yet tested if it ca= uses > any new problems. I tested importing modules in an ad-hoc environment =E2=80=94 no problems. Unfortunately, this doesn=E2=80=99t fix all reproducibility problems with n= umpy: --8<---------------cut here---------------start------------->8--- Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config__.= cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy= -1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config__.= cpython-36.pyc differ Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_comman= d.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-num= py-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_comm= and.cpython-36.pyc differ Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_info= .cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-nump= y-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_inf= o.cpython-36.pyc differ Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36= .pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/li= b/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc diffe= r Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py= c and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/p= ython3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/ut= ils.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache_= _/utils.cpython-36.pyc differ --8<---------------cut here---------------end--------------->8--- But the successes with simpler Python packages are promising. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net