From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: bug#22533: Python bytecode reproducibility Date: Mon, 05 Mar 2018 22:46:38 +0100 Message-ID: <87zi3mqjpd.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> <87woyrz8sv.fsf@elephly.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:56521) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1esxxm-0001Qp-OF for bug-guix@gnu.org; Mon, 05 Mar 2018 16:48:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1esxxi-0005ct-Lv for bug-guix@gnu.org; Mon, 05 Mar 2018 16:48:06 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:38452) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1esxxi-0005co-Gz for bug-guix@gnu.org; Mon, 05 Mar 2018 16:48:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1esxxi-0000Bt-3L for bug-guix@gnu.org; Mon, 05 Mar 2018 16:48:02 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-reply-to: List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: =?UTF-8?Q?G=C3=A1bor?= Boskovits Cc: 22533@debbugs.gnu.org G=C3=A1bor Boskovits writes: > 2018-03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits : > >> 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : >> >>> >>> Ricardo Wurmus writes: >>> >>> > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems = with numpy: >>> > >>> > --8<---------------cut here---------------start------------->8--- >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >>> pycache__/__config__.cpython-36.pyc differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >>> pycache__/exec_command.cpython-36.pyc differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >>> pycache__/system_info.cpython-36.pyc differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/__pycache__/__config__.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-3= 6.pyc >>> differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/__pycache__/version.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.p= yc >>> differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/testing/nose_ >>> tools/__pycache__/utils.cpython-36.pyc differ >>> > --8<---------------cut here---------------end--------------->8--- >>> >>> Here=E2=80=99s what diffoscope says: >>> >>> --8<---------------cut here---------------start------------->8--- >>> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache_ >>> _/version.cpython-36.pyc >>> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/ >>> version.cpython-36.pyc >>> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.p= yc >>> @@ -1,8 +1,8 @@ >>> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&....... >>> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&....... >>> 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@.. >>> 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d. >>> 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e. >>> 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0 >>> 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c >>> 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671 >>> 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)... >>> --8<---------------cut here---------------end--------------->8--- >>> >>> In other words: this is the timestamp field of the pyc file. >>> >>> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the >>> python-build-system? >>> >>> >> It seems that the deterministic build patch already landed upstream >> https://github.com/python/cpython/pull/5200, so we might consider >> applying the upstream patches. WDYT? >> > > And also this: https://github.com/python/cpython/pull/4575. > I'm now having a look at this approach. However this second one > seems quite invasive... These patches are for what will become Python 3.7. Python 3.6 does not have support for =E2=80=9Cinvalidation_mode=E2=80=9D, so at least the first= patch would not work for us. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net