From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?G=C3=A1bor?= Boskovits Subject: bug#22533: Python bytecode reproducibility Date: Mon, 5 Mar 2018 16:36:31 +0100 Message-ID: References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> <87woyrz8sv.fsf@elephly.net> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a114412361d206f0566ac1712" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:43038) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1essAm-0006Ar-Nj for bug-guix@gnu.org; Mon, 05 Mar 2018 10:37:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1essAg-0004D1-Gz for bug-guix@gnu.org; Mon, 05 Mar 2018 10:37:08 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:38200) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1essAg-0004Cn-CZ for bug-guix@gnu.org; Mon, 05 Mar 2018 10:37:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1essAg-0002D1-5M for bug-guix@gnu.org; Mon, 05 Mar 2018 10:37:02 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87woyrz8sv.fsf@elephly.net> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Ricardo Wurmus Cc: 22533@debbugs.gnu.org --001a114412361d206f0566ac1712 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : > > Ricardo Wurmus writes: > > > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems wi= th numpy: > > > > --8<---------------cut here---------------start------------->8--- > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/distutils/__pycache__/__config__.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/distutils/__pycache__/__config__.cpython-36= .pyc > differ > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/distutils/__pycache__/exec_command.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/distutils/__pycache__/exec_command.cpython-= 36.pyc > differ > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/distutils/__pycache__/system_info.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/distutils/__pycache__/system_info.cpython-3= 6.pyc > differ > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/__pycache__/__config__.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc diffe= r > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/__pycache__/version.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cpytho= n-36.pyc > differ > > --8<---------------cut here---------------end--------------->8--- > > Here=E2=80=99s what diffoscope says: > > --8<---------------cut here---------------start------------->8--- > diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0{-check,}/lib/python3.6/site-packages/ > numpy/__pycache__/version.cpython-36.pyc > --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check= / > lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc > +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc > @@ -1,8 +1,8 @@ > -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&....... > +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&....... > 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@.. > 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d. > 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e. > 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0 > 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c > 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671 > 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)... > --8<---------------cut here---------------end--------------->8--- > > In other words: this is the timestamp field of the pyc file. > > Maybe this can be avoided by setting DETERMINISTIC_BUILD in the > python-build-system? > > It seems that the deterministic build patch already landed upstream https://github.com/python/cpython/pull/5200, so we might consider applying the upstream patches. WDYT? > -- > Ricardo > > GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC > https://elephly.net > > > --001a114412361d206f0566ac1712 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-05 1:05 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:<= br>

Ricardo Wurmus <rekado@elephly.net= > writes:

> Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w= ith numpy:
>
> --8<---------------cut here---------------start-------------&g= t;8---
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutil= s/__pycache__/__config__.cpython-36.pyc and /gnu/store/kd06ql8fyn= lydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-pac= kages/numpy/distutils/__pycache__/__config__.cpython-36.pyc diffe= r
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutil= s/__pycache__/exec_command.cpython-36.pyc and /gnu/store/kd06ql8f= ynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-p= ackages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc d= iffer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutil= s/__pycache__/system_info.cpython-36.pyc and /gnu/store/kd06ql8fy= nlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-pa= ckages/numpy/distutils/__pycache__/system_info.cpython-36.pyc dif= fer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycach= e__/__config__.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnw= k2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/nump= y/__pycache__/__config__.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycach= e__/version.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2l= h0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/<= wbr>__pycache__/version.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/testing/= nose_tools/__pycache__/utils.cpython-36.pyc and /gnu/store/kd06ql= 8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site= -packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.p= yc differ
> --8<---------------cut here---------------end---------------&g= t;8---

Here=E2=80=99s what diffoscope says:

--8<---------------cut here---------------start------------->8--= -
diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-num= py-1.14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache__/= version.cpython-36.pyc
--- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14= .0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
+++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14= .0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-= 36.pyc
@@ -1,8 +1,8 @@
-00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000=C2=A0 3......Z&.....= ..
+00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000=C2=A0 3......Z&.....= ..
=C2=A000000010: 0000 0000 0000 0000 0001 0000 0040 0000=C2=A0 .............= @..
=C2=A000000020: 0073 2000 0000 6400 5a00 6400 5a01 6400=C2=A0 .s ...d.Z.d.Z= .d.
=C2=A000000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502=C2=A0 Z.d.Z.d.Z.e.s= .e.
=C2=A000000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30=C2=A0 Z.d.S.).z.1.1= 4.0
=C2=A000000050: da28 3639 3134 6262 3431 6630 6662 3363=C2=A0 .(6914bb41f0f= b3c
=C2=A000000060: 3162 6135 3030 6261 6534 6537 6436 3731=C2=A0 1ba500bae4e7d= 671
=C2=A000000070: 6461 3935 3336 3738 3666 544e 2905 da0d=C2=A0 da9536786fTN)= ...
--8<---------------cut here---------------end--------------->8--= -

In other words: this is the timestamp field of the pyc file.

Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
python-build-system?


It seems that the deterministic build patch alrea= dy landed upstream
applying the upstream patches. WDYT?
=C2=A0
--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net



--001a114412361d206f0566ac1712--