From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?G=C3=A1bor?= Boskovits Subject: bug#22533: Python bytecode reproducibility Date: Mon, 5 Mar 2018 21:33:02 +0100 Message-ID: References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> <87woyrz8sv.fsf@elephly.net> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a11446d988c8c0f0566b03bdf" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:37788) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eswo9-0005SX-LL for bug-guix@gnu.org; Mon, 05 Mar 2018 15:34:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eswo6-0006qm-EU for bug-guix@gnu.org; Mon, 05 Mar 2018 15:34:05 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:38377) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eswo6-0006qf-A1 for bug-guix@gnu.org; Mon, 05 Mar 2018 15:34:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1eswo6-0004uV-3p for bug-guix@gnu.org; Mon, 05 Mar 2018 15:34:02 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Ricardo Wurmus Cc: 22533@debbugs.gnu.org --001a11446d988c8c0f0566b03bdf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits : > 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : > >> >> Ricardo Wurmus writes: >> >> > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w= ith numpy: >> > >> > --8<---------------cut here---------------start------------->8--- >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >> pycache__/__config__.cpython-36.pyc differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >> pycache__/exec_command.cpython-36.pyc differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >> pycache__/system_info.cpython-36.pyc differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/__pycache__/__config__.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36= .pyc >> differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/__pycache__/version.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py= c >> differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/testing/nose_ >> tools/__pycache__/utils.cpython-36.pyc differ >> > --8<---------------cut here---------------end--------------->8--- >> >> Here=E2=80=99s what diffoscope says: >> >> --8<---------------cut here---------------start------------->8--- >> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache_ >> _/version.cpython-36.pyc >> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/ >> version.cpython-36.pyc >> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py= c >> @@ -1,8 +1,8 @@ >> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&....... >> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&....... >> 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@.. >> 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d. >> 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e. >> 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0 >> 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c >> 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671 >> 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)... >> --8<---------------cut here---------------end--------------->8--- >> >> In other words: this is the timestamp field of the pyc file. >> >> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the >> python-build-system? >> >> > It seems that the deterministic build patch already landed upstream > https://github.com/python/cpython/pull/5200, so we might consider > applying the upstream patches. WDYT? > And also this: https://github.com/python/cpython/pull/4575. I'm now having a look at this approach. However this second one seems quite invasive... > > >> -- >> Ricardo >> >> GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC >> https://elephly.net >> >> >> > --001a11446d988c8c0f0566b03bdf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits <boskovits@gmail.com&g= t;:
2018-03-05 1:05 GMT+01:00 Ricardo Wurmus &l= t;rekado@elephly.ne= t>:

Ricardo Wurmus <= rekado@elephly.net> writes:

> Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w= ith numpy:
>
> --8<---------------cut here---------------start-------------&g= t;8---
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__<= wbr>pycache__/__config__.cpython-36.pyc and /gnu/store/kd06ql8fynlydym= zhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-pac= kages/numpy/distutils/__pycache__/__config__.cpython-36.pyc diffe= r
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__<= wbr>pycache__/exec_command.cpython-36.pyc and /gnu/store/kd06ql8fynlyd= ymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-p= ackages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc d= iffer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__<= wbr>pycache__/system_info.cpython-36.pyc and /gnu/store/kd06ql8fynlydy= mzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-pa= ckages/numpy/distutils/__pycache__/system_info.cpython-36.pyc dif= fer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycache__/_= _config__.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0= 778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/__p= ycache__/__config__.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycache__/<= wbr>version.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778= dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/__pyca= che__/version.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/testing/nose_= tools/__pycache__/utils.cpython-36.pyc and /gnu/store/kd06ql8fynl= ydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.p= yc differ
> --8<---------------cut here---------------end---------------&g= t;8---

Here=E2=80=99s what diffoscope says:

--8<---------------cut here---------------start------------->8--= -
diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.= 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache__/= version.cpython-36.pyc
--- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14= .0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cp= ython-36.pyc
+++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14= .0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-= 36.pyc
@@ -1,8 +1,8 @@
-00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000=C2=A0 3......Z&.....= ..
+00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000=C2=A0 3......Z&.....= ..
=C2=A000000010: 0000 0000 0000 0000 0001 0000 0040 0000=C2=A0 .............= @..
=C2=A000000020: 0073 2000 0000 6400 5a00 6400 5a01 6400=C2=A0 .s ...d.Z.d.Z= .d.
=C2=A000000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502=C2=A0 Z.d.Z.d.Z.e.s= .e.
=C2=A000000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30=C2=A0 Z.d.S.).z.1.1= 4.0
=C2=A000000050: da28 3639 3134 6262 3431 6630 6662 3363=C2=A0 .(6914bb41f0f= b3c
=C2=A000000060: 3162 6135 3030 6261 6534 6537 6436 3731=C2=A0 1ba500bae4e7d= 671
=C2=A000000070: 6461 3935 3336 3738 3666 544e 2905 da0d=C2=A0 da9536786fTN)= ...
--8<---------------cut here---------------end--------------->8--= -

In other words: this is the timestamp field of the pyc file.

Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
python-build-system?


=
It seems that the deterministic build patch already landed= upstream
applying the upstream patches. WDYT?
=

I'm now having a look at this approach. However this second one
=
seems quite invasive...
=C2=A0
=C2=A0
--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net




--001a11446d988c8c0f0566b03bdf--