From: Ricardo Wurmus <rekado@elephly.net>
To: "Gábor Boskovits" <boskovits@gmail.com>
Cc: 22533@debbugs.gnu.org
Subject: bug#22533: Python bytecode reproducibility
Date: Mon, 05 Mar 2018 22:46:38 +0100 [thread overview]
Message-ID: <87zi3mqjpd.fsf@elephly.net> (raw)
In-Reply-To: <CAE4v=piEhWjz9Jei-WFgYke20QgQWvPks6rcoizagpjida-RsQ@mail.gmail.com>
Gábor Boskovits <boskovits@gmail.com> writes:
> 2018-03-05 16:36 GMT+01:00 Gábor Boskovits <boskovits@gmail.com>:
>
>> 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:
>>
>>>
>>> Ricardo Wurmus <rekado@elephly.net> writes:
>>>
>>> > Unfortunately, this doesn’t fix all reproducibility problems with numpy:
>>> >
>>> > --8<---------------cut here---------------start------------->8---
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>>> pycache__/__config__.cpython-36.pyc differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>>> pycache__/exec_command.cpython-36.pyc differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/distutils/__
>>> pycache__/system_info.cpython-36.pyc differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/__pycache__/__config__.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc
>>> differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/__pycache__/version.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
>>> differ
>>> > Binary files /gnu/store/kd06ql8fynlydymzhhn
>>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site-
>>> packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and
>>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/testing/nose_
>>> tools/__pycache__/utils.cpython-36.pyc differ
>>> > --8<---------------cut here---------------end--------------->8---
>>>
>>> Here’s what diffoscope says:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache_
>>> _/version.cpython-36.pyc
>>> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/
>>> version.cpython-36.pyc
>>> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.
>>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
>>> @@ -1,8 +1,8 @@
>>> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&.......
>>> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&.......
>>> 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@..
>>> 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d.
>>> 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e.
>>> 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0
>>> 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c
>>> 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671
>>> 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)...
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> In other words: this is the timestamp field of the pyc file.
>>>
>>> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
>>> python-build-system?
>>>
>>>
>> It seems that the deterministic build patch already landed upstream
>> https://github.com/python/cpython/pull/5200, so we might consider
>> applying the upstream patches. WDYT?
>>
>
> And also this: https://github.com/python/cpython/pull/4575.
> I'm now having a look at this approach. However this second one
> seems quite invasive...
These patches are for what will become Python 3.7. Python 3.6 does not
have support for “invalidation_mode”, so at least the first patch would
not work for us.
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
next prev parent reply other threads:[~2018-03-05 21:48 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-02 5:15 bug#22533: Non-determinism in python-3 ".pyc" bytecode Leo Famulari
2016-02-02 8:54 ` Leo Famulari
2016-02-02 20:41 ` Ludovic Courtès
2016-02-04 23:17 ` Leo Famulari
2016-03-29 23:11 ` Cyril Roelandt
2016-03-29 23:13 ` Cyril Roelandt
2016-04-06 8:29 ` Ludovic Courtès
2017-05-26 13:41 ` bug#22533: Python bytecode reproducibility Marius Bakke
2018-03-03 22:37 ` Ricardo Wurmus
2018-03-04 9:21 ` Gábor Boskovits
2018-03-04 12:46 ` Ricardo Wurmus
2018-03-04 15:30 ` Gábor Boskovits
2018-03-04 19:18 ` Ricardo Wurmus
2018-03-05 0:02 ` Ricardo Wurmus
2018-03-05 0:05 ` Ricardo Wurmus
2018-03-05 15:36 ` Gábor Boskovits
2018-03-05 20:33 ` Gábor Boskovits
2018-03-05 21:46 ` Ricardo Wurmus [this message]
2018-03-05 22:02 ` Ricardo Wurmus
2018-03-05 22:06 ` Ricardo Wurmus
2018-03-05 23:21 ` Marius Bakke
2018-03-06 13:28 ` Ricardo Wurmus
2018-03-06 14:43 ` Ricardo Wurmus
2018-03-06 14:57 ` Gábor Boskovits
2018-03-08 10:39 ` Gábor Boskovits
2019-01-14 13:40 ` Ricardo Wurmus
2019-02-03 21:22 ` Ricardo Wurmus
2019-02-04 22:39 ` Ludovic Courtès
2018-03-05 9:25 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87zi3mqjpd.fsf@elephly.net \
--to=rekado@elephly.net \
--cc=22533@debbugs.gnu.org \
--cc=boskovits@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).