From: "Gábor Boskovits" <boskovits@gmail.com>
To: Ricardo Wurmus <rekado@elephly.net>
Cc: 22533@debbugs.gnu.org
Subject: bug#22533: Python bytecode reproducibility
Date: Sun, 4 Mar 2018 16:30:59 +0100 [thread overview]
Message-ID: <CAE4v=pjmdLujBt8uSQduwft2x1-kqjWFbZqWpmam+cKbaeby1Q@mail.gmail.com> (raw)
In-Reply-To: <874llw101c.fsf@elephly.net>
[-- Attachment #1: Type: text/plain, Size: 3002 bytes --]
2018-03-04 13:46 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:
>
> Hi Gábor,
>
> > Nix had this issue, it seems they have a python 3.5 solution, which
> > should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570.
> > WDYT?
>
> Here’s the patch for Nix:
>
> https://patch-diff.githubusercontent.com/raw/
> NixOS/nixpkgs/pull/22585.diff
>
> Here are the relevant changes to the Python packages:
>
> * Python 3.4
>
> substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
> substituteInPlace "Lib/importlib/_bootstrap.py" --replace "source_mtime
> = int(source_stats['mtime'])" "source_mtime = 1"
>
> * Python 3.5
>
> substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
> substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace
> "source_mtime = int(st['mtime'])" "source_mtime = 1"
>
> * Python 3.6
> substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']"
> "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"
> substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace
> "source_mtime = int(st['mtime'])" "source_mtime = 1"
>
>
>
Nice, thanks for the summary.
Can we adopt this as is?
Do we need the 3.4 and 3.5 fix or the 3.6 one is enough?
> For all packages they set these environment variables:
>
> - set PYTHONHASHSEED=0 (for hashes of str, bytes and datetime objects)
>
> - set DETERMINISTIC_BUILD; for conditional patching of the timestamp
> for package builds. The timestamp is not patched in ad-hoc
> environments, because that would mess with Python’s ability to
> determine whether to compile source files.
>
>
Should we set these in python-build-system? What about python booststrap?
I guess we use gnu-build-system there, so bootstrap packages might need to
set these explicitly?
> They also rebuild all bytecode (with the exception of lib2to3 because it
> is Python 2 code) three times, once for each optimization level.
>
> --8<---------------cut here---------------start------------->8---
> + # Determinism: rebuild all bytecode
> + # We exclude lib2to3 because that's Python 2 code which fails
> + # We rebuild three times, once for each optimization level
> + find $out -name "*.py" | $out/bin/python -m compileall -q -f -x
> "lib2to3" -i -
> + find $out -name "*.py" | $out/bin/python -O -m compileall -q -f -x
> "lib2to3" -i -
> + find $out -name "*.py" | $out/bin/python -OO -m compileall -q -f -x
> "lib2to3" -i -
> --8<---------------cut here---------------end--------------->8---
>
>
Do we also have to do this, or should we settle with one optimization
level? Which one?
> --
> Ricardo
>
> GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
> https://elephly.net
>
>
>
[-- Attachment #2: Type: text/html, Size: 4686 bytes --]
next prev parent reply other threads:[~2018-03-04 15:32 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-02 5:15 bug#22533: Non-determinism in python-3 ".pyc" bytecode Leo Famulari
2016-02-02 8:54 ` Leo Famulari
2016-02-02 20:41 ` Ludovic Courtès
2016-02-04 23:17 ` Leo Famulari
2016-03-29 23:11 ` Cyril Roelandt
2016-03-29 23:13 ` Cyril Roelandt
2016-04-06 8:29 ` Ludovic Courtès
2017-05-26 13:41 ` bug#22533: Python bytecode reproducibility Marius Bakke
2018-03-03 22:37 ` Ricardo Wurmus
2018-03-04 9:21 ` Gábor Boskovits
2018-03-04 12:46 ` Ricardo Wurmus
2018-03-04 15:30 ` Gábor Boskovits [this message]
2018-03-04 19:18 ` Ricardo Wurmus
2018-03-05 0:02 ` Ricardo Wurmus
2018-03-05 0:05 ` Ricardo Wurmus
2018-03-05 15:36 ` Gábor Boskovits
2018-03-05 20:33 ` Gábor Boskovits
2018-03-05 21:46 ` Ricardo Wurmus
2018-03-05 22:02 ` Ricardo Wurmus
2018-03-05 22:06 ` Ricardo Wurmus
2018-03-05 23:21 ` Marius Bakke
2018-03-06 13:28 ` Ricardo Wurmus
2018-03-06 14:43 ` Ricardo Wurmus
2018-03-06 14:57 ` Gábor Boskovits
2018-03-08 10:39 ` Gábor Boskovits
2019-01-14 13:40 ` Ricardo Wurmus
2019-02-03 21:22 ` Ricardo Wurmus
2019-02-04 22:39 ` Ludovic Courtès
2018-03-05 9:25 ` Ludovic Courtès
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAE4v=pjmdLujBt8uSQduwft2x1-kqjWFbZqWpmam+cKbaeby1Q@mail.gmail.com' \
--to=boskovits@gmail.com \
--cc=22533@debbugs.gnu.org \
--cc=rekado@elephly.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).