unofficial mirror of bug-guix@gnu.org 
 help / color / mirror / code / Atom feed
From: Leo Famulari <leo@famulari.name>
To: 22533@debbugs.gnu.org
Subject: bug#22533: Non-determinism in python-3 ".pyc" bytecode
Date: Tue, 2 Feb 2016 00:15:44 -0500	[thread overview]
Message-ID: <20160202051544.GA11744@jasmine> (raw)

While preparing a package for borg [0], I found that the built output
was not reproducible. The problem is that the bytecode compiler [1] for
Python 3.4.3 (our current version) encodes the mtime of the
corresponding Python source file in the output. This is described in
PEP-3147 [2], and the responsible Python code is referenced below [3].

I tested a few of our existing python-3 packages: python-ccm,
python-pysam, and python-scripttest all exhibit the same problem.

We fixed this in python-2 with the patch
python-2.7-source-date-epoch.patch, but I don't know how to write this
patch for python-3.

Can somebody write this patch?

I asked about this on #debian-reproducible and they said that it wasn't
an issue for Debian since they don't ship bytecode, but instead generate
it at install time. Of course, that doesn't really apply to Guix.

I used diffoscope-34 to inspect the build outputs to find this, and you
can see the report here:
https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html

It's first demonstrated in the file
...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc.

The first 2 bytes are the "magic numbers" described in PEP-3147, which
specify the version of the bytecode format. The next 2 bytes are the
problematic timestamp, as described in the PEP-3147.

[0]
http://borgbackup.github.io/

[1]
https://docs.python.org/3/library/py_compile.html

[2]
https://www.python.org/dev/peps/pep-3147/

[3] Check out the Guix git commit 4efc8eb27502c, and from there:
$ tar xf $(./pre-inst-env guix build --source python-3)
$ sed -n 139,140p Python-3.4.3/Lib/py_compile.py
    bytecode = importlib._bootstrap._code_to_bytecode(
            code, source_stats['mtime'], source_stats['size'])

             reply	other threads:[~2016-02-02  5:17 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-02-02  5:15 Leo Famulari [this message]
2016-02-02  8:54 ` bug#22533: Non-determinism in python-3 ".pyc" bytecode Leo Famulari
2016-02-02 20:41 ` Ludovic Courtès
2016-02-04 23:17   ` Leo Famulari
2016-03-29 23:11     ` Cyril Roelandt
2016-03-29 23:13     ` Cyril Roelandt
2016-04-06  8:29       ` Ludovic Courtès
2017-05-26 13:41 ` bug#22533: Python bytecode reproducibility Marius Bakke
2018-03-03 22:37   ` Ricardo Wurmus
2018-03-04  9:21     ` Gábor Boskovits
2018-03-04 12:46       ` Ricardo Wurmus
2018-03-04 15:30         ` Gábor Boskovits
2018-03-04 19:18         ` Ricardo Wurmus
2018-03-05  0:02           ` Ricardo Wurmus
2018-03-05  0:05             ` Ricardo Wurmus
2018-03-05 15:36               ` Gábor Boskovits
2018-03-05 20:33                 ` Gábor Boskovits
2018-03-05 21:46                   ` Ricardo Wurmus
2018-03-05 22:02               ` Ricardo Wurmus
2018-03-05 22:06             ` Ricardo Wurmus
2018-03-05 23:21           ` Marius Bakke
2018-03-06 13:28             ` Ricardo Wurmus
2018-03-06 14:43               ` Ricardo Wurmus
2018-03-06 14:57                 ` Gábor Boskovits
2018-03-08 10:39           ` Gábor Boskovits
2019-01-14 13:40             ` Ricardo Wurmus
2019-02-03 21:22               ` Ricardo Wurmus
2019-02-04 22:39                 ` Ludovic Courtès
2018-03-05  9:25     ` Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://guix.gnu.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160202051544.GA11744@jasmine \
    --to=leo@famulari.name \
    --cc=22533@debbugs.gnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).