From mboxrd@z Thu Jan 1 00:00:00 1970 From: Leo Famulari Subject: bug#22533: Non-determinism in python-3 ".pyc" bytecode Date: Thu, 4 Feb 2016 18:17:08 -0500 Message-ID: <20160204231708.GA1297@jasmine> References: <20160202051544.GA11744@jasmine> <87powezvyo.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="LQksG6bCIzRHxTLp" Content-Transfer-Encoding: 8bit Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35120) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aRTA7-0003FO-Kj for bug-guix@gnu.org; Thu, 04 Feb 2016 18:18:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aRTA2-00080E-MV for bug-guix@gnu.org; Thu, 04 Feb 2016 18:18:07 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:52140) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aRTA2-000809-Hn for bug-guix@gnu.org; Thu, 04 Feb 2016 18:18:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84) (envelope-from ) id 1aRTA2-0007cX-5n for bug-guix@gnu.org; Thu, 04 Feb 2016 18:18:02 -0500 Sender: "Debbugs-submit" Resent-Message-ID: Content-Disposition: inline In-Reply-To: <87powezvyo.fsf@gnu.org> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 22533@debbugs.gnu.org --LQksG6bCIzRHxTLp Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit On Tue, Feb 02, 2016 at 09:41:19PM +0100, Ludovic Courtès wrote: > Could you give it a try and refine as needed? :-) I altered your example as shown in the attached patch. It causes some tests related to timestamps to fail, so I disabled them in a very crude way. The final patch should address those tests more carefully. But, the patch doesn't seem to have the desired effect so I'm asking for help! Here is how I tested the patch: I build python-3 with it, and then `export SOURCE_DATE_EPOCH=1` and enter the resulting Python shell. I manually define the '_w_long' function used by the patched function. Then: print (_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH')))) b'\x01\x00\x00\x00' But, when I leave the Python shell and issue `python3 -m compileall helloworld.py`, the timestamps are present in the compiled bytecode. I can watch the clock "tick" by doing this repeatedly: $ touch helloworld.py && rm -r __pycache__ && \ python3 -m compileall helloworld.py && \ hexdump __pycache__/helloworld.cpython-34.pyc | head -n1 I'm not much of a Python programmer, so I'm stumped. --LQksG6bCIzRHxTLp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-SOURCE_DATE_EPOCH.patch" >From d34a71e4ec4501cb53acd3e15633bc1a05665be9 Mon Sep 17 00:00:00 2001 Message-Id: From: Leo Famulari Date: Wed, 3 Feb 2016 20:44:02 -0500 Subject: [PATCH 1/1] SOURCE_DATE_EPOCH --- .../patches/python-3.4.3-source-date-epoch.patch | 21 +++++++++++++++++++++ gnu/packages/python.scm | 14 +++++++++++++- 2 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 gnu/packages/patches/python-3.4.3-source-date-epoch.patch diff --git a/gnu/packages/patches/python-3.4.3-source-date-epoch.patch b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch new file mode 100644 index 0000000..403b2df --- /dev/null +++ b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch @@ -0,0 +1,21 @@ +diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py +index 5b91c05..a87d178 100644 +--- Lib/importlib/_bootstrap.py ++++ Lib/importlib/_bootstrap.py +@@ -666,8 +666,15 @@ def _compile_bytecode(data, name=None, bytecode_path=None, source_path=None): + def _code_to_bytecode(code, mtime=0, source_size=0): + """Compile a code object into bytecode for writing out to a byte-compiled + file.""" ++ """os and locale are required for the SOURCE_DATE_EPOCH ++ deterministic timestamp conditional.""" ++ import os ++ import locale + data = bytearray(MAGIC_NUMBER) +- data.extend(_w_long(mtime)) ++ if os.getenv('SOURCE_DATE_EPOCH'): ++ data.extend(_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH')))) ++ else: ++ data.extend(_w_long(mtime)) + data.extend(_w_long(source_size)) + data.extend(marshal.dumps(code)) + return data diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 48f65b5..cd366f5 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -173,6 +173,17 @@ ;; gnu-build-system.scm. (setenv "SOURCE_DATE_EPOCH" "1") #t)) + (add-before 'configure 'disable-timestamp-tests + (lambda _ + ;; Filter for existing files, since this only affects + ;; Python-3 if the SOURCE_DATE_EPOCH patch is applied. + (substitute* (filter file-exists? + '("Lib/test/test_importlib/test_abc.py")) + (("test_code_bad_timestamp") "disable_test_code_bad_timestamp")) + (substitute* (filter file-exists? + '("Lib/test/test_importlib/source/test_file_loader.py")) + (("test_old_timestamp") "disable_test_old_timestamp")) + )) (add-before 'configure 'do-not-record-configure-flags (lambda* (#:key configure-flags #:allow-other-keys) ;; Remove configure flags from the installed '_sysconfigdata.py' @@ -268,7 +279,8 @@ data types.") ;; XXX Try removing this patch for python > 3.4.3 "python-disable-ssl-test.patch" "python-3-deterministic-build-info.patch" - "python-3-search-paths.patch"))) + "python-3-search-paths.patch" + "python-3.4.3-source-date-epoch.patch"))) (patch-flags '("-p0")) (sha256 (base32 -- 2.6.3 --LQksG6bCIzRHxTLp--