bug#22533: Non-determinism in python-3 ".pyc" bytecode

[Leo Famulari <leo@famulari.name>]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

On Tue, Feb 02, 2016 at 09:41:19PM +0100, Ludovic Courtès wrote:
> Could you give it a try and refine as needed?  :-)

I altered your example as shown in the attached patch. It causes some
tests related to timestamps to fail, so I disabled them in a very crude
way. The final patch should address those tests more carefully.

But, the patch doesn't seem to have the desired effect so I'm asking for
help!

Here is how I tested the patch:

I build python-3 with it, and then `export SOURCE_DATE_EPOCH=1` and
enter the resulting Python shell. I manually define the '_w_long'
function used by the patched function. Then: 

print (_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH'))))
b'\x01\x00\x00\x00'

But, when I leave the Python shell and issue `python3 -m compileall
helloworld.py`, the timestamps are present in the compiled bytecode. I
can watch the clock "tick" by doing this repeatedly:

$ touch helloworld.py && rm -r __pycache__ && \
python3 -m compileall helloworld.py &&  \
hexdump __pycache__/helloworld.cpython-34.pyc | head -n1

I'm not much of a Python programmer, so I'm stumped.

[-- Attachment #2: 0001-SOURCE_DATE_EPOCH.patch --]
[-- Type: text/x-diff, Size: 3447 bytes --]

From d34a71e4ec4501cb53acd3e15633bc1a05665be9 Mon Sep 17 00:00:00 2001
Message-Id: <d34a71e4ec4501cb53acd3e15633bc1a05665be9.1454625404.git.leo@famulari.name>
From: Leo Famulari <leo@famulari.name>
Date: Wed, 3 Feb 2016 20:44:02 -0500
Subject: [PATCH 1/1] SOURCE_DATE_EPOCH

---
 .../patches/python-3.4.3-source-date-epoch.patch    | 21 +++++++++++++++++++++
 gnu/packages/python.scm                             | 14 +++++++++++++-
 2 files changed, 34 insertions(+), 1 deletion(-)
 create mode 100644 gnu/packages/patches/python-3.4.3-source-date-epoch.patch

diff --git a/gnu/packages/patches/python-3.4.3-source-date-epoch.patch b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch
new file mode 100644
index 0000000..403b2df
--- /dev/null
+++ b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch
@@ -0,0 +1,21 @@
+diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
+index 5b91c05..a87d178 100644
+--- Lib/importlib/_bootstrap.py
++++ Lib/importlib/_bootstrap.py
+@@ -666,8 +666,15 @@ def _compile_bytecode(data, name=None, bytecode_path=None, source_path=None):
+ def _code_to_bytecode(code, mtime=0, source_size=0):
+     """Compile a code object into bytecode for writing out to a byte-compiled
+     file."""
++    """os and locale are required for the SOURCE_DATE_EPOCH
++    deterministic timestamp conditional."""
++    import os
++    import locale
+     data = bytearray(MAGIC_NUMBER)
+-    data.extend(_w_long(mtime))
++    if os.getenv('SOURCE_DATE_EPOCH'):
++        data.extend(_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH'))))
++    else:
++        data.extend(_w_long(mtime))
+     data.extend(_w_long(source_size))
+     data.extend(marshal.dumps(code))
+     return data
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 48f65b5..cd366f5 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -173,6 +173,17 @@
              ;; gnu-build-system.scm.
              (setenv "SOURCE_DATE_EPOCH" "1")
              #t))
+          (add-before 'configure 'disable-timestamp-tests
+            (lambda _
+              ;; Filter for existing files, since this only affects
+              ;; Python-3 if the SOURCE_DATE_EPOCH patch is applied.
+              (substitute* (filter file-exists?
+                                   '("Lib/test/test_importlib/test_abc.py"))
+                           (("test_code_bad_timestamp") "disable_test_code_bad_timestamp"))
+              (substitute* (filter file-exists?
+                                   '("Lib/test/test_importlib/source/test_file_loader.py"))
+                           (("test_old_timestamp") "disable_test_old_timestamp"))
+              ))
           (add-before 'configure 'do-not-record-configure-flags
             (lambda* (#:key configure-flags #:allow-other-keys)
               ;; Remove configure flags from the installed '_sysconfigdata.py'
@@ -268,7 +279,8 @@ data types.")
                               ;; XXX Try removing this patch for python > 3.4.3
                               "python-disable-ssl-test.patch"
                               "python-3-deterministic-build-info.patch"
-                              "python-3-search-paths.patch")))
+                              "python-3-search-paths.patch"
+                              "python-3.4.3-source-date-epoch.patch")))
               (patch-flags '("-p0"))
               (sha256
                (base32
-- 
2.6.3