From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ricardo Wurmus Subject: bug#22533: Python bytecode reproducibility Date: Mon, 05 Mar 2018 23:06:51 +0100 Message-ID: <87woyqqiro.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:33282) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1esyH7-0000Vz-C9 for bug-guix@gnu.org; Mon, 05 Mar 2018 17:08:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1esyH4-0001Am-0C for bug-guix@gnu.org; Mon, 05 Mar 2018 17:08:05 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:38465) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1esyH3-0001Ai-Se for bug-guix@gnu.org; Mon, 05 Mar 2018 17:08:01 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1esyH3-0000fP-Lu for bug-guix@gnu.org; Mon, 05 Mar 2018 17:08:01 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-reply-to: <87y3j7z8xk.fsf@elephly.net> List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: =?UTF-8?Q?G=C3=A1bor?= Boskovits Cc: 22533@debbugs.gnu.org Ricardo Wurmus writes: > Ricardo Wurmus writes: > >> I have applied this patch locally: >> >> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm >> index 5f701701a..0d1ecc3c6 100644 >> --- a/gnu/packages/python.scm >> +++ b/gnu/packages/python.scm >> @@ -359,8 +359,42 @@ data types.") >> "Lib/ctypes/test/test_win32.py" ; fails o= n aarch64 >> "Lib/test/test_fcntl.py")) ; fails on aar= ch64 >> #t)))) >> - (arguments (substitute-keyword-arguments (package-arguments python-= 2) >> - ((#:tests? _) #t))) >> + (arguments >> + (substitute-keyword-arguments (package-arguments python-2) >> + ((#:tests? _) #t) >> + ((#:phases phases) >> + `(modify-phases ,phases >> + (add-after 'unpack 'patch-timestamp-for-pyc-files >> + (lambda _ >> + ;; We set DETERMINISTIC_BUILD to only override the mtime= when >> + ;; building with Guix, lest we break auto-compilation in >> + ;; environments. >> + (setenv "DETERMINISTIC_BUILD" "1") >> + (substitute* "Lib/py_compile.py" >> + (("source_stats\\['mtime'\\]") >> + "(1 if 'DETERMINISTIC_BUILD' in os.environ else sourc= e_stats['mtime'])")) >> + >> + ;; Use deterministic hashes for strings, bytes, and date= time >> + ;; objects. >> + (setenv "PYTHONHASHSEED" "0") >> + >> + ;; Reset mtime when validating bytecode header. >> + (substitute* "Lib/importlib/_bootstrap_external.py" >> + (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)= ") >> + "source_mtime =3D 1")) >> + #t)) >> + (add-after 'unpack 'disable-timestamp-tests >> + (lambda _ >> + (substitute* "Lib/test/test_importlib/source/test_file_l= oader.py" >> + (("test_bad_marshal") >> + "disable_test_bad_marshal") >> + (("test_no_marshal") >> + "disable_test_no_marshal") >> + (("test_non_code_marshal") >> + "disable_test_non_code_marshal")) >> + #t)) >> + (add-before 'check 'allow-non-deterministic-compilation >> + (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t)))))) >> (native-search-paths >> (list (search-path-specification >> (variable "PYTHONPATH") >> >> It allows me to build python-six and python-sip reproducibly. It does >> not fix problems with Python 2, and I haven=E2=80=99t yet tested if it c= auses >> any new problems. I should also note that Python 3 itself still contains pyc files with timestamps. This could be the reason why in Nix all pyc files are rebuilt (more than once). -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net