[-- Attachment #1: Type: text/plain, Size: 884 bytes --] Hi Guix! this patch (going to core-updates) removes files from the output of python2 and python3, in an attempt to save some space. First, I noticed that python3 shipped windows binaries in its source distribution and installs them to the output, in lib/python3.8/distutils/command/*.exe, lib/python3.8/site-packages/*.exe and lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them from the snippet, so they should not be installed anymore. I also noticed that python installs files that were built during the test phase, so I remove them, but we need to check they are not actually needed. Other distros ship them in a separate package, so maybe we could have a separate test output if this is causing issues? I haven't yet tested the patch, because it needs to rebuild part of the bootstrap chain. I'll report progress and update my patch depending on the results! [-- Attachment #2: 0001-gnu-python-Remove-unnecessary-files.patch --] [-- Type: text/x-patch, Size: 2238 bytes --] From 63ebe5c610ff8587a73641e4139dc14a4d3e379e Mon Sep 17 00:00:00 2001 From: Julien Lepiller <julien@lepiller.eu> Date: Wed, 17 Mar 2021 13:33:22 +0100 Subject: [PATCH] gnu: python: Remove unnecessary files. * gnu/packages/python.scm (python2)[arguments]: Remove files built during the test phase. (python3)[arguments]: Also remove windows binaries shipped with the sources. --- gnu/packages/python.scm | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 49d43ab008..3bb30ca011 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -294,7 +294,16 @@ (not (string-prefix? "test_support." file)))))) - (call-with-output-file "__init__.py" (const #t))))))))) + (call-with-output-file "__init__.py" (const #t)))) + (let ((libdir (string-append out "/lib/" pythonX.Y))) + (for-each + (lambda (directory) + (let ((dir (string-append libdir "/" directory))) + (when (file-exists? dir) + (delete-file-recursively dir)))) + '("email/test" "ctypes/test" "unittest/test" "tkinter/test" + "sqlite3/test" "bsddb/test" "lib-tk/test" "lib2to3/tests" + "json/tests" "distutils/tests")))))))) (add-after 'remove-tests 'rebuild-bytecode (lambda* (#:key outputs #:allow-other-keys) (let ((out (assoc-ref outputs "out"))) @@ -424,6 +433,9 @@ data types.") (substitute* "Modules/Setup" ;; Link Expat instead of embedding the bundled one. (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n")) + ;; Delete windows binaries + (for-each delete-file + (find-files "Lib/distutils/command" ".*.exe$")) #t)))) (arguments (substitute-keyword-arguments (package-arguments python-2) -- 2.30.0
[-- Attachment #1: Type: text/plain, Size: 1402 bytes --] On Wed, Mar 17, 2021 at 01:41:02PM +0100, Julien Lepiller wrote: > Hi Guix! > > this patch (going to core-updates) removes files from the output of > python2 and python3, in an attempt to save some space. First, I noticed > that python3 shipped windows binaries in its source distribution and > installs them to the output, in lib/python3.8/distutils/command/*.exe, > lib/python3.8/site-packages/*.exe and > lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them > from the snippet, so they should not be installed anymore. > > I also noticed that python installs files that were built during the > test phase, so I remove them, but we need to check they are not > actually needed. Other distros ship them in a separate package, so > maybe we could have a separate test output if this is causing issues? > > I haven't yet tested the patch, because it needs to rebuild part of the > bootstrap chain. I'll report progress and update my patch depending > on the results! I got stuck on python-minimal with test-asyncio but then it didn't hang the second time I ran the build. I don't have a good pure python package I can think of to test the change. -- Efraim Flashner <efraim@flashner.co.il> אפרים פלשנר GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --]
[-- Attachment #1: Type: text/plain, Size: 1394 bytes --] Le Wed, 17 Mar 2021 13:41:02 +0100, Julien Lepiller <julien@lepiller.eu> a écrit : > Hi Guix! > > this patch (going to core-updates) removes files from the output of > python2 and python3, in an attempt to save some space. First, I > noticed that python3 shipped windows binaries in its source > distribution and installs them to the output, in > lib/python3.8/distutils/command/*.exe, > lib/python3.8/site-packages/*.exe and > lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them > from the snippet, so they should not be installed anymore. > > I also noticed that python installs files that were built during the > test phase, so I remove them, but we need to check they are not > actually needed. Other distros ship them in a separate package, so > maybe we could have a separate test output if this is causing issues? > > I haven't yet tested the patch, because it needs to rebuild part of > the bootstrap chain. I'll report progress and update my patch > depending on the results! I discovered other .exe in python3, so this updated patch deletes them. Since I modify the python from commencement.scm by changing a phase in python-2.7, I decided to also change the definition there to remove the offending files. I managed to build up to python-3.9 without any issue, but haven't tested building more than a handful of python packages. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-gnu-python-Remove-unnecessary-files.patch --] [-- Type: text/x-patch, Size: 6312 bytes --] From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00 2001 From: Julien Lepiller <julien@lepiller.eu> Date: Wed, 17 Mar 2021 13:33:22 +0100 Subject: [PATCH 1/2] gnu: python: Remove unnecessary files. * gnu/packages/python.scm (python2)[arguments]: Remove files built during the test phase. (python3)[arguments]: Also remove windows binaries shipped with the sources. --- gnu/packages/commencement.scm | 9 ++++++++- gnu/packages/python.scm | 35 +++++++++++++++++++++++++++++++++-- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm index b2921a4a48..b2b9651444 100644 --- a/gnu/packages/commencement.scm +++ b/gnu/packages/commencement.scm @@ -10,6 +10,7 @@ ;;; Copyright © 2020 Timothy Sample <samplet@ngyro.com> ;;; Copyright © 2020 Guy Fleury Iteriteka <gfleury@disroot.org> ;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com> +;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu> ;;; ;;; This file is part of GNU Guix. ;;; @@ -3098,7 +3099,12 @@ memoized as a function of '%current-system'." (delete-file-recursively "Modules/expat") (substitute* "Modules/Setup.dist" ;; Link Expat instead of embedding the bundled one. - (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))))))) + (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n")) + ;; Delete windows binaries + (for-each delete-file + (find-files "Lib/distutils/command" ".*.exe$")) + (for-each delete-file + (find-files "Lib/ensurepip" ".*.whl$"))))))) (inputs `(,@(%boot0-inputs) ("expat" ,expat-sans-tests))) ;remove OpenSSL, zlib, etc. @@ -3129,6 +3135,7 @@ memoized as a function of '%current-system'." ('add-after unpack apply-alignment-patch _)) `(modify-phases ,original-phases ,@changes)) (_ phases))) + (delete 'remove-windows-binaries) (add-before 'configure 'disable-modules (lambda _ (substitute* "setup.py" diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 49d43ab008..da9859b11a 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -31,7 +31,7 @@ ;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org> ;;; Copyright © 2016, 2017 Alex Vong <alexvong1995@gmail.com> ;;; Copyright © 2016, 2017, 2018 Arun Isaac <arunisaac@systemreboot.net> -;;; Copyright © 2016, 2017, 2018 Julien Lepiller <julien@lepiller.eu> +;;; Copyright © 2016, 2017, 2018, 2021 Julien Lepiller <julien@lepiller.eu> ;;; Copyright © 2016, 2017, 2018 Tobias Geerinckx-Rice <me@tobias.gr> ;;; Copyright © 2016, 2017 Thomas Danckaert <post@thomasdanckaert.be> ;;; Copyright © 2017 Carlo Zancanaro <carlo@zancanaro.id.au> @@ -294,7 +294,16 @@ (not (string-prefix? "test_support." file)))))) - (call-with-output-file "__init__.py" (const #t))))))))) + (call-with-output-file "__init__.py" (const #t)))) + (let ((libdir (string-append out "/lib/" pythonX.Y))) + (for-each + (lambda (directory) + (let ((dir (string-append libdir "/" directory))) + (when (file-exists? dir) + (delete-file-recursively dir)))) + '("email/test" "ctypes/test" "unittest/test" "tkinter/test" + "sqlite3/test" "bsddb/test" "lib-tk/test" "lib2to3/tests" + "json/tests" "distutils/tests")))))))) (add-after 'remove-tests 'rebuild-bytecode (lambda* (#:key outputs #:allow-other-keys) (let ((out (assoc-ref outputs "out"))) @@ -424,6 +433,9 @@ data types.") (substitute* "Modules/Setup" ;; Link Expat instead of embedding the bundled one. (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n")) + ;; Delete windows binaries + (for-each delete-file + (find-files "Lib/distutils/command" ".*.exe$")) #t)))) (arguments (substitute-keyword-arguments (package-arguments python-2) @@ -469,6 +481,23 @@ data types.") ,@(if (hurd-system?) `((delete 'patch-regen-for-hurd)) ;regen was removed after 3.5.9 '()) + (add-after 'unpack 'remove-windows-binaries + (lambda _ + ;; Delete .exe from embedded .whl (zip) files + (for-each + (lambda (whl) + (let ((dir "whl-content")) + (mkdir-p dir) + (with-directory-excursion dir + (let ((whl (string-append "../" whl))) + (invoke "unzip" whl) + (for-each delete-file + (find-files "." ".*.exe$")) + (delete-file whl) + (apply invoke "zip" "-X" whl + (find-files "." ".*" #:directories? #t)))) + (delete-file-recursively dir))) + (find-files "Lib/ensurepip" ".*.whl$")))) (add-before 'check 'set-TZDIR (lambda* (#:key inputs native-inputs #:allow-other-keys) ;; test_email requires the Olson time zone database. @@ -503,6 +532,8 @@ data types.") ,(customize-site version)))))) (native-inputs `(("tzdata" ,tzdata-for-tests) + ("unzip" ,unzip) + ("zip" ,(@ (gnu packages compression) zip)) ,@(if (%current-target-system) `(("python3" ,this-package)) '()) -- 2.30.0
Hi Julien, Julien Lepiller <julien@lepiller.eu> skribis: > Le Wed, 17 Mar 2021 13:41:02 +0100, > Julien Lepiller <julien@lepiller.eu> a écrit : > >> Hi Guix! >> >> this patch (going to core-updates) removes files from the output of >> python2 and python3, in an attempt to save some space. First, I >> noticed that python3 shipped windows binaries in its source >> distribution and installs them to the output, in >> lib/python3.8/distutils/command/*.exe, >> lib/python3.8/site-packages/*.exe and >> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them >> from the snippet, so they should not be installed anymore. >> >> I also noticed that python installs files that were built during the >> test phase, so I remove them, but we need to check they are not >> actually needed. Other distros ship them in a separate package, so >> maybe we could have a separate test output if this is causing issues? >> >> I haven't yet tested the patch, because it needs to rebuild part of >> the bootstrap chain. I'll report progress and update my patch >> depending on the results! > > I discovered other .exe in python3, so this updated patch deletes them. > Since I modify the python from commencement.scm by changing a phase in > python-2.7, I decided to also change the definition there to remove the > offending files. Woow, good catch. > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00 2001 > From: Julien Lepiller <julien@lepiller.eu> > Date: Wed, 17 Mar 2021 13:33:22 +0100 > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files. > > * gnu/packages/python.scm (python2)[arguments]: Remove files built > during the test phase. > (python3)[arguments]: Also remove windows binaries shipped with the > sources. Unless it causes bootstrapping issues, I recommend removing those files from a snippet so that ‘guix build -S python’ returns the cleaned-up source. WDYT? Thanks, Ludo’.
Le Fri, 19 Mar 2021 13:27:06 +0100,
Ludovic Courtès <ludo@gnu.org> a écrit :
> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
> > 2001 From: Julien Lepiller <julien@lepiller.eu>
> > Date: Wed, 17 Mar 2021 13:33:22 +0100
> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
> >
> > * gnu/packages/python.scm (python2)[arguments]: Remove files built
> > during the test phase.
> > (python3)[arguments]: Also remove windows binaries shipped with the
> > sources.
>
> Unless it causes bootstrapping issues, I recommend removing those
> files from a snippet so that ‘guix build -S python’ returns the
> cleaned-up source.
>
> WDYT?
>
> Thanks,
> Ludo’.
Hi, sorry for the late reply. I need to use unzip on these files to
remove some of their content, and rezip them later. How can I do that
from a snippet?
Hi, Julien Lepiller <julien@lepiller.eu> skribis: > Le Fri, 19 Mar 2021 13:27:06 +0100, > Ludovic Courtès <ludo@gnu.org> a écrit : > >> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00 >> > 2001 From: Julien Lepiller <julien@lepiller.eu> >> > Date: Wed, 17 Mar 2021 13:33:22 +0100 >> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files. >> > >> > * gnu/packages/python.scm (python2)[arguments]: Remove files built >> > during the test phase. >> > (python3)[arguments]: Also remove windows binaries shipped with the >> > sources. >> >> Unless it causes bootstrapping issues, I recommend removing those >> files from a snippet so that ‘guix build -S python’ returns the >> cleaned-up source. [...] > Hi, sorry for the late reply. I need to use unzip on these files to > remove some of their content, and rezip them later. How can I do that > from a snippet? Oh right, the bit that unzips/rezips .whl files cannot easily be done from a snippet, so better keep it as a phase. Deleting *.exe can be done from a snippet though. BTW, the patch reads: (find-files "Lib/distutils/command" ".*.exe$") but the regexp should be: "\\.exe$" (“.*” is unnecessary, but the dot before “exe” must be protected.) Likewise for: (find-files "Lib/ensurepip" ".*.whl$") Thanks, Ludo’.
Le Thu, 03 Jun 2021 11:46:44 +0200,
Ludovic Courtès <ludo@gnu.org> a écrit :
> Hi,
>
> Julien Lepiller <julien@lepiller.eu> skribis:
>
> > Le Fri, 19 Mar 2021 13:27:06 +0100,
> > Ludovic Courtès <ludo@gnu.org> a écrit :
> >
> >> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
> >> > 2001 From: Julien Lepiller <julien@lepiller.eu>
> >> > Date: Wed, 17 Mar 2021 13:33:22 +0100
> >> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
> >> >
> >> > * gnu/packages/python.scm (python2)[arguments]: Remove files
> >> > built during the test phase.
> >> > (python3)[arguments]: Also remove windows binaries shipped with
> >> > the sources.
> >>
> >> Unless it causes bootstrapping issues, I recommend removing those
> >> files from a snippet so that ‘guix build -S python’ returns the
> >> cleaned-up source.
>
> [...]
>
> > Hi, sorry for the late reply. I need to use unzip on these files to
> > remove some of their content, and rezip them later. How can I do
> > that from a snippet?
>
> Oh right, the bit that unzips/rezips .whl files cannot easily be done
> from a snippet, so better keep it as a phase.
>
> Deleting *.exe can be done from a snippet though.
>
> BTW, the patch reads:
>
> (find-files "Lib/distutils/command" ".*.exe$")
>
> but the regexp should be:
>
> "\\.exe$"
>
> (“.*” is unnecessary, but the dot before “exe” must be protected.)
>
> Likewise for:
>
> (find-files "Lib/ensurepip" ".*.whl$")
>
> Thanks,
> Ludo’.
Fixed, and pushed as 20d52562e5c943b30e48e310b58b94054c3a54e5 to
core-updates.