[bug#47214] [core-updates] gnu: python: Remove unnecessary files

[bug#47214] [core-updates] gnu: python: Remove unnecessary files

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

Hi Guix!

this patch (going to core-updates) removes files from the output of
python2 and python3, in an attempt to save some space. First, I noticed
that python3 shipped windows binaries in its source distribution and
installs them to the output, in lib/python3.8/distutils/command/*.exe,
lib/python3.8/site-packages/*.exe and
lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
from the snippet, so they should not be installed anymore.

I also noticed that python installs files that were built during the
test phase, so I remove them, but we need to check they are not
actually needed. Other distros ship them in a separate package, so
maybe we could have a separate test output if this is causing issues?

I haven't yet tested the patch, because it needs to rebuild part of the
bootstrap chain. I'll report progress and update my patch depending
on the results!

[-- Attachment #2: 0001-gnu-python-Remove-unnecessary-files.patch --]
[-- Type: text/x-patch, Size: 2238 bytes --]

From 63ebe5c610ff8587a73641e4139dc14a4d3e379e Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Wed, 17 Mar 2021 13:33:22 +0100
Subject: [PATCH] gnu: python: Remove unnecessary files.

* gnu/packages/python.scm (python2)[arguments]: Remove files built
during the test phase.
(python3)[arguments]: Also remove windows binaries shipped with the
sources.
---
 gnu/packages/python.scm | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 49d43ab008..3bb30ca011 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -294,7 +294,16 @@
                                             (not
                                              (string-prefix? "test_support."
                                                              file))))))
-                      (call-with-output-file "__init__.py" (const #t)))))))))
+                      (call-with-output-file "__init__.py" (const #t))))
+                  (let ((libdir (string-append out "/lib/" pythonX.Y)))
+                    (for-each
+                     (lambda (directory)
+                       (let ((dir (string-append libdir "/" directory)))
+                         (when (file-exists? dir)
+                           (delete-file-recursively dir))))
+                     '("email/test" "ctypes/test" "unittest/test" "tkinter/test"
+                       "sqlite3/test" "bsddb/test" "lib-tk/test" "lib2to3/tests"
+                       "json/tests" "distutils/tests"))))))))
          (add-after 'remove-tests 'rebuild-bytecode
            (lambda* (#:key outputs #:allow-other-keys)
              (let ((out (assoc-ref outputs "out")))
@@ -424,6 +433,9 @@ data types.")
                   (substitute* "Modules/Setup"
                     ;; Link Expat instead of embedding the bundled one.
                     (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
+                  ;; Delete windows binaries
+                  (for-each delete-file
+                            (find-files "Lib/distutils/command" ".*.exe$"))
                   #t))))
     (arguments
      (substitute-keyword-arguments (package-arguments python-2)
-- 
2.30.0

[bug#47214] [core-updates] gnu: python: Remove unnecessary files

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 1402 bytes --]

On Wed, Mar 17, 2021 at 01:41:02PM +0100, Julien Lepiller wrote:
> Hi Guix!
> 
> this patch (going to core-updates) removes files from the output of
> python2 and python3, in an attempt to save some space. First, I noticed
> that python3 shipped windows binaries in its source distribution and
> installs them to the output, in lib/python3.8/distutils/command/*.exe,
> lib/python3.8/site-packages/*.exe and
> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
> from the snippet, so they should not be installed anymore.
> 
> I also noticed that python installs files that were built during the
> test phase, so I remove them, but we need to check they are not
> actually needed. Other distros ship them in a separate package, so
> maybe we could have a separate test output if this is causing issues?
> 
> I haven't yet tested the patch, because it needs to rebuild part of the
> bootstrap chain. I'll report progress and update my patch depending
> on the results!

I got stuck on python-minimal with test-asyncio but then it didn't hang
the second time I ran the build. I don't have a good pure python package
I can think of to test the change.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[bug#47214] [core-updates] gnu: python: Remove unnecessary files

[Julien Lepiller]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

Le Wed, 17 Mar 2021 13:41:02 +0100,
Julien Lepiller <julien@lepiller.eu> a écrit :

> Hi Guix!
> 
> this patch (going to core-updates) removes files from the output of
> python2 and python3, in an attempt to save some space. First, I
> noticed that python3 shipped windows binaries in its source
> distribution and installs them to the output, in
> lib/python3.8/distutils/command/*.exe,
> lib/python3.8/site-packages/*.exe and
> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
> from the snippet, so they should not be installed anymore.
> 
> I also noticed that python installs files that were built during the
> test phase, so I remove them, but we need to check they are not
> actually needed. Other distros ship them in a separate package, so
> maybe we could have a separate test output if this is causing issues?
> 
> I haven't yet tested the patch, because it needs to rebuild part of
> the bootstrap chain. I'll report progress and update my patch
> depending on the results!

I discovered other .exe in python3, so this updated patch deletes them.
Since I modify the python from commencement.scm by changing a phase in
python-2.7, I decided to also change the definition there to remove the
offending files.

I managed to build up to python-3.9 without any issue, but haven't
tested building more than a handful of python packages.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-gnu-python-Remove-unnecessary-files.patch --]
[-- Type: text/x-patch, Size: 6312 bytes --]

From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Wed, 17 Mar 2021 13:33:22 +0100
Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.

* gnu/packages/python.scm (python2)[arguments]: Remove files built
during the test phase.
(python3)[arguments]: Also remove windows binaries shipped with the
sources.
---
 gnu/packages/commencement.scm |  9 ++++++++-
 gnu/packages/python.scm       | 35 +++++++++++++++++++++++++++++++++--
 2 files changed, 41 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index b2921a4a48..b2b9651444 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -10,6 +10,7 @@
 ;;; Copyright © 2020 Timothy Sample <samplet@ngyro.com>
 ;;; Copyright © 2020 Guy Fleury Iteriteka <gfleury@disroot.org>
 ;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer@gmail.com>
+;;; Copyright © 2021 Julien Lepiller <julien@lepiller.eu>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -3098,7 +3099,12 @@ memoized as a function of '%current-system'."
                    (delete-file-recursively "Modules/expat")
                    (substitute* "Modules/Setup.dist"
                      ;; Link Expat instead of embedding the bundled one.
-                     (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n")))))))
+                     (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
+                   ;; Delete windows binaries
+                   (for-each delete-file
+                             (find-files "Lib/distutils/command" ".*.exe$"))
+                   (for-each delete-file
+                             (find-files "Lib/ensurepip" ".*.whl$")))))))
     (inputs
      `(,@(%boot0-inputs)
        ("expat" ,expat-sans-tests)))              ;remove OpenSSL, zlib, etc.
@@ -3129,6 +3135,7 @@ memoized as a function of '%current-system'."
                                          ('add-after unpack apply-alignment-patch _))
                                        `(modify-phases ,original-phases ,@changes))
                                       (_ phases)))
+               (delete 'remove-windows-binaries)
                (add-before 'configure 'disable-modules
                  (lambda _
                    (substitute* "setup.py"
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 49d43ab008..da9859b11a 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -31,7 +31,7 @@
 ;;; Copyright © 2016 Dylan Jeffers <sapientech@sapientech@openmailbox.org>
 ;;; Copyright © 2016, 2017 Alex Vong <alexvong1995@gmail.com>
 ;;; Copyright © 2016, 2017, 2018 Arun Isaac <arunisaac@systemreboot.net>
-;;; Copyright © 2016, 2017, 2018 Julien Lepiller <julien@lepiller.eu>
+;;; Copyright © 2016, 2017, 2018, 2021 Julien Lepiller <julien@lepiller.eu>
 ;;; Copyright © 2016, 2017, 2018 Tobias Geerinckx-Rice <me@tobias.gr>
 ;;; Copyright © 2016, 2017 Thomas Danckaert <post@thomasdanckaert.be>
 ;;; Copyright © 2017 Carlo Zancanaro <carlo@zancanaro.id.au>
@@ -294,7 +294,16 @@
                                             (not
                                              (string-prefix? "test_support."
                                                              file))))))
-                      (call-with-output-file "__init__.py" (const #t)))))))))
+                      (call-with-output-file "__init__.py" (const #t))))
+                  (let ((libdir (string-append out "/lib/" pythonX.Y)))
+                    (for-each
+                     (lambda (directory)
+                       (let ((dir (string-append libdir "/" directory)))
+                         (when (file-exists? dir)
+                           (delete-file-recursively dir))))
+                     '("email/test" "ctypes/test" "unittest/test" "tkinter/test"
+                       "sqlite3/test" "bsddb/test" "lib-tk/test" "lib2to3/tests"
+                       "json/tests" "distutils/tests"))))))))
          (add-after 'remove-tests 'rebuild-bytecode
            (lambda* (#:key outputs #:allow-other-keys)
              (let ((out (assoc-ref outputs "out")))
@@ -424,6 +433,9 @@ data types.")
                   (substitute* "Modules/Setup"
                     ;; Link Expat instead of embedding the bundled one.
                     (("^#pyexpat.*") "pyexpat pyexpat.c -lexpat\n"))
+                  ;; Delete windows binaries
+                  (for-each delete-file
+                            (find-files "Lib/distutils/command" ".*.exe$"))
                   #t))))
     (arguments
      (substitute-keyword-arguments (package-arguments python-2)
@@ -469,6 +481,23 @@ data types.")
            ,@(if (hurd-system?)
                  `((delete 'patch-regen-for-hurd)) ;regen was removed after 3.5.9
                  '())
+           (add-after 'unpack 'remove-windows-binaries
+             (lambda _
+               ;; Delete .exe from embedded .whl (zip) files
+               (for-each
+                (lambda (whl)
+                  (let ((dir "whl-content"))
+                    (mkdir-p dir)
+                    (with-directory-excursion dir
+                      (let ((whl (string-append "../" whl)))
+                        (invoke "unzip" whl)
+                        (for-each delete-file
+                                  (find-files "." ".*.exe$"))
+                        (delete-file whl)
+                        (apply invoke "zip" "-X" whl
+                               (find-files "." ".*" #:directories? #t))))
+                    (delete-file-recursively dir)))
+                (find-files "Lib/ensurepip" ".*.whl$"))))
            (add-before 'check 'set-TZDIR
              (lambda* (#:key inputs native-inputs #:allow-other-keys)
                ;; test_email requires the Olson time zone database.
@@ -503,6 +532,8 @@ data types.")
              ,(customize-site version))))))
     (native-inputs
      `(("tzdata" ,tzdata-for-tests)
+       ("unzip" ,unzip)
+       ("zip" ,(@ (gnu packages compression) zip))
        ,@(if (%current-target-system)
              `(("python3" ,this-package))
              '())
-- 
2.30.0

[bug#47214] [core-updates] gnu: python: Remove unnecessary files

[Ludovic Courtès]

Hi Julien,

Julien Lepiller <julien@lepiller.eu> skribis:

> Le Wed, 17 Mar 2021 13:41:02 +0100,
> Julien Lepiller <julien@lepiller.eu> a écrit :
>
>> Hi Guix!
>> 
>> this patch (going to core-updates) removes files from the output of
>> python2 and python3, in an attempt to save some space. First, I
>> noticed that python3 shipped windows binaries in its source
>> distribution and installs them to the output, in
>> lib/python3.8/distutils/command/*.exe,
>> lib/python3.8/site-packages/*.exe and
>> lib/python3.8/site-packages/pip/_vendor/distlib/*.exe. I remove them
>> from the snippet, so they should not be installed anymore.
>> 
>> I also noticed that python installs files that were built during the
>> test phase, so I remove them, but we need to check they are not
>> actually needed. Other distros ship them in a separate package, so
>> maybe we could have a separate test output if this is causing issues?
>> 
>> I haven't yet tested the patch, because it needs to rebuild part of
>> the bootstrap chain. I'll report progress and update my patch
>> depending on the results!
>
> I discovered other .exe in python3, so this updated patch deletes them.
> Since I modify the python from commencement.scm by changing a phase in
> python-2.7, I decided to also change the definition there to remove the
> offending files.

Woow, good catch.

> From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien@lepiller.eu>
> Date: Wed, 17 Mar 2021 13:33:22 +0100
> Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
>
> * gnu/packages/python.scm (python2)[arguments]: Remove files built
> during the test phase.
> (python3)[arguments]: Also remove windows binaries shipped with the
> sources.

Unless it causes bootstrapping issues, I recommend removing those files
from a snippet so that ‘guix build -S python’ returns the cleaned-up
source.

WDYT?

Thanks,
Ludo’.

[bug#47214] [core-updates] gnu: python: Remove unnecessary files

[Julien Lepiller]

Le Fri, 19 Mar 2021 13:27:06 +0100,
Ludovic Courtès <ludo@gnu.org> a écrit :

> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
> > 2001 From: Julien Lepiller <julien@lepiller.eu>
> > Date: Wed, 17 Mar 2021 13:33:22 +0100
> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
> >
> > * gnu/packages/python.scm (python2)[arguments]: Remove files built
> > during the test phase.
> > (python3)[arguments]: Also remove windows binaries shipped with the
> > sources.  
> 
> Unless it causes bootstrapping issues, I recommend removing those
> files from a snippet so that ‘guix build -S python’ returns the
> cleaned-up source.
> 
> WDYT?
> 
> Thanks,
> Ludo’.

Hi, sorry for the late reply. I need to use unzip on these files to
remove some of their content, and rezip them later. How can I do that
from a snippet?

[bug#47214] [core-updates] gnu: python: Remove unnecessary files

[Ludovic Courtès]

Hi,

Julien Lepiller <julien@lepiller.eu> skribis:

> Le Fri, 19 Mar 2021 13:27:06 +0100,
> Ludovic Courtès <ludo@gnu.org> a écrit :
>
>> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
>> > 2001 From: Julien Lepiller <julien@lepiller.eu>
>> > Date: Wed, 17 Mar 2021 13:33:22 +0100
>> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
>> >
>> > * gnu/packages/python.scm (python2)[arguments]: Remove files built
>> > during the test phase.
>> > (python3)[arguments]: Also remove windows binaries shipped with the
>> > sources.  
>> 
>> Unless it causes bootstrapping issues, I recommend removing those
>> files from a snippet so that ‘guix build -S python’ returns the
>> cleaned-up source.

[...]

> Hi, sorry for the late reply. I need to use unzip on these files to
> remove some of their content, and rezip them later. How can I do that
> from a snippet?

Oh right, the bit that unzips/rezips .whl files cannot easily be done
from a snippet, so better keep it as a phase.

Deleting *.exe can be done from a snippet though.

BTW, the patch reads:

   (find-files "Lib/distutils/command" ".*.exe$")

but the regexp should be:

  "\\.exe$"

(“.*” is unnecessary, but the dot before “exe” must be protected.)

Likewise for:

  (find-files "Lib/ensurepip" ".*.whl$")

Thanks,
Ludo’.

bug#47214: [core-updates] gnu: python: Remove unnecessary files

[Julien Lepiller]

Le Thu, 03 Jun 2021 11:46:44 +0200,
Ludovic Courtès <ludo@gnu.org> a écrit :

> Hi,
> 
> Julien Lepiller <julien@lepiller.eu> skribis:
> 
> > Le Fri, 19 Mar 2021 13:27:06 +0100,
> > Ludovic Courtès <ludo@gnu.org> a écrit :
> >  
> >> > From 0d2d9a32cbbf24eadf74271b2fb1304b7a72d281 Mon Sep 17 00:00:00
> >> > 2001 From: Julien Lepiller <julien@lepiller.eu>
> >> > Date: Wed, 17 Mar 2021 13:33:22 +0100
> >> > Subject: [PATCH 1/2] gnu: python: Remove unnecessary files.
> >> >
> >> > * gnu/packages/python.scm (python2)[arguments]: Remove files
> >> > built during the test phase.
> >> > (python3)[arguments]: Also remove windows binaries shipped with
> >> > the sources.    
> >> 
> >> Unless it causes bootstrapping issues, I recommend removing those
> >> files from a snippet so that ‘guix build -S python’ returns the
> >> cleaned-up source.  
> 
> [...]
> 
> > Hi, sorry for the late reply. I need to use unzip on these files to
> > remove some of their content, and rezip them later. How can I do
> > that from a snippet?  
> 
> Oh right, the bit that unzips/rezips .whl files cannot easily be done
> from a snippet, so better keep it as a phase.
> 
> Deleting *.exe can be done from a snippet though.
> 
> BTW, the patch reads:
> 
>    (find-files "Lib/distutils/command" ".*.exe$")
> 
> but the regexp should be:
> 
>   "\\.exe$"
> 
> (“.*” is unnecessary, but the dot before “exe” must be protected.)
> 
> Likewise for:
> 
>   (find-files "Lib/ensurepip" ".*.whl$")
> 
> Thanks,
> Ludo’.

Fixed, and pushed as 20d52562e5c943b30e48e310b58b94054c3a54e5 to
core-updates.