[bug#47251] [core-updates] [patch] gnu: python: Enable optimizations.

[Julien Lepiller <julien@lepiller.eu>]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

Le Thu, 03 Jun 2021 11:48:40 +0200,
Ludovic Courtès <ludo@gnu.org> a écrit :

> Hi,
> 
> Julien Lepiller <julien@lepiller.eu> skribis:
> 
> > But I noticed another file gained a lot of weight:
> > lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs
> > 39MB, whereas on master (for 3.8.2) it weighs 6MB. Here is what
> > size has to say:  
> 
> Yeah, the .o files in there likely contain serialized GIMPLE, as a
> result of compiling with ‘-flto’.
> 
> How about removing *.a altogether in a separate patch?  Or moving it
> to a “static” output?  This would be consistent with what’s done in
> other packages.
> 
> Thanks,
> Ludo’.

Right, this is a good idea. I wrote the attached patches to: enable
optimizations, prevent building and installing the static library and
to install idle to a separate output, as it is not required in general.

With the patch from https://issues.guix.gnu.org/47214, python 3.9 on
coreupdates is ~1.7x faster than python 3.8 on master, and is ~25MB
smaller than 3.8 on master (I haven't measured the speedup for python2,
and it is ~10MB smaller with these changes than on master).

[-- Attachment #2: 0002-gnu-python-Enable-optimizations.patch --]
[-- Type: text/x-patch, Size: 1767 bytes --]

From 0c45d6e82a5f4d140840aef6b0de321122b6c4df Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Thu, 18 Mar 2021 23:41:40 +0100
Subject: [PATCH 2/4] gnu: python: Enable optimizations.

As a result, python execution time of scripts is decreased by 6 to 40%.

gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to
optimize the resulting python.
---
 gnu/packages/python.scm | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 0f20f8e7d3..031904dff6 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -184,7 +184,10 @@
              "--with-system-expat"      ;for XML support
              "--with-system-ffi"        ;build ctypes
              "--with-ensurepip=install" ;install pip and setuptools
+             "--with-computed-gotos"    ;main interpreter loop optimization
+             "--with-lto"               ;increase size by 20MB, but 15% speedup
              "--enable-unicode=ucs4"
+             "--enable-optimizations"
 
              ;; Prevent the installed _sysconfigdata.py from retaining a reference
              ;; to coreutils.
@@ -198,8 +201,10 @@
                      "ac_cv_file__dev_ptmx=no"
                      "ac_cv_file__dev_ptc=no")
                    '())
+             "CFLAGS=-fno-semantic-interposition"
              (string-append "LDFLAGS=-Wl,-rpath="
-                            (assoc-ref %outputs "out") "/lib"))
+                            (assoc-ref %outputs "out") "/lib"
+                            " -fno-semantic-interposition"))
        ;; With no -j argument tests use all available cpus, so provide one.
        #:make-flags
        (list (string-append
-- 
2.31.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0003-gnu-python-Do-not-install-static-libraries.patch --]
[-- Type: text/x-patch, Size: 9683 bytes --]

From 1faeb01becd558f0f4eec16ff32df9dbf1adda7f Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Thu, 3 Jun 2021 22:10:09 +0200
Subject: [PATCH 3/4] gnu: python: Do not install static libraries.

This is something Fedora already does since 2010, so it should be safe,
and saves a lot of space.

* gnu/packages/patches/python-2.7-no-static-lib.patch: New file.
* gnu/packages/patches/python-3-no-static-lib.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add them.
* gnu/packages/python.scm (python-2.7, python-3): Use them.
---
 gnu/local.mk                                  |  2 +
 .../patches/python-2.7-no-static-lib.patch    | 70 +++++++++++++++++
 .../patches/python-3-no-static-lib.patch      | 75 +++++++++++++++++++
 gnu/packages/python.scm                       |  6 +-
 4 files changed, 151 insertions(+), 2 deletions(-)
 create mode 100644 gnu/packages/patches/python-2.7-no-static-lib.patch
 create mode 100644 gnu/packages/patches/python-3-no-static-lib.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index d5a832146e..8370ac12e2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1551,11 +1551,13 @@ dist_patch_DATA =						\
   %D%/packages/patches/python-2.7-site-prefixes.patch		\
   %D%/packages/patches/python-2.7-source-date-epoch.patch	\
   %D%/packages/patches/python-2.7-CVE-2021-3177.patch		\
+  %D%/packages/patches/python-2.7-no-static-lib.patch		\
   %D%/packages/patches/python-3-arm-alignment.patch		\
   %D%/packages/patches/python-3-deterministic-build-info.patch	\
   %D%/packages/patches/python-3-search-paths.patch		\
   %D%/packages/patches/python-3-fix-tests.patch			\
   %D%/packages/patches/python-3-hurd-configure.patch		\
+  %D%/packages/patches/python-3-no-static-lib.patch		\
   %D%/packages/patches/python-CVE-2018-14647.patch		\
   %D%/packages/patches/python-CVE-2020-26116.patch		\
   %D%/packages/patches/python-aionotify-0.2.0-py3.8.patch	\
diff --git a/gnu/packages/patches/python-2.7-no-static-lib.patch b/gnu/packages/patches/python-2.7-no-static-lib.patch
new file mode 100644
index 0000000000..02529989be
--- /dev/null
+++ b/gnu/packages/patches/python-2.7-no-static-lib.patch
@@ -0,0 +1,70 @@
+From 92e2dcace7a3c02289b314238c5f18caec659ff1 Mon Sep 17 00:00:00 2001
+From: Julien Lepiller <julien@lepiller.eu>
+Date: Sat, 5 Jun 2021 12:41:47 +0200
+Subject: [PATCH] No static lib.
+
+This patch is adapted from Fedora. The original patch did not apply
+because of whitespace changes.
+
+00111 #
+Patch the Makefile.pre.in so that the generated Makefile doesn't try to build
+a libpythonMAJOR.MINOR.a (bug 550692):
+Downstream only: not appropriate for upstream
+---
+ Makefile.pre.in | 26 +-------------------------
+ 1 file changed, 1 insertion(+), 25 deletions(-)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 6239fc3..948f196 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -486,7 +486,7 @@ coverage-report: regen-grammar
+ 
+ 
+ # Build the interpreter
+-$(BUILDPYTHON):	Modules/python.o $(LIBRARY) $(LDLIBRARY)
++$(BUILDPYTHON):	Modules/python.o $(LDLIBRARY)
+ 		$(LINKCC) $(LDFLAGS) $(LINKFORSHARED) -o $@ \
+ 			Modules/python.o \
+ 			$(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)
+@@ -527,18 +527,6 @@ sharedmods: $(BUILDPYTHON) pybuilddir.txt Modules/_math.o
+ 		_TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \
+ 		$(PY_BUILD_ENVIRON) $(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build
+ 
+-# Build static library
+-# avoid long command lines, same as LIBRARY_OBJS
+-$(LIBRARY): $(LIBRARY_OBJS)
+-	-rm -f $@
+-	$(AR) $(ARFLAGS) $@ Modules/getbuildinfo.o
+-	$(AR) $(ARFLAGS) $@ $(PARSER_OBJS)
+-	$(AR) $(ARFLAGS) $@ $(OBJECT_OBJS)
+-	$(AR) $(ARFLAGS) $@ $(PYTHON_OBJS)
+-	$(AR) $(ARFLAGS) $@ $(MODULE_OBJS) $(SIGNAL_OBJS)
+-	$(AR) $(ARFLAGS) $@ $(MODOBJS)
+-	$(RANLIB) $@
+-
+ libpython$(VERSION).so: $(LIBRARY_OBJS)
+ 	if test $(INSTSONAME) != $(LDLIBRARY); then \
+ 		$(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+@@ -1216,18 +1204,6 @@ libainstall:	@DEF_MAKE_RULE@ python-config
+ 		else	true; \
+ 		fi; \
+ 	done
+-	@if test -d $(LIBRARY); then :; else \
+-		if test "$(PYTHONFRAMEWORKDIR)" = no-framework; then \
+-			if test "$(SO)" = .dll; then \
+-				$(INSTALL_DATA) $(LDLIBRARY) $(DESTDIR)$(LIBPL) ; \
+-			else \
+-				$(INSTALL_DATA) $(LIBRARY) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \
+-				$(RANLIB) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \
+-			fi; \
+-		else \
+-			echo Skip install of $(LIBRARY) - use make frameworkinstall; \
+-		fi; \
+-	fi
+ 	$(INSTALL_DATA) Modules/config.c $(DESTDIR)$(LIBPL)/config.c
+ 	$(INSTALL_DATA) Modules/python.o $(DESTDIR)$(LIBPL)/python.o
+ 	$(INSTALL_DATA) $(srcdir)/Modules/config.c.in $(DESTDIR)$(LIBPL)/config.c.in
+-- 
+2.31.1
+
diff --git a/gnu/packages/patches/python-3-no-static-lib.patch b/gnu/packages/patches/python-3-no-static-lib.patch
new file mode 100644
index 0000000000..b44dbc976c
--- /dev/null
+++ b/gnu/packages/patches/python-3-no-static-lib.patch
@@ -0,0 +1,75 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: David Malcolm <dmalcolm@redhat.com>
+Date: Mon, 18 Jan 2010 17:59:07 +0000
+Subject: [PATCH] 00111: Don't try to build a libpythonMAJOR.MINOR.a
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Downstream only: not appropriate for upstream.
+
+See https://bugzilla.redhat.com/show_bug.cgi?id=556092
+
+Co-authored-by: David Malcolm <dmalcolm@redhat.com>
+Co-authored-by: Bohuslav Kabrda <bkabrda@redhat.com>
+Co-authored-by: Matej Stuchlik <mstuchli@redhat.com>
+Co-authored-by: Robert Kuska <rkuska@redhat.com>
+Co-authored-by: Charalampos Stratakis <cstratak@redhat.com>
+Co-authored-by: Miro Hrončok <miro@hroncok.cz>
+---
+ Makefile.pre.in | 21 ++-------------------
+ 1 file changed, 2 insertions(+), 19 deletions(-)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 0c3981c132..5587422ec7 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -589,7 +589,7 @@ clinic: check-clean-src $(srcdir)/Modules/_blake2/blake2s_impl.c
+ 	$(PYTHON_FOR_REGEN) $(srcdir)/Tools/clinic/clinic.py --make --srcdir $(srcdir)
+ 
+ # Build the interpreter
+-$(BUILDPYTHON):	Programs/python.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS)
++$(BUILDPYTHON):	Programs/python.o $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS)
+ 	$(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS)
+ 
+ platform: $(BUILDPYTHON) pybuilddir.txt
+@@ -637,12 +637,6 @@ sharedmods: $(BUILDPYTHON) pybuilddir.txt Modules/_math.o
+ 		_TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \
+ 		$(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build
+ 
+-
+-# Build static library
+-$(LIBRARY): $(LIBRARY_OBJS)
+-	-rm -f $@
+-	$(AR) $(ARFLAGS) $@ $(LIBRARY_OBJS)
+-
+ libpython$(LDVERSION).so: $(LIBRARY_OBJS) $(DTRACE_OBJS)
+ 	if test $(INSTSONAME) != $(LDLIBRARY); then \
+ 		$(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM); \
+@@ -724,7 +718,7 @@ Makefile Modules/config.c: Makefile.pre \
+ 	@echo "The Makefile was updated, you may need to re-run make."
+ 
+ 
+-Programs/_testembed: Programs/_testembed.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS)
++Programs/_testembed: Programs/_testembed.o $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS)
+ 	$(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/_testembed.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS)
+ 
+ ############################################################################
+@@ -1652,17 +1646,6 @@ libainstall:	@DEF_MAKE_RULE@ python-config
+ 		else	true; \
+ 		fi; \
+ 	done
+-	@if test -d $(LIBRARY); then :; else \
+-		if test "$(PYTHONFRAMEWORKDIR)" = no-framework; then \
+-			if test "$(SHLIB_SUFFIX)" = .dll; then \
+-				$(INSTALL_DATA) $(LDLIBRARY) $(DESTDIR)$(LIBPL) ; \
+-			else \
+-				$(INSTALL_DATA) $(LIBRARY) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \
+-			fi; \
+-		else \
+-			echo Skip install of $(LIBRARY) - use make frameworkinstall; \
+-		fi; \
+-	fi
+ 	$(INSTALL_DATA) Modules/config.c $(DESTDIR)$(LIBPL)/config.c
+ 	$(INSTALL_DATA) Programs/python.o $(DESTDIR)$(LIBPL)/python.o
+ 	$(INSTALL_DATA) $(srcdir)/Modules/config.c.in $(DESTDIR)$(LIBPL)/config.c.in
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 031904dff6..4ac4b7c23c 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -150,7 +150,8 @@
                                 "python-2.7-source-date-epoch.patch"
                                 "python-2.7-adjust-tests.patch"
                                 "python-cross-compile.patch"
-                                "python-2.7-CVE-2021-3177.patch"))
+                                "python-2.7-CVE-2021-3177.patch"
+                                "python-2.7-no-static-lib.patch"))
        (modules '((guix build utils)))
        (snippet
         '(begin
@@ -418,7 +419,8 @@ data types.")
                         "python-3-deterministic-build-info.patch"
                         "python-3-fix-tests.patch"
                         "python-3-hurd-configure.patch"
-                        "python-3-search-paths.patch"))
+                        "python-3-search-paths.patch"
+                        "python-3-no-static-lib.patch"))
               (sha256
                (base32
                 "10vdf46q5ldnzkprm8pldvr5a9hrdpxjv7mpzgdw6vj3cl318nhc"))
-- 
2.31.1


[-- Attachment #4: 0004-gnu-python-Move-idle-to-a-separate-output.patch --]
[-- Type: text/x-patch, Size: 3301 bytes --]

From c650ac48be47b373c6c8c0b3495f30d9fe21e68b Mon Sep 17 00:00:00 2001
From: Julien Lepiller <julien@lepiller.eu>
Date: Thu, 3 Jun 2021 22:14:14 +0200
Subject: [PATCH 4/4] gnu: python: Move idle to a separate output.

idle is a programming environment that is not useful for building
packages, but it is still useful for end users. This saves 5MB in the
default output.

* gnu/packages/python.scm (python-2.7)[outputs]: Add `idle'.
[arguments]: Move idle files in the new output.
(wrap-python): Do not wrap idle since it is in a different output.
---
 gnu/packages/python.scm | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 4ac4b7c23c..72e05552d8 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -176,7 +176,8 @@
                        "Lib/test/test_subprocess.py"))
            #t))))
     (outputs '("out"
-               "tk"))                   ;tkinter; adds 50 MiB to the closure
+               "tk"                     ;tkinter; adds 50 MiB to the closure
+               "idle"))                 ;programming environment; weighs 5MB
     (build-system gnu-build-system)
     (arguments
      `(#:test-target "test"
@@ -357,6 +358,27 @@
                       (install-file tkinter.so target)
                       (delete-file tkinter.so)))))
                #t)))
+         (add-after 'install 'move-idle
+           (lambda* (#:key outputs #:allow-other-keys)
+             ;; when idle is built, move it to a separate output to save some
+             ;; space (5MB)
+             (let ((out (assoc-ref outputs "out"))
+                   (idle (assoc-ref outputs "idle")))
+               (when idle
+                 (for-each
+                  (lambda (file)
+                    (let ((target (string-append idle "/bin/" (basename file))))
+                      (install-file file (dirname target))
+                      (delete-file file)))
+                  (find-files (string-append out "/bin") "^idle"))
+                 (match (find-files out "^idlelib$" #:directories? #t)
+                   ((idlelib)
+                    (let* ((len (string-length out))
+                           (target (string-append idle "/"
+                                                  (string-drop idlelib len)
+                                                  "/site-packages")))
+                      (mkdir-p (dirname target))
+                      (rename-file idlelib target))))))))
          (add-after 'install 'install-sitecustomize.py
            ,(customize-site version)))))
     (inputs
@@ -620,8 +642,8 @@ for more information.")))
                   (lambda (old new)
                     (symlink (string-append python old)
                              (string-append bin "/" new)))
-                  `("python3" ,"pydoc3" ,"idle3" ,"pip3")
-                  `("python"  ,"pydoc"  ,"idle"  ,"pip"))
+                  `("python3" ,"pydoc3" ,"pip3")
+                  `("python"  ,"pydoc"  ,"pip"))
                 ;; python-config outputs search paths based upon its location,
                 ;; use a bash wrapper to avoid changing its outputs.
                 (let ((bash (string-append (assoc-ref %build-inputs "bash")
-- 
2.31.1