* [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations. @ 2021-03-18 23:16 Julien Lepiller 2021-03-31 12:47 ` Ludovic Courtès 0 siblings, 1 reply; 9+ messages in thread From: Julien Lepiller @ 2021-03-18 23:16 UTC (permalink / raw) To: 47251 [-- Attachment #1: Type: text/plain, Size: 1089 bytes --] Hi Guix! This is the second patch I created in my exploration of our Python package. Here, I noticed that our Python was very slow compared to other distros and I tried to find which options they used. I found some hints from Fedora. In this patch, I enable a few options and add a C/LD flag to make python run faster. I also attached a report of timing running the benchmark from pyperformance on my Fedora machine. For comparison, I have also added the results for Fedora's python3, which is somewhere in between. I noticed that this increases the size of the python package by 20MB, which is due entirely to --with-lto option, which also accounts for half of the speedup. Overall, this python is almost twice as fast on some benchmarks, and always faster on all of them. It's about 1.5 times faster on average. Strangely, the python package from Alpine is twice as small as this package, and they use the --with-lto option. I wonder how they achieve that. Note that I modified python-2, as its flags are inherited by python-3 (and also the bootstrap python from commencement.scm). [-- Attachment #2: 0002-gnu-python-Enable-optimizations.patch --] [-- Type: text/x-patch, Size: 1767 bytes --] From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001 From: Julien Lepiller <julien@lepiller.eu> Date: Thu, 18 Mar 2021 23:41:40 +0100 Subject: [PATCH 2/2] gnu: python: Enable optimizations. As a result, python execution time of scripts is decreased by 6 to 40%. gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to optimize the resulting python. --- gnu/packages/python.scm | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index da9859b11a..8de0ca343c 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -183,7 +183,10 @@ "--with-system-expat" ;for XML support "--with-system-ffi" ;build ctypes "--with-ensurepip=install" ;install pip and setuptools + "--with-computed-gotos" ;main interpreter loop optimization + "--with-lto" ;increase size by 20MB, but 15% speedup "--enable-unicode=ucs4" + "--enable-optimizations" ;; Prevent the installed _sysconfigdata.py from retaining a reference ;; to coreutils. @@ -197,8 +200,10 @@ "ac_cv_file__dev_ptmx=no" "ac_cv_file__dev_ptc=no") '()) + "CFLAGS=-fno-semantic-interposition" (string-append "LDFLAGS=-Wl,-rpath=" - (assoc-ref %outputs "out") "/lib")) + (assoc-ref %outputs "out") "/lib" + " -fno-semantic-interposition")) ;; With no -j argument tests use all available cpus, so provide one. #:make-flags (list (string-append -- 2.30.0 [-- Attachment #3: python-report --] [-- Type: application/octet-stream, Size: 5498 bytes --] | optimized c-u | current c-u | fedora (3.8.2) ------------------------+----------------------+----------------------+------------------- 2to3 | 511 ms +- 25 ms | 617 ms +- 21 ms | 578 ms +- 23 ms chameleon | 13.8 ms +- 0.3 ms | 18.0 ms +- 0.5 ms | 16.1 ms +- 0.6 ms chaos | 162 ms +- 4 ms | 232 ms +- 3 ms | 204 ms +- 4 ms crypto_pyaes | 162 ms +- 3 ms | 205 ms +- 5 ms | 218 ms +- 8 ms deltablue | 10.8 ms +- 0.3 ms | 15.3 ms +- 0.5 ms | 12.8 ms +- 0.3 ms django_template | 75.8 ms +- 1.0 ms | 104 ms +- 3 ms | 88.2 ms +- 1.8 ms dulwich_log | 111 ms +- 4 ms | 133 ms +- 2 ms | 121 ms +- 7 ms fannkuch | 695 ms +- 9 ms | 822 ms +- 8 ms | 874 ms +- 16 ms float | 171 ms +- 8 ms | 209 ms +- 4 ms | 217 ms +- 8 ms genshi_text | 43.8 ms +- 0.8 ms | 55.4 ms +- 0.8 ms | 53.5 ms +- 2.6 ms genshi_xml | 93.8 ms +- 1.5 ms | 113 ms +- 1 ms | 115 ms +- 6 ms go | 372 ms +- 7 ms | 497 ms +- 4 ms | 453 ms +- 8 ms hexiom | 14.4 ms +- 0.5 ms | 18.9 ms +- 0.2 ms | 18.2 ms +- 1.0 ms json_dumps | 19.2 ms +- 0.5 ms | 23.0 ms +- 0.3 ms | 22.7 ms +- 1.0 ms json_loads | 34.9 us +- 0.7 us | 43.5 us +- 0.7 us | 48.1 us +- 2.2 us logging_format | 14.6 us +- 0.8 us | 20.0 us +- 0.5 us | 16.9 us +- 1.0 us logging_silent | 290 ns +- 13 ns | 399 ns +- 10 ns | 343 ns +- 10 ns logging_simple | 13.1 us +- 0.3 us | 18.4 us +- 0.7 us | 15.5 us +- 0.6 us mako | 24.0 ms +- 0.4 ms | 31.2 ms +- 0.7 ms | 29.6 ms +- 1.2 ms meteor_contest | 157 ms +- 4 ms | 179 ms +- 2 ms | 189 ms +- 9 ms nbody | 184 ms +- 4 ms | 251 ms +- 4 ms | 229 ms +- 4 ms nqueens | 145 ms +- 3 ms | 180 ms +- 5 ms | 184 ms +- 5 ms pathlib | 28.4 ms +- 0.7 ms | 36.0 ms +- 1.3 ms | 35.2 ms +- 2.4 ms pickle | 14.6 us +- 0.7 us | 17.3 us +- 0.5 us | 16.4 us +- 0.6 us pickle_dict | 35.1 us +- 0.4 us | 42.3 us +- 0.6 us | 42.8 us +- 2.0 us pickle_list | 5.42 us +- 0.19 us | 6.24 us +- 0.10 us | 6.34 us +- 0.35 us pickle_pure_python | 699 us +- 15 us | 941 us +- 19 us | 800 us +- 16 us pidigits | 252 ms +- 3 ms | 267 ms +- 3 ms | 282 ms +- 4 ms pyflate | 1.01 sec +- 0.02 sec | 1.37 sec +- 0.07 sec | 1.23 sec +- 0.02 sec python_startup | 16.9 ms +- 0.6 ms | 19.6 ms +- 0.4 ms | 17.6 ms +- 0.5 ms python_startup_no_site | 11.2 ms +- 0.6 ms | 12.7 ms +- 0.2 ms | 11.5 ms +- 0.3 ms raytrace | 733 ms +- 11 ms | 1.22 sec +- 0.01 sec | 886 ms +- 10 ms regex_compile | 257 ms +- 2 ms | 353 ms +- 3 ms | 300 ms +- 4 ms regex_dna | 268 ms +- 4 ms | 296 ms +- 7 ms | 293 ms +- 15 ms regex_effbot | 4.35 ms +- 0.07 ms | 5.13 ms +- 0.39 ms | 4.93 ms +- 0.23 ms regex_v8 | 35.9 ms +- 0.7 ms | 43.3 ms +- 2.0 ms | 37.5 ms +- 0.4 ms richards | 100 ms +- 1 ms | 168 ms +- 9 ms | 124 ms +- 4 ms scimark_fft | 538 ms +- 6 ms | 773 ms +- 8 ms | 710 ms +- 33 ms scimark_lu | 235 ms +- 5 ms | 340 ms +- 3 ms | 288 ms +- 5 ms scimark_monte_carlo | 154 ms +- 2 ms | 231 ms +- 4 ms | 191 ms +- 4 ms scimark_sor | 301 ms +- 4 ms | 446 ms +- 8 ms | 363 ms +- 6 ms scimark_sparse_mat_mult | 6.65 ms +- 0.26 ms | 10.3 ms +- 0.2 ms | 8.14 ms +- 0.09 ms spectral_norm | 205 ms +- 5 ms | 296 ms +- 5 ms | 251 ms +- 2 ms sqlalchemy_declarative | 257 ms +- 3 ms | 313 ms +- 4 ms | 289 ms +- 3 ms sqlalchemy_imperative | 51.4 ms +- 1.3 ms | 63.7 ms +- 4.0 ms | 55.5 ms +- 3.3 ms sqlite_synth | 4.23 us +- 0.12 us | 5.77 us +- 0.17 us | 5.24 us +- 0.51 us sympy_expand | 766 ms +- 13 ms | 1.03 sec +- 0.04 sec | 806 ms +- 13 ms sympy_integrate | 38.4 ms +- 0.7 ms | 48.3 ms +- 0.4 ms | 42.0 ms +- 0.6 ms sympy_sum | 348 ms +- 8 ms | 433 ms +- 4 ms | 369 ms +- 6 ms sympy_str | 526 ms +- 8 ms | 676 ms +- 6 ms | 562 ms +- 6 ms telco | 8.56 ms +- 0.13 ms | 13.0 ms +- 0.7 ms | 12.0 ms +- 0.4 ms tornado_http | 293 ms +- 11 ms | 376 ms +- 23 ms | 330 ms +- 13 ms unpack_sequence | 83.8 ns +- 4.0 ns | 92.8 ns +- 5.3 ns | 89.8 ns +- 3.2 ns unpickle | 19.5 us +- 0.3 us | 27.0 us +- 0.5 us | 25.0 us +- 1.1 us unpickle_list | 5.94 us +- 0.12 us | 7.99 us +- 0.38 us | 6.82 us +- 0.53 us unpickle_pure_python | 461 us +- 10 us | 749 us +- 24 us | 566 us +- 26 us xml_etree_parse | 219 ms +- 3 ms | 259 ms +- 2 ms | 261 ms +- 13 ms xml_etree_iterparse | 155 ms +- 4 ms | 192 ms +- 2 ms | 180 ms +- 3 ms xml_etree_generate | 138 ms +- 3 ms | 185 ms +- 3 ms | 175 ms +- 9 ms xml_etree_process | 109 ms +- 4 ms | 148 ms +- 2 ms | 136 ms +- 2 ms ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations. 2021-03-18 23:16 [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations Julien Lepiller @ 2021-03-31 12:47 ` Ludovic Courtès 2021-04-10 20:55 ` Ludovic Courtès ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Ludovic Courtès @ 2021-03-31 12:47 UTC (permalink / raw) To: Julien Lepiller; +Cc: 47251 Hi Julien, Julien Lepiller <julien@lepiller.eu> skribis: >>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001 > From: Julien Lepiller <julien@lepiller.eu> > Date: Thu, 18 Mar 2021 23:41:40 +0100 > Subject: [PATCH 2/2] gnu: python: Enable optimizations. > > As a result, python execution time of scripts is decreased by 6 to 40%. > > gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to > optimize the resulting python. LGTM! Great that you found about these options. I wonder what can be done to reduce the package size. On ‘core-updates’, we’re already stripping ELF binaries a bit more than on ‘master’. What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in the executable or in libpython.so as a result of using -flto? Thanks, Ludo’. ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations. 2021-03-31 12:47 ` Ludovic Courtès @ 2021-04-10 20:55 ` Ludovic Courtès 2021-04-10 20:55 ` Ludovic Courtès 2021-06-03 2:06 ` Julien Lepiller 2 siblings, 0 replies; 9+ messages in thread From: Ludovic Courtès @ 2021-04-10 20:55 UTC (permalink / raw) To: Julien Lepiller; +Cc: 47251 Ping! :-) Ludovic Courtès <ludo@gnu.org> skribis: > Hi Julien, > > Julien Lepiller <julien@lepiller.eu> skribis: > >>>>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001 >> From: Julien Lepiller <julien@lepiller.eu> >> Date: Thu, 18 Mar 2021 23:41:40 +0100 >> Subject: [PATCH 2/2] gnu: python: Enable optimizations. >> >> As a result, python execution time of scripts is decreased by 6 to 40%. >> >> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to >> optimize the resulting python. > > LGTM! Great that you found about these options. > > I wonder what can be done to reduce the package size. On > ‘core-updates’, we’re already stripping ELF binaries a bit more than on > ‘master’. > > What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in > the executable or in libpython.so as a result of using -flto? > > Thanks, > Ludo’. ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations. 2021-03-31 12:47 ` Ludovic Courtès 2021-04-10 20:55 ` Ludovic Courtès @ 2021-04-10 20:55 ` Ludovic Courtès 2021-06-03 2:06 ` Julien Lepiller 2 siblings, 0 replies; 9+ messages in thread From: Ludovic Courtès @ 2021-04-10 20:55 UTC (permalink / raw) To: Julien Lepiller; +Cc: 47251 Ping! :-) Ludovic Courtès <ludo@gnu.org> skribis: > Hi Julien, > > Julien Lepiller <julien@lepiller.eu> skribis: > >>>>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 2001 >> From: Julien Lepiller <julien@lepiller.eu> >> Date: Thu, 18 Mar 2021 23:41:40 +0100 >> Subject: [PATCH 2/2] gnu: python: Enable optimizations. >> >> As a result, python execution time of scripts is decreased by 6 to 40%. >> >> gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to >> optimize the resulting python. > > LGTM! Great that you found about these options. > > I wonder what can be done to reduce the package size. On > ‘core-updates’, we’re already stripping ELF binaries a bit more than on > ‘master’. > > What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in > the executable or in libpython.so as a result of using -flto? > > Thanks, > Ludo’. ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations. 2021-03-31 12:47 ` Ludovic Courtès 2021-04-10 20:55 ` Ludovic Courtès 2021-04-10 20:55 ` Ludovic Courtès @ 2021-06-03 2:06 ` Julien Lepiller 2021-06-03 9:48 ` Ludovic Courtès 2 siblings, 1 reply; 9+ messages in thread From: Julien Lepiller @ 2021-06-03 2:06 UTC (permalink / raw) To: Ludovic Courtès; +Cc: 47251 Le Wed, 31 Mar 2021 14:47:44 +0200, Ludovic Courtès <ludo@gnu.org> a écrit : > Hi Julien, > > Julien Lepiller <julien@lepiller.eu> skribis: > > >>From c72e34aa0eb0db589557e047c5db0128273d5d12 Mon Sep 17 00:00:00 > >>2001 > > From: Julien Lepiller <julien@lepiller.eu> > > Date: Thu, 18 Mar 2021 23:41:40 +0100 > > Subject: [PATCH 2/2] gnu: python: Enable optimizations. > > > > As a result, python execution time of scripts is decreased by 6 to > > 40%. > > > > gnu/packages/python (python-2.7)[arguments]: Add configure and make > > flags to optimize the resulting python. > > LGTM! Great that you found about these options. > > I wonder what can be done to reduce the package size. On > ‘core-updates’, we’re already stripping ELF binaries a bit more than > on ‘master’. > > What does ‘size …/bin/python’ say? Could it be there’s GIMPLE left in > the executable or in libpython.so as a result of using -flto? > > Thanks, > Ludo’. Sorry for the late reply, I didn't see your message, and never received the pings I see on issues.guix.gnu.org. Here is the output of size on bin/python: text data bss dec hex filename 2072 704 8 2784 ae0 /gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/bin/python3 Same, for lib/libpython3.so: text data bss dec hex filename 1154 576 8 1738 6ca /gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/lib/libpython3.so Same for lib/libpython3.9.so.1.0: text data bss dec hex filename 3614304 254928 139896 4009128 3d2ca8 /gnu/store/bcggflp7zszpazgk8qa4sr0kmp5a67pg-python-minimal-3.9.5/lib/libpython3.9.so.1.0 But I noticed another file gained a lot of weight: lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs 39MB, whereas on master (for 3.8.2) it weighs 6MB. Here is what size has to say: text data bss dec hex 298 0 52 350 15e 2642 0 0 2642 a52 370 0 100 470 1d6 481 0 0 481 1e1 1828 0 0 1828 724 1909 0 0 1909 775 1817 512 0 2329 919 24497 0 16032 40529 9e51 110286 976 0 111262 1b29e 12936 0 0 12936 3288 895 0 0 895 37f 1159 0 32 1191 4a7 4311 0 0 4311 10d7 29747 56 0 29803 746b 39941 344 0 40285 9d5d 1252 0 0 1252 4e4 1209 760 16 1985 7c1 16582 0 0 16582 40c6 56468 3856 1 60325 eba5 62176 3520 2080 67776 108c0 20674 0 0 20674 50c2 3160 408 0 3568 df0 1594 488 0 2082 822 4895 1392 16 6303 189f 17197 1384 16 18597 48a5 12586 1072 0 13658 355a 20052 6096 0 26148 6624 4232 1176 0 5408 1520 33447 28560 56 62063 f26f 4950 1096 0 6046 179e 18895 4928 1344 25167 624f 5088 880 0 5968 1750 30728 1496 452 32676 7fa4 9294 1016 16 10326 2856 11512 2328 8 13848 3618 1661 728 0 2389 955 2620 1040 0 3660 e4c 48674 2280 672 51626 c9aa 102026 2024 1240 105290 19b4a 69538 6296 1352 77186 12d82 14228 3728 0 17956 4624 30454 1968 0 32422 7ea6 6484 1304 0 7788 1e6c 9917 1368 8 11293 2c1d 2185 552 0 2737 ab1 31339 1816 0 33155 8183 19061 900 584 20545 5041 1886 528 0 2414 96e 12991 2240 0 15231 3b7f 46206 3624 8 49838 c2ae 5898 1176 8 7082 1baa 5952 248 0 6200 1838 13769 1264 256 15289 3bb9 111003 11760 98432 221195 3600b 280942 5056 2168 288166 465a6 141203 0 0 141203 22793 16406 2000 8 18414 47ee 13022 944 16 13982 369e 129842 2352 1712 133906 20b12 810 0 0 810 32a 71059 72 8 71139 115e3 24372 40 0 24412 5f5c 8726 0 48 8774 2246 41512 3992 0 45504 b1c0 61876 2460 4 64340 fb54 20340 360 8 20708 50e4 74371 88 80 74539 1232b 10273 2488 24 12785 31f1 48 0 0 48 30 26200 472 408 27080 69c8 1257 0 0 1257 4e9 1404 0 0 1404 57c 55841 0 8 55849 da29 117 0 0 117 75 412 0 0 412 19c 110 0 0 110 6e 192 0 250 442 1ba 6170 25384 0 31554 7b42 10561 3280 16 13857 3621 3590 0 0 3590 e06 30434 1392 72 31898 7c9a 2079 24 0 2103 837 36616 0 96 36712 8f68 27149 384 0 27533 6b8d 22163 0 8 22171 569b 494 0 0 494 1ee 4461 0 0 4461 116d 5822 0 48 5870 16ee 7570 0 0 7570 1d92 7739 0 32 7771 1e5b 1144 0 0 1144 478 1584 0 0 1584 630 113 0 0 113 71 4422 24 24 4470 1176 20465 416 672 21553 5431 402 0 0 402 192 14654 0 0 14654 393e 16034 696 0 16730 415a 7026 0 0 7026 1b72 1754 40 4 1798 706 2920 0 0 2920 b68 28827 768 56 29651 73d3 39065 2568 1792 43425 a9a1 4590 96 452 5138 1412 7585 952 0 8537 2159 1562 52 8 1622 656 316 0 0 316 13c 4609 56 0 4665 1239 1979 0 0 1979 7bb 36752 8 2400 39160 98f8 24453 0 0 24453 5f85 11769 12 0 11781 2e05 973 32 3104 4109 100d 299 528 0 827 33b 11964 0 0 11964 2ebc 6063 24 0 6087 17c7 18488 896 0 19384 4bb8 141834 17384 8 159226 26dfa 10041 160 32 10233 27f9 3542 416 0 3958 f76 81990 4624 0 86614 15256 27982 1600 16 29598 739e 1808 320 0 2128 850 12262 2776 8 15046 3ac6 18682 3320 0 22002 55f2 19569 4368 136 24073 5e09 11529 824 0 12353 3041 44792 11312 0 56104 db28 3189 328 0 3517 dbd 16392 968 1520 18880 49c0 6380 608 0 6988 1b4c 17490 1288 444 19222 4b16 17253 3088 424 20765 511d 7382 1456 0 8838 2286 19480 576 216 20272 4f30 11320 1936 0 13256 33c8 14121 1336 0 15457 3c61 12682 1720 0 14402 3842 29184 6088 8 35280 89d0 39623 4080 0 43703 aab7 10987 1112 0 12099 2f43 10034 1864 176 12074 2f2a 17528 488 248 18264 4758 1197 352 0 1549 60d 1614 192 0 1806 70e 31873 5840 440 38153 9509 1644 1504 0 3148 c4c 91445 317 0 91762 16672 I had to remove the file name it comes from, because of width issue in email, but I don't really understand what size does, so I don't know if that's relevant or not. Do we need to keep that static library around? ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations. 2021-06-03 2:06 ` Julien Lepiller @ 2021-06-03 9:48 ` Ludovic Courtès 2021-06-05 15:43 ` Julien Lepiller 0 siblings, 1 reply; 9+ messages in thread From: Ludovic Courtès @ 2021-06-03 9:48 UTC (permalink / raw) To: Julien Lepiller; +Cc: 47251 Hi, Julien Lepiller <julien@lepiller.eu> skribis: > But I noticed another file gained a lot of weight: > lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs 39MB, > whereas on master (for 3.8.2) it weighs 6MB. Here is what size has to > say: Yeah, the .o files in there likely contain serialized GIMPLE, as a result of compiling with ‘-flto’. How about removing *.a altogether in a separate patch? Or moving it to a “static” output? This would be consistent with what’s done in other packages. Thanks, Ludo’. ^ permalink raw reply [flat|nested] 9+ messages in thread
* [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations. 2021-06-03 9:48 ` Ludovic Courtès @ 2021-06-05 15:43 ` Julien Lepiller 2021-06-06 12:34 ` Ludovic Courtès 0 siblings, 1 reply; 9+ messages in thread From: Julien Lepiller @ 2021-06-05 15:43 UTC (permalink / raw) To: Ludovic Courtès; +Cc: 47251 [-- Attachment #1: Type: text/plain, Size: 1169 bytes --] Le Thu, 03 Jun 2021 11:48:40 +0200, Ludovic Courtès <ludo@gnu.org> a écrit : > Hi, > > Julien Lepiller <julien@lepiller.eu> skribis: > > > But I noticed another file gained a lot of weight: > > lib/python3.9/config-3.9-x86_64-linux-gnu/libpython3.9.a weighs > > 39MB, whereas on master (for 3.8.2) it weighs 6MB. Here is what > > size has to say: > > Yeah, the .o files in there likely contain serialized GIMPLE, as a > result of compiling with ‘-flto’. > > How about removing *.a altogether in a separate patch? Or moving it > to a “static” output? This would be consistent with what’s done in > other packages. > > Thanks, > Ludo’. Right, this is a good idea. I wrote the attached patches to: enable optimizations, prevent building and installing the static library and to install idle to a separate output, as it is not required in general. With the patch from https://issues.guix.gnu.org/47214, python 3.9 on coreupdates is ~1.7x faster than python 3.8 on master, and is ~25MB smaller than 3.8 on master (I haven't measured the speedup for python2, and it is ~10MB smaller with these changes than on master). [-- Attachment #2: 0002-gnu-python-Enable-optimizations.patch --] [-- Type: text/x-patch, Size: 1767 bytes --] From 0c45d6e82a5f4d140840aef6b0de321122b6c4df Mon Sep 17 00:00:00 2001 From: Julien Lepiller <julien@lepiller.eu> Date: Thu, 18 Mar 2021 23:41:40 +0100 Subject: [PATCH 2/4] gnu: python: Enable optimizations. As a result, python execution time of scripts is decreased by 6 to 40%. gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to optimize the resulting python. --- gnu/packages/python.scm | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 0f20f8e7d3..031904dff6 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -184,7 +184,10 @@ "--with-system-expat" ;for XML support "--with-system-ffi" ;build ctypes "--with-ensurepip=install" ;install pip and setuptools + "--with-computed-gotos" ;main interpreter loop optimization + "--with-lto" ;increase size by 20MB, but 15% speedup "--enable-unicode=ucs4" + "--enable-optimizations" ;; Prevent the installed _sysconfigdata.py from retaining a reference ;; to coreutils. @@ -198,8 +201,10 @@ "ac_cv_file__dev_ptmx=no" "ac_cv_file__dev_ptc=no") '()) + "CFLAGS=-fno-semantic-interposition" (string-append "LDFLAGS=-Wl,-rpath=" - (assoc-ref %outputs "out") "/lib")) + (assoc-ref %outputs "out") "/lib" + " -fno-semantic-interposition")) ;; With no -j argument tests use all available cpus, so provide one. #:make-flags (list (string-append -- 2.31.1 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: 0003-gnu-python-Do-not-install-static-libraries.patch --] [-- Type: text/x-patch, Size: 9683 bytes --] From 1faeb01becd558f0f4eec16ff32df9dbf1adda7f Mon Sep 17 00:00:00 2001 From: Julien Lepiller <julien@lepiller.eu> Date: Thu, 3 Jun 2021 22:10:09 +0200 Subject: [PATCH 3/4] gnu: python: Do not install static libraries. This is something Fedora already does since 2010, so it should be safe, and saves a lot of space. * gnu/packages/patches/python-2.7-no-static-lib.patch: New file. * gnu/packages/patches/python-3-no-static-lib.patch: New file. * gnu/local.mk (dist_patch_DATA): Add them. * gnu/packages/python.scm (python-2.7, python-3): Use them. --- gnu/local.mk | 2 + .../patches/python-2.7-no-static-lib.patch | 70 +++++++++++++++++ .../patches/python-3-no-static-lib.patch | 75 +++++++++++++++++++ gnu/packages/python.scm | 6 +- 4 files changed, 151 insertions(+), 2 deletions(-) create mode 100644 gnu/packages/patches/python-2.7-no-static-lib.patch create mode 100644 gnu/packages/patches/python-3-no-static-lib.patch diff --git a/gnu/local.mk b/gnu/local.mk index d5a832146e..8370ac12e2 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -1551,11 +1551,13 @@ dist_patch_DATA = \ %D%/packages/patches/python-2.7-site-prefixes.patch \ %D%/packages/patches/python-2.7-source-date-epoch.patch \ %D%/packages/patches/python-2.7-CVE-2021-3177.patch \ + %D%/packages/patches/python-2.7-no-static-lib.patch \ %D%/packages/patches/python-3-arm-alignment.patch \ %D%/packages/patches/python-3-deterministic-build-info.patch \ %D%/packages/patches/python-3-search-paths.patch \ %D%/packages/patches/python-3-fix-tests.patch \ %D%/packages/patches/python-3-hurd-configure.patch \ + %D%/packages/patches/python-3-no-static-lib.patch \ %D%/packages/patches/python-CVE-2018-14647.patch \ %D%/packages/patches/python-CVE-2020-26116.patch \ %D%/packages/patches/python-aionotify-0.2.0-py3.8.patch \ diff --git a/gnu/packages/patches/python-2.7-no-static-lib.patch b/gnu/packages/patches/python-2.7-no-static-lib.patch new file mode 100644 index 0000000000..02529989be --- /dev/null +++ b/gnu/packages/patches/python-2.7-no-static-lib.patch @@ -0,0 +1,70 @@ +From 92e2dcace7a3c02289b314238c5f18caec659ff1 Mon Sep 17 00:00:00 2001 +From: Julien Lepiller <julien@lepiller.eu> +Date: Sat, 5 Jun 2021 12:41:47 +0200 +Subject: [PATCH] No static lib. + +This patch is adapted from Fedora. The original patch did not apply +because of whitespace changes. + +00111 # +Patch the Makefile.pre.in so that the generated Makefile doesn't try to build +a libpythonMAJOR.MINOR.a (bug 550692): +Downstream only: not appropriate for upstream +--- + Makefile.pre.in | 26 +------------------------- + 1 file changed, 1 insertion(+), 25 deletions(-) + +diff --git a/Makefile.pre.in b/Makefile.pre.in +index 6239fc3..948f196 100644 +--- a/Makefile.pre.in ++++ b/Makefile.pre.in +@@ -486,7 +486,7 @@ coverage-report: regen-grammar + + + # Build the interpreter +-$(BUILDPYTHON): Modules/python.o $(LIBRARY) $(LDLIBRARY) ++$(BUILDPYTHON): Modules/python.o $(LDLIBRARY) + $(LINKCC) $(LDFLAGS) $(LINKFORSHARED) -o $@ \ + Modules/python.o \ + $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST) +@@ -527,18 +527,6 @@ sharedmods: $(BUILDPYTHON) pybuilddir.txt Modules/_math.o + _TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \ + $(PY_BUILD_ENVIRON) $(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build + +-# Build static library +-# avoid long command lines, same as LIBRARY_OBJS +-$(LIBRARY): $(LIBRARY_OBJS) +- -rm -f $@ +- $(AR) $(ARFLAGS) $@ Modules/getbuildinfo.o +- $(AR) $(ARFLAGS) $@ $(PARSER_OBJS) +- $(AR) $(ARFLAGS) $@ $(OBJECT_OBJS) +- $(AR) $(ARFLAGS) $@ $(PYTHON_OBJS) +- $(AR) $(ARFLAGS) $@ $(MODULE_OBJS) $(SIGNAL_OBJS) +- $(AR) $(ARFLAGS) $@ $(MODOBJS) +- $(RANLIB) $@ +- + libpython$(VERSION).so: $(LIBRARY_OBJS) + if test $(INSTSONAME) != $(LDLIBRARY); then \ + $(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \ +@@ -1216,18 +1204,6 @@ libainstall: @DEF_MAKE_RULE@ python-config + else true; \ + fi; \ + done +- @if test -d $(LIBRARY); then :; else \ +- if test "$(PYTHONFRAMEWORKDIR)" = no-framework; then \ +- if test "$(SO)" = .dll; then \ +- $(INSTALL_DATA) $(LDLIBRARY) $(DESTDIR)$(LIBPL) ; \ +- else \ +- $(INSTALL_DATA) $(LIBRARY) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \ +- $(RANLIB) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \ +- fi; \ +- else \ +- echo Skip install of $(LIBRARY) - use make frameworkinstall; \ +- fi; \ +- fi + $(INSTALL_DATA) Modules/config.c $(DESTDIR)$(LIBPL)/config.c + $(INSTALL_DATA) Modules/python.o $(DESTDIR)$(LIBPL)/python.o + $(INSTALL_DATA) $(srcdir)/Modules/config.c.in $(DESTDIR)$(LIBPL)/config.c.in +-- +2.31.1 + diff --git a/gnu/packages/patches/python-3-no-static-lib.patch b/gnu/packages/patches/python-3-no-static-lib.patch new file mode 100644 index 0000000000..b44dbc976c --- /dev/null +++ b/gnu/packages/patches/python-3-no-static-lib.patch @@ -0,0 +1,75 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: David Malcolm <dmalcolm@redhat.com> +Date: Mon, 18 Jan 2010 17:59:07 +0000 +Subject: [PATCH] 00111: Don't try to build a libpythonMAJOR.MINOR.a +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Downstream only: not appropriate for upstream. + +See https://bugzilla.redhat.com/show_bug.cgi?id=556092 + +Co-authored-by: David Malcolm <dmalcolm@redhat.com> +Co-authored-by: Bohuslav Kabrda <bkabrda@redhat.com> +Co-authored-by: Matej Stuchlik <mstuchli@redhat.com> +Co-authored-by: Robert Kuska <rkuska@redhat.com> +Co-authored-by: Charalampos Stratakis <cstratak@redhat.com> +Co-authored-by: Miro Hrončok <miro@hroncok.cz> +--- + Makefile.pre.in | 21 ++------------------- + 1 file changed, 2 insertions(+), 19 deletions(-) + +diff --git a/Makefile.pre.in b/Makefile.pre.in +index 0c3981c132..5587422ec7 100644 +--- a/Makefile.pre.in ++++ b/Makefile.pre.in +@@ -589,7 +589,7 @@ clinic: check-clean-src $(srcdir)/Modules/_blake2/blake2s_impl.c + $(PYTHON_FOR_REGEN) $(srcdir)/Tools/clinic/clinic.py --make --srcdir $(srcdir) + + # Build the interpreter +-$(BUILDPYTHON): Programs/python.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS) ++$(BUILDPYTHON): Programs/python.o $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS) + $(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) + + platform: $(BUILDPYTHON) pybuilddir.txt +@@ -637,12 +637,6 @@ sharedmods: $(BUILDPYTHON) pybuilddir.txt Modules/_math.o + _TCLTK_INCLUDES='$(TCLTK_INCLUDES)' _TCLTK_LIBS='$(TCLTK_LIBS)' \ + $(PYTHON_FOR_BUILD) $(srcdir)/setup.py $$quiet build + +- +-# Build static library +-$(LIBRARY): $(LIBRARY_OBJS) +- -rm -f $@ +- $(AR) $(ARFLAGS) $@ $(LIBRARY_OBJS) +- + libpython$(LDVERSION).so: $(LIBRARY_OBJS) $(DTRACE_OBJS) + if test $(INSTSONAME) != $(LDLIBRARY); then \ + $(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM); \ +@@ -724,7 +718,7 @@ Makefile Modules/config.c: Makefile.pre \ + @echo "The Makefile was updated, you may need to re-run make." + + +-Programs/_testembed: Programs/_testembed.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS) ++Programs/_testembed: Programs/_testembed.o $(LDLIBRARY) $(PY3LIBRARY) $(EXPORTSYMS) + $(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/_testembed.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) + + ############################################################################ +@@ -1652,17 +1646,6 @@ libainstall: @DEF_MAKE_RULE@ python-config + else true; \ + fi; \ + done +- @if test -d $(LIBRARY); then :; else \ +- if test "$(PYTHONFRAMEWORKDIR)" = no-framework; then \ +- if test "$(SHLIB_SUFFIX)" = .dll; then \ +- $(INSTALL_DATA) $(LDLIBRARY) $(DESTDIR)$(LIBPL) ; \ +- else \ +- $(INSTALL_DATA) $(LIBRARY) $(DESTDIR)$(LIBPL)/$(LIBRARY) ; \ +- fi; \ +- else \ +- echo Skip install of $(LIBRARY) - use make frameworkinstall; \ +- fi; \ +- fi + $(INSTALL_DATA) Modules/config.c $(DESTDIR)$(LIBPL)/config.c + $(INSTALL_DATA) Programs/python.o $(DESTDIR)$(LIBPL)/python.o + $(INSTALL_DATA) $(srcdir)/Modules/config.c.in $(DESTDIR)$(LIBPL)/config.c.in diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 031904dff6..4ac4b7c23c 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -150,7 +150,8 @@ "python-2.7-source-date-epoch.patch" "python-2.7-adjust-tests.patch" "python-cross-compile.patch" - "python-2.7-CVE-2021-3177.patch")) + "python-2.7-CVE-2021-3177.patch" + "python-2.7-no-static-lib.patch")) (modules '((guix build utils))) (snippet '(begin @@ -418,7 +419,8 @@ data types.") "python-3-deterministic-build-info.patch" "python-3-fix-tests.patch" "python-3-hurd-configure.patch" - "python-3-search-paths.patch")) + "python-3-search-paths.patch" + "python-3-no-static-lib.patch")) (sha256 (base32 "10vdf46q5ldnzkprm8pldvr5a9hrdpxjv7mpzgdw6vj3cl318nhc")) -- 2.31.1 [-- Attachment #4: 0004-gnu-python-Move-idle-to-a-separate-output.patch --] [-- Type: text/x-patch, Size: 3301 bytes --] From c650ac48be47b373c6c8c0b3495f30d9fe21e68b Mon Sep 17 00:00:00 2001 From: Julien Lepiller <julien@lepiller.eu> Date: Thu, 3 Jun 2021 22:14:14 +0200 Subject: [PATCH 4/4] gnu: python: Move idle to a separate output. idle is a programming environment that is not useful for building packages, but it is still useful for end users. This saves 5MB in the default output. * gnu/packages/python.scm (python-2.7)[outputs]: Add `idle'. [arguments]: Move idle files in the new output. (wrap-python): Do not wrap idle since it is in a different output. --- gnu/packages/python.scm | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 4ac4b7c23c..72e05552d8 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -176,7 +176,8 @@ "Lib/test/test_subprocess.py")) #t)))) (outputs '("out" - "tk")) ;tkinter; adds 50 MiB to the closure + "tk" ;tkinter; adds 50 MiB to the closure + "idle")) ;programming environment; weighs 5MB (build-system gnu-build-system) (arguments `(#:test-target "test" @@ -357,6 +358,27 @@ (install-file tkinter.so target) (delete-file tkinter.so))))) #t))) + (add-after 'install 'move-idle + (lambda* (#:key outputs #:allow-other-keys) + ;; when idle is built, move it to a separate output to save some + ;; space (5MB) + (let ((out (assoc-ref outputs "out")) + (idle (assoc-ref outputs "idle"))) + (when idle + (for-each + (lambda (file) + (let ((target (string-append idle "/bin/" (basename file)))) + (install-file file (dirname target)) + (delete-file file))) + (find-files (string-append out "/bin") "^idle")) + (match (find-files out "^idlelib$" #:directories? #t) + ((idlelib) + (let* ((len (string-length out)) + (target (string-append idle "/" + (string-drop idlelib len) + "/site-packages"))) + (mkdir-p (dirname target)) + (rename-file idlelib target)))))))) (add-after 'install 'install-sitecustomize.py ,(customize-site version))))) (inputs @@ -620,8 +642,8 @@ for more information."))) (lambda (old new) (symlink (string-append python old) (string-append bin "/" new))) - `("python3" ,"pydoc3" ,"idle3" ,"pip3") - `("python" ,"pydoc" ,"idle" ,"pip")) + `("python3" ,"pydoc3" ,"pip3") + `("python" ,"pydoc" ,"pip")) ;; python-config outputs search paths based upon its location, ;; use a bash wrapper to avoid changing its outputs. (let ((bash (string-append (assoc-ref %build-inputs "bash") -- 2.31.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations. 2021-06-05 15:43 ` Julien Lepiller @ 2021-06-06 12:34 ` Ludovic Courtès 2021-06-11 1:03 ` bug#47251: " Julien Lepiller 0 siblings, 1 reply; 9+ messages in thread From: Ludovic Courtès @ 2021-06-06 12:34 UTC (permalink / raw) To: Julien Lepiller; +Cc: 47251 Hi, Julien Lepiller <julien@lepiller.eu> skribis: > Right, this is a good idea. I wrote the attached patches to: enable > optimizations, prevent building and installing the static library and > to install idle to a separate output, as it is not required in general. > > With the patch from https://issues.guix.gnu.org/47214, python 3.9 on > coreupdates is ~1.7x faster than python 3.8 on master, and is ~25MB > smaller than 3.8 on master (I haven't measured the speedup for python2, > and it is ~10MB smaller with these changes than on master). Yay! > From 0c45d6e82a5f4d140840aef6b0de321122b6c4df Mon Sep 17 00:00:00 2001 > From: Julien Lepiller <julien@lepiller.eu> > Date: Thu, 18 Mar 2021 23:41:40 +0100 > Subject: [PATCH 2/4] gnu: python: Enable optimizations. > > As a result, python execution time of scripts is decreased by 6 to 40%. > > gnu/packages/python (python-2.7)[arguments]: Add configure and make flags to > optimize the resulting python. [...] > + "CFLAGS=-fno-semantic-interposition" > (string-append "LDFLAGS=-Wl,-rpath=" > - (assoc-ref %outputs "out") "/lib")) > + (assoc-ref %outputs "out") "/lib" > + " -fno-semantic-interposition")) Could you add a comment for ‘-fno-semantic-interposition’? Also, please make sure that setting CFLAGS does not override useful defaults (for Autoconf-based build systems, CFLAGS defaults to ‘-O2 -g’ and overriding it means you lost both -O2 and -g; if we lose ‘-g’, ‘--with-debug-info’ won’t have any effect). > From 1faeb01becd558f0f4eec16ff32df9dbf1adda7f Mon Sep 17 00:00:00 2001 > From: Julien Lepiller <julien@lepiller.eu> > Date: Thu, 3 Jun 2021 22:10:09 +0200 > Subject: [PATCH 3/4] gnu: python: Do not install static libraries. > > This is something Fedora already does since 2010, so it should be safe, > and saves a lot of space. > > * gnu/packages/patches/python-2.7-no-static-lib.patch: New file. > * gnu/packages/patches/python-3-no-static-lib.patch: New file. > * gnu/local.mk (dist_patch_DATA): Add them. > * gnu/packages/python.scm (python-2.7, python-3): Use them. LGTM. > From c650ac48be47b373c6c8c0b3495f30d9fe21e68b Mon Sep 17 00:00:00 2001 > From: Julien Lepiller <julien@lepiller.eu> > Date: Thu, 3 Jun 2021 22:14:14 +0200 > Subject: [PATCH 4/4] gnu: python: Move idle to a separate output. > > idle is a programming environment that is not useful for building > packages, but it is still useful for end users. This saves 5MB in the > default output. > > * gnu/packages/python.scm (python-2.7)[outputs]: Add `idle'. > [arguments]: Move idle files in the new output. > (wrap-python): Do not wrap idle since it is in a different output. Good idea, LGTM. Thank you! Ludo’. ^ permalink raw reply [flat|nested] 9+ messages in thread
* bug#47251: [core-updates] [patch] gnu: python: Enable optimizations. 2021-06-06 12:34 ` Ludovic Courtès @ 2021-06-11 1:03 ` Julien Lepiller 0 siblings, 0 replies; 9+ messages in thread From: Julien Lepiller @ 2021-06-11 1:03 UTC (permalink / raw) To: Ludovic Courtès; +Cc: 47251-done Le Sun, 06 Jun 2021 14:34:31 +0200, Ludovic Courtès <ludo@gnu.org> a écrit : > > > + "CFLAGS=-fno-semantic-interposition" > > (string-append "LDFLAGS=-Wl,-rpath=" > > - (assoc-ref %outputs "out") "/lib")) > > + (assoc-ref %outputs "out") "/lib" > > + " -fno-semantic-interposition")) > > Could you add a comment for ‘-fno-semantic-interposition’? Also, > please make sure that setting CFLAGS does not override useful > defaults (for Autoconf-based build systems, CFLAGS defaults to ‘-O2 > -g’ and overriding it means you lost both -O2 and -g; if we lose > ‘-g’, ‘--with-debug-info’ won’t have any effect). > With the patch, python still builds with -g -O3. Pushed as 99d450b085b7bf44d669093f6c9964391ce1b451 to e4aa346d57f61956626032d0d1dfca6b3c96e06a to core-updates, thanks! ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-06-11 1:04 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-03-18 23:16 [bug#47251] [core-updates] [patch] gnu: python: Enable optimizations Julien Lepiller 2021-03-31 12:47 ` Ludovic Courtès 2021-04-10 20:55 ` Ludovic Courtès 2021-04-10 20:55 ` Ludovic Courtès 2021-06-03 2:06 ` Julien Lepiller 2021-06-03 9:48 ` Ludovic Courtès 2021-06-05 15:43 ` Julien Lepiller 2021-06-06 12:34 ` Ludovic Courtès 2021-06-11 1:03 ` bug#47251: " Julien Lepiller
Code repositories for project(s) associated with this external index https://git.savannah.gnu.org/cgit/guix.git This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.