bug#23967: 25.1.50; Slow compilation of ns-win.el

bug#23967: 25.1.50; Slow compilation of ns-win.el

[Lars Ingebrigtsen]

I've been looking slightly at why "make -j8" isn't as parallel as you'd
expect, and in my tests it's because one single .el compilation job is
weirdly slow.

To test:

If I say "make bootstrap" (to get rid of everything) and then halt the
compilation after it's made bootstrap-emacs

[larsi@stories ~/src/emacs/trunk/lisp]$ time ../src/bootstrap-emacs -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t)" -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../lisp/term/ns-win.el

real    0m44.097s
user    0m44.156s
sys     0m0.012s

44 seconds to compile it.  From stracing, the thing it says before it
... doesn't say anything for half a minute is:

readlinkat(AT_FDCWD, "/home/larsi/src/emacs/trunk/lisp/international/uni-decomposition.el", 0x7fff42af6240, 1024) = -1 EINVAL (Invalid argument)

So...  er...  it has something to do with Unicode?

This may not be a bug or anything, but it's perhaps something that
should be looked at to speed up the initial Emacs compilation.


In GNU Emacs 25.1.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.14.5)
 of 2016-07-12 built on stories
Repository revision: df7774be39af76d3072a0278ef815a47bf50dfe9
Windowing system distributor 'The X.Org Foundation', version 11.0.11604000
System Description:	Debian GNU/Linux 8.5 (jessie)

Recent messages:
Error reading dir-locals: (error "Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-macs.elc")
Eager macro-expansion failure: (error "Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-macs.elc") [8 times]
Error: (error "Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-macs.elc")
previous-line: Beginning of buffer
Error reading dir-locals: (error "Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-macs.elc")
Eager macro-expansion failure: (error "Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-macs.elc") [8 times]
Error: (error "Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-macs.elc")
Mark saved where search started
Mark set
Making completion list...

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  global-whitespace-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t

Load-path shadows:
/home/larsi/src/cddb.el/expect hides /home/larsi/lisp/expect
/home/larsi/src/cddb.el/captitle hides /home/larsi/lisp/captitle
/home/larsi/src/clock.el/clock hides /home/larsi/lisp/clock
~/pgnus/contrib/vcard hides /home/larsi/lisp/vcard
/home/larsi/src/pvr.el/pvr hides /home/larsi/lisp/pvr
~/lisp/zenirc-2.112/src/zenirc-example hides /home/larsi/lisp/zenirc-example
~/pgnus/contrib/compface hides /home/larsi/src/emacs/trunk/lisp/image/compface

Features:
(shadow ecomplete emacsbug sendmail vc vc-dispatcher map pp misearch
multi-isearch copyright vc-cvs shr-color color eww gnus-html help-fns
radix-tree sort gnus-cite smiley ansi-color url-queue url-cache
mm-archive gnus-async gnus-dup qp gnus-ml gmane spam-gmane dns mm-url
disp-table gnus-fun gnus-mdrtn gnus-topic pop3 nndoc nnmbox utf-7 nnml
nnfolder network-stream starttls nnir spam-report spam spam-stat gnus-uu
yenc gnus-delay gnus-draft gnus-agent gnus-srvr gnus-score score-mode
nnvirtual nntp gnus-cache gnus-msg gnus-art mm-uu mml2015 mm-view
mml-smime smime dig gnus-sum nndraft nnmh gnus-group gnus-undo
gnus-start gnus-cloud nnimap utf7 netrc nnoo parse-time gnus-spec
gnus-win nnmail gnus-int gnus-range mail-source message format-spec
rfc822 mml mml-sec epa epg mailabbrev gmm-utils mailheader gnus nnheader
gnus-util rmail rmail-loaddefs mail-utils whitespace movie mkv shr svg
imdb dom pvr debug debbugs-gnu easy-mmode derived debbugs soap-client
mm-decode mm-bodies mm-encode url-http tls gnutls url-auth mail-parse
rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr url-gw nsm puny
url url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap warnings rng-xsd rng-dt rng-util xsd-regexp
xml ido flyspell ispell benchmark w3m browse-url doc-view subr-x dired
dired-loaddefs image-mode timezone w3m-hist w3m-fb w3m-ems wid-edit
w3m-ccl ccl w3m-favicon w3m-image w3m-proc w3m-util add-log mail-extr
jka-compr cl finder-inf package epg-config url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache url-vars seq byte-opt gv bytecomp byte-compile cl-extra
help-mode easymenu cconv cl-loaddefs pcase cl-lib time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core term/tty-colors frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 1045166 126768)
 (symbols 48 170836 0)
 (miscs 40 288 729)
 (strings 32 245855 17893)
 (string-bytes 1 14892431)
 (vectors 16 36358)
 (vector-slots 8 1122185 66612)
 (floats 8 6931 746)
 (intervals 56 60845 5795)
 (buffers 976 61)
 (heap 1024 184172 181041))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#23967: 25.1.50; Slow compilation of ns-win.el

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 13 Jul 2016 14:19:21 +0200
> 
> I've been looking slightly at why "make -j8" isn't as parallel as you'd
> expect, and in my tests it's because one single .el compilation job is
> weirdly slow.
> 
> To test:
> 
> If I say "make bootstrap" (to get rid of everything) and then halt the
> compilation after it's made bootstrap-emacs
> 
> [larsi@stories ~/src/emacs/trunk/lisp]$ time ../src/bootstrap-emacs -batch --no-site-file --no-site-lisp --eval "(setq load-prefer-newer t)" -l bytecomp -f byte-compile-refresh-preloaded -f batch-byte-compile ../lisp/term/ns-win.el
> 
> real    0m44.097s
> user    0m44.156s
> sys     0m0.012s
> 
> 44 seconds to compile it.  From stracing, the thing it says before it
> ... doesn't say anything for half a minute is:
> 
> readlinkat(AT_FDCWD, "/home/larsi/src/emacs/trunk/lisp/international/uni-decomposition.el", 0x7fff42af6240, 1024) = -1 EINVAL (Invalid argument)
> 
> So...  er...  it has something to do with Unicode?
> 
> This may not be a bug or anything, but it's perhaps something that
> should be looked at to speed up the initial Emacs compilation.

This is a known problem.  The culprit is ucs-normalize.el
(uni-decomposition is loaded when ucs-normalize is compiled).  It
takes a long time to compile even with Emacs that already has the
byte-compiled byte-compiler loaded into it.  When compiling
ucs-normalize with an interpreted byte-compiler, it takes ages (11 min
on my Core i7).  And since ucs-normalize is now preloaded on OS X, we
compile it with the interpreted byte-compiler, as we do with any other
file that is preloaded on _some_ platform.

Patches to solve this conundrum in some way are welcome.

bug#23967: 25.1.50; Slow compilation of ns-win.el

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

On Wed, Jul 13, 2016 at 10:55 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> This is a known problem.  The culprit is ucs-normalize.el
> (uni-decomposition is loaded when ucs-normalize is compiled).  It
> takes a long time to compile even with Emacs that already has the
> byte-compiled byte-compiler loaded into it.

I ran the profiler on a compilation of ucs-normalize.el and found 2
easy optimizations (ucs-normalize-block-compose-chars was using
with-temp-buffer in a loop, so I lifted the it out of the loop; using
regexp-opt-charset instead of regexp-opt saves some char-to-string
conversion, sorting, and duplicate deletion). The attached patch
brings the compilation down from 2.5 seconds to 0.8 seconds in my
normal running Emacs, and using the bootstrap-emacs command posted by
Lars (swapping ../lisp/international/ucs-normalize.el for
../lisp/term/ns-win.el) from 1m30s to 7s.

>  When compiling
> ucs-normalize with an interpreted byte-compiler, it takes ages (11 min
> on my Core i7).  And since ucs-normalize is now preloaded on OS X, we
> compile it with the interpreted byte-compiler, as we do with any other
> file that is preloaded on _some_ platform.
>
> Patches to solve this conundrum in some way are welcome.

Could we call `byte-compile' on the byte-compiler functions after loading them?

[-- Attachment #2: ucs-normalize.el.diff --]
[-- Type: text/plain, Size: 2735 bytes --]

diff --git i/lisp/international/ucs-normalize.el w/lisp/international/ucs-normalize.el
index 4b364ee..c03ea50 100644
--- i/lisp/international/ucs-normalize.el
+++ w/lisp/international/ucs-normalize.el
@@ -263,7 +263,7 @@ ucs-normalize-combining-chars
 (defvar ucs-normalize-combining-chars-regexp nil
   "Regular expression to match sequence of combining characters.")
   (setq ucs-normalize-combining-chars-regexp
-  (eval-when-compile (concat (regexp-opt (mapcar 'char-to-string combining-chars)) "+")))
+        (eval-when-compile (concat (regexp-opt-charset combining-chars) "+")))
 
 (declare-function decomposition-translation-alist "ucs-normalize"
                   (decomposition-function))
@@ -396,20 +396,20 @@ ucs-normalize-block-compose-chars
 It includes Singletons, CompositionExclusions, and Non-Starter
 decomposition."
     (let (entries decomposition composition)
-      (mapc
-       (lambda (start-end)
-         (cl-do ((i (car start-end) (+ i 1))) ((> i (cdr start-end)))
-           (setq decomposition
-                 (string-to-list
-                  (with-temp-buffer
-                    (insert i)
-                    (translate-region 1 2 decomposition-translation)
-                    (buffer-string))))
-           (setq composition
-                 (ucs-normalize-block-compose-chars decomposition composition-predicate))
-           (when (not (equal composition (list i)))
-             (setq entries (cons i entries)))))
-       check-range)
+      (with-temp-buffer
+        (dolist (start-end check-range)
+          (cl-do ((i (car start-end) (+ i 1))) ((> i (cdr start-end)))
+            (setq decomposition
+                  (string-to-list
+                   (progn
+                     (erase-buffer)
+                     (insert i)
+                     (translate-region 1 2 decomposition-translation)
+                     (buffer-string))))
+            (setq composition
+                  (ucs-normalize-block-compose-chars decomposition composition-predicate))
+            (when (not (equal composition (list i)))
+              (setq entries (cons i entries))))))
       ;;(remove-duplicates
        (append entries
                ucs-normalize-composition-exclusions
@@ -431,7 +431,7 @@ ucs-normalize-block-compose-chars
     (setq hfs-nfc-quick-check-list (quick-check-list 'ucs-normalize-hfs-nfd-table t ))
 
   (defun quick-check-list-to-regexp (quick-check-list)
-    (regexp-opt (mapcar 'char-to-string (append quick-check-list combining-chars))))
+    (regexp-opt-charset (append quick-check-list combining-chars)))
 
   (defun quick-check-decomposition-list-to-regexp (quick-check-list)
     (concat (quick-check-list-to-regexp quick-check-list) "\\|[가-힣]"))

bug#23967: 25.1.50; Slow compilation of ns-win.el

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Wed, 13 Jul 2016 17:15:15 -0400
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 23967@debbugs.gnu.org
> 
> I ran the profiler on a compilation of ucs-normalize.el and found 2
> easy optimizations (ucs-normalize-block-compose-chars was using
> with-temp-buffer in a loop, so I lifted the it out of the loop; using
> regexp-opt-charset instead of regexp-opt saves some char-to-string
> conversion, sorting, and duplicate deletion). The attached patch
> brings the compilation down from 2.5 seconds to 0.8 seconds in my
> normal running Emacs, and using the bootstrap-emacs command posted by
> Lars (swapping ../lisp/international/ucs-normalize.el for
> ../lisp/term/ns-win.el) from 1m30s to 7s.

LGTM, thanks.

However, I'm worried that we have no test for ucs-normalize, so it's
hard to be sure the non-trivial functionality is unchanged, even
though your changes are pretty straightforward.

How about adding a test that uses the data in this file:

  http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt

ucs-normalize claims to have passed an old version of this, but I see
no existing way of re-running that test, did I miss something?

I'd feel much safer if the tests all pass at least as they did with
the old version.

> Could we call `byte-compile' on the byte-compiler functions after loading them?

Maybe, you will have to try.  The bootstrap of the byte compiler is
somewhat tricky, given all the dependencies (see COMPILE_FIRST in
lisp/Makefile.in).

Thanks.

bug#23967: 25.1.50; Slow compilation of ns-win.el

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 1653 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> However, I'm worried that we have no test for ucs-normalize, so it's
> hard to be sure the non-trivial functionality is unchanged, even
> though your changes are pretty straightforward.
>
> How about adding a test that uses the data in this file:
>
>   http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt
>
> ucs-normalize claims to have passed an old version of this, but I see
> no existing way of re-running that test, did I miss something?

I don't see any evidence of an existing test.  I stared writing a new
one, and it's failing with the original ucs-normalize.el (or I'm
misunderstanding the requirements).

The first invariant to test is

    c2 ==  toNFC(c1) ==  toNFC(c2) == toNFC(c3)

(cX is column X, columns numbered from 1).

Line 15131 of NormalizationTest.txt has
# c1   c2      c3
1112E;1112E;11131 11127;1112E;11131 11127; # (◌𑄮; ◌𑄮; ◌𑄱◌𑄧; ◌𑄮; ◌𑄱◌𑄧; ) CHAKMA VOWEL SIGN O

So I think toNFC(c3) == c2 is equivalent to

    (equal (ucs-normalize-NFC-string
            (string #x11131 #x11127))
           (string #x1112E))

which gives nil.

Lines 15131 to 15139 and 16149 to 16289 are failing.  To check
invariants for a single line, load the attached ucs-normalize-tests.el,
put point at the beginning of the line and evaluate

        (ucs-normalize-tests--invariants-hold-p
         (ucs-normalize-tests--parse-column)
         (ucs-normalize-tests--parse-column)
         (ucs-normalize-tests--parse-column)
         (ucs-normalize-tests--parse-column)
         (ucs-normalize-tests--parse-column))


[-- Attachment #2: initial ucs-normalize.el file --]
[-- Type: application/emacs-lisp, Size: 5935 bytes --]

bug#23967: 25.1.50; Slow compilation of ns-win.el

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 23967@debbugs.gnu.org,  larsi@gnus.org
> Date: Thu, 14 Jul 2016 23:22:54 -0400
> 
> > How about adding a test that uses the data in this file:
> >
> >   http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt
> >
> > ucs-normalize claims to have passed an old version of this, but I see
> > no existing way of re-running that test, did I miss something?
> 
> I don't see any evidence of an existing test.

Right.

> I stared writing a new one, and it's failing with the original
> ucs-normalize.el (or I'm misunderstanding the requirements).

If the failures are identical to the original ucs-normalize, let's for
now just mark them as known failures, and look into them later.  I
don't want to delay this important change that speeds up the bootstrap
due to problems unrelated to the change.

> The first invariant to test is
> 
>     c2 ==  toNFC(c1) ==  toNFC(c2) == toNFC(c3)
> 
> (cX is column X, columns numbered from 1).
> 
> Line 15131 of NormalizationTest.txt has
> # c1   c2      c3
> 1112E;1112E;11131 11127;1112E;11131 11127; # (◌𑄮; ◌𑄮; ◌𑄱◌𑄧; ◌𑄮; ◌𑄱◌𑄧; ) CHAKMA VOWEL SIGN O
> 
> So I think toNFC(c3) == c2 is equivalent to
> 
>     (equal (ucs-normalize-NFC-string
>             (string #x11131 #x11127))
>            (string #x1112E))
> 
> which gives nil.
> 
> Lines 15131 to 15139 and 16149 to 16289 are failing.

I will look into this later.  Thanks for the footwork.

bug#23967: 25.1.50; Slow compilation of ns-win.el

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> I stared writing a new one, and it's failing with the original
>> ucs-normalize.el (or I'm misunderstanding the requirements).
>
> If the failures are identical to the original ucs-normalize, let's for
> now just mark them as known failures, and look into them later.  I
> don't want to delay this important change that speeds up the bootstrap
> due to problems unrelated to the change.
>> 
>> Lines 15131 to 15139 and 16149 to 16289 are failing.

I finished writing the tests.  It does pass with
http://www.unicode.org/Public/5.2.0/ucd/NormalizationTest.txt as was
claimed in the ucs-normalize.el file (except that some of the rule 2
tests fail with that version, but that's only because version 5.2.0 has
less code points listed in Part 1).

Here are the patches, first adds the test (I elided the content of
NormalizationTest.txt to save space), second is the optimization (same
as before, except I dropped the change from mapc to dolist, because some
benchmark-runs showed me that mapc is significantly faster when not
compiled).


[-- Attachment #2: patch 1, elided content --]
[-- Type: application/octet-stream, Size: 3683 bytes --]

[-- Attachment #3: patch 2 --]
[-- Type: text/x-diff, Size: 3467 bytes --]

From 821aa63d1efd763d42cf0ae8249727aa7cb692fc Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Fri, 15 Jul 2016 22:10:33 -0400
Subject: [PATCH v1 2/2] Optimize ucs-normalize.el compilation

* lisp/international/ucs-normalize.el (ucs-normalize-combining-chars-regexp):
(quick-check-list-to-regexp): Use regexp-opt-charset instead of
regexp-opt.
* lisp/international/ucs-normalize.el (quick-check-list): Reuse a single
temp buffer for the whole loop.
---
 lisp/international/ucs-normalize.el | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)

diff --git a/lisp/international/ucs-normalize.el b/lisp/international/ucs-normalize.el
index 4b364ee..ac2a0d9 100644
--- a/lisp/international/ucs-normalize.el
+++ b/lisp/international/ucs-normalize.el
@@ -263,7 +263,7 @@ ucs-normalize-combining-chars
 (defvar ucs-normalize-combining-chars-regexp nil
   "Regular expression to match sequence of combining characters.")
   (setq ucs-normalize-combining-chars-regexp
-  (eval-when-compile (concat (regexp-opt (mapcar 'char-to-string combining-chars)) "+")))
+        (eval-when-compile (concat (regexp-opt-charset combining-chars) "+")))
 
 (declare-function decomposition-translation-alist "ucs-normalize"
                   (decomposition-function))
@@ -396,20 +396,22 @@ ucs-normalize-block-compose-chars
 It includes Singletons, CompositionExclusions, and Non-Starter
 decomposition."
     (let (entries decomposition composition)
-      (mapc
-       (lambda (start-end)
-         (cl-do ((i (car start-end) (+ i 1))) ((> i (cdr start-end)))
-           (setq decomposition
-                 (string-to-list
-                  (with-temp-buffer
-                    (insert i)
-                    (translate-region 1 2 decomposition-translation)
-                    (buffer-string))))
-           (setq composition
-                 (ucs-normalize-block-compose-chars decomposition composition-predicate))
-           (when (not (equal composition (list i)))
-             (setq entries (cons i entries)))))
-       check-range)
+      (with-temp-buffer
+        (mapc
+         (lambda (start-end)
+           (cl-do ((i (car start-end) (+ i 1))) ((> i (cdr start-end)))
+             (setq decomposition
+                   (string-to-list
+                    (progn
+                      (erase-buffer)
+                      (insert i)
+                      (translate-region 1 2 decomposition-translation)
+                      (buffer-string))))
+             (setq composition
+                   (ucs-normalize-block-compose-chars decomposition composition-predicate))
+             (when (not (equal composition (list i)))
+               (setq entries (cons i entries)))))
+         check-range))
       ;;(remove-duplicates
        (append entries
                ucs-normalize-composition-exclusions
@@ -431,7 +433,7 @@ ucs-normalize-block-compose-chars
     (setq hfs-nfc-quick-check-list (quick-check-list 'ucs-normalize-hfs-nfd-table t ))
 
   (defun quick-check-list-to-regexp (quick-check-list)
-    (regexp-opt (mapcar 'char-to-string (append quick-check-list combining-chars))))
+    (regexp-opt-charset (append quick-check-list combining-chars)))
 
   (defun quick-check-decomposition-list-to-regexp (quick-check-list)
     (concat (quick-check-list-to-regexp quick-check-list) "\\|[가-힣]"))
-- 
2.8.0

bug#23967: 25.1.50; Slow compilation of ns-win.el

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 23967@debbugs.gnu.org,  larsi@gnus.org
> Date: Fri, 15 Jul 2016 22:46:12 -0400
> 
> I finished writing the tests.  It does pass with
> http://www.unicode.org/Public/5.2.0/ucd/NormalizationTest.txt as was
> claimed in the ucs-normalize.el file (except that some of the rule 2
> tests fail with that version, but that's only because version 5.2.0 has
> less code points listed in Part 1).
> 
> Here are the patches, first adds the test (I elided the content of
> NormalizationTest.txt to save space), second is the optimization (same
> as before, except I dropped the change from mapc to dolist, because some
> benchmark-runs showed me that mapc is significantly faster when not
> compiled).

Thanks.  Please push this to master, and please also mention
NormalizationTest.txt in admin/notes/unicode, as one more file that
needs to be imported from each new Unicode standard, followed by
running this particular test, to verify compliance.

bug#23967: 25.1.50; Slow compilation of ns-win.el

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Cc: 23967@debbugs.gnu.org,  larsi@gnus.org
>> Date: Fri, 15 Jul 2016 22:46:12 -0400
>> 
>> I finished writing the tests.  It does pass with
>> http://www.unicode.org/Public/5.2.0/ucd/NormalizationTest.txt as was
>> claimed in the ucs-normalize.el file (except that some of the rule 2
>> tests fail with that version, but that's only because version 5.2.0 has
>> less code points listed in Part 1).
>> 
>> Here are the patches, first adds the test (I elided the content of
>> NormalizationTest.txt to save space), second is the optimization (same
>> as before, except I dropped the change from mapc to dolist, because some
>> benchmark-runs showed me that mapc is significantly faster when not
>> compiled).
>
> Thanks.  Please push this to master, and please also mention
> NormalizationTest.txt in admin/notes/unicode, as one more file that
> needs to be imported from each new Unicode standard, followed by
> running this particular test, to verify compliance.

Pushed as e333157, eed3b46.

bug#23967: 25.1.50; Slow compilation of ns-win.el

[Lars Ingebrigtsen]

npostavs@users.sourceforge.net writes:

> Pushed as e333157, eed3b46.

Great!  Thanks for fixing this stuff.

A "time make bootstrap -j 8" on my super-spiffy new 4 Core 4GHz i7
machine went from

real    2m48.326s
user    13m4.340s
sys     0m19.444s

to

real    2m19.142s
user    11m55.004s
sys     0m19.288s

That's a 17% improvement on the time it takes for users to do the first
build (and for developers to say "make bootstrap", of course), and is
probably a whole lot more seconds on not quite so spiffy machines.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#23967: 25.1.50; Slow compilation of ns-win.el

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Could we call `byte-compile' on the byte-compiler functions after loading them?
>
> Maybe, you will have to try.  The bootstrap of the byte compiler is
> somewhat tricky, given all the dependencies (see COMPILE_FIRST in
> lisp/Makefile.in).

So I tried moving the COMPILE_FIRST into loadup.el, which does bring
bootstrapping[1] down from 1m5s to 0m47s for me.  But IIUC it
reduces parallelism when compiling these files, so possibly it's
actually a loss overall.

[1]: Timed with compile-command = "rm -f bootstrap-emacs ../lisp/emacs-lisp/*.elc && time make bootstrap-emacs"


[-- Attachment #2: diff --]
[-- Type: text/x-diff, Size: 4199 bytes --]

diff --git a/lisp/Makefile.in b/lisp/Makefile.in
index 12bb9c7..e0d4522 100644
--- a/lisp/Makefile.in
+++ b/lisp/Makefile.in
@@ -100,20 +100,6 @@ AUTOGENEL =
 BYTE_COMPILE_FLAGS = \
   --eval '(setq load-prefer-newer t)' $(BYTE_COMPILE_EXTRA_FLAGS)
 
-# Files to compile before others during a bootstrap.  This is done to
-# speed up the bootstrap process.  They're ordered by size, so we use
-# the slowest-compiler on the smallest file and move to larger files as the
-# compiler gets faster.  'autoload.elc' comes last because it is not used by
-# the compiler (so its compilation does not speed up subsequent compilations),
-# it's only placed here so as to speed up generation of the loaddefs.el file.
-
-COMPILE_FIRST = \
-	$(lisp)/emacs-lisp/macroexp.elc \
-	$(lisp)/emacs-lisp/cconv.elc    \
-	$(lisp)/emacs-lisp/byte-opt.elc \
-	$(lisp)/emacs-lisp/bytecomp.elc \
-	$(lisp)/emacs-lisp/autoload.elc
-
 # Prevent any settings in the user environment causing problems.
 unexport EMACSDATA EMACSDOC EMACSPATH
 
@@ -281,9 +267,7 @@ .SUFFIXES:
 .el.elc:
 	$(AM_V_ELC)$(emacs) $(BYTE_COMPILE_FLAGS) -f batch-byte-compile $<
 
-.PHONY: compile-first compile-main compile compile-always
-
-compile-first: $(COMPILE_FIRST)
+.PHONY: compile-main compile compile-always
 
 # In 'compile-main' we could directly do
 #    ... | xargs $(MAKE)
@@ -336,7 +320,7 @@ semantic:
 # date.  Some .el files don't get compiled because they set the
 # local variable no-byte-compile.
 # Calling make recursively because suffix rule cannot have prerequisites.
-compile: $(LOADDEFS) autoloads compile-first
+compile: $(LOADDEFS) autoloads
 	$(MAKE) compile-main
 
 # Compile all Lisp files.  This is like 'compile' but compiles files
@@ -375,7 +359,7 @@ compile-after-backup:
 # There is no reason to use this rule unless you only have a single
 # core and CPU time is an issue.
 .PHONY: compile-one-process
-compile-one-process: $(LOADDEFS) compile-first
+compile-one-process: $(LOADDEFS)
 	$(emacs) $(BYTE_COMPILE_FLAGS) \
 	    --eval "(batch-byte-recompile-directory 0)" $(lisp)
 
diff --git a/lisp/loadup.el b/lisp/loadup.el
index 5c16464..183944b 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -245,6 +245,31 @@
 (load "progmodes/elisp-mode")
 (load "textmodes/text-mode")
 (load "textmodes/fill")
+
+;; Compile the byte compiler.  This is done to speed up the bootstrap
+;; process.  They're ordered by size, so we use the slowest-compiler
+;; on the smallest file and move to larger files as the compiler gets
+;; faster.  'autoload' comes last because it is not used by the
+;; compiler (so its compilation does not speed up subsequent
+;; compilations), it's only placed here so as to speed up generation
+;; of the loaddefs.el file.
+;;
+;; The byte compiler requires elisp-mode for parsing, and fill
+;; functions for printing warnings.
+(if (equal (member "bootstrap" command-line-args) '("bootstrap"))
+    (let (;; $HOME is not defined(!?), so (expand-file-name "~")
+          ;; crashes (called from `abbreviate-file-name').
+          (abbreviated-home-dir "/home/dir")
+          ;; dir locals needs time-date(?)
+          (enable-dir-local-variables nil))
+      (message "byte compiling the byte compiler...")
+      (setq debug-on-error t)
+      (mapc (lambda (file)
+              (setq file (locate-file file load-path '(".elc" ".el")))
+              (or (equal (substring file -4) ".elc")
+                  (byte-compile-file file t)))
+            '("macroexp" "cconv" "byte-opt" "bytecomp" "autoload"))))
+
 (load "newcomment")
 
 (load "replace")
diff --git a/lisp/progmodes/elisp-mode.el b/lisp/progmodes/elisp-mode.el
index f360791..b8d1d51 100644
--- a/lisp/progmodes/elisp-mode.el
+++ b/lisp/progmodes/elisp-mode.el
@@ -235,7 +235,7 @@ emacs-lisp-mode
               (append '((?\` . ?\') (?‘ . ?’)) electric-pair-text-pairs))
   (setq-local electric-quote-string t)
   (setq imenu-case-fold-search nil)
-  (add-function :before-until (local 'eldoc-documentation-function)
+  (add-function :before-until (local 'eldoc-documentation-functions)
                 #'elisp-el