bug#59692: 29.0.60; Reduce installation size by disabling byte-compilation for some lisp files that won't benefit

bug#59692: 29.0.60; Reduce installation size by disabling byte-compilation for some lisp files that won't benefit

[daanturo]

[-- Attachment #1: Type: text/plain, Size: 2733 bytes --]



Recently, after each rebuild, I noticed that Emacs' total installation
size has become bigger overtime. That's understandably unavoidable but
there's room for improvement. By analyzing disk usage I discovered
that:

"lisp/leim/"'s compiled elisp files are taking the most significant disk
storage.

By inspecting some of those, I don't think we are able to gain any
practical benefit by compiling them, they don't contain (~)any
function/macro definitions at all.

So I try to disable byte-compilation (by either dir-local vars or
file-local vars) in:

- lisp/leim/ : I disable for the whole directory

And some more .el(s) that I don't find so essential to compile:

- lisp/play/ : not so serious or critical to workflows

- lisp/cedet/{ede,semantic,srecode}/ : we now have LSP/eglot!
- lisp/cedet/{ede.el,semantic.el} : same as above, also a big file by
itself

(I wished pulse.el, mode-local.el, data-debug.el were moved out of this
lisp/cedet/, then putting a single .dir-local.el there would be the
simplest)

- lisp/obsolete/ : already obsolete anyway, but some packages may still
depend on them so no-native-compile only

Then build Emacs again.

The final size of my Emacs installation, compared to before applying
those no-compile local variables:
298.79 MiB -> 253.09 MiB
That's 45.7 MiB disk saved. About 15.3% of total shaved!

Please consider at least "lisp/leim/"'s gigantic files such as
"ZIRANMA.el", "ARRAY30.el", "ja-dic.el" ,etc. as I find those the most
disk-expensive when compiled, yet questionable benefit and waste of time
spending to build.


In GNU Emacs 29.0.60 (build 2, x86_64-pc-linux-gnu, GTK+ Version
3.24.35, cairo version 1.17.6) of 2022-11-30 built on dan-laptop
Repository revision: ec7908204e3d1d415a17813a1fac1a8de334d3b1
Repository branch: makepkg
System Description: Arch Linux

Configured using:
'configure --sysconfdir=/etc --prefix=/usr --libexecdir=/usr/lib
--localstatedir=/var
'--program-transform-name='\''s/\(ctags\)/\1.emacs/'\''' --with-json
--with-libsystemd --with-mailutils --with-modules
--with-native-compilation --with-pgtk --without-xaw3d --with-sound=alsa
--with-tree-sitter --with-xinput2 --with-xwidgets
--with-native-compilation=aot --without-compress-install
'CFLAGS=-march=native -O2 -pipe -fno-plt -fexceptions
-Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
-fstack-clash-protection -fcf-protection'
LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY
PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM XWIDGETS GTK3 ZLIB

-- 
Daanturo.

[-- Attachment #2: 0001-No-compile-for-some-lisp-directories-and-files.patch --]
[-- Type: text/x-patch, Size: 3882 bytes --]

From e39138e254a727cbe724760b65bafd551cadcc39 Mon Sep 17 00:00:00 2001
From: Daanturo <daanturo@gmail.com>
Date: Tue, 29 Nov 2022 23:40:19 +0700
Subject: [PATCH] No compile for some lisp directories and files

* lisp/leim/
* lisp/cedet/{ede.el,semantic.el}: produce large compilation output
* lisp/cedet/semantic/
* lisp/cedet/ede/
* lisp/cedet/srecode/
* lisp/play/
* lisp/obsolete/ : only no-native-compile
---
 lisp/cedet/ede.el                  | 4 ++++
 lisp/cedet/ede/.dir-locals.el      | 4 ++++
 lisp/cedet/semantic.el             | 4 ++++
 lisp/cedet/semantic/.dir-locals.el | 4 ++++
 lisp/cedet/srecode/.dir-locals.el  | 4 ++++
 lisp/leim/.dir-locals.el           | 4 ++++
 lisp/obsolete/.dir-locals.el       | 4 ++++
 lisp/play/.dir-locals.el           | 4 ++++
 8 files changed, 32 insertions(+)
 create mode 100644 lisp/cedet/ede/.dir-locals.el
 create mode 100644 lisp/cedet/semantic/.dir-locals.el
 create mode 100644 lisp/cedet/srecode/.dir-locals.el
 create mode 100644 lisp/leim/.dir-locals.el
 create mode 100644 lisp/obsolete/.dir-locals.el
 create mode 100644 lisp/play/.dir-locals.el

diff --git a/lisp/cedet/ede.el b/lisp/cedet/ede.el
index 1118235757..4fd8be23a9 100644
--- a/lisp/cedet/ede.el
+++ b/lisp/cedet/ede.el
@@ -1539,3 +1539,7 @@ project-root
   (ede-speedbar-file-setup))
 
 ;;; ede.el ends here
+
+;; Local Variables:
+;; no-byte-compile: t
+;; End:
diff --git a/lisp/cedet/ede/.dir-locals.el b/lisp/cedet/ede/.dir-locals.el
new file mode 100644
index 0000000000..de432a4540
--- /dev/null
+++ b/lisp/cedet/ede/.dir-locals.el
@@ -0,0 +1,4 @@
+;;; Directory Local Variables            -*- no-byte-compile: t -*-
+;;; For more information see (info "(emacs) Directory Variables")
+
+((nil . ((no-byte-compile . t))))
diff --git a/lisp/cedet/semantic.el b/lisp/cedet/semantic.el
index adb4705620..e1f78ccf55 100644
--- a/lisp/cedet/semantic.el
+++ b/lisp/cedet/semantic.el
@@ -1134,3 +1134,7 @@ semantic-mode
 ;; (require 'semantic/load)
 
 ;;; semantic.el ends here
+
+;; Local Variables:
+;; no-byte-compile: t
+;; End:
diff --git a/lisp/cedet/semantic/.dir-locals.el b/lisp/cedet/semantic/.dir-locals.el
new file mode 100644
index 0000000000..de432a4540
--- /dev/null
+++ b/lisp/cedet/semantic/.dir-locals.el
@@ -0,0 +1,4 @@
+;;; Directory Local Variables            -*- no-byte-compile: t -*-
+;;; For more information see (info "(emacs) Directory Variables")
+
+((nil . ((no-byte-compile . t))))
diff --git a/lisp/cedet/srecode/.dir-locals.el b/lisp/cedet/srecode/.dir-locals.el
new file mode 100644
index 0000000000..de432a4540
--- /dev/null
+++ b/lisp/cedet/srecode/.dir-locals.el
@@ -0,0 +1,4 @@
+;;; Directory Local Variables            -*- no-byte-compile: t -*-
+;;; For more information see (info "(emacs) Directory Variables")
+
+((nil . ((no-byte-compile . t))))
diff --git a/lisp/leim/.dir-locals.el b/lisp/leim/.dir-locals.el
new file mode 100644
index 0000000000..de432a4540
--- /dev/null
+++ b/lisp/leim/.dir-locals.el
@@ -0,0 +1,4 @@
+;;; Directory Local Variables            -*- no-byte-compile: t -*-
+;;; For more information see (info "(emacs) Directory Variables")
+
+((nil . ((no-byte-compile . t))))
diff --git a/lisp/obsolete/.dir-locals.el b/lisp/obsolete/.dir-locals.el
new file mode 100644
index 0000000000..bb7ef23866
--- /dev/null
+++ b/lisp/obsolete/.dir-locals.el
@@ -0,0 +1,4 @@
+;;; Directory Local Variables            -*- no-byte-compile: t -*-
+;;; For more information see (info "(emacs) Directory Variables")
+
+((nil . ((no-native-compile . t))))
diff --git a/lisp/play/.dir-locals.el b/lisp/play/.dir-locals.el
new file mode 100644
index 0000000000..de432a4540
--- /dev/null
+++ b/lisp/play/.dir-locals.el
@@ -0,0 +1,4 @@
+;;; Directory Local Variables            -*- no-byte-compile: t -*-
+;;; For more information see (info "(emacs) Directory Variables")
+
+((nil . ((no-byte-compile . t))))
-- 
2.38.1

bug#59692: 29.0.60; Reduce installation size by disabling byte-compilation for some lisp files that won't benefit

[Eli Zaretskii]

severity 59692 wishlist
thanks

> Date: Wed, 30 Nov 2022 03:07:47 +0700
> From: daanturo <daanturo@gmail.com>
> 
> Recently, after each rebuild, I noticed that Emacs' total installation
> size has become bigger overtime. That's understandably unavoidable but
> there's room for improvement. By analyzing disk usage I discovered
> that:
> 
> "lisp/leim/"'s compiled elisp files are taking the most significant disk
> storage.
> 
> By inspecting some of those, I don't think we are able to gain any
> practical benefit by compiling them, they don't contain (~)any
> function/macro definitions at all.
> 
> So I try to disable byte-compilation (by either dir-local vars or
> file-local vars) in:
> 
> - lisp/leim/ : I disable for the whole directory
> 
> And some more .el(s) that I don't find so essential to compile:
> 
> - lisp/play/ : not so serious or critical to workflows
> 
> - lisp/cedet/{ede,semantic,srecode}/ : we now have LSP/eglot!
> - lisp/cedet/{ede.el,semantic.el} : same as above, also a big file by
> itself
> 
> (I wished pulse.el, mode-local.el, data-debug.el were moved out of this
> lisp/cedet/, then putting a single .dir-local.el there would be the
> simplest)
> 
> - lisp/obsolete/ : already obsolete anyway, but some packages may still
> depend on them so no-native-compile only
> 
> Then build Emacs again.
> 
> The final size of my Emacs installation, compared to before applying
> those no-compile local variables:
> 298.79 MiB -> 253.09 MiB
> That's 45.7 MiB disk saved. About 15.3% of total shaved!
> 
> Please consider at least "lisp/leim/"'s gigantic files such as
> "ZIRANMA.el", "ARRAY30.el", "ja-dic.el" ,etc. as I find those the most
> disk-expensive when compiled, yet questionable benefit and waste of time
> spending to build.

Thanks.

FWIW, I consider this to be a waste of our resources and a long-term
maintenance burden that I cannot justify, even if I accept your conclusions
(and I'm not at all convinced that they are completely correct, especially
since you didn't show any measurements of code speed with and without
byte-compilation).  50 MiB of disk space is too small to justify keeping
track of all the files, making decisions which will and will not be
compiled, etc.  Disk space is cheap these days.

So I don't think we should do this.

bug#59692: 29.0.60; Reduce installation size by disabling byte-compilation for some lisp files that won't benefit

[daanturo]

[-- Attachment #1: Type: text/plain, Size: 5388 bytes --]

> measurements of code speed with and without byte-compilation

I tried to evaluate performance different:

### Load time

Helper macro:
```emacs-lisp
(defmacro my-with-deferred-gc (&rest body)
  `(dlet ((gc-cons-threshold most-positive-fixnum) (gc-cons-percentage 0.75))
     ,@body))
```

(Each of those measures was executed in a new Emacs instance, as the
second `load` is (for me) cheaper than the first `load`).

#### lisp/leim/ja-dic

(my-with-deferred-gc (benchmark-progn (load
"/usr/share/emacs/29.0.60/lisp/leim/ja-dic/ja-dic")))

> .el:
Loading /usr/share/emacs/29.0.60/lisp/leim/ja-dic/ja-dic.el (source)...done
Elapsed time: 0.707891s
> .elc:
Loading /usr/share/emacs/29.0.60/lisp/leim/ja-dic/ja-dic...done
Elapsed time: 0.104832s
> .eln:
Loading /usr/share/emacs/29.0.60/lisp/leim/ja-dic/ja-dic (native compiled elisp)...done
Elapsed time: 0.104676s


#### Other auto-generated lisp files in lisp/leim/quail, all of them are
Chinese at the moment, the largests being ZIRANMA.el, ARRAY30.el:



(my-with-deferred-gc (benchmark-progn (load
"/usr/share/emacs/29.0.60/lisp/leim/quail/ZIRANMA")))

> .el:
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ZIRANMA.el (source)...done
Elapsed time: 0.037230s
> .elc:
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ZIRANMA...done
Elapsed time: 0.058042s
> .eln:
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ZIRANMA (native compiled elisp)...
Elapsed time: 0.062904s

(my-with-deferred-gc (benchmark-progn (load
"/usr/share/emacs/29.0.60/lisp/leim/quail/ARRAY30")))

> .el
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ARRAY30.el (source)...done
Elapsed time: 0.031518s
> .elc
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ARRAY30...done
Elapsed time: 0.037412s
> .eln
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/ARRAY30 (native compiled elisp)...done
Elapsed time: 0.041110s


(my-with-deferred-gc (benchmark-progn (load
"/usr/share/emacs/29.0.60/lisp/leim/quail/QJ")))
> .el
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/QJ.el (source)...done
Elapsed time: 0.001649s
> .elc
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/QJ...done
Elapsed time: 0.004481s
> .eln
Loading /usr/share/emacs/29.0.60/lisp/leim/quail/QJ (native compiled elisp)...done
Elapsed time: 0.005001s



For "ja-dic.el", byte-compilation clearly helped speedup loading. But
native-compilation doesn't offer any more significantly. As I inspect
"ja-dic.elc", byte-compilation did by expanding macros to
`(defconst var literal-list)` forms, I think that native compilation
won't offer more about run time performance.


For auto-generated files under lisp/leim/quail (ZIRANMA, ARRAY30, QJ
tested above), sometimes compilation even slows down the loading process
compared to just interpretation.


About the run time performance, since I don't know Chinese, I will
approximately benchmark lisp/leim/quail/vnvni.el (Vietnamese) instead as
this file is quite similar to ZIRANMA.el: both has only 2 forms --
(quail-define-package ...) and (quail-define-rules ...).

ydotool is used (https://github.com/ReimuNotMoe/ydotool) to perform
auto-typing as in bench-quail.el attached. The function to be
benchmarked is #'quail-input-method.

emacs -q -l bench-quail.el --eval "(progn (my-to-new-buffer) (my-benchmark-quail nil))"
emacs -q -l bench-quail.el --eval "(progn (my-to-new-buffer) (my-benchmark-quail 'compiled))"


Benchmarking results: (note that :num means the total key pressed by ydotool).

Loading /usr/share/emacs/29.0.60/lisp/leim/quail/vnvni.el (source)...done
Elapsed time: 0.016844s (0.008795s in 1 GCs)
(:compiled nil (:num 145 :total-time 2.118776153 :average 0.014612249331034484))


Loading /usr/share/emacs/29.0.60/lisp/leim/quail/vnvni (native compiled elisp)...done
Elapsed time: 0.034253s (0.009630s in 1 GCs)
(:compiled compiled (:num 145 :total-time 2.117245182999999 :average 0.014601690917241374))



So I conclude that those quail lisp package files' compilation statuses,
whether compiled or not, doesn't affect run-time performance at all.

### lisp/international/titdic-cnv.el's pinyin-convert

It produce a single form: (defconst var literal-list) so the
corresponding produced ".elc" file is just the same with new lines
joined. Also the times taken to load lisp/language/pinyin.elc and
/lisp/language/pinyin.el aren't different anyway.

### About the patch

In the patch below, I disabled native-compile for "ja-dic" while keeping
byte-compile (by modifying `generate-lisp-file-trailer` keyword args),
and disable compilation altogether for files produced by titdic-convert,
miscdic-convert, pinyin-convert. There's no new .dir-locals files
introduced like the previous patch.

> Disk space is cheap these days.

But it's always better to be light if we don't suffer any penalties. A
little number of megabytes saved will always be appreciated by people
with metered connection and/or limited bandwidth, or people who have
multiple versions of Emacs installed. Disk space is indeed becoming
cheaper, but with varying pace among countries. I am currently using a
laptop which only has 238GiB & un-expandable, that explains my joy when
pacman shows net upgrade size being negative during system updates.

Thanks to the patch, I saved 20.20 MiB by a couple of flag flips without
making any new files. Please consider again if you think it makes more
sense than my previous justifications, thank you.

-- 
Daanturo.

[-- Attachment #2: bench-quail.el --]
[-- Type: text/x-emacs-lisp, Size: 2042 bytes --]

;; -*- lexical-binding: t; -*-

;;; Benchmark helper functions

(defvar my-benchmark-accumulate-time--num-time '(0 0))
;;;###autoload
(defun my-benchmark-accumulate-time-a (func &rest args)
  (let* ((time0 (current-time)))
    (unwind-protect
        (apply func args)
      (pcase-let* ((delta (float-time (time-since time0)))
                   (`(,num ,total-time) my-benchmark-accumulate-time--num-time))
        (setq my-benchmark-accumulate-time--num-time
              (list (+ (or num 0) 1)
                    (+ (or total-time 0) delta)))))))
(defun my-benchmark-accumulate-time-report (&optional reset)
  (prog1
      (pcase-let* ((`(,num ,total-time) my-benchmark-accumulate-time--num-time))
        (list :num num
              :total-time total-time
              :average (ignore-error arith-error (/ total-time num))))
    (when reset (setq my-benchmark-accumulate-time--num-time nil))))

(defun my-to-new-buffer ()
  (interactive)
  (switch-to-buffer
   (generate-new-buffer "new")))

;;; Perform benchmarking

(defconst my-typing
  "Emacs la2 tri2nh soa5n tha3o va8n ba3n d9a chu71c na8ng. D9a6y
la2 pha62n me62m tu75 do, cha5y d9u7o75c tre6n nhie62u he65
d9ie62u ha2nh va2 co1 the63 mo73 ro65ng d9e63 the6m va2o chu71c
na8ng mo71i."
  "https://vi.wikipedia.org/wiki/Emacs")

(defun my-benchmark-quail (&optional load-compiled-quail-package)
  (interactive "P")
  (deactivate-input-method)
  (advice-add #'quail-input-method :around #'my-benchmark-accumulate-time-a)
  (my-benchmark-accumulate-time-report 'reset)
  (benchmark-progn
    (if load-compiled-quail-package
        (load "/usr/share/emacs/29.0.60/lisp/leim/quail/vnvni")
      (load "/usr/share/emacs/29.0.60/lisp/leim/quail/vnvni.el")))
  (activate-input-method "vietnamese-vni")
  (let* ((proc (start-process "" nil "ydotool" "type" my-typing)))
    (set-process-sentinel
     proc
     (lambda (&rest _)
       (deactivate-input-method)
       (print (list :compiled load-compiled-quail-package
                    (my-benchmark-accumulate-time-report)))))))

[-- Attachment #3: 0001-Inhibit-compilation-for-some-quail-packages-generators.patch --]
[-- Type: text/x-patch, Size: 2161 bytes --]

From 1ccf494d4175f8c56ef95a0661249f42337fae77 Mon Sep 17 00:00:00 2001
From: Daanturo <daanturo@gmail.com>
Date: Wed, 30 Nov 2022 16:03:59 +0700
Subject: [PATCH] Inhibit compilation for some quail packages generators

---
 lisp/international/ja-dic-cnv.el | 3 ++-
 lisp/international/titdic-cnv.el | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/lisp/international/ja-dic-cnv.el b/lisp/international/ja-dic-cnv.el
index ec68d8c804..df70959b19 100644
--- a/lisp/international/ja-dic-cnv.el
+++ b/lisp/international/ja-dic-cnv.el
@@ -396,7 +396,8 @@ skkdic-convert
 	;; Postfix
 	(with-current-buffer buf
 	  (goto-char (point-max))
-          (generate-lisp-file-trailer ja-dic-filename :compile t)))
+          (generate-lisp-file-trailer
+           ja-dic-filename :compile t :inhibit-native-compile t)))
 
       ;; Save the working buffer.
       (set-buffer buf)
diff --git a/lisp/international/titdic-cnv.el b/lisp/international/titdic-cnv.el
index b942f5fabc..8996214dcb 100644
--- a/lisp/international/titdic-cnv.el
+++ b/lisp/international/titdic-cnv.el
@@ -527,7 +527,7 @@ titdic-convert
 	  ;; Process the body part
 	  (tit-process-body)
           (generate-lisp-file-trailer
-           filename :inhibit-provide t :compile t :coding nil))))))
+           filename :inhibit-provide t :compile nil :coding nil))))))
 
 ;;;###autoload
 (defun batch-titdic-convert (&optional force)
@@ -1161,7 +1161,7 @@ miscdic-convert
                 (with-current-buffer dstbuf
                   (funcall converter dicbuf)))))
           (generate-lisp-file-trailer
-           quailfile :inhibit-provide t :compile t :coding nil)))
+           quailfile :inhibit-provide t :compile nil :coding nil)))
       (setq tail (cdr tail)))))
 
 (defun batch-miscdic-convert ()
@@ -1218,7 +1218,7 @@ pinyin-convert
           (forward-line 1)))
       (insert ")\n\"An alist holding correspondences between pinyin syllables\
  and\nChinese characters.\")\n\n")
-      (generate-lisp-file-trailer dst-file :compile t))
+      (generate-lisp-file-trailer dst-file :compile nil))
     (kill-emacs 0)))
 
 ;;; titdic-cnv.el ends here
-- 
2.38.1

bug#59692: 29.0.60; Reduce installation size by disabling byte-compilation for some lisp files that won't benefit

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> FWIW, I consider this to be a waste of our resources and a long-term
> maintenance burden that I cannot justify, even if I accept your conclusions
> (and I'm not at all convinced that they are completely correct, especially
> since you didn't show any measurements of code speed with and without
> byte-compilation).  50 MiB of disk space is too small to justify keeping
> track of all the files, making decisions which will and will not be
> compiled, etc.  Disk space is cheap these days.
>
> So I don't think we should do this.

I agree, so I'm closing this as wontfix.

bug#59692: 29.0.60; Reduce installation size by disabling byte-compilation for some lisp files that won't benefit

[daanturo]

Please take a glance at my second attempt:
https://lists.gnu.org/archive/html/bug-gnu-emacs/2022-11/msg02818.html

I did proper benchmark, also the patch only changed a couple of lines without
creating any new files.

"""
lisp/international/ja-dic-cnv.el | 3 ++-
lisp/international/titdic-cnv.el | 6 +++---
2 files changed, 5 insertions(+), 4 deletions(-)
"""


On 13/12/2022 08:04, Stefan Kangas wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>> ...you didn't show any measurements of code speed with and without
>> byte-compilation)...
>> ...keeping
>> track of all the files, making decisions which will and will not be
>> compiled, etc. ...
> I agree, so I'm closing this as wontfix.

-- 
Daanturo.