bug#41646: Startup in Windows is very slow when load-path contains many

[Stefan Kangas <stefankangas@gmail.com>]

[I have unarchived Bug#41646 so that this discussion is archived in the
bug tracker.]

Lin Sun <sunlin7.mail@gmail.com> writes:

> On Sun, Oct 13, 2024 at 5:45 AM Eli Zaretskii <eliz@gnu.org> wrote:
>> To summarize: my point is that I think we prefer extending existing
>> libraries instead of introducing new ones with partially-overlapping
>> functionalities.  I wonder what Stefan and Andrea (CC'ed) think about
>> this.
>
> Totally understand, and thanks for inviting Stefan and Andrea in the
> conversation, to let new participants know the context quickly, I had
> post a patch to speed up Emacs for the scenario that Emacs will very
> slow on startup with hundreds packages (>300)  installed, the keypoint
> is hundreds packages will add hundreds paths into the `load-path',
> then a simple `(require X)' may trigger hundreds searching according
> to the `load-path', so the patch build a map for feature --> filepath
> from the variable `load-history' and store to disk, after that loading
> the cache and get the filepath from the cache for requiring a feature
> will significantly speed up Emacs on startup or requiring heavy
> features even there were hundreds packages. I also give the examples
> based on Spacemacs, whose startup time can be reduced from 9.703 to
> 4.175 seconds (341packages) on a Windows system, with simply add two
> lines to enable the `loadhints' from patch: (require 'loadhints)
> (loadhints-init 'startup).

Thanks, it sounds like a useful feature.

I agree with Eli that it would be better if it was not implemented as a
separate library, but integrated into the existing functionality.  For
example, I see that you use `define-advice`.  We generally frown upon
using that in core, since we could adapt our code to make it work with
the new use case.

Here's a suggestion:

Since this is about speeding up load time during startup, how about
adding a new defvar that enables a cache for `require`, and then set
that to t during startup and nil after?  That would make the feature
work as expected without any user customization.  In general, this is
preferable, as this reduces the overall complexity of Emacs, both for
users and developers.

Why should this be MS-Windows specific, BTW?  Is slow startup time with
many packages much less of an issue on other operating systems?  Is disk
access somehow slower on MS-Windows?

> From 52f79eedb1944169b6c8ac4cfde101f59345d815 Mon Sep 17 00:00:00 2001
> From: Lin Sun <sunlin7@hotmail.com>
> Date: Sat, 19 Aug 2023 06:55:59 +0000
> Subject: [PATCH] * lisp/loadhints.el: new file to cache hints for `require'
>  function
>
> ---
>  lisp/loadhints.el | 114 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 114 insertions(+)
>  create mode 100644 lisp/loadhints.el
>
> diff --git a/lisp/loadhints.el b/lisp/loadhints.el
> new file mode 100644
> index 00000000000..9befd885f0b
> --- /dev/null
> +++ b/lisp/loadhints.el
> @@ -0,0 +1,114 @@
> +;;; loadhints.el --- Give hints for `require'  -*- lexical-binding:t -*-
> +
> +;; Copyright (C) 2023-2024 Free Software Foundation, Inc.
> +
> +;; Author: Lin Sun <sunlin7@hotmail.com>
> +;; Keywords: utility
> +
> +;; This file is part of GNU Emacs.
> +
> +;; GNU Emacs is free software: you can redistribute it and/or modify
> +;; it under the terms of the GNU General Public License as published by
> +;; the Free Software Foundation, either version 3 of the License, or
> +;; (at your option) any later version.
> +
> +;; GNU Emacs is distributed in the hope that it will be useful,
> +;; but WITHOUT ANY WARRANTY; without even the implied warranty of
> +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;; GNU General Public License for more details.
> +
> +;; You should have received a copy of the GNU General Public License
> +;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
> +
> +;;; Commentary:
> +
> +;; loadhints will save the feature pathes to a cache file, and uses the
> +;; cache to speed up the `require' function, it will rapidly speed up the
> +;; `require' function when hundreds directories in `load-path' (especially
> +;; for windows). Just call `(loadhints-init 'startup)' in emacs user init
> +;; file.
> +
> +;;; Code:
> +
> +(defcustom loadhints-type (if (eq system-type 'windows-nt) 'startup)
> +  "The loadhints-type behavior.
> +A nil value means to disable this feature.
> +`startup' means to work on startup.
> +`manual' means depending on user manually update the cache.
> +`aggressive' means update the cache at emacs quit."
> +  :type '(choice (const :tag "Disable" nil)
> +                 (const :tag "Startup" startup)
> +                 (const :tag "Manual" manual)
> +                 (const :tag "Aggressive" aggressive)))
> +
> +(defcustom loadhints-cache-file
> +  (expand-file-name "loadhints-cache" user-emacs-directory)
> +  "File to save the recent list into."
> +  :version "31.0"
> +  :type 'file
> +  :initialize 'custom-initialize-default
> +  :set (lambda (symbol value)
> +         (let ((oldvalue (symbol-value symbol)))
> +           (custom-set-default symbol value)
> +           (and loadhints-type
> +                (not (equal value oldvalue))
> +                (load oldvalue t)))))
> +
> +(defvar loadhints--cache nil)
> +(defvar loadhints--modified nil)
> +
> +;;;###autoload
> +(defun loadhints-refresh-maybe (&optional force async)
> +  "(Re)generate the loadhints cache file.
> +When call with prefix, will FORCE refresh the loadhints cache."
> +  (interactive "P")
> +  (when (and force (null loadhints-type))
> +    (user-error "Loadhints not avaliable for `loadhints-type' is nil"))
> +  (when (and loadhints-type
> +             (or force
> +                 loadhints--modified
> +                 (null (locate-file loadhints-cache-file '("/")
> +                                    (get-load-suffixes)))))
> +    (let ((res (make-hash-table :test 'equal))
> +          (filepath (concat loadhints-cache-file ".el")))
> +      (cl-loop for (path . rest) in load-history
> +               do (when-let ((x (cl-find 'provide rest
> +                                         :test (lambda (a b)
> +                                                 (and (consp b)
> +                                                      (eq a (car b)))))))
> +                    (puthash (cdr x) path res)))
> +      (with-temp-file filepath
> +        (insert (format "(setq loadhints--cache %S)" res)))
> +      (if async
> +          (async-byte-compile-file filepath)
> +        (byte-compile-file filepath)))))
> +
> +;;;###autoload
> +(defun loadhints-init (&optional type)
> +  "Setup the advice for `require' and load the cached hints."
> +  (when type
> +    (setopt loadhints-type type))
> +
> +  (when loadhints-type
> +    (define-advice require (:around (fn feature &optional filename noerror))
> +      (when-let (((null filename))
> +                 ((null  (featurep feature)))
> +                 (loadhints--cache)
> +                 (path (gethash feature loadhints--cache)))
> +        (if (not (file-exists-p path))
> +            (setq loadhints--modified t)
> +          (setq filename path)))
> +      (funcall fn feature filename noerror))
> +
> +    (when-let ((filepath (locate-file loadhints-cache-file '("/")
> +                                      (get-load-suffixes))))
> +      (load filepath))
> +
> +    (cond ((eq loadhints-type 'startup)
> +           (add-hook 'after-init-hook #'(lambda ()
> +                                          (loadhints-refresh-maybe nil t))))
> +          ((eq loadhints-type 'aggressive)
> +           (add-hook 'kill-emacs-hook #'loadhints-refresh-maybe)))))
> +
> +(provide 'loadhints)
> +;;; loadhints.el ends here
> --
> 2.34.1