bug#41646: Startup in Windows is very slow when load-path contains many

bug#41646: Startup in Windows is very slow when load-path contains many

[Stefan Kangas]

[I have unarchived Bug#41646 so that this discussion is archived in the
bug tracker.]

Lin Sun <sunlin7.mail@gmail.com> writes:

> On Sun, Oct 13, 2024 at 5:45 AM Eli Zaretskii <eliz@gnu.org> wrote:
>> To summarize: my point is that I think we prefer extending existing
>> libraries instead of introducing new ones with partially-overlapping
>> functionalities.  I wonder what Stefan and Andrea (CC'ed) think about
>> this.
>
> Totally understand, and thanks for inviting Stefan and Andrea in the
> conversation, to let new participants know the context quickly, I had
> post a patch to speed up Emacs for the scenario that Emacs will very
> slow on startup with hundreds packages (>300)  installed, the keypoint
> is hundreds packages will add hundreds paths into the `load-path',
> then a simple `(require X)' may trigger hundreds searching according
> to the `load-path', so the patch build a map for feature --> filepath
> from the variable `load-history' and store to disk, after that loading
> the cache and get the filepath from the cache for requiring a feature
> will significantly speed up Emacs on startup or requiring heavy
> features even there were hundreds packages. I also give the examples
> based on Spacemacs, whose startup time can be reduced from 9.703 to
> 4.175 seconds (341packages) on a Windows system, with simply add two
> lines to enable the `loadhints' from patch: (require 'loadhints)
> (loadhints-init 'startup).

Thanks, it sounds like a useful feature.

I agree with Eli that it would be better if it was not implemented as a
separate library, but integrated into the existing functionality.  For
example, I see that you use `define-advice`.  We generally frown upon
using that in core, since we could adapt our code to make it work with
the new use case.

Here's a suggestion:

Since this is about speeding up load time during startup, how about
adding a new defvar that enables a cache for `require`, and then set
that to t during startup and nil after?  That would make the feature
work as expected without any user customization.  In general, this is
preferable, as this reduces the overall complexity of Emacs, both for
users and developers.

Why should this be MS-Windows specific, BTW?  Is slow startup time with
many packages much less of an issue on other operating systems?  Is disk
access somehow slower on MS-Windows?

> From 52f79eedb1944169b6c8ac4cfde101f59345d815 Mon Sep 17 00:00:00 2001
> From: Lin Sun <sunlin7@hotmail.com>
> Date: Sat, 19 Aug 2023 06:55:59 +0000
> Subject: [PATCH] * lisp/loadhints.el: new file to cache hints for `require'
>  function
>
> ---
>  lisp/loadhints.el | 114 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 114 insertions(+)
>  create mode 100644 lisp/loadhints.el
>
> diff --git a/lisp/loadhints.el b/lisp/loadhints.el
> new file mode 100644
> index 00000000000..9befd885f0b
> --- /dev/null
> +++ b/lisp/loadhints.el
> @@ -0,0 +1,114 @@
> +;;; loadhints.el --- Give hints for `require'  -*- lexical-binding:t -*-
> +
> +;; Copyright (C) 2023-2024 Free Software Foundation, Inc.
> +
> +;; Author: Lin Sun <sunlin7@hotmail.com>
> +;; Keywords: utility
> +
> +;; This file is part of GNU Emacs.
> +
> +;; GNU Emacs is free software: you can redistribute it and/or modify
> +;; it under the terms of the GNU General Public License as published by
> +;; the Free Software Foundation, either version 3 of the License, or
> +;; (at your option) any later version.
> +
> +;; GNU Emacs is distributed in the hope that it will be useful,
> +;; but WITHOUT ANY WARRANTY; without even the implied warranty of
> +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;; GNU General Public License for more details.
> +
> +;; You should have received a copy of the GNU General Public License
> +;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
> +
> +;;; Commentary:
> +
> +;; loadhints will save the feature pathes to a cache file, and uses the
> +;; cache to speed up the `require' function, it will rapidly speed up the
> +;; `require' function when hundreds directories in `load-path' (especially
> +;; for windows). Just call `(loadhints-init 'startup)' in emacs user init
> +;; file.
> +
> +;;; Code:
> +
> +(defcustom loadhints-type (if (eq system-type 'windows-nt) 'startup)
> +  "The loadhints-type behavior.
> +A nil value means to disable this feature.
> +`startup' means to work on startup.
> +`manual' means depending on user manually update the cache.
> +`aggressive' means update the cache at emacs quit."
> +  :type '(choice (const :tag "Disable" nil)
> +                 (const :tag "Startup" startup)
> +                 (const :tag "Manual" manual)
> +                 (const :tag "Aggressive" aggressive)))
> +
> +(defcustom loadhints-cache-file
> +  (expand-file-name "loadhints-cache" user-emacs-directory)
> +  "File to save the recent list into."
> +  :version "31.0"
> +  :type 'file
> +  :initialize 'custom-initialize-default
> +  :set (lambda (symbol value)
> +         (let ((oldvalue (symbol-value symbol)))
> +           (custom-set-default symbol value)
> +           (and loadhints-type
> +                (not (equal value oldvalue))
> +                (load oldvalue t)))))
> +
> +(defvar loadhints--cache nil)
> +(defvar loadhints--modified nil)
> +
> +;;;###autoload
> +(defun loadhints-refresh-maybe (&optional force async)
> +  "(Re)generate the loadhints cache file.
> +When call with prefix, will FORCE refresh the loadhints cache."
> +  (interactive "P")
> +  (when (and force (null loadhints-type))
> +    (user-error "Loadhints not avaliable for `loadhints-type' is nil"))
> +  (when (and loadhints-type
> +             (or force
> +                 loadhints--modified
> +                 (null (locate-file loadhints-cache-file '("/")
> +                                    (get-load-suffixes)))))
> +    (let ((res (make-hash-table :test 'equal))
> +          (filepath (concat loadhints-cache-file ".el")))
> +      (cl-loop for (path . rest) in load-history
> +               do (when-let ((x (cl-find 'provide rest
> +                                         :test (lambda (a b)
> +                                                 (and (consp b)
> +                                                      (eq a (car b)))))))
> +                    (puthash (cdr x) path res)))
> +      (with-temp-file filepath
> +        (insert (format "(setq loadhints--cache %S)" res)))
> +      (if async
> +          (async-byte-compile-file filepath)
> +        (byte-compile-file filepath)))))
> +
> +;;;###autoload
> +(defun loadhints-init (&optional type)
> +  "Setup the advice for `require' and load the cached hints."
> +  (when type
> +    (setopt loadhints-type type))
> +
> +  (when loadhints-type
> +    (define-advice require (:around (fn feature &optional filename noerror))
> +      (when-let (((null filename))
> +                 ((null  (featurep feature)))
> +                 (loadhints--cache)
> +                 (path (gethash feature loadhints--cache)))
> +        (if (not (file-exists-p path))
> +            (setq loadhints--modified t)
> +          (setq filename path)))
> +      (funcall fn feature filename noerror))
> +
> +    (when-let ((filepath (locate-file loadhints-cache-file '("/")
> +                                      (get-load-suffixes))))
> +      (load filepath))
> +
> +    (cond ((eq loadhints-type 'startup)
> +           (add-hook 'after-init-hook #'(lambda ()
> +                                          (loadhints-refresh-maybe nil t))))
> +          ((eq loadhints-type 'aggressive)
> +           (add-hook 'kill-emacs-hook #'loadhints-refresh-maybe)))))
> +
> +(provide 'loadhints)
> +;;; loadhints.el ends here
> --
> 2.34.1

bug#41646: Startup in Windows is very slow when load-path contains many

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sun, 13 Oct 2024 09:50:42 +0000
> Cc: Andrea Corallo <acorallo@gnu.org>, 41646@debbugs.gnu.org, 
> 	Stefan Monnier <monnier@gnu.org>
> 
> Why should this be MS-Windows specific, BTW?  Is slow startup time with
> many packages much less of an issue on other operating systems?  Is disk
> access somehow slower on MS-Windows?

I don't think this is Windows-specific, it's just that on Windows the
gains might be higher.  Disk access is somewhat slower on Windows,
yes.

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

On Sun, Oct 13, 2024 at 10:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Stefan Kangas <stefankangas@gmail.com>
> > Date: Sun, 13 Oct 2024 09:50:42 +0000
> > Cc: Andrea Corallo <acorallo@gnu.org>, 41646@debbugs.gnu.org,
> >       Stefan Monnier <monnier@gnu.org>
> >
> > Why should this be MS-Windows specific, BTW?  Is slow startup time with
> > many packages much less of an issue on other operating systems?  Is disk
> > access somehow slower on MS-Windows?
>
> I don't think this is Windows-specific, it's just that on Windows the
> gains might be higher.  Disk access is somewhat slower on Windows,
> yes.

Thanks, and append the result on Ubuntu: from 2.132 to 1.573seconds
(383 packages).


On Sun, Oct 13, 2024 at 9:50 AM Stefan Kangas <stefankangas@gmail.com> wrote:
>
> Here's a suggestion:
>
> Since this is about speeding up load time during startup, how about
> adding a new defvar that enables a cache for `require`, and then set
> that to t during startup and nil after?  That would make the feature
> work as expected without any user customization.  In general, this is
> preferable, as this reduces the overall complexity of Emacs, both for
> users and developers.

The "loadhints-cache" is not only for startup, actually it also works
anytime when requiring a feature but the `exec-path' has hundreds of
entries.

Agree, using the "loadhints-cache" in the `require' function native
code will be better than defeine-advice, I'm implementing it now, and
will post a patch later.

bug#41646: Startup in Windows is very slow when load-path contains many

[Eli Zaretskii]

> From: Lin Sun <sunlin7.mail@gmail.com>
> Date: Sun, 13 Oct 2024 14:47:46 +0000
> Cc: Stefan Kangas <stefankangas@gmail.com>, acorallo@gnu.org, 41646@debbugs.gnu.org, 
> 	monnier@gnu.org
> 
> On Sun, Oct 13, 2024 at 10:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Stefan Kangas <stefankangas@gmail.com>
> > > Date: Sun, 13 Oct 2024 09:50:42 +0000
> > > Cc: Andrea Corallo <acorallo@gnu.org>, 41646@debbugs.gnu.org,
> > >       Stefan Monnier <monnier@gnu.org>
> > >
> > > Why should this be MS-Windows specific, BTW?  Is slow startup time with
> > > many packages much less of an issue on other operating systems?  Is disk
> > > access somehow slower on MS-Windows?
> >
> > I don't think this is Windows-specific, it's just that on Windows the
> > gains might be higher.  Disk access is somewhat slower on Windows,
> > yes.
> 
> Thanks, and append the result on Ubuntu: from 2.132 to 1.573seconds
> (383 packages).
> 
> 
> On Sun, Oct 13, 2024 at 9:50 AM Stefan Kangas <stefankangas@gmail.com> wrote:
> >
> > Here's a suggestion:
> >
> > Since this is about speeding up load time during startup, how about
> > adding a new defvar that enables a cache for `require`, and then set
> > that to t during startup and nil after?  That would make the feature
> > work as expected without any user customization.  In general, this is
> > preferable, as this reduces the overall complexity of Emacs, both for
> > users and developers.
> 
> The "loadhints-cache" is not only for startup, actually it also works
> anytime when requiring a feature but the `exec-path' has hundreds of
> entries.

Is it possible to extend filecache.el to cover this use case as well?
For example, by adding to filecache.el the ability to save the cache,
and perhaps also the ability to collect the cached files by hooking
into 'load'?

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Is it possible to extend filecache.el to cover this use case as well?
> For example, by adding to filecache.el the ability to save the cache,
> and perhaps also the ability to collect the cached files by hooking
> into 'load'?
It's doable but I'm not sure how to maintain the feature-->filepath
maps to cover both file system changes and user preferred exec-path
order, or if it could break the user experience?

bug#41646: Startup in Windows is very slow when load-path contains many

[Eli Zaretskii]

> From: Lin Sun <sunlin7.mail@gmail.com>
> Date: Sun, 13 Oct 2024 15:43:39 +0000
> Cc: stefankangas@gmail.com, acorallo@gnu.org, 41646@debbugs.gnu.org, 
> 	monnier@gnu.org
> 
> On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Is it possible to extend filecache.el to cover this use case as well?
> > For example, by adding to filecache.el the ability to save the cache,
> > and perhaps also the ability to collect the cached files by hooking
> > into 'load'?
> It's doable but I'm not sure how to maintain the feature-->filepath
> maps to cover both file system changes and user preferred exec-path
> order, or if it could break the user experience?

Sorry, I don't understand: what does exec-path have to do with this?
I thought this was about speeding up loading of Lisp files at startup?

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

On Sun, Oct 13, 2024 at 3:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lin Sun <sunlin7.mail@gmail.com>
> >
> > On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > Is it possible to extend filecache.el to cover this use case as well?
> > > For example, by adding to filecache.el the ability to save the cache,
> > > and perhaps also the ability to collect the cached files by hooking
> > > into 'load'?
> > It's doable but I'm not sure how to maintain the feature-->filepath
> > maps to cover both file system changes and user preferred exec-path
> > order, or if it could break the user experience?
>
> Sorry, I don't understand: what does exec-path have to do with this?
> I thought this was about speeding up loading of Lisp files at startup?
Sorry I mean the `load-path' (not the exec-path), my typo. Thanks

bug#41646: Startup in Windows is very slow when load-path contains many

[Eli Zaretskii]

> From: Lin Sun <sunlin7.mail@gmail.com>
> Date: Sun, 13 Oct 2024 16:03:16 +0000
> Cc: stefankangas@gmail.com, acorallo@gnu.org, 41646@debbugs.gnu.org, 
> 	monnier@gnu.org
> 
> On Sun, Oct 13, 2024 at 3:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Lin Sun <sunlin7.mail@gmail.com>
> > >
> > > On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > > >
> > > > Is it possible to extend filecache.el to cover this use case as well?
> > > > For example, by adding to filecache.el the ability to save the cache,
> > > > and perhaps also the ability to collect the cached files by hooking
> > > > into 'load'?
> > > It's doable but I'm not sure how to maintain the feature-->filepath
> > > maps to cover both file system changes and user preferred exec-path
> > > order, or if it could break the user experience?
> >
> > Sorry, I don't understand: what does exec-path have to do with this?
> > I thought this was about speeding up loading of Lisp files at startup?
> Sorry I mean the `load-path' (not the exec-path), my typo. Thanks

OK, but please elaborate on the difficulty, because I don't think I
understand it.

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

Hi Eli,

I wrote the code and tested it, the patch file attached.
In the patch, a new "load-hints" is introduced with a default value nil.
When its value is nil, everything works as before.
When its value is set as a list of hints, the openp function calling
count is reduced a lot.
I had an example for require "org", the "load-hints" can reduce the
access attempts from 123 to 6, here is the test command lines, strace
the emacs in batch mode, a single (require 'org) trigger more than
100+ open attempts; with correct load-hints, the count reduce to 6.
> strace src/emacs -batch -eval "(require 'org)" 2>&1| grep 'open.*/org\.' | wc -l
>    123
> strace src/emacs -batch -eval "(let ((load-hints '((\"org*\" \"~/tmp/emacs.debug/lisp/org/\"))))(require 'org))" 2>&1| grep 'open.*/org\.' | wc -l
>      6

And you had mentioned scan the load-path to build a full hints list,
it absolutely should work; I'm looking to change package.el to
generate the "<package>-autoloads.el" work with "load-hints",
currently the autoloads.el will add its folder into "load-path",
just change it to add its  path into "load-hints" should work, then we
do NOT need to provide a function to build the "load-hints" for the
packages installed by package.el won't bother the "load-path".

[-- Attachment #2: 0001-New-variable-load-hints-to-speedup-searching-file-fo.patch --]
[-- Type: text/x-patch, Size: 7048 bytes --]

From f194d69445c1ef0e08db29b07570ee3ed7b2ed62 Mon Sep 17 00:00:00 2001
From: Lin Sun <sunlin7@hotmail.com>
Date: Wed, 16 Oct 2024 07:31:59 +0000
Subject: [PATCH] New variable load-hints to speedup searching file for (load)
 function

* lisp/subr.el: (locate-library) support the `load-hints' variable
* src/lread.c: (load) function support the `load-hints' variable
---
 lisp/subr.el |  9 ++---
 src/lread.c  | 93 ++++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 81 insertions(+), 21 deletions(-)

diff --git a/lisp/subr.el b/lisp/subr.el
index 2eaed682406..3d9599270ed 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -3141,10 +3141,11 @@ locate-library
 string.  When run interactively, the argument INTERACTIVE-CALL is t,
 and the file name is displayed in the echo area."
   (interactive (list (read-library-name) nil nil t))
-  (let ((file (locate-file library
-			   (or path load-path)
-			   (append (unless nosuffix (get-load-suffixes))
-				   load-file-rep-suffixes))))
+  (let ((file (locate-file-internal
+               library (or path load-path)
+               (append (unless nosuffix (get-load-suffixes))
+                       load-file-rep-suffixes)
+               nil (unless path load-hints))))
     (if interactive-call
 	(if file
 	    (message "Library is file %s" (abbreviate-file-name file))
diff --git a/src/lread.c b/src/lread.c
index 95c6891c205..6de1c5be5eb 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1271,6 +1271,53 @@ close_file_unwind_android_fd (void *ptr)
 
 #endif
 
+static bool
+complete_filename_p (Lisp_Object pathname)
+{
+  const unsigned char *s = SDATA (pathname);
+  return (IS_DIRECTORY_SEP (s[0])
+	  || (SCHARS (pathname) > 2 && IS_DEVICE_SEP (s[1])
+	      && IS_DIRECTORY_SEP (s[2])));
+}
+
+/* search the file in load hints to get a path list */
+static Lisp_Object
+search_load_hints(Lisp_Object load_hints, Lisp_Object file) {
+  Lisp_Object load_path = Qnil;
+  Lisp_Object tail = load_hints;
+  FOR_EACH_TAIL_SAFE (tail)
+    {
+      bool fullmatch = false;
+      ptrdiff_t len = -1;
+      Lisp_Object row = XCAR (tail);
+      Lisp_Object key = XCAR (row);
+      CHECK_STRING (key);
+
+      if (SBYTES (key) - 1 <= SBYTES (file))
+	{
+	  if (SBYTES (key) >= 1
+	      && SDATA (key)[SBYTES (key) - 1] == '*')
+	    len = SBYTES (key) - 1; /* "file-*" format */
+	  else if (SBYTES (key) == SBYTES (file))
+	    {
+	      len = SBYTES (key);
+	      fullmatch = true;
+	    }
+	}
+
+      if (len >= 0 && 0 == memcmp (SDATA (key), SDATA (file), len))
+	{
+	  if (fullmatch)
+	    {
+	      load_path = CALLN (Fappend, XCDR (row));
+	      break;
+	    }
+	  load_path = CALLN (Fappend, load_path, XCDR (row));
+	}
+    }
+  return load_path;
+}
+
 DEFUN ("load", Fload, Sload, 1, 5, 0,
        doc: /* Execute a file of Lisp code named FILE.
 First try FILE with `.elc' appended, then try with `.el', then try
@@ -1278,7 +1325,9 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 then try FILE unmodified (the exact suffixes in the exact order are
 determined by `load-suffixes').  Environment variable references in
 FILE are replaced with their values by calling `substitute-in-file-name'.
-This function searches the directories in `load-path'.
+This function searches the entry in `load-hints` first, if some entries
+matched, searches in the matched pathes. Otherwise, searches directories
+in `load-path'.
 
 If optional second arg NOERROR is non-nil,
 report no error if FILE doesn't exist.
@@ -1327,7 +1376,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 #endif
   specpdl_ref fd_index UNINIT;
   specpdl_ref count = SPECPDL_INDEX ();
-  Lisp_Object found, efound, hist_file_name;
+  Lisp_Object found, efound, hist_file_name, load_path = Qnil;
   /* True means we printed the ".el is newer" message.  */
   bool newer = 0;
   /* True means we are loading a compiled file.  */
@@ -1409,12 +1458,18 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 	    suffixes = CALLN (Fappend, suffixes, Vload_file_rep_suffixes);
 	}
 
+      if (!complete_filename_p (file))
+	load_path = search_load_hints(Vload_hints, file);
+
+      if (NILP (load_path))
+	load_path = Vload_path;
+
 #if !defined USE_ANDROID_ASSETS
-      fd = openp (Vload_path, file, suffixes, &found, Qnil,
+      fd = openp (load_path, file, suffixes, &found, Qnil,
 		  load_prefer_newer, no_native, NULL);
 #else
       asset = NULL;
-      rc = openp (Vload_path, file, suffixes, &found, Qnil,
+      rc = openp (load_path, file, suffixes, &found, Qnil,
 		  load_prefer_newer, no_native, &asset);
       fd.fd = rc;
       fd.asset = asset;
@@ -1780,16 +1835,7 @@ save_match_data_load (Lisp_Object file, Lisp_Object noerror,
   return unbind_to (count, result);
 }
 \f
-static bool
-complete_filename_p (Lisp_Object pathname)
-{
-  const unsigned char *s = SDATA (pathname);
-  return (IS_DIRECTORY_SEP (s[0])
-	  || (SCHARS (pathname) > 2
-	      && IS_DEVICE_SEP (s[1]) && IS_DIRECTORY_SEP (s[2])));
-}
-
-DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2, 4, 0,
+DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2, 5, 0,
        doc: /* Search for FILENAME through PATH.
 Returns the file's name in absolute form, or nil if not found.
 If SUFFIXES is non-nil, it should be a list of suffixes to append to
@@ -1797,12 +1843,18 @@ DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2,
 If non-nil, PREDICATE is used instead of `file-readable-p'.
 PREDICATE can also be an integer to pass to the faccessat(2) function,
 in which case file-name-handlers are ignored.
+LOAD-HINTS is a list same as `load-hints'.
 This function will normally skip directories, so if you want it to find
 directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
-  (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate)
+  (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate,
+   Lisp_Object load_hints)
 {
-  Lisp_Object file;
-  int fd = openp (path, filename, suffixes, &file, predicate, false, true,
+  Lisp_Object file, dirs = Qnil;
+  if (!NILP(load_hints))
+    dirs = search_load_hints(load_hints, filename);
+  if (NILP(dirs))
+    dirs = path;
+  int fd = openp (dirs, filename, suffixes, &file, predicate, false, true,
 		  NULL);
   if (NILP (predicate) && fd >= 0)
     emacs_close (fd);
@@ -5851,6 +5903,13 @@ syms_of_lread (void)
 	       doc: /* Non-nil means read recursive structures using #N= and #N# syntax.  */);
   Vread_circle = Qt;
 
+  DEFVAR_LISP ("load-hints", Vload_hints,
+	       doc: /* A list for name to directory-list to search for files
+to load, before the load-path.  Eache entry is a file name to directory list,
+file name ends with a '*' means prefix matching. Example:
+  '(("name1-*" "/path1" "/path2")).  */);
+  Vload_hints = Qnil;
+
   DEFVAR_LISP ("load-path", Vload_path,
 	       doc: /* List of directories to search for files to load.
 Each element is a string (directory file name) or nil (meaning
-- 
2.34.1

bug#41646: Startup in Windows is very slow when load-path contains many

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> [I have unarchived Bug#41646 so that this discussion is archived in the
> bug tracker.]

[ Note: I haven't seen the earlier part of this discussion (but I did read the
  old bug#41646 thread).  ]

>> Totally understand, and thanks for inviting Stefan and Andrea in the
>> conversation, to let new participants know the context quickly, I had
>> post a patch to speed up Emacs for the scenario that Emacs will very
>> slow on startup with hundreds packages (>300)  installed, the keypoint
>> is hundreds packages will add hundreds paths into the `load-path',
>> then a simple `(require X)' may trigger hundreds searching according
>> to the `load-path',

Yup, that's a known problem which we've been not solving for
a long time.
The "latest" workaround is `package-quickstart`, which reduces some of
the associated pain.

If you're not using `package-quickstart` and are bothered by a slow
startup, then I'd recommend you go back and enable `package-quickstart`.

Similarly in the old bug#41646 thread, I see mentions of 40s startup
time, 8s of which are *not* spent in openp: 8s is still quite slow, so
it might be worth looking at what the startup file is doing and see if
we could do less at startup (e.g. load things more lazily).

Of course, the "long load-path" problem will still end up biting in
some cases.  Until now we've managed to make those cases rare enough
that we haven't had to actually solve it.

>> so the patch build a map for feature --> filepath from the variable
>> `load-history' and store to disk, after that loading

I don't think we want a cache that's stored on disk: it would take too
much effort to create it, load it, ensure it's not damaged if several
Emacs sessions try to write it at the same time, make sure it's fresh,
etc...

Especially since I believe that building the cache shouldn't take very
long: longer than a single "look for file F in `load-path`", maybe, but
not by much.

IOW we could keep a cache that's populated on-the-fly the first time we
`load` something, and that's then automatically refreshed when `load`
sees a new `load-path`.


        Stefan