bug#41646: Startup in Windows is very slow when load-path contains many

bug#41646: Startup in Windows is very slow when load-path contains many

[Stefan Kangas]

[I have unarchived Bug#41646 so that this discussion is archived in the
bug tracker.]

Lin Sun <sunlin7.mail@gmail.com> writes:

> On Sun, Oct 13, 2024 at 5:45 AM Eli Zaretskii <eliz@gnu.org> wrote:
>> To summarize: my point is that I think we prefer extending existing
>> libraries instead of introducing new ones with partially-overlapping
>> functionalities.  I wonder what Stefan and Andrea (CC'ed) think about
>> this.
>
> Totally understand, and thanks for inviting Stefan and Andrea in the
> conversation, to let new participants know the context quickly, I had
> post a patch to speed up Emacs for the scenario that Emacs will very
> slow on startup with hundreds packages (>300)  installed, the keypoint
> is hundreds packages will add hundreds paths into the `load-path',
> then a simple `(require X)' may trigger hundreds searching according
> to the `load-path', so the patch build a map for feature --> filepath
> from the variable `load-history' and store to disk, after that loading
> the cache and get the filepath from the cache for requiring a feature
> will significantly speed up Emacs on startup or requiring heavy
> features even there were hundreds packages. I also give the examples
> based on Spacemacs, whose startup time can be reduced from 9.703 to
> 4.175 seconds (341packages) on a Windows system, with simply add two
> lines to enable the `loadhints' from patch: (require 'loadhints)
> (loadhints-init 'startup).

Thanks, it sounds like a useful feature.

I agree with Eli that it would be better if it was not implemented as a
separate library, but integrated into the existing functionality.  For
example, I see that you use `define-advice`.  We generally frown upon
using that in core, since we could adapt our code to make it work with
the new use case.

Here's a suggestion:

Since this is about speeding up load time during startup, how about
adding a new defvar that enables a cache for `require`, and then set
that to t during startup and nil after?  That would make the feature
work as expected without any user customization.  In general, this is
preferable, as this reduces the overall complexity of Emacs, both for
users and developers.

Why should this be MS-Windows specific, BTW?  Is slow startup time with
many packages much less of an issue on other operating systems?  Is disk
access somehow slower on MS-Windows?

> From 52f79eedb1944169b6c8ac4cfde101f59345d815 Mon Sep 17 00:00:00 2001
> From: Lin Sun <sunlin7@hotmail.com>
> Date: Sat, 19 Aug 2023 06:55:59 +0000
> Subject: [PATCH] * lisp/loadhints.el: new file to cache hints for `require'
>  function
>
> ---
>  lisp/loadhints.el | 114 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 114 insertions(+)
>  create mode 100644 lisp/loadhints.el
>
> diff --git a/lisp/loadhints.el b/lisp/loadhints.el
> new file mode 100644
> index 00000000000..9befd885f0b
> --- /dev/null
> +++ b/lisp/loadhints.el
> @@ -0,0 +1,114 @@
> +;;; loadhints.el --- Give hints for `require'  -*- lexical-binding:t -*-
> +
> +;; Copyright (C) 2023-2024 Free Software Foundation, Inc.
> +
> +;; Author: Lin Sun <sunlin7@hotmail.com>
> +;; Keywords: utility
> +
> +;; This file is part of GNU Emacs.
> +
> +;; GNU Emacs is free software: you can redistribute it and/or modify
> +;; it under the terms of the GNU General Public License as published by
> +;; the Free Software Foundation, either version 3 of the License, or
> +;; (at your option) any later version.
> +
> +;; GNU Emacs is distributed in the hope that it will be useful,
> +;; but WITHOUT ANY WARRANTY; without even the implied warranty of
> +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;; GNU General Public License for more details.
> +
> +;; You should have received a copy of the GNU General Public License
> +;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
> +
> +;;; Commentary:
> +
> +;; loadhints will save the feature pathes to a cache file, and uses the
> +;; cache to speed up the `require' function, it will rapidly speed up the
> +;; `require' function when hundreds directories in `load-path' (especially
> +;; for windows). Just call `(loadhints-init 'startup)' in emacs user init
> +;; file.
> +
> +;;; Code:
> +
> +(defcustom loadhints-type (if (eq system-type 'windows-nt) 'startup)
> +  "The loadhints-type behavior.
> +A nil value means to disable this feature.
> +`startup' means to work on startup.
> +`manual' means depending on user manually update the cache.
> +`aggressive' means update the cache at emacs quit."
> +  :type '(choice (const :tag "Disable" nil)
> +                 (const :tag "Startup" startup)
> +                 (const :tag "Manual" manual)
> +                 (const :tag "Aggressive" aggressive)))
> +
> +(defcustom loadhints-cache-file
> +  (expand-file-name "loadhints-cache" user-emacs-directory)
> +  "File to save the recent list into."
> +  :version "31.0"
> +  :type 'file
> +  :initialize 'custom-initialize-default
> +  :set (lambda (symbol value)
> +         (let ((oldvalue (symbol-value symbol)))
> +           (custom-set-default symbol value)
> +           (and loadhints-type
> +                (not (equal value oldvalue))
> +                (load oldvalue t)))))
> +
> +(defvar loadhints--cache nil)
> +(defvar loadhints--modified nil)
> +
> +;;;###autoload
> +(defun loadhints-refresh-maybe (&optional force async)
> +  "(Re)generate the loadhints cache file.
> +When call with prefix, will FORCE refresh the loadhints cache."
> +  (interactive "P")
> +  (when (and force (null loadhints-type))
> +    (user-error "Loadhints not avaliable for `loadhints-type' is nil"))
> +  (when (and loadhints-type
> +             (or force
> +                 loadhints--modified
> +                 (null (locate-file loadhints-cache-file '("/")
> +                                    (get-load-suffixes)))))
> +    (let ((res (make-hash-table :test 'equal))
> +          (filepath (concat loadhints-cache-file ".el")))
> +      (cl-loop for (path . rest) in load-history
> +               do (when-let ((x (cl-find 'provide rest
> +                                         :test (lambda (a b)
> +                                                 (and (consp b)
> +                                                      (eq a (car b)))))))
> +                    (puthash (cdr x) path res)))
> +      (with-temp-file filepath
> +        (insert (format "(setq loadhints--cache %S)" res)))
> +      (if async
> +          (async-byte-compile-file filepath)
> +        (byte-compile-file filepath)))))
> +
> +;;;###autoload
> +(defun loadhints-init (&optional type)
> +  "Setup the advice for `require' and load the cached hints."
> +  (when type
> +    (setopt loadhints-type type))
> +
> +  (when loadhints-type
> +    (define-advice require (:around (fn feature &optional filename noerror))
> +      (when-let (((null filename))
> +                 ((null  (featurep feature)))
> +                 (loadhints--cache)
> +                 (path (gethash feature loadhints--cache)))
> +        (if (not (file-exists-p path))
> +            (setq loadhints--modified t)
> +          (setq filename path)))
> +      (funcall fn feature filename noerror))
> +
> +    (when-let ((filepath (locate-file loadhints-cache-file '("/")
> +                                      (get-load-suffixes))))
> +      (load filepath))
> +
> +    (cond ((eq loadhints-type 'startup)
> +           (add-hook 'after-init-hook #'(lambda ()
> +                                          (loadhints-refresh-maybe nil t))))
> +          ((eq loadhints-type 'aggressive)
> +           (add-hook 'kill-emacs-hook #'loadhints-refresh-maybe)))))
> +
> +(provide 'loadhints)
> +;;; loadhints.el ends here
> --
> 2.34.1

bug#41646: Startup in Windows is very slow when load-path contains many

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sun, 13 Oct 2024 09:50:42 +0000
> Cc: Andrea Corallo <acorallo@gnu.org>, 41646@debbugs.gnu.org, 
> 	Stefan Monnier <monnier@gnu.org>
> 
> Why should this be MS-Windows specific, BTW?  Is slow startup time with
> many packages much less of an issue on other operating systems?  Is disk
> access somehow slower on MS-Windows?

I don't think this is Windows-specific, it's just that on Windows the
gains might be higher.  Disk access is somewhat slower on Windows,
yes.

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

On Sun, Oct 13, 2024 at 10:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Stefan Kangas <stefankangas@gmail.com>
> > Date: Sun, 13 Oct 2024 09:50:42 +0000
> > Cc: Andrea Corallo <acorallo@gnu.org>, 41646@debbugs.gnu.org,
> >       Stefan Monnier <monnier@gnu.org>
> >
> > Why should this be MS-Windows specific, BTW?  Is slow startup time with
> > many packages much less of an issue on other operating systems?  Is disk
> > access somehow slower on MS-Windows?
>
> I don't think this is Windows-specific, it's just that on Windows the
> gains might be higher.  Disk access is somewhat slower on Windows,
> yes.

Thanks, and append the result on Ubuntu: from 2.132 to 1.573seconds
(383 packages).


On Sun, Oct 13, 2024 at 9:50 AM Stefan Kangas <stefankangas@gmail.com> wrote:
>
> Here's a suggestion:
>
> Since this is about speeding up load time during startup, how about
> adding a new defvar that enables a cache for `require`, and then set
> that to t during startup and nil after?  That would make the feature
> work as expected without any user customization.  In general, this is
> preferable, as this reduces the overall complexity of Emacs, both for
> users and developers.

The "loadhints-cache" is not only for startup, actually it also works
anytime when requiring a feature but the `exec-path' has hundreds of
entries.

Agree, using the "loadhints-cache" in the `require' function native
code will be better than defeine-advice, I'm implementing it now, and
will post a patch later.

bug#41646: Startup in Windows is very slow when load-path contains many

[Eli Zaretskii]

> From: Lin Sun <sunlin7.mail@gmail.com>
> Date: Sun, 13 Oct 2024 14:47:46 +0000
> Cc: Stefan Kangas <stefankangas@gmail.com>, acorallo@gnu.org, 41646@debbugs.gnu.org, 
> 	monnier@gnu.org
> 
> On Sun, Oct 13, 2024 at 10:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Stefan Kangas <stefankangas@gmail.com>
> > > Date: Sun, 13 Oct 2024 09:50:42 +0000
> > > Cc: Andrea Corallo <acorallo@gnu.org>, 41646@debbugs.gnu.org,
> > >       Stefan Monnier <monnier@gnu.org>
> > >
> > > Why should this be MS-Windows specific, BTW?  Is slow startup time with
> > > many packages much less of an issue on other operating systems?  Is disk
> > > access somehow slower on MS-Windows?
> >
> > I don't think this is Windows-specific, it's just that on Windows the
> > gains might be higher.  Disk access is somewhat slower on Windows,
> > yes.
> 
> Thanks, and append the result on Ubuntu: from 2.132 to 1.573seconds
> (383 packages).
> 
> 
> On Sun, Oct 13, 2024 at 9:50 AM Stefan Kangas <stefankangas@gmail.com> wrote:
> >
> > Here's a suggestion:
> >
> > Since this is about speeding up load time during startup, how about
> > adding a new defvar that enables a cache for `require`, and then set
> > that to t during startup and nil after?  That would make the feature
> > work as expected without any user customization.  In general, this is
> > preferable, as this reduces the overall complexity of Emacs, both for
> > users and developers.
> 
> The "loadhints-cache" is not only for startup, actually it also works
> anytime when requiring a feature but the `exec-path' has hundreds of
> entries.

Is it possible to extend filecache.el to cover this use case as well?
For example, by adding to filecache.el the ability to save the cache,
and perhaps also the ability to collect the cached files by hooking
into 'load'?

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Is it possible to extend filecache.el to cover this use case as well?
> For example, by adding to filecache.el the ability to save the cache,
> and perhaps also the ability to collect the cached files by hooking
> into 'load'?
It's doable but I'm not sure how to maintain the feature-->filepath
maps to cover both file system changes and user preferred exec-path
order, or if it could break the user experience?

bug#41646: Startup in Windows is very slow when load-path contains many

[Eli Zaretskii]

> From: Lin Sun <sunlin7.mail@gmail.com>
> Date: Sun, 13 Oct 2024 15:43:39 +0000
> Cc: stefankangas@gmail.com, acorallo@gnu.org, 41646@debbugs.gnu.org, 
> 	monnier@gnu.org
> 
> On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Is it possible to extend filecache.el to cover this use case as well?
> > For example, by adding to filecache.el the ability to save the cache,
> > and perhaps also the ability to collect the cached files by hooking
> > into 'load'?
> It's doable but I'm not sure how to maintain the feature-->filepath
> maps to cover both file system changes and user preferred exec-path
> order, or if it could break the user experience?

Sorry, I don't understand: what does exec-path have to do with this?
I thought this was about speeding up loading of Lisp files at startup?

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

On Sun, Oct 13, 2024 at 3:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lin Sun <sunlin7.mail@gmail.com>
> >
> > On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > Is it possible to extend filecache.el to cover this use case as well?
> > > For example, by adding to filecache.el the ability to save the cache,
> > > and perhaps also the ability to collect the cached files by hooking
> > > into 'load'?
> > It's doable but I'm not sure how to maintain the feature-->filepath
> > maps to cover both file system changes and user preferred exec-path
> > order, or if it could break the user experience?
>
> Sorry, I don't understand: what does exec-path have to do with this?
> I thought this was about speeding up loading of Lisp files at startup?
Sorry I mean the `load-path' (not the exec-path), my typo. Thanks

bug#41646: Startup in Windows is very slow when load-path contains many

[Eli Zaretskii]

> From: Lin Sun <sunlin7.mail@gmail.com>
> Date: Sun, 13 Oct 2024 16:03:16 +0000
> Cc: stefankangas@gmail.com, acorallo@gnu.org, 41646@debbugs.gnu.org, 
> 	monnier@gnu.org
> 
> On Sun, Oct 13, 2024 at 3:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: Lin Sun <sunlin7.mail@gmail.com>
> > >
> > > On Sun, Oct 13, 2024 at 3:24 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > > >
> > > > Is it possible to extend filecache.el to cover this use case as well?
> > > > For example, by adding to filecache.el the ability to save the cache,
> > > > and perhaps also the ability to collect the cached files by hooking
> > > > into 'load'?
> > > It's doable but I'm not sure how to maintain the feature-->filepath
> > > maps to cover both file system changes and user preferred exec-path
> > > order, or if it could break the user experience?
> >
> > Sorry, I don't understand: what does exec-path have to do with this?
> > I thought this was about speeding up loading of Lisp files at startup?
> Sorry I mean the `load-path' (not the exec-path), my typo. Thanks

OK, but please elaborate on the difficulty, because I don't think I
understand it.

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

Hi Eli,

I wrote the code and tested it, the patch file attached.
In the patch, a new "load-hints" is introduced with a default value nil.
When its value is nil, everything works as before.
When its value is set as a list of hints, the openp function calling
count is reduced a lot.
I had an example for require "org", the "load-hints" can reduce the
access attempts from 123 to 6, here is the test command lines, strace
the emacs in batch mode, a single (require 'org) trigger more than
100+ open attempts; with correct load-hints, the count reduce to 6.
> strace src/emacs -batch -eval "(require 'org)" 2>&1| grep 'open.*/org\.' | wc -l
>    123
> strace src/emacs -batch -eval "(let ((load-hints '((\"org*\" \"~/tmp/emacs.debug/lisp/org/\"))))(require 'org))" 2>&1| grep 'open.*/org\.' | wc -l
>      6

And you had mentioned scan the load-path to build a full hints list,
it absolutely should work; I'm looking to change package.el to
generate the "<package>-autoloads.el" work with "load-hints",
currently the autoloads.el will add its folder into "load-path",
just change it to add its  path into "load-hints" should work, then we
do NOT need to provide a function to build the "load-hints" for the
packages installed by package.el won't bother the "load-path".

[-- Attachment #2: 0001-New-variable-load-hints-to-speedup-searching-file-fo.patch --]
[-- Type: text/x-patch, Size: 7048 bytes --]

From f194d69445c1ef0e08db29b07570ee3ed7b2ed62 Mon Sep 17 00:00:00 2001
From: Lin Sun <sunlin7@hotmail.com>
Date: Wed, 16 Oct 2024 07:31:59 +0000
Subject: [PATCH] New variable load-hints to speedup searching file for (load)
 function

* lisp/subr.el: (locate-library) support the `load-hints' variable
* src/lread.c: (load) function support the `load-hints' variable
---
 lisp/subr.el |  9 ++---
 src/lread.c  | 93 ++++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 81 insertions(+), 21 deletions(-)

diff --git a/lisp/subr.el b/lisp/subr.el
index 2eaed682406..3d9599270ed 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -3141,10 +3141,11 @@ locate-library
 string.  When run interactively, the argument INTERACTIVE-CALL is t,
 and the file name is displayed in the echo area."
   (interactive (list (read-library-name) nil nil t))
-  (let ((file (locate-file library
-			   (or path load-path)
-			   (append (unless nosuffix (get-load-suffixes))
-				   load-file-rep-suffixes))))
+  (let ((file (locate-file-internal
+               library (or path load-path)
+               (append (unless nosuffix (get-load-suffixes))
+                       load-file-rep-suffixes)
+               nil (unless path load-hints))))
     (if interactive-call
 	(if file
 	    (message "Library is file %s" (abbreviate-file-name file))
diff --git a/src/lread.c b/src/lread.c
index 95c6891c205..6de1c5be5eb 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1271,6 +1271,53 @@ close_file_unwind_android_fd (void *ptr)
 
 #endif
 
+static bool
+complete_filename_p (Lisp_Object pathname)
+{
+  const unsigned char *s = SDATA (pathname);
+  return (IS_DIRECTORY_SEP (s[0])
+	  || (SCHARS (pathname) > 2 && IS_DEVICE_SEP (s[1])
+	      && IS_DIRECTORY_SEP (s[2])));
+}
+
+/* search the file in load hints to get a path list */
+static Lisp_Object
+search_load_hints(Lisp_Object load_hints, Lisp_Object file) {
+  Lisp_Object load_path = Qnil;
+  Lisp_Object tail = load_hints;
+  FOR_EACH_TAIL_SAFE (tail)
+    {
+      bool fullmatch = false;
+      ptrdiff_t len = -1;
+      Lisp_Object row = XCAR (tail);
+      Lisp_Object key = XCAR (row);
+      CHECK_STRING (key);
+
+      if (SBYTES (key) - 1 <= SBYTES (file))
+	{
+	  if (SBYTES (key) >= 1
+	      && SDATA (key)[SBYTES (key) - 1] == '*')
+	    len = SBYTES (key) - 1; /* "file-*" format */
+	  else if (SBYTES (key) == SBYTES (file))
+	    {
+	      len = SBYTES (key);
+	      fullmatch = true;
+	    }
+	}
+
+      if (len >= 0 && 0 == memcmp (SDATA (key), SDATA (file), len))
+	{
+	  if (fullmatch)
+	    {
+	      load_path = CALLN (Fappend, XCDR (row));
+	      break;
+	    }
+	  load_path = CALLN (Fappend, load_path, XCDR (row));
+	}
+    }
+  return load_path;
+}
+
 DEFUN ("load", Fload, Sload, 1, 5, 0,
        doc: /* Execute a file of Lisp code named FILE.
 First try FILE with `.elc' appended, then try with `.el', then try
@@ -1278,7 +1325,9 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 then try FILE unmodified (the exact suffixes in the exact order are
 determined by `load-suffixes').  Environment variable references in
 FILE are replaced with their values by calling `substitute-in-file-name'.
-This function searches the directories in `load-path'.
+This function searches the entry in `load-hints` first, if some entries
+matched, searches in the matched pathes. Otherwise, searches directories
+in `load-path'.
 
 If optional second arg NOERROR is non-nil,
 report no error if FILE doesn't exist.
@@ -1327,7 +1376,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 #endif
   specpdl_ref fd_index UNINIT;
   specpdl_ref count = SPECPDL_INDEX ();
-  Lisp_Object found, efound, hist_file_name;
+  Lisp_Object found, efound, hist_file_name, load_path = Qnil;
   /* True means we printed the ".el is newer" message.  */
   bool newer = 0;
   /* True means we are loading a compiled file.  */
@@ -1409,12 +1458,18 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 	    suffixes = CALLN (Fappend, suffixes, Vload_file_rep_suffixes);
 	}
 
+      if (!complete_filename_p (file))
+	load_path = search_load_hints(Vload_hints, file);
+
+      if (NILP (load_path))
+	load_path = Vload_path;
+
 #if !defined USE_ANDROID_ASSETS
-      fd = openp (Vload_path, file, suffixes, &found, Qnil,
+      fd = openp (load_path, file, suffixes, &found, Qnil,
 		  load_prefer_newer, no_native, NULL);
 #else
       asset = NULL;
-      rc = openp (Vload_path, file, suffixes, &found, Qnil,
+      rc = openp (load_path, file, suffixes, &found, Qnil,
 		  load_prefer_newer, no_native, &asset);
       fd.fd = rc;
       fd.asset = asset;
@@ -1780,16 +1835,7 @@ save_match_data_load (Lisp_Object file, Lisp_Object noerror,
   return unbind_to (count, result);
 }
 \f
-static bool
-complete_filename_p (Lisp_Object pathname)
-{
-  const unsigned char *s = SDATA (pathname);
-  return (IS_DIRECTORY_SEP (s[0])
-	  || (SCHARS (pathname) > 2
-	      && IS_DEVICE_SEP (s[1]) && IS_DIRECTORY_SEP (s[2])));
-}
-
-DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2, 4, 0,
+DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2, 5, 0,
        doc: /* Search for FILENAME through PATH.
 Returns the file's name in absolute form, or nil if not found.
 If SUFFIXES is non-nil, it should be a list of suffixes to append to
@@ -1797,12 +1843,18 @@ DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2,
 If non-nil, PREDICATE is used instead of `file-readable-p'.
 PREDICATE can also be an integer to pass to the faccessat(2) function,
 in which case file-name-handlers are ignored.
+LOAD-HINTS is a list same as `load-hints'.
 This function will normally skip directories, so if you want it to find
 directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
-  (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate)
+  (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate,
+   Lisp_Object load_hints)
 {
-  Lisp_Object file;
-  int fd = openp (path, filename, suffixes, &file, predicate, false, true,
+  Lisp_Object file, dirs = Qnil;
+  if (!NILP(load_hints))
+    dirs = search_load_hints(load_hints, filename);
+  if (NILP(dirs))
+    dirs = path;
+  int fd = openp (dirs, filename, suffixes, &file, predicate, false, true,
 		  NULL);
   if (NILP (predicate) && fd >= 0)
     emacs_close (fd);
@@ -5851,6 +5903,13 @@ syms_of_lread (void)
 	       doc: /* Non-nil means read recursive structures using #N= and #N# syntax.  */);
   Vread_circle = Qt;
 
+  DEFVAR_LISP ("load-hints", Vload_hints,
+	       doc: /* A list for name to directory-list to search for files
+to load, before the load-path.  Eache entry is a file name to directory list,
+file name ends with a '*' means prefix matching. Example:
+  '(("name1-*" "/path1" "/path2")).  */);
+  Vload_hints = Qnil;
+
   DEFVAR_LISP ("load-path", Vload_path,
 	       doc: /* List of directories to search for files to load.
 Each element is a string (directory file name) or nil (meaning
-- 
2.34.1

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

Hi Eli,

I had patched the package.el also, and now the `load-hints` can boost
the emacs, especially for the emacs on windows.

The `load-hints` can reduce searching attempts by putting the matched
paths on the top of `load-path`, it won't break the original
load-path; and the patch for the `package.el` will put the installed
files into the `load-hints`.

I tested on both Ubuntu 20.04 and Windows 11, I tested the patches
with `load-hints` enabled/disabled, based on the Spacemacs
distribution, 276 packages installed, test cli is: emacs -nw --eval
"(require 'helm)",

1. On my local linux test env, disable load-hints on the package.el,
the test cli spends 6.327s; and enable load-hints then it spends
5.392s.

2. On my local Windows test env, disable load-hints on the package.el,
the test cli spends 11.769s, and enable load-hints then it spends
7.279s.

It's very effective for windows, without any break changes.

Please review the patches.  Thanks.

Best Regards, Lin

[-- Attachment #2: 0002-lisp-emacs-lisp-package.el-Support-the-load-hints.patch --]
[-- Type: text/x-patch, Size: 4666 bytes --]

From 927b3e7dc1097cfa4ce9f681b4438f9d99476739 Mon Sep 17 00:00:00 2001
From: Lin Sun <sunlin7@hotmail.com>
Date: Sat, 19 Oct 2024 06:43:15 +0000
Subject: [PATCH 2/2] * lisp/emacs-lisp/package.el: Support the load-hints

---
 lisp/emacs-lisp/package.el | 79 +++++++++++++++++++++++++++++++-------
 1 file changed, 66 insertions(+), 13 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 90d6150ed0b..1c4d47b71f2 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -208,6 +208,22 @@ package-load-list
   :risky t
   :version "24.1")
 
+(defcustom package-enable-load-hints t
+  "Non-nil means enable the `load-hints' for the packages.
+
+The value can be one of:
+
+  t             Add package dir into both `load-hints' and `load-path'.
+
+  `aggressive'  If all files in a package dir were covered by the `load-hints'
+                then will not add the package dir into `load-path'.
+
+  nil           Don't used the `load-hints'."
+  :type '(choice (const :value nil            :tag "Disable")
+                 (const :value t              :tag "Enable(safe)")
+                 (const :value aggressive     :tag "Enable(agressive)"))
+  :version "31.1")
+
 (defcustom package-archives `(("gnu" .
                                ,(format "http%s://elpa.gnu.org/packages/"
                                         (if (gnutls-available-p) "s" "")))
@@ -1095,21 +1111,58 @@ package-generate-autoloads
          ;; We don't need 'em, and this makes the output reproducible.
          (autoload-timestamps nil)
          (backup-inhibited t)
-         (version-control 'never))
+         (version-control 'never)
+         hints-list hints-covered-all)
+    ;; if package-enabled-load-hints is non-nil then collecting loadable
+    ;; files in pkg-dir and generating the load-hints list.
+    (when-let* (package-enable-load-hints
+                (name (symbol-name name))
+                (files (cl-set-difference (directory-files pkg-dir)
+                                          '("." "..") :test #'string=))
+                ;; list of files basename, the load-suffixes was removed
+                (bases
+                 (remove nil
+                         (mapcar
+                          (lambda (f)
+                            (cl-some
+                             (lambda (s)
+                               (if-let* ((n (length s))
+                                         ((length> f n))
+                                         ((string= s (substring f (- n)))))
+                                   (substring f 0 (- n))))
+                             (get-load-suffixes)))
+                          files))))
+      (setq hints-covered-all (length= bases (length files))
+            hints-list
+            (cl-remove-duplicates
+             (mapcar (lambda (s)
+                       (format "(add-to-list 'load-hints '(%S %S))"
+                               (if (string-prefix-p name s)
+                                   (concat name "*")
+                                 s)
+                               pkg-dir))
+                     bases)
+             :test 'string=)))
     (loaddefs-generate
      pkg-dir output-file nil
-     (prin1-to-string
-      '(add-to-list
-        'load-path
-        ;; Add the directory that will contain the autoload file to
-        ;; the load path.  We don't hard-code `pkg-dir', to avoid
-        ;; issues if the package directory is moved around.
-        ;; `loaddefs-generate' has code to do this for us, but it's
-        ;; not currently exposed.  (Bug#63625)
-        (or (and load-file-name
-                 (directory-file-name
-                  (file-name-directory load-file-name)))
-             (car load-path)))))
+     (concat
+      (when hints-list
+        (string-join hints-list "\n"))
+      "\n"
+      (unless (and hints-covered-all
+                   (eq package-enable-load-hints 'aggressive))
+        (prin1-to-string
+         '(add-to-list
+           'load-path
+           ;; Add the directory that will contain the autoload file to
+           ;; the load path.  We don't hard-code `pkg-dir', to avoid
+           ;; issues if the package directory is moved around.
+           ;; `loaddefs-generate' has code to do this for us, but it's
+           ;; not currently exposed.  (Bug#63625)
+           (or (and load-file-name
+                    (directory-file-name
+                     (file-name-directory load-file-name)))
+               (car load-path)))))))
     (let ((buf (find-buffer-visiting output-file)))
       (when buf (kill-buffer buf)))
     auto-name))
-- 
2.34.1


[-- Attachment #3: 0001-New-variable-load-hints-to-speedup-searching-file-fo.patch --]
[-- Type: text/x-patch, Size: 7490 bytes --]

From ef7fdb972bbdd49769af6551b434b17ec62d0951 Mon Sep 17 00:00:00 2001
From: Lin Sun <sunlin7@hotmail.com>
Date: Wed, 16 Oct 2024 07:31:59 +0000
Subject: [PATCH 1/2] New variable load-hints to speedup searching file for
 (load) function

* lisp/subr.el: (locate-library) support the `load-hints' variable
* src/lread.c: (load) function support the `load-hints' variable
---
 lisp/subr.el |  9 ++---
 src/lread.c  | 97 ++++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 84 insertions(+), 22 deletions(-)

diff --git a/lisp/subr.el b/lisp/subr.el
index 2eaed682406..3d9599270ed 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -3141,10 +3141,11 @@ locate-library
 string.  When run interactively, the argument INTERACTIVE-CALL is t,
 and the file name is displayed in the echo area."
   (interactive (list (read-library-name) nil nil t))
-  (let ((file (locate-file library
-			   (or path load-path)
-			   (append (unless nosuffix (get-load-suffixes))
-				   load-file-rep-suffixes))))
+  (let ((file (locate-file-internal
+               library (or path load-path)
+               (append (unless nosuffix (get-load-suffixes))
+                       load-file-rep-suffixes)
+               nil (unless path load-hints))))
     (if interactive-call
 	(if file
 	    (message "Library is file %s" (abbreviate-file-name file))
diff --git a/src/lread.c b/src/lread.c
index 95c6891c205..587681ef36c 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1271,6 +1271,53 @@ close_file_unwind_android_fd (void *ptr)
 
 #endif
 
+static bool
+complete_filename_p (Lisp_Object pathname)
+{
+  const unsigned char *s = SDATA (pathname);
+  return (IS_DIRECTORY_SEP (s[0])
+	  || (SCHARS (pathname) > 2 && IS_DEVICE_SEP (s[1])
+	      && IS_DIRECTORY_SEP (s[2])));
+}
+
+/* search the file in load hints to get a path list */
+static Lisp_Object
+search_load_hints(Lisp_Object load_hints, Lisp_Object file) {
+  Lisp_Object load_path = Qnil;
+  Lisp_Object tail = load_hints;
+  FOR_EACH_TAIL_SAFE (tail)
+    {
+      bool fullmatch = false;
+      ptrdiff_t len = -1;
+      Lisp_Object row = XCAR (tail);
+      Lisp_Object key = XCAR (row);
+      CHECK_STRING (key);
+
+      if (SBYTES (key) - 1 <= SBYTES (file))
+	{
+	  if (SBYTES (key) >= 1
+	      && SDATA (key)[SBYTES (key) - 1] == '*')
+	    len = SBYTES (key) - 1; /* "file-*" format */
+	  else if (SBYTES (key) == SBYTES (file))
+	    {
+	      len = SBYTES (key);
+	      fullmatch = true;
+	    }
+	}
+
+      if (len >= 0 && 0 == memcmp (SDATA (key), SDATA (file), len))
+	{
+	  if (fullmatch)
+	    {
+	      load_path = CALLN (Fappend, XCDR (row));
+	      break;
+	    }
+	  load_path = CALLN (Fappend, load_path, XCDR (row));
+	}
+    }
+  return load_path;
+}
+
 DEFUN ("load", Fload, Sload, 1, 5, 0,
        doc: /* Execute a file of Lisp code named FILE.
 First try FILE with `.elc' appended, then try with `.el', then try
@@ -1278,7 +1325,9 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 then try FILE unmodified (the exact suffixes in the exact order are
 determined by `load-suffixes').  Environment variable references in
 FILE are replaced with their values by calling `substitute-in-file-name'.
-This function searches the directories in `load-path'.
+This function searches the entry in `load-hints` first, if some entries
+matched, searches in the matched pathes. Otherwise, searches directories
+in `load-path'.
 
 If optional second arg NOERROR is non-nil,
 report no error if FILE doesn't exist.
@@ -1327,7 +1376,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 #endif
   specpdl_ref fd_index UNINIT;
   specpdl_ref count = SPECPDL_INDEX ();
-  Lisp_Object found, efound, hist_file_name;
+  Lisp_Object found, efound, hist_file_name, load_path = Qnil;
   /* True means we printed the ".el is newer" message.  */
   bool newer = 0;
   /* True means we are loading a compiled file.  */
@@ -1409,12 +1458,20 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 	    suffixes = CALLN (Fappend, suffixes, Vload_file_rep_suffixes);
 	}
 
+      if (! (NILP (Vload_hints) || complete_filename_p (file)))
+	load_path = search_load_hints(Vload_hints, file);
+
+      if (NILP (load_path))
+	load_path = Vload_path;
+      else
+	load_path = CALLN (Fappend, load_path, Vload_path);
+
 #if !defined USE_ANDROID_ASSETS
-      fd = openp (Vload_path, file, suffixes, &found, Qnil,
+      fd = openp (load_path, file, suffixes, &found, Qnil,
 		  load_prefer_newer, no_native, NULL);
 #else
       asset = NULL;
-      rc = openp (Vload_path, file, suffixes, &found, Qnil,
+      rc = openp (load_path, file, suffixes, &found, Qnil,
 		  load_prefer_newer, no_native, &asset);
       fd.fd = rc;
       fd.asset = asset;
@@ -1780,16 +1837,7 @@ save_match_data_load (Lisp_Object file, Lisp_Object noerror,
   return unbind_to (count, result);
 }
 \f
-static bool
-complete_filename_p (Lisp_Object pathname)
-{
-  const unsigned char *s = SDATA (pathname);
-  return (IS_DIRECTORY_SEP (s[0])
-	  || (SCHARS (pathname) > 2
-	      && IS_DEVICE_SEP (s[1]) && IS_DIRECTORY_SEP (s[2])));
-}
-
-DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2, 4, 0,
+DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2, 5, 0,
        doc: /* Search for FILENAME through PATH.
 Returns the file's name in absolute form, or nil if not found.
 If SUFFIXES is non-nil, it should be a list of suffixes to append to
@@ -1797,12 +1845,18 @@ DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2,
 If non-nil, PREDICATE is used instead of `file-readable-p'.
 PREDICATE can also be an integer to pass to the faccessat(2) function,
 in which case file-name-handlers are ignored.
+LOAD-HINTS is a list same as `load-hints'.
 This function will normally skip directories, so if you want it to find
 directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
-  (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate)
+  (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate,
+   Lisp_Object load_hints)
 {
-  Lisp_Object file;
-  int fd = openp (path, filename, suffixes, &file, predicate, false, true,
+  Lisp_Object file, dirs = Qnil;
+  if (!NILP(load_hints))
+    dirs = search_load_hints(load_hints, filename);
+  if (NILP(dirs))
+    dirs = path;
+  int fd = openp (dirs, filename, suffixes, &file, predicate, false, true,
 		  NULL);
   if (NILP (predicate) && fd >= 0)
     emacs_close (fd);
@@ -1882,7 +1936,7 @@ maybe_swap_for_eln (bool no_native, Lisp_Object *filename, int *fd,
 		 can't find even central .el files.  */
 	      if (NILP (Flocate_file_internal (build_string ("simple.el"),
 					       Vload_path,
-					       Qnil, Qnil)))
+					       Qnil, Qnil, Qnil)))
 		return;
 	      Vdelayed_warnings_list
 		= Fcons (list2
@@ -5851,6 +5905,13 @@ syms_of_lread (void)
 	       doc: /* Non-nil means read recursive structures using #N= and #N# syntax.  */);
   Vread_circle = Qt;
 
+  DEFVAR_LISP ("load-hints", Vload_hints,
+	       doc: /* A list for name to directory-list to search for files
+to load, before the load-path.  Eache entry is a file name to directory list,
+file name ends with a '*' means prefix matching. Example:
+  '(("name1-*" "/path1" "/path2")).  */);
+  Vload_hints = Qnil;
+
   DEFVAR_LISP ("load-path", Vload_path,
 	       doc: /* List of directories to search for files to load.
 Each element is a string (directory file name) or nil (meaning
-- 
2.34.1

bug#41646: Startup in Windows is very slow when load-path contains many

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The `load-hints` can reduce searching attempts by putting the matched
> paths on the top of `load-path`, it won't break the original
> load-path; and the patch for the `package.el` will put the installed
> files into the `load-hints`.

The downside is that it can break existing setups for users who use
`package.el` but also modify their `load-path` "by hand" in the
init file, and it doesn't help users who don't use `package.el`.

Note also that your `load-hints` could grow large, so scanning it could
take a significant amount of time.  Maybe it would make sense to turn it
into a hash table for those entries that don't use the "*" special
thingy (and maybe use a radix-tree for those entries using the "*"
special thingy)?

But your prefix idea makes me think maybe we can aim for a significantly
smaller table, where we basically record only one entry per
package/directory, like for "~/.emacs.d/elpa/helm-core-VERSION/" we just
record "helm" because all the `.el` files share the "helm" prefix.
I.e. keep for each dir the corresponding longest-common-prefix.
If we're careful to consider only those files with a `.el` suffix, then
I think we can reduce the hint to such a longest-common-prefix.  I.e. an
info which doesn't say just "you can find FOO* files here" but "you can
find *only* FOO* files here".

Then we should be able to create quickly (so it can be recomputed on the
fly whenever `load-path` changes) a radix-tree that maps a relative
file name to the list of directories from `load-path` where it is
worthwhile to look (by filtering out those dirs whose
longest-common-prefix doesn't match).  We'd only do it for MUST_SUFFIX
is specified, of course.

> 1. On my local linux test env, disable load-hints on the package.el,
> the test cli spends 6.327s; and enable load-hints then it spends
> 5.392s.
>
> 2. On my local Windows test env, disable load-hints on the package.el,
> the test cli spends 11.769s, and enable load-hints then it spends
> 7.279s.

Is that with or without using `package-quickstart`?

BTW, in your patch, you change `locate-file-internal` which seems wrong,
since that function is not specific to loading ELisp files, it's also
used for $MANPATH, $PATH, and things like that.

Similarly, I wasn't able to convince myself that your patch does the
right thing when `require` or `load` is used such that MUST_SUFFIX is
not specified.


        Stefan

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

On Mon, Oct 21, 2024 at 2:34 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > The `load-hints` can reduce searching attempts by putting the matched
> > paths on the top of `load-path`, it won't break the original
> > load-path; and the patch for the `package.el` will put the installed
> > files into the `load-hints`.
>
> The downside is that it can break existing setups for users who use
> `package.el` but also modify their `load-path` "by hand" in the
> init file, and it doesn't help users who don't use `package.el`.

The `load-hints' do nothing with its default value nil.

The package manager can use the `load-hints' or ignore it. The
"package.el" is emacs builtin package manager, can support or ignore
the `load-hints' by setting `package-enable-load-hints` to t or nil.
Other package managers can continue without any change, or do some
work to get performance benefits by supporting the `load-hints'.

> Note also that your `load-hints` could grow large, so scanning it could
> take a significant amount of time.  Maybe it would make sense to turn it
> into a hash table for those entries that don't use the "*" special
> thingy (and maybe use a radix-tree for those entries using the "*"
> special thingy)?

The `load-hints` could grow as the `load-path', or larger than the
`load-path', but it should not be too much.

For the hash table, it can not support the '*', or the hash-table
requires every file to have an entry explicitly, which costs a lot on
building the table.

I had checked the radix-tree at the beginning, it's not user-friendly,
or it's not easy to dump the radix tree for an end user to understand
which is obviously matching the entry or not.

The `load-hints' in the list are easy to understand / maintain by the end user.

> But your prefix idea makes me think maybe we can aim for a significantly
> smaller table, where we basically record only one entry per
> package/directory, like for "~/.emacs.d/elpa/helm-core-VERSION/" we just
> record "helm" because all the `.el` files share the "helm" prefix.
> I.e. keep for each dir the corresponding longest-common-prefix.
> If we're careful to consider only those files with a `.el` suffix, then
> I think we can reduce the hint to such a longest-common-prefix.  I.e. an
> info which doesn't say just "you can find FOO* files here" but "you can
> find *only* FOO* files here".

I had searched all 200+ packages in my test env, most of the packages
use their feature name as the prefix, only 11 packages have
exceptions. But I didn't understand how it works toward the
`load-hints'.

> Then we should be able to create quickly (so it can be recomputed on the
> fly whenever `load-path` changes) a radix-tree that maps a relative
> file name to the list of directories from `load-path` where it is
> worthwhile to look (by filtering out those dirs whose
> longest-common-prefix doesn't match).  We'd only do it for MUST_SUFFIX
> is specified, of course.
>
> > 1. On my local linux test env, disable load-hints on the package.el,
> > the test cli spends 6.327s; and enable load-hints then it spends
> > 5.392s.
> >
> > 2. On my local Windows test env, disable load-hints on the package.el,
> > the test cli spends 11.769s, and enable load-hints then it spends
> > 7.279s.
>
> Is that with or without using `package-quickstart`?

The `package-quickstart' does not help in this scenario, the key point
is the `load-path` count will times the read attempt.
A simple "(require 'X)" will lead the emacs walks through the
`load-path' to attempt opening the "X.so, X.so.gz, X.elc, X.elc.gz,
X.el, X.el.gz" one by one, if the `load-path' has 200 entries, emacs
will try search 200x6=1200 times for the worst case.

The `load-hints` will help put the matched path to the top of
`load-path` then emacs can find the X on the top entries of
`load-path' then returns shortly.

> BTW, in your patch, you change `locate-file-internal` which seems wrong,
> since that function is not specific to loading ELisp files, it's also
> used for $MANPATH, $PATH, and things like that.
>
> Similarly, I wasn't able to convince myself that your patch does the
> right thing when `require` or `load` is used such that MUST_SUFFIX is
> not specified.

I'm going to search the cases carefully. Thank you for all the
comments, appreciate it.

bug#41646: Startup in Windows is very slow when load-path contains many

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> The downside is that it can break existing setups for users who use
>> `package.el` but also modify their `load-path` "by hand" in the
>> init file, and it doesn't help users who don't use `package.el`.
> The `load-hints' do nothing with its default value nil.

No, but if `package.el` adds elements to `load-hints` and then in your
init file you add more elements to `load-path`, the `load-hints` may end
up taking precedence over the new elements of `load-path` in some cases
and thus change which file shadows which.

> I had checked the radix-tree at the beginning, it's not user-friendly,
> or it's not easy to dump the radix tree for an end user to understand
> which is obviously matching the entry or not.

You're not "supposed" to look at the representation of the radix tree,
indeed (unless you're working on `radix-tree.el`, of course).
Feel free to ask for help using it.

> The `load-hints' in the list are easy to understand / maintain by the
> end user.

I don't think we need to burden end users with load hints.

>> But your prefix idea makes me think maybe we can aim for a significantly
>> smaller table, where we basically record only one entry per
>> package/directory, like for "~/.emacs.d/elpa/helm-core-VERSION/" we just
>> record "helm" because all the `.el` files share the "helm" prefix.
>> I.e. keep for each dir the corresponding longest-common-prefix.
>> If we're careful to consider only those files with a `.el` suffix, then
>> I think we can reduce the hint to such a longest-common-prefix.  I.e. an
>> info which doesn't say just "you can find FOO* files here" but "you can
>> find *only* FOO* files here".
>
> I had searched all 200+ packages in my test env, most of the packages
> use their feature name as the prefix, only 11 packages have
> exceptions. But I didn't understand how it works toward the
> `load-hints'.

Here's the idea: the `<PKG>-autoloads.el` file can registers the longest
common prefix of all the `.el` files for its own `load-path` entry, with
say:

    (load-prefix-register <DIR> <PREFIX>)

where we'd define this function along the lines of

    (defconst load-prefix-directories (make-hash-table :test 'equal))
      "Set of entries from `load-path` for which we have prefix info.")

    (defconst load-prefix-map radix-tree-empty
      "Table associating file prefixes to directories.")

    (defun load-prefix-register (dir prefix)
      (puthash dir t load-prefix-directories)
      (let ((dirs (radix-tree-lookup load-prefix-map prefix)))
        (unless (member dir dirs)
          (setq load-prefix-map (radix-tree-insert load-prefix-map prefix
                                                   (cons dir dirs))))))

    (defun load-prefix-trim-load-path (file)
      "Return a trimmed `load-path` to use for FILE."
      (if (file-name-directory file)
          ;; If there's a `/` in FILE, fallback on the safe default.
          load-path
        (let* ((prefixes (radix-tree-prefixes load-prefix-map file))
               (dirs (apply #'append (mapcar #'cdr prefixes))))
          ;; Remove from `load-path` the entries which can't possibly
          ;; have FILE because their prefixes doesn't match.
          (cl-remove-if (lambda (dir)
                          (and (gethash dir load-prefix-directories)
                               (not (member dir dirs))))
                        load-path))))

and then `load` can use `load-prefix-trim-load-path` to iterate on
a much shorter `load-path`.

I'm not completely sure if it's a good idea, tho: I'd really prefer
a solution that doesn't require any change to any package management
code, which instead uses a cache (updated/filled automatically) of all
the files found in all the `load-path` directories.

>> Is that with or without using `package-quickstart`?
> The `package-quickstart' does not help in this scenario,

AFAICT your scenario includes Emacs startup with packages installed, so
`package-quickstart' can definitely make a difference.  But maybe you're
right that it will affect both cases equally.

>> Similarly, I wasn't able to convince myself that your patch does the
>> right thing when `require` or `load` is used such that MUST_SUFFIX is
>> not specified.
> The `load-hints` just put the matched paths on the top of `load-path`,
> still following the `load-path' mechanism, and won't affect any other
> features (Or someone already has some code to adjust the `load-path'
> orders, can just ignore the `load-hints', everything work like before,
> no break changes).

But if FOO is in one dir and FOO.el is in another dir, adding entries
to the load path can change which file we end up loading.


        Stefan

bug#41646: Startup in Windows is very slow when load-path contains many

[Lin Sun]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

On Mon, Oct 21, 2024 at 2:34 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> BTW, in your patch, you change `locate-file-internal` which seems wrong,
> since that function is not specific to loading ELisp files, it's also
> used for $MANPATH, $PATH, and things like that.

You're right, the change in `locate-file-internal' didn't merge the
original path.
I attached the patch to append the `dirs' with `path' together.

> Similarly, I wasn't able to convince myself that your patch does the
> right thing when `require` or `load` is used such that MUST_SUFFIX is
> not specified.

The `load-hints` just put the matched paths on the top of `load-path`,
still following the `load-path' mechanism, and won't affect any other
features (Or someone already has some code to adjust the `load-path'
orders, can just ignore the `load-hints', everything work like before,
no break changes).

[-- Attachment #2: 0001-New-variable-load-hints-to-speedup-searching-file-fo.patch --]
[-- Type: text/x-patch, Size: 7539 bytes --]

From 65c54e882a0aab28fc5697a64fd1fdf15a266440 Mon Sep 17 00:00:00 2001
From: Lin Sun <sunlin7@hotmail.com>
Date: Wed, 16 Oct 2024 07:31:59 +0000
Subject: [PATCH 1/2] New variable load-hints to speedup searching file for
 (load) function

* lisp/subr.el: (locate-library) support the `load-hints' variable
* src/lread.c: (load) function support the `load-hints' variable
---
 lisp/subr.el |  9 ++---
 src/lread.c  | 99 ++++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 86 insertions(+), 22 deletions(-)

diff --git a/lisp/subr.el b/lisp/subr.el
index 2eaed682406..3d9599270ed 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -3141,10 +3141,11 @@ locate-library
 string.  When run interactively, the argument INTERACTIVE-CALL is t,
 and the file name is displayed in the echo area."
   (interactive (list (read-library-name) nil nil t))
-  (let ((file (locate-file library
-			   (or path load-path)
-			   (append (unless nosuffix (get-load-suffixes))
-				   load-file-rep-suffixes))))
+  (let ((file (locate-file-internal
+               library (or path load-path)
+               (append (unless nosuffix (get-load-suffixes))
+                       load-file-rep-suffixes)
+               nil (unless path load-hints))))
     (if interactive-call
 	(if file
 	    (message "Library is file %s" (abbreviate-file-name file))
diff --git a/src/lread.c b/src/lread.c
index 95c6891c205..8d558ad8c66 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1271,6 +1271,53 @@ close_file_unwind_android_fd (void *ptr)
 
 #endif
 
+static bool
+complete_filename_p (Lisp_Object pathname)
+{
+  const unsigned char *s = SDATA (pathname);
+  return (IS_DIRECTORY_SEP (s[0])
+	  || (SCHARS (pathname) > 2 && IS_DEVICE_SEP (s[1])
+	      && IS_DIRECTORY_SEP (s[2])));
+}
+
+/* search the file in load hints to get a path list */
+static Lisp_Object
+search_load_hints(Lisp_Object load_hints, Lisp_Object file) {
+  Lisp_Object load_path = Qnil;
+  Lisp_Object tail = load_hints;
+  FOR_EACH_TAIL_SAFE (tail)
+    {
+      bool fullmatch = false;
+      ptrdiff_t len = -1;
+      Lisp_Object row = XCAR (tail);
+      Lisp_Object key = XCAR (row);
+      CHECK_STRING (key);
+
+      if (SBYTES (key) - 1 <= SBYTES (file))
+	{
+	  if (SBYTES (key) >= 1
+	      && SDATA (key)[SBYTES (key) - 1] == '*')
+	    len = SBYTES (key) - 1; /* "file-*" format */
+	  else if (SBYTES (key) == SBYTES (file))
+	    {
+	      len = SBYTES (key);
+	      fullmatch = true;
+	    }
+	}
+
+      if (len >= 0 && 0 == memcmp (SDATA (key), SDATA (file), len))
+	{
+	  if (fullmatch)
+	    {
+	      load_path = CALLN (Fappend, XCDR (row));
+	      break;
+	    }
+	  load_path = CALLN (Fappend, load_path, XCDR (row));
+	}
+    }
+  return load_path;
+}
+
 DEFUN ("load", Fload, Sload, 1, 5, 0,
        doc: /* Execute a file of Lisp code named FILE.
 First try FILE with `.elc' appended, then try with `.el', then try
@@ -1278,7 +1325,9 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 then try FILE unmodified (the exact suffixes in the exact order are
 determined by `load-suffixes').  Environment variable references in
 FILE are replaced with their values by calling `substitute-in-file-name'.
-This function searches the directories in `load-path'.
+This function searches the entry in `load-hints` first, if some entries
+matched, searches in the matched pathes. Otherwise, searches directories
+in `load-path'.
 
 If optional second arg NOERROR is non-nil,
 report no error if FILE doesn't exist.
@@ -1327,7 +1376,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 #endif
   specpdl_ref fd_index UNINIT;
   specpdl_ref count = SPECPDL_INDEX ();
-  Lisp_Object found, efound, hist_file_name;
+  Lisp_Object found, efound, hist_file_name, load_path = Qnil;
   /* True means we printed the ".el is newer" message.  */
   bool newer = 0;
   /* True means we are loading a compiled file.  */
@@ -1409,12 +1458,20 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
 	    suffixes = CALLN (Fappend, suffixes, Vload_file_rep_suffixes);
 	}
 
+      if (! (NILP (Vload_hints) || complete_filename_p (file)))
+	load_path = search_load_hints(Vload_hints, file);
+
+      if (NILP (load_path))
+	load_path = Vload_path;
+      else
+	load_path = CALLN (Fappend, load_path, Vload_path);
+
 #if !defined USE_ANDROID_ASSETS
-      fd = openp (Vload_path, file, suffixes, &found, Qnil,
+      fd = openp (load_path, file, suffixes, &found, Qnil,
 		  load_prefer_newer, no_native, NULL);
 #else
       asset = NULL;
-      rc = openp (Vload_path, file, suffixes, &found, Qnil,
+      rc = openp (load_path, file, suffixes, &found, Qnil,
 		  load_prefer_newer, no_native, &asset);
       fd.fd = rc;
       fd.asset = asset;
@@ -1780,16 +1837,7 @@ save_match_data_load (Lisp_Object file, Lisp_Object noerror,
   return unbind_to (count, result);
 }
 \f
-static bool
-complete_filename_p (Lisp_Object pathname)
-{
-  const unsigned char *s = SDATA (pathname);
-  return (IS_DIRECTORY_SEP (s[0])
-	  || (SCHARS (pathname) > 2
-	      && IS_DEVICE_SEP (s[1]) && IS_DIRECTORY_SEP (s[2])));
-}
-
-DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2, 4, 0,
+DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2, 5, 0,
        doc: /* Search for FILENAME through PATH.
 Returns the file's name in absolute form, or nil if not found.
 If SUFFIXES is non-nil, it should be a list of suffixes to append to
@@ -1797,12 +1845,20 @@ DEFUN ("locate-file-internal", Flocate_file_internal, Slocate_file_internal, 2,
 If non-nil, PREDICATE is used instead of `file-readable-p'.
 PREDICATE can also be an integer to pass to the faccessat(2) function,
 in which case file-name-handlers are ignored.
+LOAD-HINTS is a list same as `load-hints'.
 This function will normally skip directories, so if you want it to find
 directories, make sure the PREDICATE function returns `dir-ok' for them.  */)
-  (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate)
+  (Lisp_Object filename, Lisp_Object path, Lisp_Object suffixes, Lisp_Object predicate,
+   Lisp_Object load_hints)
 {
-  Lisp_Object file;
-  int fd = openp (path, filename, suffixes, &file, predicate, false, true,
+  Lisp_Object file, dirs = Qnil;
+  if (!NILP(load_hints))
+    dirs = search_load_hints(load_hints, filename);
+  if (NILP(dirs))
+    dirs = path;
+  else
+    dirs = CALLN (Fappend, dirs, path);
+  int fd = openp (dirs, filename, suffixes, &file, predicate, false, true,
 		  NULL);
   if (NILP (predicate) && fd >= 0)
     emacs_close (fd);
@@ -1882,7 +1938,7 @@ maybe_swap_for_eln (bool no_native, Lisp_Object *filename, int *fd,
 		 can't find even central .el files.  */
 	      if (NILP (Flocate_file_internal (build_string ("simple.el"),
 					       Vload_path,
-					       Qnil, Qnil)))
+					       Qnil, Qnil, Qnil)))
 		return;
 	      Vdelayed_warnings_list
 		= Fcons (list2
@@ -5851,6 +5907,13 @@ syms_of_lread (void)
 	       doc: /* Non-nil means read recursive structures using #N= and #N# syntax.  */);
   Vread_circle = Qt;
 
+  DEFVAR_LISP ("load-hints", Vload_hints,
+	       doc: /* A list for name to directory-list to search for files
+to load, before the load-path.  Eache entry is a file name to directory list,
+file name ends with a '*' means prefix matching. Example:
+  '(("name1-*" "/path1" "/path2")).  */);
+  Vload_hints = Qnil;
+
   DEFVAR_LISP ("load-path", Vload_path,
 	       doc: /* List of directories to search for files to load.
 Each element is a string (directory file name) or nil (meaning
-- 
2.34.1


[-- Attachment #3: 0002-lisp-emacs-lisp-package.el-Support-the-load-hints.patch --]
[-- Type: text/x-patch, Size: 4666 bytes --]

From 4efb3b689ef6f97cab82e5b34cf9dabc3f3d7ee0 Mon Sep 17 00:00:00 2001
From: Lin Sun <sunlin7@hotmail.com>
Date: Sat, 19 Oct 2024 06:43:15 +0000
Subject: [PATCH 2/2] * lisp/emacs-lisp/package.el: Support the load-hints

---
 lisp/emacs-lisp/package.el | 79 +++++++++++++++++++++++++++++++-------
 1 file changed, 66 insertions(+), 13 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 90d6150ed0b..1c4d47b71f2 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -208,6 +208,22 @@ package-load-list
   :risky t
   :version "24.1")
 
+(defcustom package-enable-load-hints t
+  "Non-nil means enable the `load-hints' for the packages.
+
+The value can be one of:
+
+  t             Add package dir into both `load-hints' and `load-path'.
+
+  `aggressive'  If all files in a package dir were covered by the `load-hints'
+                then will not add the package dir into `load-path'.
+
+  nil           Don't used the `load-hints'."
+  :type '(choice (const :value nil            :tag "Disable")
+                 (const :value t              :tag "Enable(safe)")
+                 (const :value aggressive     :tag "Enable(agressive)"))
+  :version "31.1")
+
 (defcustom package-archives `(("gnu" .
                                ,(format "http%s://elpa.gnu.org/packages/"
                                         (if (gnutls-available-p) "s" "")))
@@ -1095,21 +1111,58 @@ package-generate-autoloads
          ;; We don't need 'em, and this makes the output reproducible.
          (autoload-timestamps nil)
          (backup-inhibited t)
-         (version-control 'never))
+         (version-control 'never)
+         hints-list hints-covered-all)
+    ;; if package-enabled-load-hints is non-nil then collecting loadable
+    ;; files in pkg-dir and generating the load-hints list.
+    (when-let* (package-enable-load-hints
+                (name (symbol-name name))
+                (files (cl-set-difference (directory-files pkg-dir)
+                                          '("." "..") :test #'string=))
+                ;; list of files basename, the load-suffixes was removed
+                (bases
+                 (remove nil
+                         (mapcar
+                          (lambda (f)
+                            (cl-some
+                             (lambda (s)
+                               (if-let* ((n (length s))
+                                         ((length> f n))
+                                         ((string= s (substring f (- n)))))
+                                   (substring f 0 (- n))))
+                             (get-load-suffixes)))
+                          files))))
+      (setq hints-covered-all (length= bases (length files))
+            hints-list
+            (cl-remove-duplicates
+             (mapcar (lambda (s)
+                       (format "(add-to-list 'load-hints '(%S %S))"
+                               (if (string-prefix-p name s)
+                                   (concat name "*")
+                                 s)
+                               pkg-dir))
+                     bases)
+             :test 'string=)))
     (loaddefs-generate
      pkg-dir output-file nil
-     (prin1-to-string
-      '(add-to-list
-        'load-path
-        ;; Add the directory that will contain the autoload file to
-        ;; the load path.  We don't hard-code `pkg-dir', to avoid
-        ;; issues if the package directory is moved around.
-        ;; `loaddefs-generate' has code to do this for us, but it's
-        ;; not currently exposed.  (Bug#63625)
-        (or (and load-file-name
-                 (directory-file-name
-                  (file-name-directory load-file-name)))
-             (car load-path)))))
+     (concat
+      (when hints-list
+        (string-join hints-list "\n"))
+      "\n"
+      (unless (and hints-covered-all
+                   (eq package-enable-load-hints 'aggressive))
+        (prin1-to-string
+         '(add-to-list
+           'load-path
+           ;; Add the directory that will contain the autoload file to
+           ;; the load path.  We don't hard-code `pkg-dir', to avoid
+           ;; issues if the package directory is moved around.
+           ;; `loaddefs-generate' has code to do this for us, but it's
+           ;; not currently exposed.  (Bug#63625)
+           (or (and load-file-name
+                    (directory-file-name
+                     (file-name-directory load-file-name)))
+               (car load-path)))))))
     (let ((buf (find-buffer-visiting output-file)))
       (when buf (kill-buffer buf)))
     auto-name))
-- 
2.34.1

bug#41646: Startup in Windows is very slow when load-path contains many

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> [I have unarchived Bug#41646 so that this discussion is archived in the
> bug tracker.]

[ Note: I haven't seen the earlier part of this discussion (but I did read the
  old bug#41646 thread).  ]

>> Totally understand, and thanks for inviting Stefan and Andrea in the
>> conversation, to let new participants know the context quickly, I had
>> post a patch to speed up Emacs for the scenario that Emacs will very
>> slow on startup with hundreds packages (>300)  installed, the keypoint
>> is hundreds packages will add hundreds paths into the `load-path',
>> then a simple `(require X)' may trigger hundreds searching according
>> to the `load-path',

Yup, that's a known problem which we've been not solving for
a long time.
The "latest" workaround is `package-quickstart`, which reduces some of
the associated pain.

If you're not using `package-quickstart` and are bothered by a slow
startup, then I'd recommend you go back and enable `package-quickstart`.

Similarly in the old bug#41646 thread, I see mentions of 40s startup
time, 8s of which are *not* spent in openp: 8s is still quite slow, so
it might be worth looking at what the startup file is doing and see if
we could do less at startup (e.g. load things more lazily).

Of course, the "long load-path" problem will still end up biting in
some cases.  Until now we've managed to make those cases rare enough
that we haven't had to actually solve it.

>> so the patch build a map for feature --> filepath from the variable
>> `load-history' and store to disk, after that loading

I don't think we want a cache that's stored on disk: it would take too
much effort to create it, load it, ensure it's not damaged if several
Emacs sessions try to write it at the same time, make sure it's fresh,
etc...

Especially since I believe that building the cache shouldn't take very
long: longer than a single "look for file F in `load-path`", maybe, but
not by much.

IOW we could keep a cache that's populated on-the-fly the first time we
`load` something, and that's then automatically refreshed when `load`
sees a new `load-path`.


        Stefan