bug#41646: Startup in Windows is very slow when load-path contains many entries.

["Nicolas Bértolo" <nicolasbertolo@gmail.com>]

I have an issue regarding startup times in Windows. My configuration is
Spacemacs with many layers enabled. My load-path contains 380 entries.

I have profiled Emacs in Windows and found that it spends most of the startup
time calling wopen(). This is because when calling (load "foo") it checks all
directories in load-path for ("foo.el" "foo.elc" "foo.el.gz" "foo.elc.gz"
"foo.dll"). It gets worse when load-prefer-newer is t.

In my case `load-path` contains 380 entries, so every call to load will perform
380 * 5 = 1900 calls to wopen. This is very slow in Windows because its
filesystem is not optimized for so many accesses to small files.

I thought that a caching mechanism would help.

This "cache" would consist of a mapping of files that would get probed when
(load "foo") runs. It would be implemented as a hash table. The contents of this
hash table could be loaded from load-cache.el files in each package directory.

The directory foo-pkg could have a file load-cache.el with:

foo -> ("foo-pkg/foo.el"
        "foo-pkg/foo.elc")
[...]

The directory bar-pkg could have a file load-cache.el with:

bar -> ("bar-pkg/bar.el"
        "bar-pkg/bar.elc")
[...]

When a package is activated we could update the in-memory hash table by loading
its load-cache.el file. Then, when (require 'foo) runs, the loading code could
look at the hash table and only fopen() the files associated with the feature we
are loading. This would reduce the number of calls to fopen() from thousands to
~2 in the worst case.

Or we could have a big load-cache.el in `package-user-dir'. I prefer many small
files because maintaining the big file when many Emacs instances could be
installing or removing packages is synchronization nightmare.

Of course, this feature would be disabled by default.

What do you think?

Nicolas