* bug#41646: Startup in Windows is very slow when load-path contains many entries.
@ 2020-06-01 14:26 Nicolas Bértolo
2020-06-01 16:05 ` Eli Zaretskii
0 siblings, 1 reply; 12+ messages in thread
From: Nicolas Bértolo @ 2020-06-01 14:26 UTC (permalink / raw)
To: 41646
I have an issue regarding startup times in Windows. My configuration is
Spacemacs with many layers enabled. My load-path contains 380 entries.
I have profiled Emacs in Windows and found that it spends most of the startup
time calling wopen(). This is because when calling (load "foo") it checks all
directories in load-path for ("foo.el" "foo.elc" "foo.el.gz" "foo.elc.gz"
"foo.dll"). It gets worse when load-prefer-newer is t.
In my case `load-path` contains 380 entries, so every call to load will perform
380 * 5 = 1900 calls to wopen. This is very slow in Windows because its
filesystem is not optimized for so many accesses to small files.
I thought that a caching mechanism would help.
This "cache" would consist of a mapping of files that would get probed when
(load "foo") runs. It would be implemented as a hash table. The contents of this
hash table could be loaded from load-cache.el files in each package directory.
The directory foo-pkg could have a file load-cache.el with:
foo -> ("foo-pkg/foo.el"
"foo-pkg/foo.elc")
[...]
The directory bar-pkg could have a file load-cache.el with:
bar -> ("bar-pkg/bar.el"
"bar-pkg/bar.elc")
[...]
When a package is activated we could update the in-memory hash table by loading
its load-cache.el file. Then, when (require 'foo) runs, the loading code could
look at the hash table and only fopen() the files associated with the feature we
are loading. This would reduce the number of calls to fopen() from thousands to
~2 in the worst case.
Or we could have a big load-cache.el in `package-user-dir'. I prefer many small
files because maintaining the big file when many Emacs instances could be
installing or removing packages is synchronization nightmare.
Of course, this feature would be disabled by default.
What do you think?
Nicolas
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-06-01 14:26 bug#41646: Startup in Windows is very slow when load-path contains many entries Nicolas Bértolo
@ 2020-06-01 16:05 ` Eli Zaretskii
2020-06-01 16:46 ` Nicolas Bértolo
0 siblings, 1 reply; 12+ messages in thread
From: Eli Zaretskii @ 2020-06-01 16:05 UTC (permalink / raw)
To: Nicolas Bértolo; +Cc: 41646
> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Mon, 1 Jun 2020 11:26:35 -0300
>
> I have an issue regarding startup times in Windows. My configuration is
> Spacemacs with many layers enabled. My load-path contains 380 entries.
>
> I have profiled Emacs in Windows and found that it spends most of the startup
> time calling wopen(). This is because when calling (load "foo") it checks all
> directories in load-path for ("foo.el" "foo.elc" "foo.el.gz" "foo.elc.gz"
> "foo.dll"). It gets worse when load-prefer-newer is t.
>
> In my case `load-path` contains 380 entries, so every call to load will perform
> 380 * 5 = 1900 calls to wopen. This is very slow in Windows because its
> filesystem is not optimized for so many accesses to small files.
So this is not specific to Windows, it's just that Windows has slower
file access. IOW, if load-path becomes significantly larger, the slow
startup will show on Posix systems as well, right?
Next question: are the 'wopen' calls coming from 'openp'? if so,
perhaps we could first try a cheaper call, like 'chmod' (or its Win32
API equivalent), and save the 'wopen' call if 'chmod' fails? Did you
try that?
> I thought that a caching mechanism would help.
My main concern with a cache is how to make sure it reflects what's on
the disk, when files are added or removed.
Thanks.
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-06-01 16:05 ` Eli Zaretskii
@ 2020-06-01 16:46 ` Nicolas Bértolo
2020-06-01 17:17 ` Eli Zaretskii
0 siblings, 1 reply; 12+ messages in thread
From: Nicolas Bértolo @ 2020-06-01 16:46 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: 41646
> So this is not specific to Windows, it's just that Windows has slower
> file access. IOW, if load-path becomes significantly larger, the slow
> startup will show on Posix systems as well, right?
Exactly, but not nearly as bad. I profiled the same Spacemacs configuration in
Ubuntu 20.04 and startup takes 10 seconds, with 5 seconds spent inside openp().
Compare this to 40 seconds in total and 32 seconds inside openp() in Windows.
> Next question: are the 'wopen' calls coming from 'openp'? if so,
> perhaps we could first try a cheaper call, like 'chmod' (or its Win32
> API equivalent), and save the 'wopen' call if 'chmod' fails? Did you
> try that?
They come from openp(), yes. I haven't tried it. I think the issue is related to
the IO architecture in Windows, so I don't expect major speedups, but it could
certainly help.
> My main concern with a cache is how to make sure it reflects what's on
> the disk, when files are added or removed.
That is my main concern as well. A good option would be to rely on package.el to
generate the load-cache.el file for a package when it generates autoloads. If we
can't find any of the files mentioned in the cache then we fallback to the
current load-path code. That is a simple solution that would take care of
addition and removal of files, I think.
Nico.
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-06-01 16:46 ` Nicolas Bértolo
@ 2020-06-01 17:17 ` Eli Zaretskii
2020-06-01 19:51 ` Nicolas Bértolo
0 siblings, 1 reply; 12+ messages in thread
From: Eli Zaretskii @ 2020-06-01 17:17 UTC (permalink / raw)
To: Nicolas Bértolo; +Cc: 41646
> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Mon, 1 Jun 2020 13:46:01 -0300
> Cc: 41646@debbugs.gnu.org
>
> > So this is not specific to Windows, it's just that Windows has slower
> > file access. IOW, if load-path becomes significantly larger, the slow
> > startup will show on Posix systems as well, right?
>
> Exactly, but not nearly as bad. I profiled the same Spacemacs configuration in
> Ubuntu 20.04 and startup takes 10 seconds, with 5 seconds spent inside openp().
>
> Compare this to 40 seconds in total and 32 seconds inside openp() in Windows.
You may be surprised how many users of GNU/Linux consider 5 sec during
startup an unacceptable slowdown.
> > Next question: are the 'wopen' calls coming from 'openp'? if so,
> > perhaps we could first try a cheaper call, like 'chmod' (or its Win32
> > API equivalent), and save the 'wopen' call if 'chmod' fails? Did you
> > try that?
>
> They come from openp(), yes. I haven't tried it. I think the issue is related to
> the IO architecture in Windows, so I don't expect major speedups, but it could
> certainly help.
'chmod' doesn't access the file, it only accesses the directory. So
it could be much faster.
> > My main concern with a cache is how to make sure it reflects what's on
> > the disk, when files are added or removed.
>
> That is my main concern as well. A good option would be to rely on package.el to
> generate the load-cache.el file for a package when it generates autoloads. If we
> can't find any of the files mentioned in the cache then we fallback to the
> current load-path code. That is a simple solution that would take care of
> addition and removal of files, I think.
Is this supposed to work only when packages are installed by
package.el? What about manual installation?
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-06-01 17:17 ` Eli Zaretskii
@ 2020-06-01 19:51 ` Nicolas Bértolo
2020-06-02 2:17 ` Eli Zaretskii
0 siblings, 1 reply; 12+ messages in thread
From: Nicolas Bértolo @ 2020-06-01 19:51 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: 41646
> You may be surprised how many users of GNU/Linux consider 5 sec during
> startup an unacceptable slowdown.
Me too. But I had given up on it until I read how it was implemented.
> 'chmod' doesn't access the file, it only accesses the directory. So
> it could be much faster.
As far as I can see there is no way to issue a call to 'chmod' that does not
try to change the permissions of the file passed. I will try to do it
with access().
> Is this supposed to work only when packages are installed by
> package.el? What about manual installation?
Everything should work OK as long as you provide a load-cache.el file in the
directory you add to load-path. We could provide a function to generate it
automatically.
Thanks, Nico
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-06-01 19:51 ` Nicolas Bértolo
@ 2020-06-02 2:17 ` Eli Zaretskii
2020-06-03 1:07 ` Nicolas Bértolo
0 siblings, 1 reply; 12+ messages in thread
From: Eli Zaretskii @ 2020-06-02 2:17 UTC (permalink / raw)
To: Nicolas Bértolo; +Cc: 41646
> From: Nicolas Bértolo <nicolasbertolo@gmail.com>
> Date: Mon, 1 Jun 2020 16:51:11 -0300
> Cc: 41646@debbugs.gnu.org
>
> > 'chmod' doesn't access the file, it only accesses the directory. So
> > it could be much faster.
>
> As far as I can see there is no way to issue a call to 'chmod' that does not
> try to change the permissions of the file passed. I will try to do it
> with access().
Sorry, I meant 'access', of course. More accurately,
GetFileAttributes.
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-06-02 2:17 ` Eli Zaretskii
@ 2020-06-03 1:07 ` Nicolas Bértolo
2020-08-14 17:22 ` Lars Ingebrigtsen
0 siblings, 1 reply; 12+ messages in thread
From: Nicolas Bértolo @ 2020-06-03 1:07 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: 41646
[-- Attachment #1: Type: text/plain, Size: 206 bytes --]
> Sorry, I meant 'access', of course. More accurately,
> GetFileAttributes.
That was a very good idea. The attached patch reduces time spent
in openp() during startup by 5 seconds, or 15%.
Thanks, Nico.
[-- Attachment #2: 0001-In-Windows-check-if-file-exists-before-opening-it.patch --]
[-- Type: application/octet-stream, Size: 1299 bytes --]
From 497d2bccffa0a9b04870307ec5cd85d81cf65d89 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Nicol=C3=A1s=20B=C3=A9rtolo?= <nicolasbertolo@gmail.com>
Date: Tue, 2 Jun 2020 22:02:27 -0300
Subject: [PATCH] In Windows check if file exists before opening it.
This reduces time spent in 'openp' by 15% during startup in my tests.
* src/lread.c (openp): Use faccessat to check that a file exists
before opening it.
---
src/lread.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/src/lread.c b/src/lread.c
index 29deddaf15f..8ea19dd8d08 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1742,7 +1742,17 @@ openp (Lisp_Object path, Lisp_Object str, Lisp_Object suffixes,
}
else
{
- fd = emacs_open (pfn, O_RDONLY, 0);
+ /* In some systems (like Windows) finding out if a
+ file exists is cheaper to do than actually opening
+ it. Only open the file when we are sure that it
+ exists. */
+#ifdef WINDOWSNT
+ if (faccessat (AT_FDCWD, pfn, R_OK, AT_EACCESS))
+ fd = -1;
+ else
+#endif
+ fd = emacs_open (pfn, O_RDONLY, 0);
+
if (fd < 0)
{
if (! (errno == ENOENT || errno == ENOTDIR))
--
2.25.1.windows.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-06-03 1:07 ` Nicolas Bértolo
@ 2020-08-14 17:22 ` Lars Ingebrigtsen
2020-08-14 19:07 ` Eli Zaretskii
0 siblings, 1 reply; 12+ messages in thread
From: Lars Ingebrigtsen @ 2020-08-14 17:22 UTC (permalink / raw)
To: Nicolas Bértolo; +Cc: 41646
Nicolas Bértolo <nicolasbertolo@gmail.com> writes:
>> Sorry, I meant 'access', of course. More accurately,
>> GetFileAttributes.
>
> That was a very good idea. The attached patch reduces time spent
> in openp() during startup by 5 seconds, or 15%.
That's a huge speed-up indeed.
[...]
> - fd = emacs_open (pfn, O_RDONLY, 0);
> + /* In some systems (like Windows) finding out if a
> + file exists is cheaper to do than actually opening
> + it. Only open the file when we are sure that it
> + exists. */
> +#ifdef WINDOWSNT
> + if (faccessat (AT_FDCWD, pfn, R_OK, AT_EACCESS))
> + fd = -1;
> + else
> +#endif
> + fd = emacs_open (pfn, O_RDONLY, 0);
> +
This is a Windows-specific patch, so I have no opinion on it. Eli?
Does this make sense to you?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-08-14 17:22 ` Lars Ingebrigtsen
@ 2020-08-14 19:07 ` Eli Zaretskii
2021-05-13 9:17 ` Lars Ingebrigtsen
0 siblings, 1 reply; 12+ messages in thread
From: Eli Zaretskii @ 2020-08-14 19:07 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: nicolasbertolo, 41646
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, 41646@debbugs.gnu.org
> Date: Fri, 14 Aug 2020 19:22:26 +0200
>
> > That was a very good idea. The attached patch reduces time spent
> > in openp() during startup by 5 seconds, or 15%.
>
> That's a huge speed-up indeed.
>
> [...]
>
> > - fd = emacs_open (pfn, O_RDONLY, 0);
> > + /* In some systems (like Windows) finding out if a
> > + file exists is cheaper to do than actually opening
> > + it. Only open the file when we are sure that it
> > + exists. */
> > +#ifdef WINDOWSNT
> > + if (faccessat (AT_FDCWD, pfn, R_OK, AT_EACCESS))
> > + fd = -1;
> > + else
> > +#endif
> > + fd = emacs_open (pfn, O_RDONLY, 0);
> > +
>
> This is a Windows-specific patch, so I have no opinion on it. Eli?
> Does this make sense to you?
Not sure. 15% sounds too little for having OS-specific code in that
place. I hoped it will be much more.
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2020-08-14 19:07 ` Eli Zaretskii
@ 2021-05-13 9:17 ` Lars Ingebrigtsen
2021-05-13 10:20 ` Eli Zaretskii
0 siblings, 1 reply; 12+ messages in thread
From: Lars Ingebrigtsen @ 2021-05-13 9:17 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: nicolasbertolo, 41646
Eli Zaretskii <eliz@gnu.org> writes:
>> This is a Windows-specific patch, so I have no opinion on it. Eli?
>> Does this make sense to you?
>
> Not sure. 15% sounds too little for having OS-specific code in that
> place. I hoped it will be much more.
So would your preference be to not apply the patch?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2021-05-13 9:17 ` Lars Ingebrigtsen
@ 2021-05-13 10:20 ` Eli Zaretskii
2021-05-13 11:31 ` Lars Ingebrigtsen
0 siblings, 1 reply; 12+ messages in thread
From: Eli Zaretskii @ 2021-05-13 10:20 UTC (permalink / raw)
To: Lars Ingebrigtsen; +Cc: nicolasbertolo, 41646
> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: nicolasbertolo@gmail.com, 41646@debbugs.gnu.org
> Date: Thu, 13 May 2021 11:17:07 +0200
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> >> This is a Windows-specific patch, so I have no opinion on it. Eli?
> >> Does this make sense to you?
> >
> > Not sure. 15% sounds too little for having OS-specific code in that
> > place. I hoped it will be much more.
>
> So would your preference be to not apply the patch?
No, I guess it's okay after all, so let's install it.
TIA
^ permalink raw reply [flat|nested] 12+ messages in thread
* bug#41646: Startup in Windows is very slow when load-path contains many entries.
2021-05-13 10:20 ` Eli Zaretskii
@ 2021-05-13 11:31 ` Lars Ingebrigtsen
0 siblings, 0 replies; 12+ messages in thread
From: Lars Ingebrigtsen @ 2021-05-13 11:31 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: nicolasbertolo, 41646
Eli Zaretskii <eliz@gnu.org> writes:
> No, I guess it's okay after all, so let's install it.
OK; I tested the patch on a Windows VM, and Emacs seemed to start
normally, as far as I could tell (but I didn't test it further than
that).
Pushed to Emacs 28 now.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-05-13 11:31 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-01 14:26 bug#41646: Startup in Windows is very slow when load-path contains many entries Nicolas Bértolo
2020-06-01 16:05 ` Eli Zaretskii
2020-06-01 16:46 ` Nicolas Bértolo
2020-06-01 17:17 ` Eli Zaretskii
2020-06-01 19:51 ` Nicolas Bértolo
2020-06-02 2:17 ` Eli Zaretskii
2020-06-03 1:07 ` Nicolas Bértolo
2020-08-14 17:22 ` Lars Ingebrigtsen
2020-08-14 19:07 ` Eli Zaretskii
2021-05-13 9:17 ` Lars Ingebrigtsen
2021-05-13 10:20 ` Eli Zaretskii
2021-05-13 11:31 ` Lars Ingebrigtsen
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).