bug#30854: 27.0.50; Speeding up package.el startup

bug#30854: 27.0.50; Speeding up package.el startup

[arthur.miller]

> There are many ways to make Emacs's startup slow.
> One of them is to have many packages installed.  The growing popularity
> of ELPA means that it's now common for users to have hundreds of ELPA
> packages installed, which will easily cause Emacs's startup to take at
> least 1s even with an empty ~/.emacs.  For users like me who (re)start
> their Emacs session only rarely, this is not an issue, but for others it
> can be an annoyance that's significant enough to try and circumvent it
> by not using package.el (e.g. installing all their packages by hand or
> using other packaging like DOOM).
> 
> If there's no objection, I plan on installing the patch below which
> below lets users cut down package.el startup time by skipping package's
> initialization and instead loading a single precomputed file which is
> the concatenation of all the installed <pkg>-autoloads.el.  In my
> experience this speeds up activation of package.el by a factor 5.
> We could speed it up even further by byte-compiling this file, but this
> has bumped into some corner cases problems so I'm sticking to
> a non-compiled file for now.
> 
> 
> 
>         Stefan

Which corner cases apply to this? If it is OK to ask you?

I have been playing with my own init, and I see quite a speed-up by
compiling and baking package-quickstart.el into my init file (with
disabled package-quickstart).

I have also optimized it slightly: every autoload file adds it's own
path to load-path via add-to-list at startup time. But all those paths
are known when package-quickstart-refresh runs, so we can simply
concatenate them to one list and then emit code to just nconc emitted
that list to load-path at startup (this is what I do in my init file).

Together with that optimization and byte-compiling init file with baked
package-quickstart.el I see quite a speed-up, compared to non
byte-compiled package-quickstart.el as it is in Emacs.

Init time for daemon process decreased from ~0.06 to ~0.02 by just doing
this. My Gui emacs starts at almost same speed as Emacs with -Q flag
with this. My init file is not the worlds largest, currently I have only
~60 packages but it might be of interest to other Emacs users if those
corner cases were maybe worked out so that package-quickstart.el can be
byte compiled by default?

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

>> We could speed it up even further by byte-compiling this file, but this
>> has bumped into some corner cases problems so I'm sticking to
>> a non-compiled file for now.
> Which corner cases apply to this? If it is OK to ask you?

[ You quoted a lot of text, so I'm not completely sure, but `C-s` seemed
  to indicate that you were referring to just those above three lines
  I kept.  ]

I can't remember off-hand.  Actually, I have a vague recollection that
the problems turned out having some other origin.  So indeed we should
byte-compile the resulting file.

> compiling and baking package-quickstart.el into my init file (with
> disabled package-quickstart).

I don't know what this means.

> I have also optimized it slightly: every autoload file adds it's own
> path to load-path via add-to-list at startup time. But all those paths
> are known when package-quickstart-refresh runs, so we can simply
> concatenate them to one list and then emit code to just nconc emitted
> that list to load-path at startup (this is what I do in my init file).

I'm surprised a single call to `nconc` would be noticeably faster than
a hundred calls to `add-to-list`.  I mean, obviously it's a lot faster,
but unless you do that in a loop (which is not the case here),
I wouldn't expect the difference to be measurable.


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[arthur.miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> We could speed it up even further by byte-compiling this file, but this
>>> has bumped into some corner cases problems so I'm sticking to
>>> a non-compiled file for now.
>> Which corner cases apply to this? If it is OK to ask you?
>
> [ You quoted a lot of text, so I'm not completely sure, but `C-s` seemed
>   to indicate that you were referring to just those above three lines
>   I kept.  ]
Yes indeed, that was what I was mostly interested about.

> I can't remember off-hand.  Actually, I have a vague recollection that
> the problems turned out having some other origin.  So indeed we should
> byte-compile the resulting file.
That sounds great! Maybe more people could try it before you enable it
by default? It seems to work quite well for me, but people's setups
are so different.

>> compiling and baking package-quickstart.el into my init file (with
>> disabled package-quickstart).
>
> I don't know what this means.
Ah; sorry. That ment that I simply copy-pasted the content of
package-quickstart.el (minus some) into my init file, and compiled init file.

You can see here an excerpt as illustration:

( ...
      (when init-file-bake-autoloads
        (with-temp-buffer
          (when init-file-bake-autoloads
            (insert "(defvar package-activated-list nil)"))
          (insert-file-contents-literally "autoloads.el")
          (write-region (point-min) (point-max) "init.el" t 0)))
 ... )

(autoloads.el is just package-quickstart.el where I have prepended paths
and removed add-to-list lines)

>> I have also optimized it slightly: every autoload file adds it's own
>> path to load-path via add-to-list at startup time. But all those paths
>> are known when package-quickstart-refresh runs, so we can simply
>> concatenate them to one list and then emit code to just nconc emitted
>> that list to load-path at startup (this is what I do in my init file).
>
> I'm surprised a single call to `nconc` would be noticeably faster than
> a hundred calls to `add-to-list`.  I mean, obviously it's a lot faster,
> but unless you do that in a loop (which is not the case here),
> I wouldn't expect the difference to be measurable.

I don't know for sure what is going on here to be honest. Maybe it is
just result of byte-compilation alone, maybe there is more. I goofed a
little bit and didn't disabled 'quickstart' correctly when I was testing
this and I got same results with and without pre-calculated paths, so
you are probably correct there. The real speed boost came when I merged
content of package-quickstart.el with my init file and disabled
package-quickstart. It means package-initialize is not called so that
saves some time too I guess.

But byte compilation of that file gives boost in itself; just not as
much as I got when I disabled package-quickstart. So maybe could be
something for 28.0 if tested by more people?

Thanks for the answer.
/a

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

> But byte compilation of that file gives boost in itself;

[ BTW, I just pushed to `master` a patch that byte-compiles the
  quickstart file.  ]

> just not as much as I got when I disabled package-quickstart.

I'm surprised there's much difference: you mentioned
`package-initialize` but nowadays `package-initialize` is not called in
a normal Emacs startup: `package-activate-all` is used instead.
And `package-activate-all` reduces to loading the quickstart file if
that file exists.

Any chance you could try and dig deeper to try and find more precisely
the origin of the performance difference you see?


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Basil L. Contovounesios]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But byte compilation of that file gives boost in itself;
>
> [ BTW, I just pushed to `master` a patch that byte-compiles the
>   quickstart file.  ]

The "no-byte-compile: t" in package-quickstart-refresh disagrees.  :)

-- 
Basil

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

>>> But byte compilation of that file gives boost in itself;
>> [ BTW, I just pushed to `master` a patch that byte-compiles the
>>   quickstart file.  ]
> The "no-byte-compile: t" in package-quickstart-refresh disagrees.  :)

Damned little bastard!


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[arthur.miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> But byte compilation of that file gives boost in itself;
>
> [ BTW, I just pushed to `master` a patch that byte-compiles the
>   quickstart file.  ]
>
>> just not as much as I got when I disabled package-quickstart.
>
> I'm surprised there's much difference: you mentioned
> `package-initialize` but nowadays `package-initialize` is not called in
> a normal Emacs startup: `package-activate-all` is used instead.
> And `package-activate-all` reduces to loading the quickstart file if
> that file exists.
>
> Any chance you could try and dig deeper to try and find more precisely
> the origin of the performance difference you see?
>
Unfortunately I got less time then I thought I would have today.

I have tested some more with following:

Quick start enabled:

early-init.el
(setq package-quickstart t)

init.el

(package-activate-all)

package-quickstart.el is byte- compiled

startup time ~0.28 to ~0.30

Quick start disabled:

early-init.el:

(setq package-quickstart nil
      package-quickstart nil
      package-enable-at-startup nil
      package--init-file-ensured t)

init.el:

(defvar package-activated-list nil)
(setq load-path (append 
'("/home/arthur/.emacs.d/elpa/yasnippet-20200604.246" ....  "/home/arthur/.emacs.d/elpa/async-20200809.501") load-path))

followed by pasted package-quickstart.el with removed add-to-list
for load-path statements

What I see is that startup time is consistently somewhat faster with
"disabled" quickstart. I am getting ~0.24 with my version; and ~0.28
with "original" quickstart.

All other things equal.

See if there is something wrong in how I enable quicstart above.

best regards
/a

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

> Unfortunately I got less time then I thought I would have today.

There's no hurry.

> early-init.el
> (setq package-quickstart t)

You don't need it there: `package-activate-all` will use a quickstart
file if found, regardless of the value of `package-quickstart`.
IOW the value of `package-quickstart` onlt matters to decide whether we
*generate* the quickstart file.

> init.el
> (package-activate-all)

This should not be needed (and can be harmful) since
`package-activate-all` should have been run by Emacs between
`early-init.el` and `init.el`

> package-quickstart.el is byte- compiled

Good.

> startup time ~0.28 to ~0.30

OK.

> Quick start disabled:
>
> early-init.el:
>
> (setq package-quickstart nil
>       package-quickstart nil
>       package-enable-at-startup nil
>       package--init-file-ensured t)
[...]
> What I see is that startup time is consistently somewhat faster with
> "disabled" quickstart.  I am getting ~0.24 with my version; and ~0.28
> with "original" quickstart.

Maybe the difference is just due to loading `package`?
IOW, what happens if you add (require 'package) to your "Quick start
disabled" version?  Does the startup time go back up to ~0.28?


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Maybe the difference is just due to loading `package`?
> IOW, what happens if you add (require 'package) to your "Quick start
> disabled" version?  Does the startup time go back up to ~0.28?

Why not just preload package.el?  A good 75 percent of our users seem to
use it, according to the Emacs Survey.

28.8 %  built-in package.el
46.3 %  use-package [uses package.el behind the scenes]
------
75.1 %  total

Source: https://emacssurvey.org/2020/

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

>> Maybe the difference is just due to loading `package`?
>> IOW, what happens if you add (require 'package) to your "Quick start
>> disabled" version?  Does the startup time go back up to ~0.28?
> Why not just preload package.el?

First, I'd like to understand if that's indeed the source of the
timing difference.

But yes, I think it would make a lot of sense to split package.el into the part
that manages installation/updates/... and the part that just activates
the packages installed and to preload the second part.
Also exposing the API of that second part might encourage other package
managers to make use of it.


        Stefan

RE: bug#30854: 27.0.50; Speeding up package.el startup

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

Personally I would Peter it out of "normal" session completely. In my opinion package administration is relatively rarely performed operation. What I mean with that is that, compared to something like some other functionality, package.el functions are called very few times and on user demand, I.e. they are not performance critical.

So package can be initialized when user asks for some of it "entrance " functions, via auotoload. What has to be done is of course bring in paths and auotoload into user session. That was my idea with merging auotoloads and paths in my experiment.


-------- Originalmeddelande --------
Från: Stefan Kangas <stefankangas@gmail.com>
Datum: 2020-12-19 19:24 (GMT+01:00)
Till: Stefan Monnier <monnier@iro.umontreal.ca>, arthur.miller@live.com
Kopia: emacs-devel@gnu.org
Ämne: Re: bug#30854: 27.0.50; Speeding up package.el startup

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Maybe the difference is just due to loading `package`?
> IOW, what happens if you add (require 'package) to your "Quick start
> disabled" version?  Does the startup time go back up to ~0.28?

Why not just preload package.el?  A good 75 percent of our users seem to
use it, according to the Emacs Survey.

28.8 %  built-in package.el
46.3 %  use-package [uses package.el behind the scenes]
------
75.1 %  total

Source: https://emacssurvey.org/2020/

[-- Attachment #2: Type: text/html, Size: 2219 bytes --]

RE: bug#30854: 27.0.50; Speeding up package.el startup

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Ok. I'll do changes tomorrow and test. Requiring package.el will bring start up time up, will be interesting to see by how much.


-------- Originalmeddelande --------
Från: Stefan Kangas <stefankangas@gmail.com>
Datum: 2020-12-19 19:24 (GMT+01:00)
Till: Stefan Monnier <monnier@iro.umontreal.ca>, arthur.miller@live.com
Kopia: emacs-devel@gnu.org
Ämne: Re: bug#30854: 27.0.50; Speeding up package.el startup

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Maybe the difference is just due to loading `package`?
> IOW, what happens if you add (require 'package) to your "Quick start
> disabled" version?  Does the startup time go back up to ~0.28?

Why not just preload package.el?  A good 75 percent of our users seem to
use it, according to the Emacs Survey.

28.8 %  built-in package.el
46.3 %  use-package [uses package.el behind the scenes]
------
75.1 %  total

Source: https://emacssurvey.org/2020/

[-- Attachment #2: Type: text/html, Size: 1667 bytes --]

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Pankaj Jangid]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> But yes, I think it would make a lot of sense to split package.el into
> the part that manages installation/updates/... and the part that just
> activates the packages installed and to preload the second part.  Also
> exposing the API of that second part might encourage other package
> managers to make use of it.

I think something like ‘use-package’ will be part of next major
release. At that time we can roll out a package installation part for
use-package to depend upon. And of course for backward compatibility of
user scripts.

Yes package installation/updates should be independent of package
loading infrastruction.

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Maybe the difference is just due to loading `package`?
>>> IOW, what happens if you add (require 'package) to your "Quick start
>>> disabled" version?  Does the startup time go back up to ~0.28?
>> Why not just preload package.el?
>
> First, I'd like to understand if that's indeed the source of the
> timing difference.
>
> But yes, I think it would make a lot of sense to split package.el into the part
> that manages installation/updates/... and the part that just activates
> the packages installed and to preload the second part.
> Also exposing the API of that second part might encourage other package
> managers to make use of it.
>
>
>         Stefan
It seems you were correct.

I just added (require 'package) and the startup time went t0 0.29+; I
had my baked int autoloads file, so that is why it is probably somewhat
higher since it is doing extra now. Measured several times and it didn't
go below 0.29+ secs. You can see below en excerpt of init.el (just for
the illustration):


;;; init.el -*- lexical-binding: t; -*-
;;
;; This file is machine generated by init-file generator, don't edit
;; manually, edit instead file init.org and generate new init file from it

(defvar old-file-name-handler file-name-handler-alist)
(setq file-name-handler-alist nil)

(let ((default-directory  (expand-file-name "lisp" user-emacs-directory)))
      (normal-top-level-add-to-load-path '("."))
      (normal-top-level-add-subdirs-to-load-path))

(define-prefix-command 'C-z-map)
(global-set-key (kbd "C-z") 'C-z-map)
(define-prefix-command 'C-f-map)
(global-set-key (kbd "C-f") 'C-f-map)
(global-unset-key (kbd "C-v"))
(require 'package)
(setq load-path (append 
'("/home/arthur/.emacs.d/elpa/yasnippet-20200604.246" "/home/arthur/.emacs.d/elpa/wrap-region-20140117.720" "/home/arthur/.emacs.d/elpa/which-key-20200908.2301" "/home/arthur/.emacs.d/elpa/tablist-20200427.2205" "/home/arthur/.emacs.d/elpa/spinner-1.7.3" "/home/arthur/.emacs.d/elpa/solarized-theme-20201207.1431" "/home/arthur/.emacs.d/elpa/s-20180406.808" "/home/arthur/.emacs.d/elpa/projectile-20201210.927" "/home/arthur/.emacs.d/elpa/pos-tip-20191227.1356" "/home/arthur/.emacs.d/elpa/popup-20200610.317" "/home/arthur/.emacs.d/elpa/pkg-info-20150517.1143" "/home/arthur/.emacs.d/elpa/pdf-tools-20200512.1524" "/home/arthur/.emacs.d/elpa/org-pdftools-20200929.2241" "/home/arthur/.emacs.d/elpa/org-noter-pdftools-20200929.2241" "/home/arthur/.emacs.d/elpa/org-noter-20191020.1212" "/home/arthur/.emacs.d/elpa/org-20201216" "/home/arthur/.emacs.d/elpa/nov-20201207.3" "/home/arthur/.emacs.d/elpa/modern-cpp-font-lock-20200530.1010" "/home/arthur/.emacs.d/elpa/math-symbol-lists-20200131.2333" "/home/arthur/.emacs.d/elpa/markdown-mode-20201211.329" "/home/arthur/.emacs.d/elpa/lv-20200507.1518" "/home/arthur/.emacs.d/elpa/lsp-ui-20201209.332" "/home/arthur/.emacs.d/elpa/lsp-mode-20201211.1939" "/home/arthur/.emacs.d/elpa/imenu-anywhere-20190512.1939" "/home/arthur/.emacs.d/elpa/hydra-20201115.1055" "/home/arthur/.emacs.d/elpa/ht-20201119.518" "/home/arthur/.emacs.d/elpa/helpful-20201012.614" "/home/arthur/.emacs.d/elpa/helm-xref-20201004.1817" "/home/arthur/.emacs.d/elpa/helm-swoop-20200814.448" "/home/arthur/.emacs.d/elpa/helm-projectile-20201208.1837" "/home/arthur/.emacs.d/elpa/helm-org-20200311.633" "/home/arthur/.emacs.d/elpa/helm-make-20200620.27" "/home/arthur/.emacs.d/elpa/helm-lsp-20200910.518" "/home/arthur/.emacs.d/elpa/helm-ls-git-20200519.912" "/home/arthur/.emacs.d/elpa/helm-flyspell-20170210.1901" "/home/arthur/.emacs.d/elpa/helm-emms-20201205.1635" "/home/arthur/.emacs.d/elpa/helm-dired-history-20170524.1046" "/home/arthur/.emacs.d/elpa/helm-descbinds-20190501.935" "/home/arthur/.emacs.d/elpa/helm-core-20201202.907" "/home/arthur/.emacs.d/elpa/helm-c-yasnippet-20201118.1009" "/home/arthur/.emacs.d/elpa/helm-ag-20200915.1650" "/home/arthur/.emacs.d/elpa/helm-20201211.1044" "/home/arthur/.emacs.d/elpa/google-c-style-20180130.1736" "/home/arthur/.emacs.d/elpa/gnupg" "/home/arthur/.emacs.d/elpa/flycheck-20201214.2154" "/home/arthur/.emacs.d/elpa/f-20191110.1357" "/home/arthur/.emacs.d/elpa/expand-region-20200304.1839" "/home/arthur/.emacs.d/elpa/esxml-20201130.809" "/home/arthur/.emacs.d/elpa/esup-20200814.1400" "/home/arthur/.emacs.d/elpa/epl-20180205.2049" "/home/arthur/.emacs.d/elpa/emms-20201105.2351" "/home/arthur/.emacs.d/elpa/elisp-refs-20200815.2357" "/home/arthur/.emacs.d/elpa/dired-subtree-20180922.1615" "/home/arthur/.emacs.d/elpa/dired-hacks-utils-20201005.2318" "/home/arthur/.emacs.d/elpa/diminish-20191127.1326" "/home/arthur/.emacs.d/elpa/dash-functional-20200617.702" "/home/arthur/.emacs.d/elpa/dash-20200803.1520" "/home/arthur/.emacs.d/elpa/company-quickhelp-20201208.2308" "/home/arthur/.emacs.d/elpa/company-math-20200131.2337" "/home/arthur/.emacs.d/elpa/company-lsp-20190612.1553" "/home/arthur/.emacs.d/elpa/company-c-headers-20190825.1631" "/home/arthur/.emacs.d/elpa/company-20201214.1620" "/home/arthur/.emacs.d/elpa/beacon-20190104.1931" "/home/arthur/.emacs.d/elpa/auto-package-update-20200826.2227" "/home/arthur/.emacs.d/elpa/async-20200809.501") load-path))
(let ((load-true-file-name "/home/arthur/.emacs.d/elpa/yasnippet-20200604.246/yasnippet-autoloads.el")(load-file-name "/home/arthur/.emacs.d/elpa/yasnippet-20200604.246/yasnippet-autoloads.el"))

(autoload 'yas-minor-mode "yasnippet" "\
Toggle YASnippet mode.

...

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

>> But yes, I think it would make a lot of sense to split package.el into the part
>> that manages installation/updates/... and the part that just activates
>> the packages installed and to preload the second part.
>> Also exposing the API of that second part might encourage other package
>> managers to make use of it.
>>
>>
>>         Stefan
> It seems you were correct.
> I just added (require 'package) and the startup time went t0 0.29+;

Thanks for confirming,

Would you be interested in trying to split package.el into the "core
part" needed for `package-activate-all` (and which we could then
preload), and the rest?


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> But yes, I think it would make a lot of sense to split package.el into the part
>>> that manages installation/updates/... and the part that just activates
>>> the packages installed and to preload the second part.
>>> Also exposing the API of that second part might encourage other package
>>> managers to make use of it.
>>>
>>>
>>>         Stefan
>> It seems you were correct.
>> I just added (require 'package) and the startup time went t0 0.29+;
>
> Thanks for confirming,
>
> Would you be interested in trying to split package.el into the "core
> part" needed for `package-activate-all` (and which we could then
> preload), and the rest?
>
>
>         Stefan
I have arrived to this from slightly different direction: I did some
optimization as described in Emacs Doom Faq, and just noticed quickstart
file took some time, and looked and it's contents; so I am not really
introduced in what package.el does. It will take me some time to get
into it.

In my own init I do rude directory-files listing to get all directories
in Elpa directory and I just add those to the path; but packages can be
installed in different places etc; so I will have to take a look at how it
is done and what is safe to do/assuem etc.

If it is no rush with this, I'll look at it tomorrow and see if I can do
it.

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

>> Would you be interested in trying to split package.el into the "core
>> part" needed for `package-activate-all` (and which we could then
>> preload), and the rest?
>>
>>
>>         Stefan
> I have arrived to this from slightly different direction: I did some
> optimization as described in Emacs Doom Faq, and just noticed quickstart
> file took some time, and looked and it's contents; so I am not really
> introduced in what package.el does. It will take me some time to get
> into it.
>
> In my own init I do rude directory-files listing to get all directories
> in Elpa directory and I just add those to the path; but packages can be
> installed in different places etc; so I will have to take a look at how it
> is done and what is safe to do/assuem etc.

What I'm proposing doesn't involve any of that: no need to
reimplement anything.
Just look at the existing code and move the relevant parts to
a separate file.


        Stefan

RE: bug#30854: 27.0.50; Speeding up package.el startup

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1393 bytes --]

Yeah, I know, just saying I don't know how package.el does what it does. Will take a peek and look if I understand it.


-------- Originalmeddelande --------
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Datum: 2020-12-20 18:16 (GMT+01:00)
Till: Arthur Miller <arthur.miller@live.com>
Kopia: Stefan Kangas <stefankangas@gmail.com>, emacs-devel@gnu.org
Ämne: Re: bug#30854: 27.0.50; Speeding up package.el startup

>> Would you be interested in trying to split package.el into the "core
>> part" needed for `package-activate-all` (and which we could then
>> preload), and the rest?
>>
>>
>>         Stefan
> I have arrived to this from slightly different direction: I did some
> optimization as described in Emacs Doom Faq, and just noticed quickstart
> file took some time, and looked and it's contents; so I am not really
> introduced in what package.el does. It will take me some time to get
> into it.
>
> In my own init I do rude directory-files listing to get all directories
> in Elpa directory and I just add those to the path; but packages can be
> installed in different places etc; so I will have to take a look at how it
> is done and what is safe to do/assuem etc.

What I'm proposing doesn't involve any of that: no need to
reimplement anything.
Just look at the existing code and move the relevant parts to
a separate file.


        Stefan


[-- Attachment #2: Type: text/html, Size: 2226 bytes --]

RE: bug#30854: 27.0.50; Speeding up package.el startup

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

I don't think there is anything needed to refactor out of package.el; I don't see what is relevant to Emacs startup. As I see the parts for quickstart are just for creation of quickstart file. I maybe misunderstand, I am just glancing on my phone

Is there any reason to require package.el at startup at all? I am not sure where do I find how Emacs reads startup files and how entire startup goes, will have to investigate :-).

Btw, seems like quick starts does similar as I did, takes all auotoloads regardless if a package is activated or not. Maybe it could concatenatejust activated packages only?


-------- Originalmeddelande --------
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Datum: 2020-12-20 18:16 (GMT+01:00)
Till: Arthur Miller <arthur.miller@live.com>
Kopia: Stefan Kangas <stefankangas@gmail.com>, emacs-devel@gnu.org
Ämne: Re: bug#30854: 27.0.50; Speeding up package.el startup

>> Would you be interested in trying to split package.el into the "core
>> part" needed for `package-activate-all` (and which we could then
>> preload), and the rest?
>>
>>
>>         Stefan
> I have arrived to this from slightly different direction: I did some
> optimization as described in Emacs Doom Faq, and just noticed quickstart
> file took some time, and looked and it's contents; so I am not really
> introduced in what package.el does. It will take me some time to get
> into it.
>
> In my own init I do rude directory-files listing to get all directories
> in Elpa directory and I just add those to the path; but packages can be
> installed in different places etc; so I will have to take a look at how it
> is done and what is safe to do/assuem etc.

What I'm proposing doesn't involve any of that: no need to
reimplement anything.
Just look at the existing code and move the relevant parts to
a separate file.


        Stefan


[-- Attachment #2: Type: text/html, Size: 2816 bytes --]

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

> Is there any reason to require package.el at startup at all?

Yes: `package-activate-all` (called from `startup.el`).


        Stefan

RE: bug#30854: 27.0.50; Speeding up package.el startup

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 1652 bytes --]

Only thing it would need to refactor out is the packuage-quickstart variable. It would have to be somewhere where Emacs can find it so it can generate quickstart file if it dies not exists. And if it does not exists it can require package.el and create file, otherwise it is just to proceed with loading the quickstart file?

Is it that simple or I misunderstand something?



-------- Originalmeddelande --------
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Datum: 2020-12-20 18:16 (GMT+01:00)
Till: Arthur Miller <arthur.miller@live.com>
Kopia: Stefan Kangas <stefankangas@gmail.com>, emacs-devel@gnu.org
Ämne: Re: bug#30854: 27.0.50; Speeding up package.el startup

>> Would you be interested in trying to split package.el into the "core
>> part" needed for `package-activate-all` (and which we could then
>> preload), and the rest?
>>
>>
>>         Stefan
> I have arrived to this from slightly different direction: I did some
> optimization as described in Emacs Doom Faq, and just noticed quickstart
> file took some time, and looked and it's contents; so I am not really
> introduced in what package.el does. It will take me some time to get
> into it.
>
> In my own init I do rude directory-files listing to get all directories
> in Elpa directory and I just add those to the path; but packages can be
> installed in different places etc; so I will have to take a look at how it
> is done and what is safe to do/assuem etc.

What I'm proposing doesn't involve any of that: no need to
reimplement anything.
Just look at the existing code and move the relevant parts to
a separate file.


        Stefan


[-- Attachment #2: Type: text/html, Size: 2571 bytes --]

RE: bug#30854: 27.0.50; Speeding up package.el startup

[arthur miller]

[-- Attachment #1: Type: text/plain, Size: 519 bytes --]

Aha. Ok. Thx.



Skickat från min Samsung Galaxy-smartphone.



-------- Originalmeddelande --------
Från: Stefan Monnier <monnier@iro.umontreal.ca>
Datum: 2020-12-21 00:44 (GMT+01:00)
Till: arthur miller <arthur.miller@live.com>
Kopia: Stefan Kangas <stefankangas@gmail.com>, emacs-devel@gnu.org
Ämne: Re: bug#30854: 27.0.50; Speeding up package.el startup

> Is there any reason to require package.el at startup at all?

Yes: `package-activate-all` (called from `startup.el`).


        Stefan


[-- Attachment #2: Type: text/html, Size: 1414 bytes --]

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

[-- Attachment #1: Type: text/plain, Size: 691 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Is there any reason to require package.el at startup at all?
>
> Yes: `package-activate-all` (called from `startup.el`).

See if this is acceptable. I tried to make as few chagnes to package.el
as possible, and I didn't wont to introduce a new file, so I just put
the relevant files into startup.el.

I am not sure about package--activated; if quicstart should set it to 't
or not. I don't think is needed, and am not sure if it is straight wrong
to do too in quickstart, so I don't do that.

Seems to work for me, but I am never sure with Emacs :-). If that this
change is acceptable I can write something short for the news file too.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: quickstart.patch --]
[-- Type: text/x-patch, Size: 5728 bytes --]

From 8c80ed0349a3c505d8410480885ffb46a4417ebc Mon Sep 17 00:00:00 2001
From: Arthur Miller <arthur.miller@live.com>
Date: Mon, 21 Dec 2020 17:10:59 +0100
Subject: [PATCH] Refactored parts of package.el to startup.el for faster
 quickstart.

---
 lisp/emacs-lisp/package.el | 17 ---------
 lisp/startup.el            | 73 +++++++++++++++++++++++++-------------
 2 files changed, 49 insertions(+), 41 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index fa93ffd0cc..36c1787216 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -609,11 +609,6 @@ package-alist
 loaded and/or activated, customize `package-load-list'.")
 (put 'package-alist 'risky-local-variable t)
 
-(defvar package-activated-list nil
-  ;; FIXME: This should implicitly include all builtin packages.
-  "List of the names of currently activated packages.")
-(put 'package-activated-list 'risky-local-variable t)
-
 ;;;; Populating `package-alist'.
 
 ;; The following functions are called on each installed package by
@@ -1574,10 +1569,6 @@ package-read-all-archive-contents
 (defvar package--initialized nil
   "Non-nil if `package-initialize' has been run.")
 
-;;;###autoload
-(defvar package--activated nil
-  "Non-nil if `package-activate-all' has been run.")
-
 ;;;###autoload
 (defun package-initialize (&optional no-activate)
   "Load Emacs Lisp packages, and activate them.
@@ -1608,8 +1599,6 @@ package-initialize
   ;; `package--initialized' is t.
   (package--build-compatibility-table))
 
-(defvar package-quickstart-file)
-
 ;;;###autoload
 (defun package-activate-all ()
   "Activate all installed packages.
@@ -4034,12 +4023,6 @@ package-quickstart
   :type 'boolean
   :version "27.1")
 
-(defcustom package-quickstart-file
-  (locate-user-emacs-file "package-quickstart.el")
-  "Location of the file used to speed up activation of packages at startup."
-  :type 'file
-  :version "27.1")
-
 (defun package--quickstart-maybe-refresh ()
   (if package-quickstart
       ;; FIXME: Delay refresh in case we're installing/deleting
diff --git a/lisp/startup.el b/lisp/startup.el
index b1128f6d02..25f949bf44 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1224,34 +1224,59 @@ command-line
       "early-init.el"
       startup-init-directory)))
   (setq early-init-file user-init-file)
-
+  
+  (defcustom package-quickstart-file
+    (locate-user-emacs-file "package-quickstart.el")
+    "Location of the file used to speed up activation of packages at startup."
+    :type 'file
+    :version "27.1")
+  
+  (defvar package--activated nil
+    "Non-nil if `package-activate-all' has been run.")
+
+  (defvar package-activated-list nil
+    ;; FIXME: This should implicitly include all builtin packages.
+    "List of the names of currently activated packages.")
+  (put 'package-activated-list 'risky-local-variable t)
+  
   ;; If any package directory exists, initialize the package system.
   (and user-init-file
        package-enable-at-startup
        (not (bound-and-true-p package--activated))
-       (catch 'package-dir-found
-	 (let (dirs)
-	   (if (boundp 'package-directory-list)
-	       (setq dirs package-directory-list)
-	     (dolist (f load-path)
-	       (and (stringp f)
-		    (equal (file-name-nondirectory f) "site-lisp")
-		    (push (expand-file-name "elpa" f) dirs))))
-	   (push (if (boundp 'package-user-dir)
-		     package-user-dir
-		   (locate-user-emacs-file "elpa"))
-		 dirs)
-	   (dolist (dir dirs)
-	     (when (file-directory-p dir)
-	       (dolist (subdir (directory-files dir))
-		 (when (let ((subdir (expand-file-name subdir dir)))
-                         (and (file-directory-p subdir)
-                              (file-exists-p
-                               (expand-file-name
-                                (package--description-file subdir)
-                                subdir))))
-		   (throw 'package-dir-found t)))))))
-       (package-activate-all))
+         ;; If quickstart file exists, just load it
+         (if (file-readable-p (expand-file-name package-quickstart-file))
+             ;; Skip load-source-file-function which would slow us down by a factor
+             ;; 2 (this assumes we were careful to save this file so it doesn't need
+             ;; any decoding).
+             (let ((load-source-file-function nil))
+               ;;(setq package--activated t)
+               (load package-quickstart-file nil 'nomessage))
+           ;; When quickstart file is not found, we do business as
+           ;; usual
+           (progn
+             (catch 'package-dir-found
+	       (let (dirs)
+	         (if (boundp 'package-directory-list)
+	             (setq dirs package-directory-list)
+	           (dolist (f load-path)
+	             (and (stringp f)
+		          (equal (file-name-nondirectory f) "site-lisp")
+		          (push (expand-file-name "elpa" f) dirs))))
+	         (push (if (boundp 'package-user-dir)
+		           package-user-dir
+		         (locate-user-emacs-file "elpa"))
+		       dirs)
+	         (dolist (dir dirs)
+	           (when (file-directory-p dir)
+	             (dolist (subdir (directory-files dir))
+		       (when (let ((subdir (expand-file-name subdir dir)))
+                               (and (file-directory-p subdir)
+                                    (file-exists-p
+                                     (expand-file-name
+                                      (package--description-file subdir)
+                                      subdir))))
+		         (throw 'package-dir-found t)))))))
+             (package-activate-all))))
 
   ;; Make sure window system's init file was loaded in loadup.el if
   ;; using a window system.
-- 
2.29.2

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Kangas]

Arthur Miller <arthur.miller@live.com> writes:

>> Yes: `package-activate-all` (called from `startup.el`).
>
> See if this is acceptable. I tried to make as few chagnes to package.el
> as possible, and I didn't wont to introduce a new file, so I just put
> the relevant files into startup.el.

FWIW, I think a new file would be better.  We could then also move the
bits and pieces from subr.el to this more logical place.

(On a related note, I'm not sure that making `package-activate-all'
disappear is TRT.  I think it is useful to keep that function.)

> I am not sure about package--activated; if quicstart should set it to 't
> or not. I don't think is needed, and am not sure if it is straight wrong
> to do too in quickstart, so I don't do that.

It is still useful if we keep `package-activate-all', I think.

But my preference would in any case be to make any functional changes
separately from this refactoring.

> Seems to work for me, but I am never sure with Emacs :-).

Does your patch avoid loading package.el?

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Kangas <stefankangas@gmail.com> writes:

> Arthur Miller <arthur.miller@live.com> writes:
>
>>> Yes: `package-activate-all` (called from `startup.el`).
>>
>> See if this is acceptable. I tried to make as few chagnes to package.el
>> as possible, and I didn't wont to introduce a new file, so I just put
>> the relevant files into startup.el.
>
> FWIW, I think a new file would be better.  We could then also move the
> bits and pieces from subr.el to this more logical place.
Aha. I didn't know there are pieces in subr.el also in play here. I am
not so well introduced into internals.

> (On a related note, I'm not sure that making `package-activate-all'
> disappear is TRT.  I think it is useful to keep that function.)
When you say "to keep that function" you mean to run it at startup, even
when quickstart is activated or you mean to keep it at all as
functionality in package.el?

For the latter, I haven't removed anything. When quickstart is active,
and the package-quickstart.el exists, then we don't need to run it. We
just need to make Emacs aware of autoloads and load-paths where to find
those files refered by autoloads. That is qhat package-quickstart.el
does, so we can just loaded. At least what I think. Seems to work.

User will have to run package-initialize at later time for any
package.el functions (refreshing contenst, installing, unisntalling
etc). For me it is quite acceptable cost for the performance gain.

>> I am not sure about package--activated; if quicstart should set it to 't
>> or not. I don't think is needed, and am not sure if it is straight wrong
>> to do too in quickstart, so I don't do that.
>
> It is still useful if we keep `package-activate-all', I think.
>
> But my preference would in any case be to make any functional changes
> separately from this refactoring.
This does not do any functional change other but avoiding to load package.el
when quickstart is active. If a user would really like to have
package.el loaded from the start time, than it is just to call
(package-initialize) in user init file?

> Does your patch avoid loading package.el?
Yes. When package-quickstart.el exists in user-emacs-directory.
If that file does not exists then everything is loaded just as before,
and (package-activate-all) will be run as normally.

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Arthur Miller <arthur.miller@live.com> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> Arthur Miller <arthur.miller@live.com> writes:
>>
>>>> Yes: `package-activate-all` (called from `startup.el`).
>>>
>>> See if this is acceptable. I tried to make as few chagnes to package.el
>>> as possible, and I didn't wont to introduce a new file, so I just put
>>> the relevant files into startup.el.
>>
>> FWIW, I think a new file would be better.  We could then also move the
>> bits and pieces from subr.el to this more logical place.
> Aha. I didn't know there are pieces in subr.el also in play here. I am
> not so well introduced into internals.
>
>> (On a related note, I'm not sure that making `package-activate-all'
>> disappear is TRT.  I think it is useful to keep that function.)
> When you say "to keep that function" you mean to run it at startup, even
> when quickstart is activated or you mean to keep it at all as
> functionality in package.el?
>
> For the latter, I haven't removed anything. When quickstart is active,
> and the package-quickstart.el exists, then we don't need to run it. We
> just need to make Emacs aware of autoloads and load-paths where to find
> those files refered by autoloads. That is qhat package-quickstart.el
> does, so we can just loaded. At least what I think. Seems to work.
>
> User will have to run package-initialize at later time for any
> package.el functions (refreshing contenst, installing, unisntalling
> etc). For me it is quite acceptable cost for the performance gain.
>
>>> I am not sure about package--activated; if quicstart should set it to 't
>>> or not. I don't think is needed, and am not sure if it is straight wrong
>>> to do too in quickstart, so I don't do that.
>>
>> It is still useful if we keep `package-activate-all', I think.
>>
>> But my preference would in any case be to make any functional changes
>> separately from this refactoring.
> This does not do any functional change other but avoiding to load package.el
> when quickstart is active. If a user would really like to have
> package.el loaded from the start time, than it is just to call
> (package-initialize) in user init file?
>
>> Does your patch avoid loading package.el?
> Yes. When package-quickstart.el exists in user-emacs-directory.
> If that file does not exists then everything is loaded just as before,
> and (package-activate-all) will be run as normally.
I forgott also to say,

I can also udate "user" functions of package.el (package-list,
package-refresh-contents etc) to check if package.el is initialized, and
if not they could call package-initialize on behalf of user, so end users
does not need to call pacakge-initialize themselves, neither in init
file or interactively. As I see now, some of them already do so, some don't.

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

>> (On a related note, I'm not sure that making `package-activate-all'
>> disappear is TRT.  I think it is useful to keep that function.)
> When you say "to keep that function" you mean to run it at startup, even
> when quickstart is activated or you mean to keep it at all as
> functionality in package.el?

`package-activate-all` is not an internal function (e.g. it's
documented in `etc/NEWS.27`) so we have to preserve it.

It can be very useful also for users who want to have more control about
which packages are activated when (e.g. activating packages before the
end of `early-init.el` or from within `init.el`).

> This does not do any functional change other but avoiding to load
> package.el when quickstart is active.  If a user would really like to
> have package.el loaded from the start time, than it is just to call
> (package-initialize) in user init file?

`package-initialize` is a thing of the past.  Nowadays users should
never need to call it (that doesn't mean it's never useful: it's used
internally in various `package.el` functions and there may still be
cases where it's needed outside of `package.el` but these should be
considered as bugs, IMO).

>> Does your patch avoid loading package.el?
> Yes. When package-quickstart.el exists in user-emacs-directory.
> If that file does not exists then everything is loaded just as before,
> and (package-activate-all) will be run as normally.

Oh, indeed, that's another option I had not considered.

I was thinking of arranging for `package.el` not to be loaded regardless
of the use of `package-quickstart` (i.e. preload all of
`package-activate-all`), but maybe you're right and the only case where
it's worthwhile avoiding to load `package.el` is when we use
`package-quickstart`.


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

> I can also udate "user" functions of package.el (package-list,
> package-refresh-contents etc) to check if package.el is initialized, and
> if not they could call package-initialize on behalf of user, so end users
> does not need to call pacakge-initialize themselves, neither in init
> file or interactively. As I see now, some of them already do so, some don't.

Indeed some calls to `package-initialize` are probably still missing.
[ I don't know what you mean by `package-list`, OTOH.  ]
Sometimes rather than add a call to it, it's better to rework the code
so it doesn't require `package-initialize` to run before, tho.

E.g. I can't see why `package-refresh-contents` would need
`package-initialize` to run before it can do its job.


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> (On a related note, I'm not sure that making `package-activate-all'
>>> disappear is TRT.  I think it is useful to keep that function.)
>> When you say "to keep that function" you mean to run it at startup, even
>> when quickstart is activated or you mean to keep it at all as
>> functionality in package.el?
>
> `package-activate-all` is not an internal function (e.g. it's
> documented in `etc/NEWS.27`) so we have to preserve it.
>
> It can be very useful also for users who want to have more control about
> which packages are activated when (e.g. activating packages before the
> end of `early-init.el` or from within `init.el`).
I have to say I am little bit unsure of how this works.

Maybe I missunderstand how it works; but can user activate packages
before the end of early-init.el if they don't require package.el?
Wouldn't it be contradictory to the idea of quickstart?

Docs says new file has to be generated every time activation of a
package is changed:

  "Precompute activation actions to speed up startup.
This requires the use of `package-quickstart-refresh' every time the
activations need to be changed, such as when `package-load-list' is
modified."

>> This does not do any functional change other but avoiding to load
>> package.el when quickstart is active.  If a user would really like to
>> have package.el loaded from the start time, than it is just to call
>> (package-initialize) in user init file?
>
> `package-initialize` is a thing of the past.  Nowadays users should
> never need to call it (that doesn't mean it's never useful: it's used
> internally in various `package.el` functions and there may still be
> cases where it's needed outside of `package.el` but these should be
> considered as bugs, IMO).
Ok; I didn't know. I just checked some interactive functions without
calling package-initialize in a fresh Emacs process and they work, so
it's super.

>>> Does your patch avoid loading package.el?
>> Yes. When package-quickstart.el exists in user-emacs-directory.
>> If that file does not exists then everything is loaded just as before,
>> and (package-activate-all) will be run as normally.
>
> Oh, indeed, that's another option I had not considered.
>
> I was thinking of arranging for `package.el` not to be loaded regardless
> of the use of `package-quickstart` (i.e. preload all of
> `package-activate-all`), but maybe you're right and the only case where
> it's worthwhile avoiding to load `package.el` is when we use
> `package-quickstart`.

I just hacked things with the least effort possible :-). This was the
smallest change I was able to come up with atm.

Are you sure it is worth the efforth? It would move things in two
different places, would need more care to work with later on.

I don't see why someone would not want to use quickstart? So as a
thought: why not make quickstart normal mode how package loading works?

We could move quickstart file into package archives directory, the
"elpa" dir in .emacs.d and when packages are installed instead of emitting
autoloads.el in every directory, we could append them into an autoloads.el
database, i.e. quickstart file.

That would need some extra care to remove and update those correctly
when a package is uninstalled and updated; but if we insert start and
end markers for each package in an elisp comment, it wouldn't be
difficult to implemented.

Alternatively just scratch the file and re-generate on every
update. Does not seem to take awfull lot of time to generate it, so it
would be even easier strategy.

Is that too much or could it work?

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I can also udate "user" functions of package.el (package-list,
>> package-refresh-contents etc) to check if package.el is initialized, and
>> if not they could call package-initialize on behalf of user, so end users
>> does not need to call pacakge-initialize themselves, neither in init
>> file or interactively. As I see now, some of them already do so, some don't.
>
> Indeed some calls to `package-initialize` are probably still missing.
> [ I don't know what you mean by `package-list`, OTOH.  ]
> Sometimes rather than add a call to it, it's better to rework the code
> so it doesn't require `package-initialize` to run before, tho.
>
> E.g. I can't see why `package-refresh-contents` would need
> `package-initialize` to run before it can do its job.
I was just looking hastily through the list of autloaded interactive
functions, but you are correct, it does not need it.

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

> Maybe I missunderstand how it works; but can user activate packages
> before the end of early-init.el if they don't require package.el?

Of course, they can just call `package-activate-all`.

> Wouldn't it be contradictory to the idea of quickstart?

This is orthogonal to `package-quickstart`.

> I just hacked things with the least effort possible :-). This was the
> smallest change I was able to come up with atm.

Yes, I still have to sleep on it, I think, but I'm beginning to like the idea.

> I don't see why someone would not want to use quickstart?  So as a
> thought: why not make quickstart normal mode how package loading works?

Yes, I think we should.  But for that we need to make sure the
quickstart file is properly recreated every time it's needed.

There might also be situations where that can't be done, tho: e.g. if
`package-directory-list` includes a directory maintained by the
sysadmin, then we won't be told when packages get
installed/removed/upgraded in that directory.  To handle such a case
we'd have to either disable quickstart altogether or add some kind of
"freshness" check at startup before using the quickstart file.


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I don't see why someone would not want to use quickstart?  So as a
>> thought: why not make quickstart normal mode how package loading works?
>
> Yes, I think we should.  But for that we need to make sure the
> quickstart file is properly recreated every time it's needed.
>
> There might also be situations where that can't be done, tho: e.g. if
> `package-directory-list` includes a directory maintained by the
> sysadmin, then we won't be told when packages get
> installed/removed/upgraded in that directory.  To handle such a case
> we'd have to either disable quickstart altogether or add some kind of
> "freshness" check at startup before using the quickstart file.

Aha; I hope I understand correct:

Sysadmin can install packages for many users to share them, users have
read priviledge, not write of course. Sysadmin would install/uninstall
packages and a user would need to read autoloads.el from that shared
system dirctory, and then maybe even from it's own?

Could something like "site-autoloads.el" do? Similar to how
site-start.el works?

That file would be read only of course, and could be loaded before
user's autoloads are loaded. That way if user install same package but
different versions, user's would take precedence since it would override
site-autoloads. It would still be just two files to check for and
load. And hopefully path to it could be set in site-start.el or
elsewhere in setup so Emacs does not need to search entire load path to
find it?

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

> Sysadmin can install packages for many users to share them, users have
> read priviledge, not write of course. Sysadmin would install/uninstall
> packages and a user would need to read autoloads.el from that shared
> system dirctory, and then maybe even from it's own?

The users may have some of the packages installed in their
~/.emacs.d/elpa which take precedence over the versions installed by the
sysadmin, and also their `package-load-list` may disable some of the
sysadmin-installed packages.  So you can't have a single concatenated
autoloads file for all the sysadmin-installed packages shared between
all the users.

So I think instead the package-quickstart file should come with a bit of
extra freshness info, e.g. the `mtime` of all the directories in
`package-directory-list` when it was created, so we can skip the
quickstart and mark it "needs-refresh" when one of the directories has
been modified (by (un)installing a package).


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

I think we are having multiple problems here. For the one there is
problem of quickstart: bringing autoloads into emacs and package paths
to load-path, for the second it is management of packages and users
deciding which packages to activate or not and third is discovering when
new packages are avialable (in site packages).

>> Sysadmin can install packages for many users to share them, users have
>> read priviledge, not write of course. Sysadmin would install/uninstall
>> packages and a user would need to read autoloads.el from that shared
>> system dirctory, and then maybe even from it's own?
>
> The users may have some of the packages installed in their
> ~/.emacs.d/elpa which take precedence over the versions installed by the
> sysadmin, and also their `package-load-list` may disable some of the
> sysadmin-installed packages.  So you can't have a single concatenated
> autoloads file for all the sysadmin-installed packages shared between
> all the users.

Could this strategy work: "site-autoloads.el" is used just to create
"user-autoloads.el" (I am just making up names, hope it is clear what I
mean with them) on first start. User can now disable/enable whichever
from sysadmin installed ones, as if they were present in users packages,
and regenerate autoloads.el as the docs describe. Emacs checks mtime of
site-autoloads.el and if it is changed it recreates new autoloads file
with respect to disabled packages.

I am not sure how that would be implemented, but it is probably not
impossible :-).

>
> So I think instead the package-quickstart file should come with a bit of
> extra freshness info, e.g. the `mtime` of all the directories in
> `package-directory-list` when it was created, so we can skip the
> quickstart and mark it "needs-refresh" when one of the directories has
> been modified (by (un)installing a package).

You mean to automate, so the user does not need to run
package-quickstart-refresh manually? It means Emacs would need to
iterate all the directories in site packages and user packages somewhere
at some time; probably in some idle timer function or do you have
something else in mind?

If sysadmin updated site packages then autoloads for the site would
change, so we can just check that file, if we use concatenated
autoloads.el, and we do want to use it for the speed? User autoloads
are fixed automatically at time of package management.

That would work with above idea to use site autoloads just to create
user autoloads and that is not so far away how package.el works
now (conceptually).

Checking mtime of site autoloads would address discoverability
of sysadmin installed packages, user inactivating package and
recreateing autoloads would address package management and we would stil
have one concatenated autoload file per user to quickstart emacs.

It is one idea; I am just trying to think of something efficient.

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

> Could this strategy work: "site-autoloads.el" is used just to create
> "user-autoloads.el" (I am just making up names, hope it is clear what I
> mean with them) on first start. User can now disable/enable whichever
> from sysadmin installed ones, as if they were present in users packages,
> and regenerate autoloads.el as the docs describe. Emacs checks mtime of
> site-autoloads.el and if it is changed it recreates new autoloads file
> with respect to disabled packages.

The code in package.el already handles the case I described.
What doesn't work yet is to automatically detect when
`package-quickstart.el` needs to be recreated.

> You mean to automate, so the user does not need to run
> package-quickstart-refresh manually?

Exactly.  That's what needs to happen before we can consider enabling
`package-quickstart` by default.

> It means Emacs would need to iterate all the directories in site
> packages and user packages somewhere at some time; probably in some
> idle timer function or do you have something else in mind?

To be correct, it needs to be done before we load
`package-quickstart.el`.

> If sysadmin updated site packages then autoloads for the site would

There's no such thing as "autoloads for the site" currently.
But I don't think we need to create that either.
Installing/removing/upgrading a package will change the mtime of the
parent directory (i.e. the directory that's in
`package-directory-list`), so we just need to check the mtime of
those directories.  While there can easily be hundreds of packages,
`package-directory-list` is usually a short list, holding typically less
than 5 elements, so it should not impact startup time noticeably.


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Could this strategy work: "site-autoloads.el" is used just to create
>> "user-autoloads.el" (I am just making up names, hope it is clear what I
>> mean with them) on first start. User can now disable/enable whichever
>> from sysadmin installed ones, as if they were present in users packages,
>> and regenerate autoloads.el as the docs describe. Emacs checks mtime of
>> site-autoloads.el and if it is changed it recreates new autoloads file
>> with respect to disabled packages.
>
> The code in package.el already handles the case I described.
> What doesn't work yet is to automatically detect when
> `package-quickstart.el` needs to be recreated.
>
>> You mean to automate, so the user does not need to run
>> package-quickstart-refresh manually?
>
> Exactly.  That's what needs to happen before we can consider enabling
> `package-quickstart` by default.
>
>> It means Emacs would need to iterate all the directories in site
>> packages and user packages somewhere at some time; probably in some
>> idle timer function or do you have something else in mind?
>
> To be correct, it needs to be done before we load
> `package-quickstart.el`.
>
>> If sysadmin updated site packages then autoloads for the site would
>
> There's no such thing as "autoloads for the site" currently.
Yes, I know. That was a proposal to structure it that way; also to
introduce site-autoloads.el for potential sysadmins. Similar to
site-start/load/init.el.

> But I don't think we need to create that either.
> Installing/removing/upgrading a package will change the mtime of the
> parent directory (i.e. the directory that's in
> `package-directory-list`), so we just need to check the mtime of
> those directories.  While there can easily be hundreds of packages,
> `package-directory-list` is usually a short list, holding typically less
> than 5 elements, so it should not impact startup time noticeably.
Indeed; we are on same page there.

What I was thinking of was that autoloads would be generated when
packages are updated anyway, so checking just mtime for that file is
synonymous to checking directories in that list. Discovering change is
only needed for site installed packages. If there existed one central
autoload in system elpa directory, then autoloads for all those packages
would be calculated only once: when sysadmin update packages. User Emacs
could just check that file, and if it is newer then some last time stamp,
simply update the user autoloads. But what I am thinking of now is
that sysadmin my mount storage locations under different path than how
it is mounted for a user; so site autoloads would need to be emited with
relative paths, while user's would need to be absolute and updated when
user file is generated. Not hard but but I scratch the idea if there is
extra work performed anyway. Just check mtime of the repo directory,
and generate new user autoloads; works just fine too :-). 

For the user directories there is no need to check them, since autoloads
file would be touched automatically when packages are updated.

I have come up to think of a thing when quickstart is creating
autoloads. It currently loops through all packages avialable and
generates autoloads for them all, and adds them all to the path, it does
not seem to care about what user activated or not, at least how I
understand package-quickstart-refresh function.

If there was a blacklist of some kind, then it could be checked when
autoloads file is created and those packages simply not emitted to
autoloads so emacs does not read them on the start, or do I
missunderstand how active/inactive packages works? I think it would be
needed if user is to disable site installed packages.

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

> I have come up to think of a thing when quickstart is creating
> autoloads. It currently loops through all packages avialable and
> generates autoloads for them all, and adds them all to the path, it does
> not seem to care about what user activated or not, at least how I
> understand package-quickstart-refresh function.

I does: it processes the packages using the exact same code as
`package-activate-all`, so it obeys `package-load-list` (at least last
time I checked).


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I have come up to think of a thing when quickstart is creating
>> autoloads. It currently loops through all packages avialable and
>> generates autoloads for them all, and adds them all to the path, it does
>> not seem to care about what user activated or not, at least how I
>> understand package-quickstart-refresh function.
>
> I does: it processes the packages using the exact same code as
> `package-activate-all`, so it obeys `package-load-list` (at least last
> time I checked).

Via   (package-activate (car elt)) ?

Sorry I was looking through and trying to understand how it works, I
am not 100% how it does it's thing yet.

What is the workflow for the user to disable a package from loading? I
don't see any fields where this information is saved for the future.

Do you have any plan to implement this?

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

>> I does: it processes the packages using the exact same code as
>> `package-activate-all`, so it obeys `package-load-list` (at least last
>> time I checked).
>
> Via   (package-activate (car elt)) ?

Yes.

> What is the workflow for the user to disable a package from loading?

Set `package-load-list`.

> Do you have any plan to implement this?

Don't know what you're referring to.


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I does: it processes the packages using the exact same code as
>>> `package-activate-all`, so it obeys `package-load-list` (at least last
>>> time I checked).
>>
>> Via   (package-activate (car elt)) ?
>
> Yes.
>
>> What is the workflow for the user to disable a package from loading?
>
> Set `package-load-list`.
Is it up to user to serialize this list to disk for next session or
is it somehow done on the user behalf? If user disabled a package, and
generated new autoloads, and then there were new packages installed in
site repository, how would Emacs know which are disabled and which are
just new ones? There would be need for some kind of list/info that a
package is disabled so it is not loaded a new when packages are
refreshed from the site archive.

>> Do you have any plan to implement this?
>
> Don't know what you're referring to.

My suggestion:

1. let's implement auto creation of quickstart file
(can we rename it to package-autoloads.el or something more
descriptive?) without user needing to take action.

2. turn quicstart into opt-out instead of opt-in so more users test it
and can repport if there are problems

3. add suppoort for site archive(s)

4. add support for blacklist so user can ensure Emacs is not re-enabling
packages from site archive just because the archive got updated



1. is probably trivial, since package.el already can do that. It
just needs to call package-quickstart-refresh from few more places.

2. as well

3. will need some extra code, mostly or only in startup.el

4. will need some extra code in package.el, maybe in startup.el too, not sure


Just a suggestion.

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Stefan Monnier]

>> Set `package-load-list`.
> Is it up to user to serialize this list to disk for next session or
> is it somehow done on the user behalf?

It's assumed it's manually set in the user's init file or via Custom, yes.

For example, I locally disable AUCTeX as follows:

    (setq package-load-list '((auctex nil) all))

> If user disabled a package, and generated new autoloads, and then
> there were new packages installed in site repository, how would Emacs
> know which are disabled and which are just new ones?

If `package-load-list` says it's disabled, then it's disabled, and if
not, then it's not.  I guess I don't understand the question.

> 1. is probably trivial, since package.el already can do that.  It
> just needs to call package-quickstart-refresh from few more places.

Right.

> 2. as well

Indeed.

> 3. will need some extra code, mostly or only in startup.el

It also needs extra code in `package.el` when creating
`package-quickstart.el` to record the mtimes.

BTW, other cases where the `package-quickstart.el` needs to be recreated
is when the user changes `package-load-list` or
`package-directory-list`, so we may also want to remember the values of
those vars when creating the `package-quickstart.el`.  But this part is
more tricky: it can be perfectly normal for the users to set those vars
in their init file, i.e. after `package-quickstart.el` is loaded (it's
actually one of the benefits of using `package-quickstart.el`: you don't
need to set `package.el`'s vars in your `early-init.el`), so we should
not check those vars for freshness before loading
`package-quickstart.el` but rather later.

> 4. will need some extra code in package.el, maybe in startup.el too, not sure

AFAIK there's nothing to be done for 4.


        Stefan

Re: bug#30854: 27.0.50; Speeding up package.el startup

[Arthur Miller]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Set `package-load-list`.
>> Is it up to user to serialize this list to disk for next session or
>> is it somehow done on the user behalf?
>
> It's assumed it's manually set in the user's init file or via Custom, yes.
>
> For example, I locally disable AUCTeX as follows:
>
>     (setq package-load-list '((auctex nil) all))
>
>> If user disabled a package, and generated new autoloads, and then
>> there were new packages installed in site repository, how would Emacs
>> know which are disabled and which are just new ones?
>
> If `package-load-list` says it's disabled, then it's disabled, and if
> not, then it's not.  I guess I don't understand the question.
>
>> 1. is probably trivial, since package.el already can do that.  It
>> just needs to call package-quickstart-refresh from few more places.
>
> Right.
>
>> 2. as well
>
> Indeed.
>
>> 3. will need some extra code, mostly or only in startup.el
>
> It also needs extra code in `package.el` when creating
> `package-quickstart.el` to record the mtimes.
>
> BTW, other cases where the `package-quickstart.el` needs to be recreated
> is when the user changes `package-load-list` or
> `package-directory-list`, so we may also want to remember the values of
> those vars when creating the `package-quickstart.el`.  But this part is
> more tricky: it can be perfectly normal for the users to set those vars
> in their init file, i.e. after `package-quickstart.el` is loaded (it's
> actually one of the benefits of using `package-quickstart.el`: you don't
> need to set `package.el`'s vars in your `early-init.el`), so we should
> not check those vars for freshness before loading
> `package-quickstart.el` but rather later.
>
>> 4. will need some extra code in package.el, maybe in startup.el too, not sure
>
> AFAIK there's nothing to be done for 4.
I think I like 4. best so far.