Re: Loading a package applies automatically to future sessions?

[Stefan Monnier <monnier@IRO.UMontreal.CA>]

> (setq package-enable-at-startup nil)
> (load "~/.emacs.d/package-fastpath.el")
> (setq package--initialized t)

I think setting package--initialized to t is wrong, here (IIRC
package--initialized means that we've filled the tables keeping track
of which packages are available and such).  Not a problem when you're
just trying to test the speed of my proof-of-concept patch, of course.

> Unfortunately, I couldn't byte-compile my package-fastpath.el file due
> to some packages which already byte-compile their autoload files, such
> as SLIME.

Hmm... are you saying that my code ends up fetching code for
slime-autoloads.elc instead of slime-autoloads.el?
Or that slime-autoloads.el contains (pre)compiled code?

Also, I'm not 100% surprised that byte-compiling byte-compiled code would
fail, but I can't see any immediate reason why it should fail, so it's
quite possible to it'd be easy to make it work.

> Now, I do see some improvement in my startup time, but to reiterate, it
> is not huge.  When I run Emacs with my startup commented out,
> `emacs-init-time' yields 1.2 seconds.

Not quite sure what "my startup commented out" means.  I'll assume it
means a more or less empty ~/.emacs, or maybe the use of -Q ?
In any case I guess it means that 1.2s is the "speed of light", i.e. the
holy grail of package.el on your machine.

> When I run it with my startup intact, it yields 1.3 or 1.4
> seconds. This contrasts with my previous `emacs-init-time' which was
> 1.6 or 1.7 seconds, but as you can see, it's not a really
> big difference.

If you subtract the 1.2s taken by "other things", it means that your
config used to take 0.4-0.5s of startup time, and that the use of a big
precomputed autoloads file reduced that to 0.1-0.2s, so sped it up by
factor between 2.5 and 4, which I find a bit disappointing but still
respectable (and it depends which proportion was taken by
package-initialize vs which proportion was taken by your particular
customizations).

> It is possible that this sub-par time is due to something on my own
> system, but I'm not sure nor convinced of that. Moreover, Stefan's patch
> certainly can use some improvements and some notes on how to use his
> changes, as it is a bit vexing to figure it out by reading the diff ;)

Yes, it needs work.  It's just a quick experiment to see the
potential gains.

> I did notice a few bugs with Stefan's changes.  One bug was that
> `package-installed-p' no longer yields correct results on installed
> packages.

As it stands, my patch only pre-sets package-activated-list, whereas
package-installed-p requires other things set up by package-initialize.
If you don't set package--initialized to t, you'll see that
package-installed-p will rightfully complain that you haven't called
package-initialize.

I guess my patch could also add a setting for `package-alist` to the
package-fastpath.el, but I'm not sure I like the idea (it increases the
consequences of having an out-of-date package-fastpath.el).

> I believe that this is because `package-desc-p' is also broken
> by these changes. This broke some code in my startup which I used to
> check whether I need to install new packages.

Wouldn't it be better for your code to assume that if there's
a package-fastpath.el, then those checks have already been performed
(and to re-execute those checks in package-fastpath-refresh instead)?

Or maybe package-installed-p should be changed such that when it's
called with a symbol argument, it should first check if it's in
package-activated-list, and if it's not, then it should call
package-initialize before checking package-alist.

> Another bug was that some packages were placed in bad order in
> package-fastpath.el. In other words, if a dependency's autoloads are
> written to this file after its dependent package, the dependent package
> will err, saying that it couldn't require one of its dependencies.

Hmm... I thought this can't happen because the files are concatenated in
the same order that they are normally loaded by `package-initialize`.
I guess something doesn't work the way I thought it does.
Can you investigate to see when or even why it happens?

> As a side note and a bit of an opinion, Radon Rosborough made an
> interesting remark in one of his messages. He mentioned pip and how
> things are done in Python, which really struck me. You never really
> think about using a package in a language like Python or Ruby. You just
> `require' or `import' it and that's that. It's really simple and the
> amount of packages that you have never hurts the startup of the main
> program. I know that Emacs is a bit more complicated since it's more of
> a text editor than a language and we have somewhat more intricacies to
> worry about. But I think that this kind of a model is the kind we need
> to be headed for.

The fact that it's a text editor shouldn't make any difference in this
respect.  I'm not sure exactly what you mean by "the amount of packages
that you have never hurts the startup of the main program", I guess you
mean that program A is not slowed down when you install extra packages.

In Emacs the corresponding behavior happens if you install a package by
hand and don't activate/use it.  `package.el` by default tries to
auto-activate all your packages, so indeed it's slowed down by the mere
presence of extra packages.

I think a key element that lets Python and Ruby work that way is the
namespacing and its tight binding to the filesystem.  Currently, while
conventions are "similar" (e.g. function toto-titi is likely found in
file toto*.el in package toto), they're just vague conventions.

Making them more strict would indeed make it possible to change
`require` so that a (require 'toto-tata) will automatically look for
a file toto-tata.el in a package toto without having to "activate" that
package beforehand and it would let us have a "missing function handler"
which would automatically try to find the function toto-titi in one of
the files of the titi package (e.g. by activating this package).

This might make it possible to activate some packages lazily, but
There'd still be aspects of auto-activation that would go missing, tho:
e.g. registering jgraph-mode as a handler for *.jgr files.
So some packages would still need some form of auto-activation.

> A user should be able to have a million packages and not have to worry
> about the subsequent startup time. Heck, in my own Ruby installation I
> have 436 gems and I've never even thought about startup time till now.

The downside is that you have to explicitly `import` those gems when you
need them.  In Emacs you could do the same: call (package-initialize t)
in your ~/.emacs and then call `package-activate` when you need
a particular package.

> In any case, I think Stefan's ideas and proposed changes are a good idea
> and I am in much the same boat as John Wiegley in that I restart Emacs
> often enough that startup time gets on my nerves.

IIUC there are 1.2s of your startup time which are spent without even
running a single line of package.el code, so speeding up package.el
might not be good enough (at least it seems to me that if 1.6s is too
slow, then 1.2s is likely to also be too slow).

The usual answer in Emacs for those problem is "don't do that"
(i.e. use emacsclient instead).  I understand it's not always an option.


        Stefan