Friendly discussion about (package-initialize)

Friendly discussion about (package-initialize)

[Radon Rosborough]

Hi all,

I'd like to have a friendly discussion about (package-initialize).
Specifically, about this comment that I'm sure we've all seen from
time to time:

    ;; Added by Package.el.  This must come before configurations of
    ;; installed packages.  Don't delete this line.  If you don't want it,
    ;; just comment it out by adding a semicolon to the start of the line.
    ;; You may delete these explanatory comments.
    (package-initialize)

In this email, I'm going to explain why I think the current behavior
of package.el is flawed, and propose some alternative behaviors that I
believe would be better. If the community thinks that these
alternatives are reasonable, I would be interested in contributing
relevant patches. I know that this argument has the potential to be
contentious, so I'm going to try to be as objective as possible, and
stick strictly to the technical points.

Since this is a rather long email, I've arranged it into sections.
Feel free to skip or skim the parts that you already know. I've tried
to be quite verbose so as to create a comprehensive and authoritative
summary of the issue for people who may not be as familiar with Emacs
packaging.

################################################################################
# WHAT DOES PACKAGE.EL DO?

Since version 24.1 [1], Emacs has shipped with a built-in package
manager called package.el [2]. Like other package managers such as
Quelpa [3], Cask [4], el-get [5], Borg [6], and straight.el [7], the
primary function of package.el is to acquire Elisp code, preprocess it
(e.g. byte-compilation, autoload generation), and make it available to
Emacs.

The acquiry and preprocessing of Elisp code is not relevant to this
discussion. What is important is how the code is made available to
Emacs. The general process (called "activating" a package) is
two-fold:

* add the package's files to the load-path and Info-directory-list
* evaluate the package's autoloads, so that running an autoloaded
  function loads the appropriate files from the package

Importantly, these steps must be run every time Emacs is initialized.
Installing Elisp code on the filesystem does nothing, unless it is
somehow added to one of Emacs' default load-path entries. Before a
package is activated, attempts to `require' its features will fail,
and its autoloads will not be available.

Different package managers have different opinions on how and when
packages ought to be activated. For example, in straight.el [7],
packages are activated one at a time as they are referenced in the
user's init-file. However, the most common approach is for all
packages to be activated at the same time. This is what package.el
does; it provides a `package-initialize' function which fulfills this
purpose.

Therefore, absent any special treatment, one might expect a user's
init-file to go something like this:

    ... code to configure package.el ...
    (package-initialize '(... some list of packages ...))
    ... code to configure the packages that were initialized ...

But users do not typically think in this way. They expect that
installing a package is a permanent operation, and that no further
action is required in order for that package to continue to be
available. For this reason, package.el keeps track of which packages
have been installed already, and it does this outside the init-file
(in the state of the ~/.emacs.d/elpa directory). When
`package-initialize' is called, all the installed packages are
activated. You can request package.el to activate only a subset of the
installed packages, but this is not the usual mode of operation. Thus,
the user's init-file actually looks like this:

    ... code to configure package.el ...
    (package-initialize)
    ... code to configure the packages that were initialized ...

Notice that the call to `package-initialize' must come *after* any
code that configures package.el, but *before* any code that configures
the packages that were initialized. This is important! If you try to
configure package.el after `package-initialize', your configurations
will not affect how package activation takes place. And if you try to
configure packages before `package-initialize', then variables,
functions, and so on are not going to be properly defined. (This might
not be a problem, depending on how you handle your customizations. But
it certainly does make things more complicated.)

################################################################################
#### WHAT IS THE PROBLEM?

The problem is when users forget to initialize the package management
system before configuring their packages. In other words, their
init-file just looks like this:

    ... code to configure packages ...

with no mention of `package-initialize' anywhere. Obviously this will
not work, because the packages are never activated. The "correct"
solution is to just call `package-initialize' in the init-file. But it
is hard to teach users to do this, and the resulting errors have been
a common problem for new Emacs users [8].

Thus, there was a discussion [8] on emacs-devel about how to arrange
for things to work "correctly" in the absence of a user who knows what
they are doing. The eventual outcome of that discussion is the subject
of the next section.

################################################################################
#### WHAT IS THE CURRENT SOLUTION TO THE PROBLEM?

The initial attempt was to have Emacs call `package-initialize'
automatically during initialization. However, this is not so easy as
it sounds. If Emacs calls `package-initialize' before loading the
init-file, then users cannot customize package.el in their init-file
anymore. And if Emacs calls `package-initialize' after loading the
init-file, then users cannot customize their installed packages in
their init-file anymore, unless they cause their customizations to be
loaded after the end of initialization using `after-init-hook'.

The only reasonable way for things to be set up is for
`package-initialize' to be called in the user's init-file, but before
package customizations are done. Thus, a patch [9] was suggested that
causes Emacs to write a call to `package-initialize' into the user's
init-file at startup, if it was not there to begin with. This was
shortly merged and released in Emacs 25.1. Note that the behavior is
inhibited in 'emacs -Q' mode.

################################################################################
#### JUSTIFICATIONS AND COUNTERARGUMENTS

> So what are you saying is wrong with the current behavior?

It modifies the user's init-file without asking.

> What's wrong with modifying the user's init-file? After all, Custom
> has been doing that for decades.

The difference is that Custom only modifies your init-file when you
ask it to save your customizations. In other words, it only modifies
your init-file when you ask it to. On the other hand, package.el does
it at Emacs startup without confirmation!

> OK, so it's different than Custom. What's the problem, though?

There are several issues:

* Many users want to write their init-files by hand. If they want to
  initialize the package management system, they want to do it
  themselves, and they want to put in the code by hand so that they
  know it will run in the right place.
* Many users have a modular Emacs configuration in which init.el loads
  their other customization files, one of which is usually deals with
  the package management system (including calling
  `package-initialize'). The result is that package.el sticks a
  superfluous, duplicate `package-initialize' into the root init-file.
* It's confusing. New users have no idea why Emacs is modifying their
  init-file, and a proper explanation is not given.
* It's extremely error-prone. Modifying the init-file is an
  irreversible operation, and it means that any bugs in this part of
  package.el are immediately 100x more frustrating.
* Not everyone uses package.el. It's annoying to have to explicitly
  put in configuration to disable the existing package management
  system, if you want to use an alternative package manager.

> Slow down there! Let's take this one point at a time. You said that
> some users want to write their init-files by hand. But if they do
> that, then can't they just put a call to `package-initialize' in
> their init-files, and that will prevent the issue?

No, because (1) they might be using a modular Emacs configuration that
calls `package-initialize' somewhere other than init.el, and (2) they
might be using a package manager other than package.el, and therefore
have no need of calling `package-initialize' at all.

> Well, what's wrong with them just putting a commented-out call to
> `package-initialize' in their init-file, like the inserted comments
> suggest?

This is ugly and looks ridiculous. Is requiring a magic comment in the
init-file to prevent unnecessary code from being inserted on Emacs
startup really the best possible user experience?

> Then what would be a better user experience?

Let's wait until the ALTERNATIVES section for that :)

> Wait, you said that it was a problem if `package-initialize' was put
> somewhere other than in the init-file. But if it's called during
> startup, then package.el doesn't run the logic to insert the a call
> into the init-file. So what's the issue?

The problem is that if there is an error while loading the init-file,
then the call to `package-initialize' still gets inserted, if init
hadn't gotten to the point where the actual call to
`package-initialize' was supposed to happen. This is actually
terrible, since it immediately makes debugging much more complicated:
your init-file is being changed without your knowing it, while you're
trying to debug initialization!

> OK, but that only happens when there's an error during init. We
> could make it so that the `package-initialize' happens only after a
> successful init.

That doesn't solve the problem, because there are all kinds of
circumstances where you have a "successful" init even when the user's
init-file was not loaded. For example, many modular Emacs
configurations will sustain errors in one or more modules using
`condition-case', and report the errors as warnings for an improved
debugging experience. We're back to the beginning.

Besides, often one wants to test something in a plain Emacs but not in
'emacs -Q'. In this case, the call to `package-initialize' will still
get inserted.

> How often does this really happen?

At least 20 times in my use of Emacs, and I get progressively more
annoyed every time. The existence of threads like [10] [11] indicates
that there are other people who don't like this kind of behavior.

Regardless of how we set up this system, it will never be possible to
cover all the edge cases. Once in a while, there will be some
sufficiently unexpected use case where an errant `package-initialize'
gets inserted and screws something up. Since there are better
solutions (see ALTERNATIVES), I think it's best to sidestep the whole
problem by not taking this approach in the first place.

> You said the system was confusing. Why?

The comments that come with the inserted `package-initialize' tell the
user "don't delete this line" without explaining the consequences if
you do. The user inevitably will try deleting it, since they didn't
ask for it to be added in the first place. But it will come back next
time, prompting annoyance and consternation.

The comments also advise "this must come before configurations of
installed packages", without explaining why or anything about how
package activation works. Also, the whole idea of "if you don't want
this, comment it out, but don't remove it" is bizarre; I've never seen
anything like it in any other program's configuration system.

> Can't that be fixed by clearer comments?

Yes, but the fact remains that a program inserting text into its own
configuration file (except in the case where the program has a
semantic understanding of the whole file, like for YAML) is highly
nonstandard. I don't know of any other program that does something
like this. As I said before, there are ALTERNATIVES that are more
standard, and these will not violate the principle of least surprise
as badly.

> Why are you so concerned about bugs in package.el? Is that really
> likely?

I trust that the folks here on emacs-devel will do a good job in
preventing bugs from creeping in. But nevertheless, I think putting in
this kind of logic is simply a bad idea from a software engineering
perspective. There will always be bugs, and we want to minimize the
impact of the ones that inevitably do pop up. Having code that
automatically modifies the init-file at startup is setting up the
potential for a bug with a huge impact that will be extremely annoying
and dangerous to work around. I think this is especially important to
keep in mind since Emacs releases are extremely infrequent, and so
we're going to have to deal with bugs in any release version for a
long, long time.

> What's this about alternative package managers? Shouldn't Emacs
> include special support for the built-in one? After all, that's why
> it's built in.

Sure, I think it's reasonable for Emacs to provide special support for
packages which are built in. But there's such a thing as going too
far. And I personally think that you've gone too far in providing
special support when that support actively makes it *more* difficult
to swap out an alternative implementation.

If you use an alternative package manager (and many people do), then
you really don't want package.el to be automatically activating things
when it's not asked to. You'll end up with duplicate load-path
entries, conflicting versions, needlessly slow init times, and more:
fun times (I've been there).

And you *really* don't want package.el to be modifying your init-file
that configures your alternative package manager so that it also
activates package.el! Having such code inserted when it's actively
harmful is a really frustrating experience.

################################################################################
#### ALTERNATIVES

There is lots we can do to improve the situation. Some changes will be
harmless and backwards compatible, while others would have to be
rolled out carefully if we decide to follow them.

* Don't have package.el insert `package-initialize' into the init-file
  under any circumstances. Instead, when Emacs is started up for the
  first time (when not in 'emacs -Q' mode), if no file exists at
  ~/.emacs.d/init.el, create that file with a basic template.

  This template would have a call to `package-initialize', and also
  some explanatory comments saying where you should put configuration
  of package.el versus configuration of packages, and why.

  This is my favorite option. It is a well-established (and,
  importantly, *unsurprising*) tradition for programs to come with
  skeleton configuration files that the user can customize as they
  desire. Besides, this would allow us to make a better experience for
  new users (remember all the complaints about "bad defaults"?)
  without making actual changes to the defaults and breaking backward
  compatibility.

* Don't have package.el activate packages at startup. I would actually
  prefer this, but I understand that it's not likely to happen. There
  are two advantages: (1) users of alternative package managers don't
  have to explicitly disable package.el all the time (when testing
  without their configuration file, for example), and (2) the behavior
  is much less "magic".

  This would break backward compatibility, but mostly of expectations:
  only people who put their entire package configuration code into
  `after-init-hook' would have their configurations broken (and the
  fix would of course be trivial). If we implemented the skeleton init
  file, then the experience for new users would not change.

* Improve the explanatory value of the comments that are inserted by
  package.el. I really, really think that the whole behavior of
  automatically inserting code into the init-file should be killed
  (with fire), but if the community really disagrees with me here, at
  least the comments should make an attempt at explaining what's going
  on and what the potential caveats are. This would be a trivial
  change with no downsides.

* Make it trivial to disable package.el. Right now, the only foolproof
  way to prevent the init-file from being modified by package.el is to
  place multiple advices on internal functions, and to do this as
  early as possible during init just in case there is an error, and to
  also be really careful any time you load Emacs without your
  init-file, and to double-check for the inevitable times when you
  can't prevent package.el from doing stuff.

################################################################################
#### CONCLUSION AND NEXT STEPS

I've argued strongly for removing the auto-insertion of
`package-initialize' code into the init-file by package.el, in favor
of providing a skeleton init-file. Although I've addressed a number of
counterarguments, I'm sure there are other factors I haven't
considered. Let me know what you think. Is this reasonable? If not,
why not? If so, how can I help move it forward?

Best regards,
Radon Rosborough

[1]: https://lists.gnu.org/archive/html/info-gnu-emacs/2012-06/msg00000.html
[2]: https://www.gnu.org/software/emacs/manual/html_node/emacs/Packages.html
[3]: https://github.com/quelpa/quelpa
[4]: https://github.com/cask/cask
[5]: https://github.com/dimitri/el-get
[6]: https://github.com/emacscollective/borg
[7]: https://github.com/raxod502/straight.el
[8]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html
[9]: https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01055.html
[10]: https://www.reddit.com/r/emacs/comments/56fvgd/is_there_a_way_to_stop_emacs_from_adding_the/?ref=share&ref_source=link
[11]: https://www.reddit.com/r/emacs/comments/4x655n/packageselectedpackages_always_appear_after/
[12]: https://github.com/raxod502/straight.el/issues/73

Re: Friendly discussion about (package-initialize)

[Stefan Monnier]

> Sure, I think it's reasonable for Emacs to provide special support for
> packages which are built in. But there's such a thing as going too
> far.  And I personally think that you've gone too far in providing
> special support when that support actively makes it *more* difficult
> to swap out an alternative implementation.

You wrote lots and lots of lines of text just to complain about the
addition of a single "(package-initialize)" in the user's ~/.emacs,
which should do absolutely nothing in the case where the user doesn't
use package.el (i.e. doesn't have anything inside ~/.emacs.d/elpa).

So, it's not that big of a deal, really.

We could arrange for "(package-initialize)" to only be added if there is
at least one package inside ~/.emacs.d/elpa.

More to the point, I think that's already the case.  So users of other
package managers should simply never bump into this text (unless they
also user package.el, of course).  If they do, they should report it as
a bug.

> * Make it trivial to disable package.el. Right now, the only foolproof
>   way to prevent the init-file from being modified by package.el is to
>   place multiple advices on internal functions, and to do this as

Multiple?  Doesn't

    (advice-add 'package--ensure-init-file :override #'ignore)

do the trick?

I suspect that

    (setq package-enable-at-startup nil)

might also do the trick.

This said, the main motivation for calling package--ensure-init-file
from package-initialize was to fix existing user's config where they had
packages installed yet their .emacs didn't call package-initialize, so
they had trouble configuring their packages.  One might argue that this
situation is now mostly fixed and we could change tactic: only call
package--ensure-init-file when the user installs a package.

Another thing is that rather then look for "(package-initialize)" in
~/.emacs we could keep track of whether package-initialize was called
during initialization.  This will avoid the problem when the user placed
his call in another file.


        Stefan


PS: BTW, I'd be very interested to work with maintainers of other
package managers to see how we could make them better interoperate
(e.g. make it possible to install with one tool, but activate&config
with another).

Re: Friendly discussion about (package-initialize)

[Radon Rosborough]

> You wrote lots and lots of lines of text just to complain about the
> addition of a single "(package-initialize)" in the user's ~/.emacs,
> which should do absolutely nothing in the case where the user
> doesn't use package.el (i.e. doesn't have anything inside
> ~/.emacs.d/elpa).

The problem is that even if I don't use package.el, there may be some
stuff left in ~/.emacs.d/elpa from previous times. For example, let's
say that I am trying to write a comparison between my package manager
and package.el, so even though I don't usually use package.el, I use
it temporarily.

Then, after I'm done testing, I remove all traces of package.el from
my init-file. But if I don't remember to also remove the data
directory ~/.emacs.d/elpa, then (package-initialize) is *not* a no-op:
it also activates duplicate copies of a bunch of old packages.

> So, it's not that big of a deal, really.

Only if you're OK with leaving a superfluous line in your init-file
for no reason other than that package.el wants it there. I know for a
fact that I'm not the only person who is annoyed by this kind of
behavior: see [1] for example.

Also, if it's not a big deal to have packages automatically insert
code into the user's init-file, then why don't more packages do this?
If every package did this, it would be a disaster. Somehow every
package other than package.el has managed without this mechanism; I
think package.el can do the same.

> We could arrange for "(package-initialize)" to only be added if
> there is at least one package inside ~/.emacs.d/elpa.

That's a good idea but it won't fix the problem, in my opinion. My
expectation of editor plugins is that if I remove the code configuring
them, then they should no longer affect me. This is not the case with
package.el; you have to specifically jump through some extra hoops to
prevent its side-effects from resurrecting themselves in future Emacs
sessions (e.g. deleting ~/.emacs.d/elpa).

> So users of other package managers should simply never bump into
> this text (unless they also user package.el, of course). If they do,
> they should report it as a bug.

Users of other package managers will still encounter this text unless
they delete ~/.emacs.d/elpa. In light of this, I think we have a
documentation bug, since the manual doesn't display a clear notice
that if you want to use an alternative package manager, you !!must!!
delete ~/.emacs.d/elpa first. More to the point, I don't think such a
step should be necessary: what if I wanted to switch between package
managers? That shouldn't be hard; there aren't these kinds of
persistent side effects with any of the other Emacs package managers I
tried, only for package.el.

> Multiple?  Doesn't
>
>     (advice-add 'package--ensure-init-file :override #'ignore)
>
> do the trick?

The second advice I was referring to was preventing
`package--save-selected-packages' from modifying the init-file.
However, the question of `package-selected-packages` is an entirely
separate issue and I shouldn't have snuck it in here. Sorry about
that; pretend I said just one advice.

> I suspect that
>
>     (setq package-enable-at-startup nil)
>
> might also do the trick.

I don't consider `package-enable-at-startup' to be an acceptable
solution, since I might want to use a function from package.el just to
try it out. In that case, my init-file will still get modified without
my permission. The name of the game here is "persistent side effects".
In my opinion, a package manager shouldn't have any. Again, for all
the other Emacs package managers I've seen (including the one I
wrote), you can perform any operations you'd like with them, and after
restarting Emacs, you'll be back where you started provided that you
didn't put anything in your init-file. This also happens to be the
case with every Emacs *package* I've seen. The only exception is
package.el, and I think we should consider why (if?) package.el needs
to be an exception.

> We could change tactic: only call package--ensure-init-file when the
> user installs a package.

I don't think I would enjoy this either. Would you consider it an
acceptable solution to pop up a window or display a message telling
the user that they should put `package-initialize' in their init-file,
provided that we didn't have it get called during init?

> Another thing is that rather then look for "(package-initialize)" in
> ~/.emacs we could keep track of whether package-initialize was
> called during initialization. This will avoid the problem when the
> user placed his call in another file.

Actually, Emacs already does this. And it doesn't solve the problem;
quoting from above:

    > Wait, you said that it was a problem if `package-initialize' was
    > put somewhere other than in the init-file. But if it's called
    > during startup, then package.el doesn't run the logic to insert
    > the a call into the init-file. So what's the issue?

    The problem is that if there is an error while loading the
    init-file, then the call to `package-initialize' still gets
    inserted, if init hadn't gotten to the point where the actual call
    to `package-initialize' was supposed to happen. This is actually
    terrible, since it immediately makes debugging much more
    complicated: your init-file is being changed without your knowing
    it, while you're trying to debug initialization!

    > OK, but that only happens when there's an error during init. We
    > could make it so that the `package-initialize' happens only
    > after a successful init.

    That doesn't solve the problem, because there are all kinds of
    circumstances where you have a "successful" init even when the
    user's init-file was not loaded. For example, many modular Emacs
    configurations will sustain errors in one or more modules using
    `condition-case', and report the errors as warnings for an
    improved debugging experience. We're back to the beginning.

    Besides, often one wants to test something in a plain Emacs but
    not in 'emacs -Q'. In this case, the call to `package-initialize'
    will still get inserted.

> I'd be very interested to work with maintainers of other package
> managers to see how we could make them better interoperate (e.g.
> make it possible to install with one tool, but activate&config with
> another).

Sounds like a great idea. However, do note that this will only be
useful for package managers that use the package.el format (e.g.
Quelpa, Cask, Pallet) and not for others (e.g. el-get, Borg,
straight.el).

Best,
Radon

[1]: https://www.reddit.com/r/emacs/comments/56fvgd/is_there_a_way_to_stop_emacs_from_adding_the/

Re: Friendly discussion about (package-initialize)

[Stefan Monnier]

> The problem is that even if I don't use package.el, there may be some
> stuff left in ~/.emacs.d/elpa from previous times.

Then don't do that: we want to make it as straightforward as possible for
users to install&use packages from GNU ELPA, so automatic activation of
those packages that were installed is a prerequisite.

Inevitably there will be situations where this design goal will clash
with the end-user who wants to use something else and will want to
explicitly "disable" package.el.

>> We could change tactic: only call package--ensure-init-file when the
>> user installs a package.
> I don't think I would enjoy this either.

Do you mean that it would be worse, or that it would be better but still
not good enough?  Based on your earlier message, I'm pretty sure I will
not come up with a solution which you really like, so I'm only aiming to
get "better" rather than "good enough".

> Would you consider it an acceptable solution to pop up a window or
> display a message telling the user that they should put
> `package-initialize' in their init-file, provided that we didn't have
> it get called during init?

No, I think this will reduce your annoyance level a tiny bit, along with
that of a handful of other users (all of whom likely already know
several different ways to work around the current annoyance), while
significantly increasing the annoyance level of many more new users who
just want to get on with their business without having to know anything
about how package.el works.

IOW, a bad trade-off.

>     The problem is that if there is an error while loading the
>     init-file, then the call to `package-initialize' still gets
>     inserted, if init hadn't gotten to the point where the actual call
>     to `package-initialize' was supposed to happen.

But if the call is only inserted during package-install it means that
you'd need both:
- use package-install
- in a session where your init failed to call package-initialize
for the problem to show up.  This combination seems a lot less likely
than the current one.

Another thing we could consider is to drop the automatic call to
package-initialize in lisp/startup.el (again, based on the idea that
this has now been made unnecessary by package--ensure-init-file).

>     Besides, often one wants to test something in a plain Emacs but
>     not in 'emacs -Q'. In this case, the call to `package-initialize'
>     will still get inserted.

I don't know what is a "plain Emacs but not in `emacs -Q`".  When I need
something like that I do something like

    mv ~/.emacs ~/tmp/
    <runemacs>
    mv ~/tmp/.emacs ~/

so any edits to ~/.emacs during the test will be happily thrown away
without bothering anyone.

So it seems like you're thinking of another kind of "plain Emacs but not
in `emacs -Q`".

>> I'd be very interested to work with maintainers of other package
>> managers to see how we could make them better interoperate (e.g.
>> make it possible to install with one tool, but activate&config with
>> another).
> Sounds like a great idea. However, do note that this will only be
> useful for package managers that use the package.el format (e.g.
> Quelpa, Cask, Pallet) and not for others (e.g. el-get, Borg,
> straight.el).

Part of "make them interoperate" may involve changing the formats
accepted&|used by the various tools, indeed (on-disk, on-the-server,
...).


        Stefan

Re: Friendly discussion about (package-initialize)

[Radon Rosborough]

> > The problem is that even if I don't use package.el, there may be
> > some stuff left in ~/.emacs.d/elpa from previous times.
>
> Then don't do that:

I'm actually not sure what "that" is referring to here. Sorry if it is
obvious.

> we want to make it as straightforward as possible for users to
> install&use packages from GNU ELPA, so automatic activation of those
> packages that were installed is a prerequisite.

Agreed, but it doesn't have to be done in this way. We could
accomplish the same thing simply by providing a template init-file,
without any of the problems I've mentioned.

Is there any particular reason why providing a template init-file
would be a worse solution than modifying the init-file on the fly? If
so, could you explain to me what it is?

Let me reiterate: providing a template configuration file is an
*extremely common pattern*, and it's even common for the default
behavior of a program (when run with no configuration file) to be
generally considered unacceptable. The principle of least surprise
suggests that Emacs should use the same system as every other piece of
software with this kind of problem uses.

> Inevitably there will be situations where this design goal will
> clash with the end-user who wants to use something else and will
> want to explicitly "disable" package.el.

I'm fine with disabling package.el being an explicit step. Not with it
being an ongoing battle (where package.el strikes back every time I
accidentally use one of its functions without the proper advices
defined). I would be totally satisfied with there being package.el
code in the template init-file and no mention of alternative package
managers, as long as it ends with the template init-file (i.e. it's a
one-time thing).

> Do you mean that it would be worse, or that it would be better but
> still not good enough?

Better but still not good enough. I would consider modifying the
init-file automatically at startup as "atrocious" and doing it at
other times "undesirable but acceptable if it's really the only
solution".

> Based on your earlier message, I'm pretty sure I will not come up
> with a solution which you really like, so I'm only aiming to get
> "better" rather than "good enough".

That's OK. I certainly don't believe it's the responsibility of the
Emacs community to make me happy; sorry if I gave that implication.

> > Would you consider it an acceptable solution to pop up a window or
> > display a message telling the user that they should put
> > `package-initialize' in their init-file, provided that we didn't
> > have it get called during init?

> No,

Yeah, a popup window would be very annoying. It was just an offhand
suggestion. But consider that package.el doesn't currently display any
useful message in the echo area after it's finished (just whatever
byte-compilation message happened to be last). I don't think it would
be a degredation to user experience to have it say

    Finished installing 2 packages

and then if package-initialize wasn't called during init,

    Finished installing 2 packages, but please run
    M-x package-ensure-init-file to finalize installation

or something like that. Perhaps a *Warning* would be appropriate? As
long as the message conveyed a way to find out more details, and a
quick fix (I'm fine with advising to use a function that modifies the
init-file, as long as that's an intentional action).

I'm partially playing devil's advocate here; I really think the
template init-file is the way to go. But even if we do use the
template init-file, some better messaging would still be nice, I
think.

> (all of whom likely already know several different ways to work
> around the current annoyance)

Based on the Reddit thread I linked, most people who encounter this
problem don't know how to fix it. The ones who do spent an hour or two
figuring out how to do so because there's no documentation.

> But if the call is only inserted during package-install it means
> that you'd need both:
> - use package-install
> - in a session where your init failed to call package-initialize
> for the problem to show up. This combination seems a lot less likely
> than the current one.

This happened to me repeatedly when I was testing package managers,
since I obviously wanted to test installing packages in an environment
where they weren't already activated by my primary package manager. I
do agree that this situation is uncommon; it just strikes me that the
mechanism currently in use is rather fragile if it "breaks" in such a
situation.

> Another thing we could consider is to drop the automatic call to
> package-initialize in lisp/startup.el (again, based on the idea that
> this has now been made unnecessary by package--ensure-init-file).

I am strongly in support of this unless it means that we will be more
reluctant to eventually remove `package--ensure-init-file'. Almost all
of the tangible problems I mentioned in my original email stem from
package.el modifying the init-file on the fly; arguing for the
elimination of this behavior is therefore my highest priority.

> So it seems like you're thinking of another kind of "plain Emacs but
> not in `emacs -Q`".

No, I was thinking of the same situation. I know I have been bitten by
this before, but thinking back on it I can't say that any of my use
cases there are actually common in any way. Let's pretend I didn't
make that point; I don't think it is useful.

> >> I'd be very interested to work with maintainers of other package
> >> managers to see how we could make them better interoperate (e.g.
> >> make it possible to install with one tool, but activate&config with
> >> another).
> > Sounds like a great idea. However, do note that this will only be
> > useful for package managers that use the package.el format (e.g.
> > Quelpa, Cask, Pallet) and not for others (e.g. el-get, Borg,
> > straight.el).
>
> Part of "make them interoperate" may involve changing the formats
> accepted&|used by the various tools, indeed (on-disk, on-the-server,
> ...).

Note that source-based package managers such as Borg and straight.el
are fundamentally incompatible with the package.el format (or,
depending on your perspective, the package.el format is fundamentally
incompatible with source-based package managers such as Borg and
straight.el). el-get might benefit from this improved
interoperability, though it is an interesting case as it's the only
package manager I know of that uses both package.el and non-package.el
formats for packages.

Best,
Radon

Re: Friendly discussion about (package-initialize)

[Stefan Monnier]

>> > The problem is that even if I don't use package.el, there may be
>> > some stuff left in ~/.emacs.d/elpa from previous times.
>> Then don't do that:
> I'm actually not sure what "that" is referring to here.

"That" refers to leaving stuff in ~/.emacs.d/elpa when you're not using
package.el.

> Agreed, but it doesn't have to be done in this way. We could
> accomplish the same thing simply by providing a template init-file,
> without any of the problems I've mentioned.

Maybe we should provide a template init file.
[ This file should be basically empty (tho could contain lots of
comments.  ]
But auto-creating this file just because you start Emacs is also
a problem (similar to the auto-editing of this file you're complaining
about).

> Is there any particular reason why providing a template init-file
> would be a worse solution than modifying the init-file on the fly?

We also want to cater to old users of Emacs (who already have
their own ~/.emacs) who start to use package.el.

>> Inevitably there will be situations where this design goal will
>> clash with the end-user who wants to use something else and will
>> want to explicitly "disable" package.el.
> I'm fine with disabling package.el being an explicit step.  Not with it
> being an ongoing battle (where package.el strikes back every time I
> accidentally use one of its functions without the proper advices
> defined).

Which functions do you use accidentally?

>> Do you mean that it would be worse, or that it would be better but
>> still not good enough?
> Better but still not good enough.

Great.  Then let's try to do that.  I suggest you M-x report-emacs-bug
and request this change (so it gets a tracking number, and its
dedicated discussion).

> do agree that this situation is uncommon; it just strikes me that the
> mechanism currently in use is rather fragile if it "breaks" in such a
> situation.

I wonder what you mean by "break".

>> Another thing we could consider is to drop the automatic call to
>> package-initialize in lisp/startup.el (again, based on the idea that
>> this has now been made unnecessary by package--ensure-init-file).
> I am strongly in support of this

Great.  Then let's try to do that.  I suggest you M-x report-emacs-bug
and request this change (so it gets a tracking number, and its
dedicated discussion).

> unless it means that we will be more reluctant to eventually remove
> `package--ensure-init-file'.

I don't see why it would have such an effect.


        Stefan

Re: Friendly discussion about (package-initialize)

[Radon Rosborough]

> > > > The problem is that even if I don't use package.el, there may
> > > > be some stuff left in ~/.emacs.d/elpa from previous times.
> > >
> > > Then don't do that:
> >
> > I'm actually not sure what "that" is referring to here.
>
> "That" refers to leaving stuff in ~/.emacs.d/elpa when you're not
> using package.el.

As an end user, why should I have to know how package.el stores its
data in ~/.emacs.d? Isn't that an implementation detail? I'm concerned
that knowledge of this information is apparently required to prevent
package.el from taking actions all on its own (without being asked)
after you've switched to another package manager.

At a bare minimum, there needs to be a prominent notice about this in
the documentation [1]. But I still maintain that the entire situation
is very sketchy.

> Maybe we should provide a template init file.

I think this is a good idea not just to solve this problem with
package.el, but also to make it easier to nudge new users towards
Emacs configuration best practices without having to change defaults
(which, as we all know, is extremely contentious).

> [ This file should be basically empty (tho could contain lots of
> comments. ]

That can be debated later. At the minimum, though, it would have to
contain a (package-initialize), with lots of accompanying comments to
explain what it's doing there and where you should put your code to
configure packages.

> But auto-creating this file just because you start Emacs is also a
> problem (similar to the auto-editing of this file you're complaining
> about).

Agreed. However, it can't possibly be worse that the current
situation, right? Currently, a new file is created automatically,
*and* any existing one is modified. My proposal would limit this
behavior to automatic creation, and not modification.

Of course, the init-file shouldn't be created at startup in 'emacs -Q'
mode.

> We also want to cater to old users of Emacs (who already have their
> own ~/.emacs) who start to use package.el.

I do like it when things "just work", but there's only so much magic
you can do to that end before you start losing things like robustness,
predictability, and your sanity.

I know I've brought this up before, but can you name a single other
program that deals with the problem of bad default behavior by
automatically modifying a free-form user-written configuration file on
startup, depending on a number of undocumented heuristics? If nobody
else does it, I'm sure there's a reason for that.

I like to think that most everyone who uses Emacs is intelligent
enough to understand a simple concept like "the package manager must
be initialized before you can use the packages". The problem is that
this simple concept is made quite complicated by all the magic of
package.el, and it is not clearly documented at all. If we had better
communication with users, then there would be no need for magic.

And before you say that some people don't want to modify their
init-file, and would prefer to rely on interactive commands only, let
me point out that this problem only arises if you try to put package
configuration in your init-file.

> Which functions do you use accidentally?

I didn't mean that I use package.el functions accidentally. I mean
that I use package.el functions intentionally (when I am testing other
people's package managers, for example), but I accidentally forget to
define advices on internal package.el functions before doing so, with
the result that my init-file is modified for no reason and I have to
go and revert it.

> I wonder what you mean by "break".

I mean that it inserts a call to `package-initialize' in my init-file
when that call is both superfluous and actively harmful due to the
particulars of my setup. It objectively does the wrong thing in my
case; that is why I said "break". Not to imply that this isn't the
right thing to do in general, but it's definitely not always the right
thing to do.

> > > > > We could change tactic: only call package--ensure-init-file
> > > > > when the user installs a package.
> > > >
> > > > I don't think I would enjoy this either.
> > >
> > > Do you mean that it would be worse, or that it would be better but
> > > still not good enough?
> >
> > Better but still not good enough.
>
> Great. Then let's try to do that. I suggest you M-x report-emacs-bug
> and request this change

Will do, although I'll probably wait to see where this discussion goes
first, so I have the proper context.

> I don't see why it would have such an effect.

Because if we removed `package--ensure-init-file' subsequently, then
there would no longer be any automatic activation of packages during
init, except for new users who had a template init-file generated.

But I'm glad to hear that my concern was unfounded, and the removal of
init-file modification via `package--ensure-init-file' will not face
any more resistance than it does currently.

===== Something to consider =====

Let's say that one day, another package manager is accepted into Emacs
core. Users would have a choice between package.el and alternative.el,
whatever the latter may be. Does it still make sense for Emacs to
inject a whole bunch of magic into the init sequence, including
modifying the user's init-file without their permission under several
different conditions, catering specifically to package.el? Should
Emacs do this for both package managers? That doesn't make a lot of
sense, since of course the packages installed by the two might
conflict, and the user only needs one package manager. It's up to the
user to explicitly tell Emacs which package manager to use, by placing
(package-initialize) or (alternative-initialize) in their init-file.
That's the only reasonable solution; anything else is piling hacks on
top of hacks. Does this sound familiar?

===== An allegory =====

Let us travel to an alternate universe in which the snippet package
YASnippet was designed according to the same principles as package.el,
and it was included into Emacs core. What would this mean?

Well, firstly, there would be a function called `yas-initialize', and
you'd have to call it in order to load all the currently defined
snippets, and rebind <TAB> so that it would expand snippets if
appropriate.

Whenever you started Emacs, `yas-initialize' would be run
automatically, which would rebind <TAB> without asking you, and then
you'd get a comment inserted into your init-file:

    ;; Added by Yasnippet.  This must come before configurations of
    ;; installed snippets.  Don't delete this line.  If you don't want it,
    ;; just comment it out by adding a semicolon to the start of the line.
    ;; You may delete these explanatory comments.
    (yas-initialize)

You could prevent this by setting the variable `yas-enable-at-startup'
to nil, but this wouldn't help you in the case that your init-file
encountered an error somewhere, or any number of other interesting
edge cases. It would only take one time for that comment to get
inserted, and then Yasnippet would start getting activated at startup
again, even if you had previously tried to disable it.

Of course, being a highly capable Emacs user, you are very careful to
never encounter any of those situations... somehow. But one day, you
decide to try out Yasnippet, to see if you like it. You set up the
configuration just how you like it, in a sub-module of your
modularized init-file. But as soon as you load Yasnippet, that comment
is back again, and in the wrong place! After an hour or two of poring
through the source code, it looks like the only way to permanently
disable this behavior is to place an advice on an internal function.
You can hardly believe your eyes.

"Surely the Emacs developers didn't think that everyone needed
Yasnippet activated in just the same way, or even that everyone wants
to use Yasnippet, right? This must be a mistake. How could anybody
think that globally rebinding <TAB> on startup is a good idea?"

So you go and complain about the situation, only to be told things
like:

> Well, the rebinding of <TAB> won't make any difference if the user
> doesn't use Yasnippet (i.e., doesn't have any snippets defined).

> We want to make it as straightforward as possible for users to
> define and use snippets.

> People who object to this kind of thing are in the minority; it's
> more important to make the experience as seamless as possible for
> Yasnippet users.

> Just make sure to delete all your snippets whenever you're done
> using Yasnippet, that will solve the problem.

I'm being intentionally facetious here; I certainly don't intend to
give offense. I just think that looking at the situation from outside
the lens of package.el will lend some good perspective. After all, no
package (and even no package *manager*) other than package.el does
this kind of thing, and I've yet to hear a reason why package.el just
*has* to be different.

Best regards,
Radon

P.S. If you think that comparing package initialization to rebinding
<TAB> globally is unfair, I'd say on the contrary it's quite an
egalitarian comparison:

slow init time due to loading unneeded snippets
  <==>
slow init time due to loading duplicate packages

conflicts between <TAB> rebinding and user's bindings
  <==>
conflicts between stale package.el packages and user's packages

unpredictable behavior of <TAB> because installed snippets are not
synced between machines
  <==>
unpredictable behavior of Emacs config because installed packages are
not synced between machines

disrespect of the user's desire to use another snippet manager
  <==>
disrespect of the user's desire to use another package manager

P.P.S. I originally started this comparison as a joke, but I'd
honestly be less offended by the <TAB> thing.

[1]: https://www.gnu.org/software/emacs/manual/html_node/emacs/Package-Files.html

Re: Friendly discussion about (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Wed, 9 Aug 2017 20:32:59 -0700
> Cc: emacs-devel@gnu.org
> 
> > [ This file should be basically empty (tho could contain lots of
> > comments. ]
> 
> That can be debated later. At the minimum, though, it would have to
> contain a (package-initialize), with lots of accompanying comments to
> explain what it's doing there and where you should put your code to
> configure packages.

Can someone explain, preferably in concise form, why are we having
this discussion about in which file to have the call to
package-initialize, given that startup.el already calls that function
(as IMO it should)?

As you might guess, I don't use package.el, but this issue, which to
my opinion seems to be already solved The Right Way, continues to
generate such prolonged discussions that I wonder where did we make
the wrong turn with package.el.

Thanks.

Re: Friendly discussion about (package-initialize)

[Radon Rosborough]

> Can someone explain, preferably in concise form, why are we having
> this discussion about in which file to have the call to
> package-initialize

Gladly.

Currently, Emacs has the opinion that you should call
`package-initialize' in your init-file, and it enforces this opinion
by automatically inserting such a call directly into your init-file at
startup, or whenever you initialize the package management system.

I view this behavior as wrong. Other people feel that it is the best
solution to the problem of users putting package configuration in
their init-file, which runs before `package-initialize' is called in
startup.el.

That is why we are having this discussion.

> startup.el already calls that function (as IMO it should)

I disagree, but won't speak further unless you think it will be
relevant to the discussion.

> where did we make the wrong turn with package.el.

When Emacs started modifying the init-file specifically for package.el.

Best,
Radon

Re: Friendly discussion about (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Wed, 9 Aug 2017 21:39:44 -0700
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> Currently, Emacs has the opinion that you should call
> `package-initialize' in your init-file, and it enforces this opinion
> by automatically inserting such a call directly into your init-file at
> startup, or whenever you initialize the package management system.
> 
> I view this behavior as wrong. Other people feel that it is the best
> solution to the problem of users putting package configuration in
> their init-file, which runs before `package-initialize' is called in
> startup.el.
> 
> That is why we are having this discussion.
> 
> > startup.el already calls that function (as IMO it should)
> 
> I disagree, but won't speak further unless you think it will be
> relevant to the discussion.

This sounds like a contradiction.  If we call package-initialize from
startup.el, why does it have to also be called from the init files?
And since you are opposed to Emacs putting tyhis call to init files,
why do you disagree that calling package-initialize in startup.el is
wrong?  After all, that's where Emacs performs all the initialization
for the upcoming session, and that should include initializing the
packages which need initialization.

> > where did we make the wrong turn with package.el.
> 
> When Emacs started modifying the init-file specifically for package.el.

So having the call in startup.el is correct after all?  Now I'm
utterly confused.

Re: Friendly discussion about (package-initialize)

[Radon Rosborough]

> This sounds like a contradiction. [...] Now I'm utterly confused.

Sorry, it looks like I failed to provide enough context in the name of
being concise. Let me try again.

The issue is complicated because there are actually a number of
different issues at play, which all interact in different ways.

Firstly, let me describe my ideal world. This would be a world in
which `package-initialize' was not called automatically anywhere, or
inserted into the init-file, or anything. However, new users could
still benefit from package.el because Emacs would generate a template
init-file which would contain a call to `package-initialize', if no
init-file already existed.

Now there is also the question of where it is appropriate to call
`package-initialize'. IMO, the only appropriate place to call it is in
the user's init-file. Doing it anywhere else smells like unnecessary
magic, and limits customizability. For example, calling it in
startup.el after loading the init-file means that package
customizations cannot be put in the init-file (unless you use
`after-init-hook', an advanced and rather nonstandard approach), but
the package management system still works after init. Can you possibly
think of any setup that would be *more* confusing to new users?

Then there is the issue of Emacs automatically modifying the init-file
to stick a call to `package-initialize' in it. I think this behavior
is disgusting because of how fragile and nonstandard it is.

Now let me address your specific concerns.

> If we call package-initialize from startup.el, why does it have to
> also be called from the init files?

IMO, it should not be called anywhere. I think having it only called
in startup.el would be a reasonable compromise. Having Emacs insist on
putting the call into the init-file, but then *also* calling it in
startup.el, makes no sense.

> since you are opposed to Emacs putting tyhis call to init files, why
> do you disagree that calling package-initialize in startup.el is
> wrong?

I do think that the correct place to call `package-initialize' is in
the init-file, but I object to Emacs taking care of this
automatically. It might seem like the best solution at first glance,
but doing it this way has a laundry list of horrible side effects (see
my original email).

> that's where Emacs performs all the initialization for the upcoming
> session, and that should include initializing the packages which
> need initialization.

Like I said, having the call be only in startup.el would be a
perfectly reasonable compromise IMO. My main issue is with Emacs
modifying the init-file automatically.

Does this make more sense? I can elaborate, but briefly:

1. I want `package--ensure-init-file' to go away ASAP.
2. I think Emacs should generate a template init-file; in addition to
   making `package--ensure-init-file' mostly superfluous, this would
   have other advantages in terms of improving default user
   experience.
3. If possible, I'd like startup.el to not call `package-initialize'
   either, but this is an order of magnitude less important than (1)
   and (2).

Best,
Radon

Re: Friendly discussion about (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Thu, 10 Aug 2017 10:06:34 -0700
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> Now there is also the question of where it is appropriate to call
> `package-initialize'. IMO, the only appropriate place to call it is in
> the user's init-file. Doing it anywhere else smells like unnecessary
> magic, and limits customizability. For example, calling it in
> startup.el after loading the init-file means that package
> customizations cannot be put in the init-file (unless you use
> `after-init-hook', an advanced and rather nonstandard approach), but
> the package management system still works after init. Can you possibly
> think of any setup that would be *more* confusing to new users?

I'm probably missing some details here, but in principle I don't see
here anything that should be confusing or hard to get right.  We do
that with other tricky stuff, like customizations of the basic faces,
which we read from .emacs after Emacs already started and displayed
its first frame.  And using hooks is not such scary stuff for new
users, either.  To say nothing of the fact that new users aren't
expected to mess with this anyway, they should just use what package
installation procedure arranged for.

> > If we call package-initialize from startup.el, why does it have to
> > also be called from the init files?
> 
> IMO, it should not be called anywhere. I think having it only called
> in startup.el would be a reasonable compromise. Having Emacs insist on
> putting the call into the init-file, but then *also* calling it in
> startup.el, makes no sense.

Thanks, but that doesn't really answer my question.  I asked why do we
put a call to package-initialize into user init file when we already
have that very call in startup.el.

Re: Friendly discussion about (package-initialize)

[Radon Rosborough]

> I'm probably missing some details here

I'm missing some details too, since I don't use the Custom system, and
that seems to have been part of this discussion in the past. I expect
that the fact there are so many details is what makes this discussion
keep coming up, because nobody's solution caters to every use case.

> using hooks is not such scary stuff for new users

Fine, but do we really want to tell users to put the entirety of their
init-file inside `after-init-hook'? That seems like an anti-pattern to
me. It's either `package-initialize' which has to move, or the entire
rest of the init-file. It makes more sense to me to move
`package-initialize'.

> new users aren't expected to mess with this anyway

So if we implemented a template init-file, then new users who didn't
have a init-file previously would not need to do anything.
Furthermore, existing users who are comfortable with customizing Emacs
would not have a problem either, since the concept of "you must
initialize the package management system before you can use packages"
is trivially simple. The only people disadvantaged are new users who
already had an existing init-file.

If we keep calling `package-initialize' in startup.el, then things
will work as expected for these users as well, unless they happen to
manually write some customizations of packages into their init-file.
But now the set of users who are disadvantaged is limited to:

1. new users
2. who aren't comfortable with modifying their init-file
3. who nevertheless had an existing init-file
4. and nevertheless add Lisp code directly to their init-file anyway

In my opinion, this is a rather narrow intersection, and the problems
introduced by automatic modification of the init-file far outweigh the
inconvenience to this group of users.

> that doesn't really answer my question. I asked why do we put a call
> to package-initialize into user init file when we already have that
> very call in startup.el.

You'll have to ask Stefan. I am the one arguing that this behavior
makes no sense and should be eliminated ASAP.

In short, however, this measure was introduced in an effort to allow
for package customizations to be put in the init-file without
`package-initialize' also being put in the init-file (by the user).

best,
Radon

Re: Friendly discussion about (package-initialize)

[Mark Oteiza]

Eli Zaretskii <eliz@gnu.org> writes:
>> > If we call package-initialize from startup.el, why does it have to
>> > also be called from the init files?
>> 
>> IMO, it should not be called anywhere. I think having it only called
>> in startup.el would be a reasonable compromise. Having Emacs insist on
>> putting the call into the init-file, but then *also* calling it in
>> startup.el, makes no sense.
>
> Thanks, but that doesn't really answer my question.  I asked why do we
> put a call to package-initialize into user init file when we already
> have that very call in startup.el.

The form is put there in order to coddle users who copy-paste
configurations a) into their init files, or b) in Customize settings,
that use symbols from packages before the packages are (auto-)loaded.

To quote the implementer (from this thread[2]):

> Another consequence is that users can't paste
> configurations into their init file in the way that 90% of the
> (emacs-related) internet is telling them to.
> By adding the chosen snippet to the user's init file we are preventing
> this sort of problem for new users, instead of telling them it's their
> fault for not understanding package.el.

FWIW here are some other links to threads surrounding the issue when it
was being debated/implemented:

[0] https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg00897.html
[1] https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html
[2] https://lists.gnu.org/archive/html/emacs-devel/2015-04/msg00002.html

Re: Friendly discussion about (package-initialize)

[Eli Zaretskii]

> From: Mark Oteiza <mvoteiza@udel.edu>
> Cc: Radon Rosborough <radon.neon@gmail.com>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Thu, 10 Aug 2017 16:00:58 -0400
> 
> The form is put there in order to coddle users who copy-paste
> configurations a) into their init files, or b) in Customize settings,
> that use symbols from packages before the packages are (auto-)loaded.
> 
> To quote the implementer (from this thread[2]):
> 
> > Another consequence is that users can't paste
> > configurations into their init file in the way that 90% of the
> > (emacs-related) internet is telling them to.
> > By adding the chosen snippet to the user's init file we are preventing
> > this sort of problem for new users, instead of telling them it's their
> > fault for not understanding package.el.
> 
> FWIW here are some other links to threads surrounding the issue when it
> was being debated/implemented:
> 
> [0] https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg00897.html
> [1] https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01016.html
> [2] https://lists.gnu.org/archive/html/emacs-devel/2015-04/msg00002.html

Thanks for the pointers, I will read those discussions.

Re: Friendly discussion about (package-initialize)

[Nick Helm]

On Fri 11 Aug 2017 at  5.06 AM, Radon Rosborough wrote:

> Now there is also the question of where it is appropriate to call
> `package-initialize'. IMO, the only appropriate place to call it is in
> the user's init-file.

I'm likely missing something, but why is `package-initialize' in user
space at all?

Can it not simply be called internally with the first instance of a
package customisation in the user's init.el file (or startup.el,
interactively or wherever)?

Wouldn't that mean there's no need to automatically write to init.el or
maintain a template file, package--ensure-init-file would become
redundant and copy-pasted (use-package blah...) code would work wherever
the user puts it?

Re: Friendly discussion about (package-initialize)

[Stefan Monnier]

> I'm likely missing something, but why is `package-initialize' in user
> space at all?
> Can it not simply be called internally with the first instance of a
> package customisation in the user's init.el file (or startup.el,
> interactively or wherever)?

Yes, that would be ideal.  This needs a more declarative way to
customize packages, tho, and currently I think we still want to support
the more "imperative" form of customizations.


        Stefan

Interoperation between package managers (was: Friendly discussion about (package-initialize))

[Stefan Monnier]

>> Part of "make them interoperate" may involve changing the formats
>> accepted&|used by the various tools, indeed (on-disk, on-the-server,
>> ...).
> Note that source-based package managers such as Borg and straight.el
> are fundamentally incompatible with the package.el format (or,
> depending on your perspective, the package.el format is fundamentally
> incompatible with source-based package managers such as Borg and
> straight.el).

Hmm... really?  Could you expand on why that is?


        Stefan

Re: Interoperation between package managers (was: Friendly discussion about (package-initialize))

[Radon Rosborough]

> > > Part of "make them interoperate" may involve changing the formats
> > > accepted&|used by the various tools, indeed (on-disk,
> > > on-the-server, ...).
> >
> > Note that source-based package managers such as Borg and straight.el
> > are fundamentally incompatible with the package.el format (or,
> > depending on your perspective, the package.el format is
> > fundamentally incompatible with source-based package managers such
> > as Borg and straight.el).
>
> Hmm... really?  Could you expand on why that is?

Perhaps I should not have said the package.el format is fundamentally
incompatible with source-based package managers. What I should have
said is that source-based package managers would gain little by being
compatible with the package.el format. This is because the package.el
format is inherently unable to provide for the basic features that
define a source-based package manager.

In particular, packages that are installed via package.el cannot be
version-controlled, since the package.el format dictates a flat
collection of *.el (and other) files, with no version-control
information attached other than a version number. As a corollary, it
is impossible to use the package.el format to make local modifications
to a package in a controlled way, or to contribute changes upstream
using package.el.

All of these actions (version-control operations, local changes,
upstream contributions) are foundational to a source-based package
manager. The only way to achieve them is to run packages directly from
their VCS repositories (Borg, see [1]) or via symlinks that point at
said repositories (straight.el, see [2]). Thus, from a purely
technical standpoint, it is not useful for a source-based package
manager to interoperate with package.el in any meaningful way.

I do not wish to say that package.el is flawed for failing to provide
these features, since package.el is designed to work in the absence of
a local VCS installed, and it is impossible to provide these features
without making that assumption. But for package managers which do make
that assumption, the package.el format (in my opinion) is no longer as
relevant.

Best,
Radon

[1]: https://github.com/emacscollective/borg
[2]: https://github.com/raxod502/straight.el

Re: Interoperation between package managers

[Stefan Monnier]

> Perhaps I should not have said the package.el format is fundamentally
> incompatible with source-based package managers. What I should have
> said is that source-based package managers would gain little by being
> compatible with the package.el format.

I don't know what you mean by "the package.el format".  AFAIK there are
2 "formats": one on the web-server and one in ~/.emacs.d/elpa.  I agree
that the web-server one wouldn't make much sense for "source-based"
package managers, but the installed-format doesn't seem incompatible
at all.

> In particular, packages that are installed via package.el cannot be
> version-controlled, since the package.el format dictates a flat
> collection of *.el (and other) files, with no version-control
> information attached other than a version number.

Not sure.  For example, the way I use elpa.git with package.el is:

    git clone .../elpa.git
    cd elpa
    make

and then add that directory to package-directory-list.
That lets me use packages directly from the local Git repository (which
I find important in order for `C-h f` and such to directly jump to
the real source files that I can edit, with VCS metadata).

> As a corollary, it is impossible to use the package.el format to make
> local modifications to a package in a controlled way, or to contribute
> changes upstream using package.el.

I don't see that.


        Stefan

Re: Interoperation between package managers

[Radon Rosborough]

> I don't know what you mean by "the package.el format".

I meant the one in ~/.emacs.d/elpa.

> the way I use elpa.git with package.el is:
>
>     git clone .../elpa.git
>     cd elpa
>     make
>
> and then add that directory to package-directory-list.
> That lets me use packages directly from the local Git repository (which
> I find important in order for `C-h f` and such to directly jump to
> the real source files that I can edit, with VCS metadata).

I had no idea that package.el was capable of this. Thank you for
informing me; I will have to amend some of my published criticisms of
package.el. I have actually never seen anybody else doing this with
package.el, or indeed any mention that such a thing was possible.
Thus, I have a couple of follow-up questions:

- Is this use case documented in any way? I looked at the docstring
  for `package-directory-list' and it just said that the variable is
  for system-wide use only, and that you should use `package-user-dir'
  for your personal packages. Notably absent was any mention of what
  "directories containing Emacs Lisp packages" means, since all sorts
  of different formats could be expected.

- I had originally assumed that `package-directory-list' included
  packages in an ELPA-server-compatible format, and that package.el
  would display these packages in `package-list-packages', and you
  could install them into ~/.emacs.d/elpa (thus making copies of the
  files). You are saying this is not the case, right?

- Is this the intended way to use package.el? Every tutorial I've seen
  only covers installing packages from a local or remote ELPA
  repository.

- When you modify a package, how is byte-compilation and autoload
  generation handled? Do you just have to run 'make' again? Does that
  mean you have to hope that every package's Git repository provides a
  byte-compilation and autoload generation mechanism, and then
  remember how to use each of them? (Here I am talking about packages
  that are not in GNU ELPA, i.e. the majority of them.)

- Does package.el officially support installing packages from
  version-control, or do you have to clone and manage the repositories
  manually? If not currently, is such support a future development
  target for package.el?

So package.el interop may indeed be useful for Borg. It will not be
useful for straight.el regardless, since straight.el makes
compatibility with package.el an explicit non-goal. But in any case, I
think the standardization you have proposed is a great idea, since it
appears that every package manager other than straight.el could
benefit from it.

Best,
Radon

Re: Interoperation between package managers

[Stefan Monnier]

> - Is this use case documented in any way?

It's documented in elpa.git's README file.

>   I looked at the docstring for `package-directory-list' and it just
>   said that the variable is for system-wide use only,

It's a slightly crude way to describe it, indeed.  In reality the
difference between `package-directory-list' and `package-user-dir' is
that package.el will never write to `package-directory-list', it will
only add/remove packages from `package-user-dir' (so packages in
`package-directory-list' are assumed to be installed/removed some other
way).

>   Notably absent was any mention of what "directories containing Emacs
>   Lisp packages" means, since all sorts of different formats could
>   be expected.

The installed format of a package is very close to the ELPA format,
except that it additionally has a <pkg>-autoloads.el file (well, it's
also expected that it's been compiled, but that's not strictly
necessary).

It's probably not really documented, tho, indeed.

> - I had originally assumed that `package-directory-list' included
>   packages in an ELPA-server-compatible format, and that package.el
>   would display these packages in `package-list-packages', and you
>   could install them into ~/.emacs.d/elpa (thus making copies of the
>   files). You are saying this is not the case, right?

No, indeed, they're assumed to be installed just like those of
~/.emacs.d/elpa.  To control whether or not they're activated, you need
to use package-load-list.

> - Is this the intended way to use package.el? Every tutorial I've seen
>   only covers installing packages from a local or remote
>   ELPA repository.

It's the way I use it, so it's definitely a way I want to support.
It's not advertised very loudly, tho (I do mention it on this list
every once in a while, but IIUC very few other people use it this way).

My intention was/is to try and bring GNU ELPA packages closer to bundled
packages for Emacs maintainers:

    (cd emacs; git pull; make);
    (cd elpa; git pull; make externals; make)

updates both the bundled and the "unbundled" packages at the same time,
and both kinds are activated in my Emacs sessions.  And you get to see
the compilation warnings of both kinds of packages as well ;-)

> - When you modify a package, how is byte-compilation and autoload
>   generation handled? Do you just have to run 'make' again?

Yes.

>   Does that mean you have to hope that every package's Git repository
>   provides a byte-compilation and autoload generation mechanism, and
>   then remember how to use each of them? (Here I am talking about
>   packages that are not in GNU ELPA, i.e. the majority of them.)

This is not applicable to packages that are not in GNU ELPA.

This said, I also do the following:

   cd .../elisp; git clone .../otherpackage.git
   cd .../elpa/packages;
   ln -s .../elisp/otherpackage ./

so as to let "make" treat "otherpackage" as if it were part of elpa.git.
In most cases it mostly works (the autoloads generation mechanism as
well as the compilation rules are those of elpa/GNUmakefile rather than
those provided by "otherpackage") and depending on the changes needed
locally, I try to send them upstream or I keep them as local
modifications.  This is fine for my own use, but obviously not something
I'd recommend to the uninitiated.

> - Does package.el officially support installing packages from
>   version-control, or do you have to clone and manage the repositories
>   manually?

You have to clone&manage manually.

>   If not currently, is such support a future development
>   target for package.el?

Hasn't been so far.  Rather than integrate it into package.el, it could
be a completely separate package that helps automate what I do manually.
But for best results, it would likely benefit from some adjustments in
package.el.  I haven't thought about any of it.

Currently, the main way package.el was adjusted to help support the
"elpa.git checkout in package-directory-list" was to make it so that
packages in ~/.emacs.d/elpa don't have to use a subdirectory named
<pkg>-<vers> but it can be named just <pkg> (otherwise we'd need to
rename all those dirs after "git pull" to reflect the new version
numbers).

> So package.el interop may indeed be useful for Borg. It will not be
> useful for straight.el regardless, since straight.el makes
> compatibility with package.el an explicit non-goal. But in any case, I
> think the standardization you have proposed is a great idea, since it
> appears that every package manager other than straight.el could
> benefit from it.

As mentioned above, I haven't thought about what package.el could do to
make it easier to automate what I do manually, so if you have
suggestions for things that we could change in (or add to) package.el to
make it easier for Borg to interoperate with it, please send them along,


        Stefan

Re: Interoperation between package managers

[Radon Rosborough]

> You have to clone&manage manually.
>
> >   If not currently, is such support a future development target
> >   for package.el?
>
> Hasn't been so far. Rather than integrate it into package.el, it
> could be a completely separate package that helps automate what I do
> manually.

I feel like I should point out that this support is exactly what is
already provided by straight.el. Not that there is anything wrong with
adding it to package.el as well, but you might find a reduced market
for it since straight.el provides better support for this use case
(because it makes this use case the *only* use case, unlike package.el
which also supports installation of tarballs from an ELPA server).

> Currently, the main way package.el was adjusted to help support the
> "elpa.git checkout in package-directory-list" was to make it so that
> packages in ~/.emacs.d/elpa don't have to use a subdirectory named
> <pkg>-<vers> but it can be named just <pkg> (otherwise we'd need to
> rename all those dirs after "git pull" to reflect the new version
> numbers).

This is fantastic! Those version numbers always annoyed me to no end
and were in fact one of the major reasons I didn't like package.el.
Glad to know they are now optional. As far as I can tell, this fact
remains completely undocumented?

Honestly, setting aside my philosophical differences with package.el,
I think the biggest problem is the documentation. User education about
`package-initialize' and stuff like that would be much less of a
problem if the documentation were clear, obvious, and concise.

(This means that elpa.gnu.org should *NOT* link to the "Package
Installation" wall-of-text from the Elisp manual and the plain-text,
developer-oriented README from the Git repo, without also providing
some more comprehensible sources of information.)

> As mentioned above, I haven't thought about what package.el could do
> to make it easier to automate what I do manually, so if you have
> suggestions for things that we could change in (or add to)
> package.el to make it easier for Borg to interoperate with it,
> please send them along,

You should talk to Jonas (cc'd) about this as he's the author of Borg.
I can't really provide much useful help beyond this point since I have
no desire to use package.el for anything.

Best,
Radon

Re: Interoperation between package managers

[Jonas Bernoulli]

>> As mentioned above, I haven't thought about what package.el could do
>> to make it easier to automate what I do manually, so if you have
>> suggestions for things that we could change in (or add to)
>> package.el to make it easier for Borg to interoperate with it,
>> please send them along,
>
> You should talk to Jonas (cc'd) about this as he's the author of Borg.
> I can't really provide much useful help beyond this point since I have
> no desire to use package.el for anything.

(This has gotten a bit long, you may
 skip to "What Borg needs from package.el".)

I would like to eventually allow users to use Borg alongside package.el.

The idea behind Borg is that for some people using a package is only the
first step, the second is to contribute to it.

Borg installs packages by cloning the respective git repository and then
byte-compiling the libraries, generating the autoload file, generating
the info files, and adding the appropriate directories to `load-path'
and `Info-directory-list'.

That works surprisingly well, because most packages follow some informal
common sense conventions.  But Borg also has features to deal with
packages that stray for the common path.  The main difference from other
package managers is that in the case of Borg, those packages are not
brought into line at the time a package is created (because there is
neither a "package" not a "package maintainer" just the upstream git
repository).  But like Melpa or El-Get, Bort does support "recipes"; the
only difference is when and where the recipe is "applied".  In the case
of Borg it is applied on the users machine at the time of installation
(and update).

So as I said, Borg is primarily intended for people who contribute to
many of the package that they use.  In that sense it is similar to
adding the local clone of elpa.git to `package-directory-list.  One big
difference though is that in the case of Borg each package continues to
live in its own repository.

It is already possible to use Borg and package at the same time, but I
intend to make it a fully supported use-case.  All you have to (and can)
do currently to use both package managers at the same time is to
preserve this order in the init file:

  (package-initialize)
  (borg-initialize)

This guarantees that Borg wins in case a package is installed using both
package.el and Borg.  This allows users to install most packages using
package.el and only install the packages that they contribute to using
Borg.  The main benefit of using Borg to *install* some packages instead
of just *using* the package.el-installed version and *working* in a
manually cloned git repository, is that after restarting Emacs, the
locally modified version is loaded without having to manually adjust
the `load-path'.

Of course there are other benefits, for example `borg-install' only
needs the name of a package because it knows the location of thousands
of upstream repositories.  `magit-clone' on the other hand expects an
url.

In addition to cloning package repositories, it adds the clones as git
submodules of the ~/.emacs.d repository.  This makes sense if one only
uses Borg, but when also using package.el then probably not so much.

This is the main change required on Borg's side to make it more useful
for people who also use package.el.  (It is already possible to clone a
package repository without adding it as a submodule, but currently this
also means that Borg forgoes actually installing the package by
compiling it and adding it to the `load-path'.

---

What Borg needs from package.el?

If you initialize package.el before Borg, then the Borg-installed
version of a package shadows the package.el-installed version.  It would
be nicer if package.el could be told "this is a list of packages that
are somehow installed, do not install those packages, just assume they
are available".

I don't think that adding the directory that contains the Borg-installed
packages to `package-directory-list' would do because:

* That would cause package.el to load those packages, resulting in them
  being loaded twice.  (At least they would be added to the `load-path'
  twice.)

* While some of these directories follow the format expected by
  package.el, others do not.  For example in many repositories libraries
  live in ./lisp/ instead of ./.

So really, all that Borg needs is option `package-assume-installed'.
Maybe that already exists, I haven't checked, because I haven't actually
gotten around to making Borg package.el compatible yet.

---

And since that is what started this whole conversation, I should note
that I don't really care that users who use Borg exclusively have to put

  (setq package-enable-at-startup nil)

in their init file.  While I haven't followed this conversation, I have
thought a bit about how package.el is initialized, and can appreciate
how the current solution has grown out of necessity.  It tries to
satisfy users with very different needs and especially varying levels of
understanding.

The only thing that annoys me a bit is that when I use `emacs -Q' to
debug some issue without my configuration, then I end up with
~/.emacs.d/elpa being created.  It would nice if `package-initialize'
did not do that and would only *update* the local metadata if it is
already there.  Later when the user calls `package-list-package' or
`package-install', then the initial local copy of the metadata could be
downloaded.

But package.el is not unique in this regard.  When I use `emacs -Q',
then some other build in packages also create files in ~/.emacs.d that
would be different places if my personal init.el had been loaded.  I
can live with that and just cleanup manually after an `emacs -Q'.

Re: Interoperation between package managers

[Stefan Monnier]

> Borg installs packages by cloning the respective git repository and then
> byte-compiling the libraries, generating the autoload file, generating
> the info files, and adding the appropriate directories to `load-path'
> and `Info-directory-list'.

Right, that's similar to what I do by adding the git clone as a symlink
in .../elpa/packages.

> That works surprisingly well, because most packages follow some informal
> common sense conventions.

That's indeed my experience as well.

> I don't think that adding the directory that contains the Borg-installed
> packages to `package-directory-list' would do because:

[ Which is indeed the way I'd naturally/naively see it integrated  ]

> * That would cause package.el to load those packages, resulting in them
>   being loaded twice.  (At least they would be added to the `load-path'
>   twice.)

Why loaded twice?  Do you mean because package-initialize does it once
and then borg-initialize does it a second time?  If so, then I'd suggest
to change Borg so it doesn't do it (and relies on package.el to do it
instead).

> * While some of these directories follow the format expected by
>   package.el, others do not.  For example in many repositories libraries
>   live in ./lisp/ instead of ./.

Currently I deal with this by making the symlink point to the
.../<pkg>/lisp rather than to .../<pkg>/lisp.  This also works for cases
where git clone gives me a large package which includes some Elisp
support package inside (typically the git repository for a new
language, which includes a compiler along with some support code for
various editors).

There's been some efforts to extend package.el so it can better handle .el
files in subdirectories, but there's more work to do there.

But normally the way a package is activated by package.el is just by
loading <pkg>-autoloads.el so that can deal just fine with the case
where the .el files are in .../lisp: it's the responsibility of the code
that installs the package to compile the files in .../lisp and to setup
the autoloas in <pkg>-autoloads.el; and the code in <pkg>-autoloads.el
can add the .../lisp dir to load-path rather than the top-level dir.

> The only thing that annoys me a bit is that when I use `emacs -Q' to
> debug some issue without my configuration, then I end up with
> ~/.emacs.d/elpa being created.

Sounds like a bug, indeed.  Could you M-x report-emacs-bug with a recipe
to reproduce it?

> It would nice if `package-initialize' did not do that and would only
> *update* the local metadata if it is already there.

AFAIK package-initialize doesn't update any local metadata.

> Later when the user calls `package-list-package' or `package-install',
> then the initial local copy of the metadata could be downloaded.

That's what happens, AFAIK.


        Stefan

Re: Interoperation between package managers

[Stefan Monnier]

> This is fantastic! Those version numbers always annoyed me to no end
> and were in fact one of the major reasons I didn't like package.el.
> Glad to know they are now optional. As far as I can tell, this fact
> remains completely undocumented?

Indeed.  Actually, the directory naming is indirectly due to the way the
internals is structured: since an installed ELPA package needs to have
a file <pkg>-pkg.el (to describe the meta-info) as well as a file
<pkg>-autoloads.el, we need to figure out the <pkg> name from the
directory, hence the need for a naming convention.

I'd like to change that to use constant file names (that don't include
<pkg>), so we don't need to know the package's name to find the
metainfo file.  And I'd probably want to also unify the two files into
one (which would likely hold the concatenation of <pkg>-pkg.el and
<pkg>-autoloads.el).

As for why we have <pkg>-<vers> instead of just <pkg>: that was
something I insisted on because I think it's very important to be able
to have several versions of a given package installed at the same time.

> Honestly, setting aside my philosophical differences with package.el,
> I think the biggest problem is the documentation.

To me, all of that is pretty "obvious" because I've spent enough time
both in the design and in the code, which makes it difficult to figure
out what people might need to know.

Specific requests (especially patches) are very welcome here.

> (This means that elpa.gnu.org should *NOT* [...], without also providing
> some more comprehensible sources of information.)

What (w|c)ould be such a "more comprehensible source of information"?


        Stefan

Re: Interoperation between package managers

[Radon Rosborough]

> I'd probably want to also unify the two files into one (which would
> likely hold the concatenation of <pkg>-pkg.el and
> <pkg>-autoloads.el).

I'm a little wary of this idea. It seems to me like <pkg>-autoloads.el
is purely an implementation detail (or "build artifact") of the
package manager, whereas <pkg>-pkg.el contains author-maintained
metadata that should be checked into version control in the upstream
repository.

To give some perspective on why this might be important, consider how
source-based package managers work:

* clone upstream repository
* inspect <pkg>-pkg.el (or in-file headers) to determine dependencies,
  etc.
* generate autoloads and byte-compile on the client-side

In this workflow it makes no sense to combine the package metadata
with the autoloads.

Now, straight.el doesn't make any demands on how the upstream formats
things, which I consider one of its strengths. So as long as the
metadata is available in some kind of standard format, I don't think
there will be a problem. Most likely the situation will be similar for
other source-based package managers.

> I think it's very important to be able to have several versions of a
> given package installed at the same time.

This is another concept which makes a lot of sense in the
tarball-from-ELPA workflow, but not much sense in the source-based
workflow (where you don't need to save old versions because you can
revert to any version at any time).

I don't know what the best solution is for package.el to accommodate
both of these workflows (presuming that this is desired).

> > I think the biggest problem is the documentation.
>
> To me, all of that is pretty "obvious" because I've spent enough
> time both in the design and in the code, which makes it difficult to
> figure out what people might need to know.

Yes, that's the problem. I wrote a package manager too, and I have
exactly the same experience of everything seeming obvious to me. I
think the best solution is to just sit down and explain everything.
Have you seen the README for straight.el [1]? That is the kind of
documentation that I would like to see from every library I use. (I
know it's long; splitting into multiple documents is on my to-do
list.)

> Specific requests (especially patches) are very welcome here.

Unfortunately, I can't contribute patches since I am already too busy
maintaining my own package manager. However, I can give some
suggestions for things that need to be explained.

* What is the format of a package on the server? What is the format on
  the client? Be specific.
* What does package.el do to a package? I know that it generates
  autoloads and byte-compiles, but most people don't know that.
* What does activating a package mean? Again, I know that it entails
  evaluating the autoloads file and adding a directory to the load
  path, but most people don't.
* Why is package-initialize so slow? What are the bottlenecks when you
  activate lots of packages?
* What exactly is package-refresh-contents doing? Where is it saving
  things? What are the error conditions if you forget to run it? What
  commands is it automatically run by?
* Why does package-initialize keep getting inserted into my init-file?
  What's up with package-selected-packages?
* How am I supposed to load a single package and its dependencies for
  a bug report, without loading everything else?
* What happens when I need to run a package directly from its Git
  repository? How do I make local changes to a package installed from
  package.el? How do I contribute changes upstream?
* How do I sync my configuration between machines? How can I make sure
  that package.el creates a reproducible configuration that won't
  break if I move my configuration to a new machine?
* What's this business with package-install-file? How does it differ
  from other ways of loading a file into my Emacs configuration?
* How would I host my own ELPA server? Would I want to do such a
  thing? How do I deal with needing to install packages that aren't
  available in the default archives?
* How do I install old versions of packages? Is this even possible?
* What's the deal with built-in packages? What does that mean? Are all
  the lisp/ files built-in packages? None of them? An arbitrary
  subset?
* How do I install a package if there's also a built-in version of it?
* What are the known limitations of package.el? Clearly there are a
  lot of them, judging by the number of FIXME notices in the code.
* What are the interactions between installing different versions of
  packages with different dependencies from different archives? What
  rules are used to resolve all of this?
* What are the various states the package management system can be in?
  How does calling package-activate and/or package-initialize in
  various places during init or later behave, especially if the
  backing data on disk has changed in the interim?
* How strict are the version requirement rules? Are they always
  guaranteed to find a working set of versions if such a set exists?
  (No, as I found out while looking at FIXME notices in the source
  code.)

This is just a few simple things I thought of. I'm sure if I had a
better understanding of package.el, I would have more questions, and
more specific ones. Especially given all the features you told me
about in the last few emails, which I had no idea even existed, and
therefore didn't even consider asking questions about.

Maybe package.el is simple, but unless there's a clear statement that
"this documentation covers exactly what package.el does", there's
always the concern that more magic is going on, given how opaque the
source code is.

> What (w|c)ould be such a "more comprehensible source of
> information"?

The problem is that no such source exists at the moment. We need to
create one. What I mean by "comprehensible" is that any random Emacs
user can visit a nicely formatted modern web page and understand in
just a few minutes:

* what package.el does
* where packages live (on the server, in ~/.emacs.d, etc.)
* what package-initialize does
* where they should put code to configure packages
* commonly-asked questions and troubleshooting information

Then after the short summary, there should be comprehensive
documentation on everything. Again, see my documentation [1] which I
think is a pretty comprehensive description of both the concepts and
usage of straight.el.

I want my source code to be as readable as prose, but nevertheless try
to write my documentation to be such that my users never *have* to
look at the source code. I don't think it would be a bad thing if
package.el did the same.

Best,
Radon

[1]: https://github.com/raxod502/straight.el

Re: Interoperation between package managers

[Stefan Monnier]

>> I'd probably want to also unify the two files into one (which would
>> likely hold the concatenation of <pkg>-pkg.el and
>> <pkg>-autoloads.el).
> I'm a little wary of this idea. It seems to me like <pkg>-autoloads.el
> is purely an implementation detail (or "build artifact") of the
> package manager, whereas <pkg>-pkg.el contains author-maintained
> metadata that should be checked into version control in the upstream
> repository.

My view is that <pkg>-pkg.el should be built from metadata stored
elsewhere (that's what we do in elpa.git).

>> I think it's very important to be able to have several versions of a
>> given package installed at the same time.
> This is another concept which makes a lot of sense in the
> tarball-from-ELPA workflow, but not much sense in the source-based
> workflow (where you don't need to save old versions because you can
> revert to any version at any time).

I was thinking about it in the context of packages which can come from
the user's config as well as from the system (i.e. installed by the
sysadmin).

There are other cases where it can make sense.  E.g. have both foo-1.0
and foo-2.0 installed at the same time, because foo-2.0 requires
Emacs-25 and you use both Emacs-25 and Emacs-24.

I agree that these needs aren't the most common ones, but you don't gain
anything by disallowing them, really.

>> Specific requests (especially patches) are very welcome here.
> Unfortunately, I can't contribute patches since I am already too busy
> maintaining my own package manager.  However, I can give some
> suggestions for things that need to be explained.

These look like info about the internals, and indeed we don't document
them very much.  Some of them are purposefully not documented ("use the
source, 'cause it might change").  But there's a a lot of room for
improvement anyway.  I'll see if I can improve on that.

> * What is the format of a package on the server?

AFAIK this is documented somewhere.

> * What does activating a package mean? Again, I know that it entails
>   evaluating the autoloads file and adding a directory to the load
>   path, but most people don't.

Actually "adding a directory to the load path" is only there by
accident: it should be done by the autoloads file.

> * Why is package-initialize so slow?

I don't know that it is, so I can't answer the question.

>   What are the bottlenecks when you
>   activate lots of packages?

I haven't looked into.  If you find performance problems,
M-x report-emacs-bug is probably the best option, so we can try to find
how to improve the situation.

> Maybe package.el is simple, but unless there's a clear statement that
> "this documentation covers exactly what package.el does", there's
> always the concern that more magic is going on, given how opaque the
> source code is.

The source code of package.el is not supposed to be opaque.  I wasn't
the original author and I didn't find it opaque when I looked at it.
So if it's opaque, maybe it's my fault, but I'm probably not the right
guy to fix it, then ;-)


        Stefan

Re: Interoperation between package managers

[Radon Rosborough]

> My view is that <pkg>-pkg.el should be built from metadata stored
> elsewhere (that's what we do in elpa.git).

That contradicts common usage, as reflected in the following Emacs
packages which ship a <pkg>-pkg.el file containing their metadata:

$ find . -name '*-pkg.el'
./cucumber.el/feature-mode-pkg.el
./ecukes/ecukes-pkg.el
./elisp-finalize/finalize-pkg.el
./emacs-async/async-pkg.el
./emacs-eclim/eclim-pkg.el
./ESS/lisp/ess-pkg.el
./evil/evil-pkg.el
./haskell-mode/haskell-mode-pkg.el
./markdown-toc/markdown-toc-pkg.el
./multiple-cursors.el/multiple-cursors-pkg.el
./skewer-mode/skewer-mode-pkg.el
./smartparens/smartparens-pkg.el
./zone-matrix/zone-matrix-pkg.el

The current manual [1] implies that it is the responsibility of the
upstream package maintainer to create a <pkg>-pkg.el file when
packaging their package. If that wasn't the intention, the
documentation should make this clear. But at this point, I would
advise against trying to enforce a new convention, given how useful it
is to have a consistent one established.

> I was thinking about it in the context of packages which can come
> from the user's config as well as from the system (i.e. installed by
> the sysadmin).

In this case, wouldn't the multiple versions be installed in entirely
different directories anyway?

> There are other cases where it can make sense. E.g. have both
> foo-1.0 and foo-2.0 installed at the same time, because foo-2.0
> requires Emacs-25 and you use both Emacs-25 and Emacs-24.

Again, in package.el, what you say makes perfect sense. But in a
source-based package manager, it seems better to handle this by having
the package manager check out the correct revision.

> I agree that these needs aren't the most common ones, but you don't
> gain anything by disallowing them, really.

I think you gain something by enforcing a simpler directory naming
convention. But of course this is still only relevant to source-based
package managers, and not to package.el.

> These look like info about the internals, and indeed we don't
> document them very much. Some of them are purposefully not
> documented ("use the source, 'cause it might change"). But there's a
> a lot of room for improvement anyway. I'll see if I can improve on
> that.

Sounds good. I agree that these things are sort-of internal, but the
problem is that users frequently encounter problems with package.el
("try deleting your ~/.emacs.d/elpa directory"; "try running
package-refresh-contents again"; "you put package-initialize in the
wrong place"), so it's important to have documentation that allows
debugging such problems. IMO, the package installation and loading
process doesn't need to be complicated. Everyone should be able to
understand it.

(In other words, "it should only be a black box if it always works".)

> AFAIK this is documented somewhere.

The "somewhere" part is a big part of the issue. If I want to know
something about package.el, maybe I should go to the Packages section
of the manual, or maybe the Packaging section, or maybe elpa.gnu.org,
or maybe ELPA's README, or maybe the Commentary in the source code of
package.el, or the docstrings, or the comments ???

This is very confusing for a new user who only wants to know why their
packages aren't being loaded.

Regarding the format of packages on the server, I think this
documentation is supposed to be at [2], but it doesn't seem to
actually be there.

> Actually "adding a directory to the load path" is only there by
> accident: it should be done by the autoloads file.

Does that mean M-x update-directory-autoloads will start inserting
load-path manipulation calls into the generated autoload files? Or
should package.el roll its own autoload generation routine?

> > * Why is package-initialize so slow?
>
> I don't know that it is, so I can't answer the question.
>
> >   What are the bottlenecks when you
> >   activate lots of packages?
>
> I haven't looked into.  If you find performance problems,
> M-x report-emacs-bug is probably the best option, so we can try to find
> how to improve the situation.

Sorry, I didn't mean to imply there was a bug. I just meant that
package-initialize is slow when you have lots and lots of packages,
simply because a lot of work is being done. The problem is that
package-initialize is a black box, so the user doesn't know where to
start if they want to optimize their init. It'd be helpful to know if
the bottleneck was:

* evaluating autoloads
* processing dependencies
* loading caches
* activating packages that are no longer referenced in the init-file ??
* etc

I know at least one person who was frustrated that a quarter of their
init-file was a black-box package-initialize call (at least before
they switched to straight.el).

> The source code of package.el is not supposed to be opaque. I wasn't
> the original author and I didn't find it opaque when I looked at it.
> So if it's opaque, maybe it's my fault, but I'm probably not the
> right guy to fix it, then ;-)

When I look at package.el, I see things like:

* not all the functions and variables have docstrings
* quite a few FIXME notices
* a method called "package--make-autoloads-and-stuff" (???)
* a lack of high-level explanation as to how things are organized and
  what the various formats are

Maybe if I worked with package.el more, I would not be as confused.
And maybe I have unrealistic expectations. But I think it would really
help if there were a comprehensive explanation of exactly how
package.el thinks about package installation and loading. For
reference, [3] is the corresponding explanation for straight.el.
Things like the meaning of "package", "installation", "deletion",
"version", "activate", etc. most likely seem obvious to everyone
working on package.el, but in fact there are many different ways to
handle these things (they are done very differently in straight.el
than in package.el, for example), so documentation is really
necessary.

[1]: https://www.gnu.org/software/emacs/manual/html_node/elisp/Multi_002dfile-Packages.html
[2]: https://www.gnu.org/software/emacs/manual/html_node/elisp/Package-Archives.html#Package-Archives
[3]: https://github.com/raxod502/straight.el#conceptual-overview

Re: Interoperation between package managers

[Stefan Monnier]

> The current manual [1] implies that it is the responsibility of the
> upstream package maintainer to create a <pkg>-pkg.el file when
> packaging their package.  If that wasn't the intention, the
> documentation should make this clear.  But at this point, I would
> advise against trying to enforce a new convention, given how useful it
> is to have a consistent one established.

Using a new, fixed name which doesn't include the package's name is an
incompatible change in itself anyway, so it can break various other
aspects of compatibility at the same time.

>> I was thinking about it in the context of packages which can come
>> from the user's config as well as from the system (i.e. installed by
>> the sysadmin).
> In this case, wouldn't the multiple versions be installed in entirely
> different directories anyway?

Yes.

>> There are other cases where it can make sense. E.g. have both
>> foo-1.0 and foo-2.0 installed at the same time, because foo-2.0
>> requires Emacs-25 and you use both Emacs-25 and Emacs-24.
> Again, in package.el, what you say makes perfect sense.  But in a
> source-based package manager, it seems better to handle this by having
> the package manager check out the correct revision.

That would misbehave if the user runs Emacs-24 and Emacs-25 at the
same time (and I'm one of the users who does that pretty much all the
time).

>> I agree that these needs aren't the most common ones, but you don't
>> gain anything by disallowing them, really.
> I think you gain something by enforcing a simpler directory naming
> convention.

Agreed.  But you can simplify the naming convention by using a fixed
name for the metadata, instead of by disallowing version numbers in the
directory name.

>> AFAIK this is documented somewhere.
> The "somewhere" part is a big part of the issue. If I want to know
> something about package.el, maybe I should go to the Packages section
> of the manual, or maybe the Packaging section,

One of those two, yes.

> or maybe elpa.gnu.org, or maybe ELPA's README,

That should only be relevant for packaging issues related specifically
to GNU ELPA (and elpa.git).

> or maybe the Commentary in the source code of package.el, or the
> docstrings, or the comments ???

If it's sufficiently technical, yes.

> Regarding the format of packages on the server, I think this
> documentation is supposed to be at [2],

Sounds right.

> but it doesn't seem to actually be there.

Looks like a bug, then.

>> Actually "adding a directory to the load path" is only there by
>> accident: it should be done by the autoloads file.
> Does that mean M-x update-directory-autoloads will start inserting
> load-path manipulation calls into the generated autoload files?

No.

> Or should package.el roll its own autoload generation routine?

It does, indeed: autoload.el was designed to maintain autoloads within
an existing file, which can have any arbitrary "preamble".

> start if they want to optimize their init. It'd be helpful to know if
> the bottleneck was:
>
> * evaluating autoloads
> * processing dependencies
> * loading caches
> * activating packages that are no longer referenced in the init-file ??
> * etc

I don't know of anyone who has investigated this, so AFAIK noone knows
the answer.  Maybe the time is spent in a silly useless routine that's
easy to optimize (I doubt it's the case, FWIW, but it's likely that it
can be sped up if needed).


        Stefan

Re: Interoperation between package managers

[Nikolay Kudryavtsev]

Hello.

For long term, I think Emacs would need to implement something along 
those lines:

Let's say we have feature-lookup-list, that contains feature lookup 
functions. Each function returns either the feature, or nil. When 
looking for a feature we call each lookup function one by one until we 
get a positive or run out of handlers. So the first function would be 
file-feature-lookup, using load-path. Then we get one for package.el. 
And of course the first time package.el lookup handler is called, it 
does (package-initialize). In such system you can just cut out 
package.el lookup handler from feature-lookup-list to disable it, and 
each alternative package manager would just need to provide such function.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Interoperation between package managers

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

> Let's say we have feature-lookup-list, that contains feature lookup
> functions.

This is an interesting idea with a lot of potential. Is the primary
advantage that `package-initialize' would not need to be called
directly anymore, or are there other benefits as well?

My other question would be whether this could be implemented without
performance regressions, since `require' does get called an awful lot.

[-- Attachment #2: Type: text/html, Size: 599 bytes --]

Re: Interoperation between package managers

[Nikolay Kudryavtsev]

The main benefit of such API would be interoperability between package 
managers. Since it's just functions you can always wrap them into 
functions of your own, allowing for complex cases like when you want 
some package from package.el, then other packages from some other 
package manager, then other packages from package.el.

Getting rid of package-initialize is a good thing, but it would not be 
worth it, if it was the only motivation.

Optimizing performance would be a little bit tricky, that's true, so is 
dealing with autoloads, but this seems to me quite solvable.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Interoperation between package managers

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 2326 bytes --]

> The main benefit of such API would be interoperability between
> package managers.

So to be clear, I like this proposal. But I'd like to ask whether it
will really offer a lot of benefit over the current situation.
Specifically:

> allowing for complex cases like when you want some package from
> package.el, then other packages from some other package manager,
> then other packages from package.el.

Why not just say it is the responsibility of the package manager to
let the user say which packages it is to make available and which
packages it is to leave for another package manager? That seems more
flexible and natural to me.

> dealing with autoloads

Indeed, this seems like a big problem. Because a package can run
arbitrary Lisp code in its autoloads, and the user expects this code
to be run unconditionally when the package manager makes the package
available (and in particular, *not* when a feature is required). If we
have the package installed via two different package managers, then
the only way to do this correctly seems to be to tell the package
managers which one of them is supposed to activate the autoloads --
and then we are right back at the current situation.

Would this require a restructuring of how autoloading works in
general? If so, the resulting loss of backwards compatibility
indicates that there need to be some concrete benefits to the new
approach.

                              ~~~~~

Another thing I worry about is whether this is the right abstraction
for package managers that operate in different ways than package.el.
For example, in the package manager straight.el, there is no concept
of an "installed package". Instead, you specify to load a package
declaratively in your init-file, and it is downloaded, built, and
loaded automatically if necessary. One important point here is that
the package is automatically rebuilt if it has changed since the last
init, and the process of checking for modifications takes some time.
If you care about your init-time, you probably would only want
straight.el checking for modifications to packages that it actually
loads (rather than ones loaded by a different package manager), so
straight.el must be made directly aware of which packages it is
actually supposed to load. Once again, this brings us back to the
current situation.

[-- Attachment #2: Type: text/html, Size: 2923 bytes --]

Re: Interoperation between package managers

[Nikolay Kudryavtsev]

>
> Why not just say it is the responsibility of the package manager to
> let the user say which packages it is to make available and which
> packages it is to leave for another package manager?
With a properly structured API there would be no difference between a 
function provided by a package manager and something you just hacked 
together. As such this question would be left to the user. An advantage 
for having it on this level would be that you can do such configuration 
in a uniform way regardless of the package manager you use.

> Would this require a restructuring of how autoloading works in
> general?
Yes, seems like it would require restructuring autoloading, but the 
change would probably be backwards compatible.

> Another thing I worry about is whether this is the right abstraction
> for package managers that operate in different ways than package.el.
I don't see how it wouldn't be. We're asking "hey, can anyone load 
cool.el?" Package.el would answer no, since cool.el is not installed, 
straight.el would answer yes. And if it decides to download cool.el, 
it's entirely a decision left to it and how it's configured. This of 
course opens a possible attack angle of me having (require 
'malevolent-el) somewhere in my legit code.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Interoperation between package managers

[Ted Zlatanov]

On Wed, 23 Aug 2017 22:39:14 +0300 Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> wrote: 

NK> Let's say we have feature-lookup-list, that contains feature lookup functions.
NK> Each function returns either the feature, or nil. When looking for a feature we
NK> call each lookup function one by one until we get a positive or run out of
NK> handlers. So the first function would be file-feature-lookup, using load-path.
NK> Then we get one for package.el. And of course the first time package.el lookup
NK> handler is called, it does (package-initialize). In such system you can just cut
NK> out package.el lookup handler from feature-lookup-list to disable it, and each
NK> alternative package manager would just need to provide such function.

That sounds a lot like https://wiki.debian.org/DebianAlternatives

It's a good direction for "plugin packages", but many packages are not
plugins: they augment existing functionality or provide global
facilities that don't exist yet or can't be abstracted into function
calls.

Ted

Re: Interoperation between package managers

[Nikolay Kudryavtsev]

Hello Ted.

I'm not proposing that we touch the feature layer. The change would only 
be in feature loading and the only "plugins" here would be package 
mangers or simple functions written by the end user.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Friendly discussion about (package-initialize)

[Noam Postavsky]

On Sun, Aug 6, 2017 at 9:39 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> Another thing is that rather then look for "(package-initialize)" in
> ~/.emacs we could keep track of whether package-initialize was called
> during initialization.  This will avoid the problem when the user placed
> his call in another file.

I believe we already do this in Emacs 26 (see Bug#24643/25819).

Re: Friendly discussion about (package-initialize)

[Mark Oteiza]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Sure, I think it's reasonable for Emacs to provide special support for
>> packages which are built in. But there's such a thing as going too
>> far.  And I personally think that you've gone too far in providing
>> special support when that support actively makes it *more* difficult
>> to swap out an alternative implementation.
>
> You wrote lots and lots of lines of text just to complain about the
> addition of a single "(package-initialize)"

Not every user of package.el needs (package-initialize) in their init
file.  Not every user needs it at the beginning of the file, either.

> This said, the main motivation for calling package--ensure-init-file
> from package-initialize was to fix existing user's config where they had
> packages installed yet their .emacs didn't call package-initialize, so
> they had trouble configuring their packages.  One might argue that this
> situation is now mostly fixed and we could change tactic: only call
> package--ensure-init-file when the user installs a package.

The existing behaviour:

- subverts `package-load-list' settings
- breaks setting of package-archives that would normally happen before
  calling `package-initialize'
- writes init.el twice for some reason?
- does nothing to solve the disparity between packages and customize,
  which is the topic that opened the can of worms in the first place
- is basically undocumented

I can think of one other instance of a fragile init file lisp parser in
Emacs, and that one only exists for political reasons.

Re: Friendly discussion about (package-initialize)

[Stefan Monnier]

> - subverts `package-load-list' settings

I don't know what this problem is.

> - breaks setting of package-archives that would normally happen before
>   calling `package-initialize'

How so?


        Stefan

Re: Friendly discussion about (package-initialize)

[Mark Oteiza]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> - subverts `package-load-list' settings
>
> I don't know what this problem is.

package-load-list controls which autoload files for installed packages
get loaded.  Setting it in init.el works until package.el clobbers
the init file, putting package-initialize first.

>> - breaks setting of package-archives that would normally happen before
>>   calling `package-initialize'
>
> How so?

Same as above, calling package-initialize before setting archives, one
won't see the correct archives.

This is even documented in (info "(emacs) Package Installation"), albeit
not thoroughly--only package-load-list is mentioned explicitly in this
context.

Re: Friendly discussion about (package-initialize)

[Stefan Monnier]

>>> - subverts `package-load-list' settings
>> I don't know what this problem is.
> package-load-list controls which autoload files for installed packages
> get loaded.  Setting it in init.el works until package.el clobbers
> the init file, putting package-initialize first.

I see.

I doubt this corner case affects many people, since I think users who mess
with package-load-list are likely to have a .emacs complex enough that
it already includes a call to package-initialize.

In any case it's a one-time problem.

>>> - breaks setting of package-archives that would normally happen before
>>> calling `package-initialize'
>> How so?
> Same as above

Indeed.


        Stefan

Re: Friendly discussion about (package-initialize)

[Mark Oteiza]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> - subverts `package-load-list' settings
>>> I don't know what this problem is.
>> package-load-list controls which autoload files for installed packages
>> get loaded.  Setting it in init.el works until package.el clobbers
>> the init file, putting package-initialize first.
>
> I see.
>
> I doubt this corner case affects many people, since I think users who mess
> with package-load-list are likely to have a .emacs complex enough that
> it already includes a call to package-initialize.
>
> In any case it's a one-time problem.

It's a one-time problem _if_ the user figures out that
package-initialize needs to go after it--so ensure-init-file still fails
to solve the problem that there are cases where the user needs to
understand package.el

It's not a corner case.  It's a documented use case of package.el.

Sure, `package-load-list' is arguably less likely to be changed, but
tons of people change `package-archives' to use 3rdp packages, and these
users are at the core of the argument for this misfeature.  Either
variable being in the wrong position relative to package-initialize
breaks things.

Re: Friendly discussion about (package-initialize)

[Clément Pit-Claudel]

On 2017-08-11 03:14, Mark Oteiza wrote:
> Sure, `package-load-list' is arguably less likely to be changed, but
> tons of people change `package-archives' to use 3rdp packages, and these
> users are at the core of the argument for this misfeature.  Either
> variable being in the wrong position relative to package-initialize
> breaks things.

I'm not well-versed in these issues, so apologies if this is misguided. Would the following work?

* Call package-initialize *before* loading init.el
* Add two new functions package-set-load-list and package-set-archives that call package-initialize after setting the relevant variables.  This could be extended to any other variable that affect the behavior of package-initialize.  If there are many, a larger package-configure function or macro might be more useful.

The hope it to settle this package.el trouble by providing something that works without requiring users to add a package-initialize form in their .emacs.

I can see the following downsides, and potential fixes:

* What about existing configurations? Presumably they currently call package-initialize, so this wouldn't break things for them.
* What if people set those variables directly and expect things to work? We could add watchpoints on these variables, or keep the package-initialize call that runs after loading init.el.  I don't think people would expect setting a variable to do much, so I don't think thatwould be much of an issue.
* What about performance? Won't these extra (implicit) calls (from package-set-archives etc.) make things slow? We could make them reload only what's needed. IOW, package-set-archives would be akin to (progn (setq package-archives …) (package-refresh)) with package-refresh a new function that builds atop a previous package-initialize call.
* What if I *don't want* to use package.el at all? We could recognize a special dot-file in .emacs.d to disable package.el entirely (say, .nopackage).

On the upside, we get a working configuration for everyone without having to require or insert package-initialize calls in the user's config.

Clément.

Re: Friendly discussion about (package-initialize)

[Colin Baxter]

Dear Radon,

I would like to thank you for your extensive post. I found it very
useful and informative. This was not your motivation, I know, but it
went some way to confirm my own prejudice against using package
manager. I prefer to install external packages 'by hand'.

Best wishes,

Colin.

-- 
--
Colin Baxter
m43cap@yandex.com
GnuPG fingerprint: 68A8 799C 0230 16E7 BF68  2A27 BBFA 2492 91F5 41C8