Summary and next steps for (package-initialize)

Summary and next steps for (package-initialize)

[Radon Rosborough]

SUBJECT: Summary and next steps for (package-initialize)
FROM: radon.neon@gmail.com
TO: emacs-devel@gnu.org

Hi all,

Last week I posted an inquiry [1] about package.el's auto-insertion of
code into the init-file. Six people weighed in, but no definitive
conclusion was reached. I would like to propose concrete next steps,
summarizing relevant parts of the discussion in the process.

==> The proposal

The `package--ensure-init-file' logic will be removed, and package.el
will not ever directly modify the user's init-file. However, if Emacs
is started with no init-file, then a default one will be generated and
loaded (except in 'emacs -Q'). This template init-file would include a
call to (package-initialize) and some comments explaining that package
configuration must be put after (package-initialize).

==> The disadvantages of this proposal

* If somebody (1) has an existing init-file, (2) uses package.el to
  install a package, (3) adds configuration code directly to their
  init-file, (4) fails to read the documentation about the need for
  `package-initialize', and (5) has not used package.el in the last
  year or two, then their package configurations will not work.

  I argue that this specific use case is virtually nonexistent. In
  particular, the vast majority of people already have
  (package-initialize) in their init-file due to the previous behavior
  of package.el, and new users will get to use the template init-file.

  Thus, I do not think this is a real disadvantage. Please tell me if
  you disagree.

* Emacs is creating a whole file without being asked.

  I argue that the current behavior is worse in this regard than the
  proposed behavior. The current behavior is that user-created files
  are edited automatically by Emacs. The proposed behavior is that
  nonexistent files are created automatically by Emacs. It seems clear
  to me that automatic file creation is much safer than automatic file
  editing.

  Thus, I do not think this is a real disadvantage. Please tell me if
  you disagree.

==> The advantages of this proposal

* Emacs does not automatically modify the user's init-file without
  asking. This eliminates a wide range of unfortunate and annoying
  side-effects, as you can imagine. Here are two:

    - People who don't want to use package.el don't get irrelevant and
      damaging (because of duplicate loading) code stuck in their
      config.

    - People who use package.el but call (package-initialize) in some
      file other than init.el will not get a superfluous call inserted
      (which might well break their config) if there happens to be an
      error during init.

* In future, if we wish to improve the "out-of-the-box" user
  experience, we can do so without needing to break backwards
  compatibility, by simply modifying the template init-file.

* It's consistent with standard best practices. All other programs
  which have a similar problem to package.el solve it by providing a
  template config file. The reason that all these other programs avoid
  modifying their config files is the same reason that package.el
  should avoid modifying the init-file as well.

* It will never accidentally place (package-initialize) in the wrong
  place, which happens frequently with the current system and defeats
  the entire purpose of an aggressive hack to make things "just work".
  In fact, the current system *always* places (package-initialize) in
  the wrong place if the user happened to customize anything like
  `package-archives' in their init-file (which is extremely common,
  and I'd go so far as to say that *not* doing this is the uncommon
  case).

==> Things that are unaffected by this proposal

* New user experience is unaffected. Package installation and
  configuration continues to work out-of-the-box.

* Emacs will still automatically call `package-initialize' in
  startup.el after loading the init-file, unless it was already
  called, or `package-enable-at-startup' is set to nil.

==> Conclusion

If you object to this proposal, please tell me what the concrete
disadvantages to it would be, or explain why the advantages I have
listed are not valid. Otherwise, let's start the process of figuring
out the best implementation.

Best,
Radon Rosborough

[1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> However, if Emacs is started with no init-file, then a default
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
> one will be generated and loaded (except in 'emacs -Q').
                                    ^^^^^^^^^^^^^^^^^^^^

What does that mean?  Are you perhaps distinguishing between
switches -q and -Q?  If so, why would the former lead to
generating an init file but not the latter?

Personally, I don't think Emacs should be generating init files
in any case.  And certainly not generating an init file that
does something with the package system.  Emacs should continue
with the point of view that you start it, by default, with no
init file - no code at all.  Nada.

I also do not think that Emacs should modify a user's existing
init file to initialize intended use of the package system.
I stated that in an earlier thread.

If the package system needs to write code to a file for you
to be able to use it properly then let it use its own file for
that, just like Customize does (if you define `custom-file').
It is a bad idea for Emacs to write programmatically to a file
that users normally edit manually.

The simple case of a user not using packages at all should
be preserved, with no monkeying with init files.

If a user chooses to use packages, and if the package system
needs to save code persistently for the user in order for it
to work smoothly, then it should do so in a separate user
file, not the init file.  (And that should of course only be
after a user opts in, saying that s?he wants that.)

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> > However, if Emacs is started with no init-file, then a default
>                        ^^^^^^^^^^^^^^^^^^^^^^^^^
> > one will be generated and loaded (except in 'emacs -Q').
>                                     ^^^^^^^^^^^^^^^^^^^^
>
> What does that mean?  Are you perhaps distinguishing between
> switches -q and -Q?  If so, why would the former lead to
> generating an init file but not the latter?

I mean that if user-init-file is nil, then no template init-file is
generated. But if user-init-file is non-nil, and the corresponding
file does not exist, then the template init-file will be generated
there.

It is my understanding that 'emacs -Q' (and 'emacs -q') skip loading
the init-file, so I was using 'emacs -Q' as a shorthand for "init-file
is not loaded, and user-init-file is nil".

> Personally, I don't think Emacs should be generating init files
> in any case.  And certainly not generating an init file that
> does something with the package system.  Emacs should continue
> with the point of view that you start it, by default, with no
> init file - no code at all.  Nada.

I agree wholeheartedly with everything you have said. However, a
significant number of people on this list disagree, which is why I
have made a middle-of-the-road proposal that is more likely to be
accepted.

Do you agree that my proposal (only an init-file generated if one
doesn't exist) is at least better than the current situation
(hand-written init-file is modified programmatically by Emacs)? I'd
like to fix the brokenness of the current situation before trying to
move towards the best possible solution. If we accept this proposal,
we could later think about things like removing (package-initialize)
from the template init-file (although I doubt this will happen anytime
soon, if ever, based on the feedback I got).

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> > Personally, I don't think Emacs should be generating init files
> > in any case.  And certainly not generating an init file that
> > does something with the package system.  Emacs should continue
> > with the point of view that you start it, by default, with no
> > init file - no code at all.  Nada.
> 
> I agree wholeheartedly with everything you have said. However, a
> significant number of people on this list disagree,

Hm.  I haven't read this thread very carefully.  But I also
haven't gotten the impression that a significant number of
people here have argued for generating init files.  Not at all.
And I haven't gotten the impression that the default behavior
should be to start Emacs with no init file.  So I don't quite
follow you there.

> which is why I have made a middle-of-the-road proposal
> that is more likely to be accepted.

By "middle-of-the-road" are you perhaps referring to the fact
that you, like I, don't like Emacs messing with an existing
init file in order to try to tame package.el?  Is that what
you meant?  If not, I don't see anything one might call
"middle" about what (I understand of) your proposal.

> Do you agree that my proposal (only an init-file generated if one
> doesn't exist) is at least better than the current situation
> (hand-written init-file is modified programmatically by Emacs)?

Honestly, I'm not qualified to speak about solutions for
package.el problems.  I don't use the package system, myself.
So I can't say whether your proposal is better or is a
compromise of some sort.

I don't like the idea of Emacs generating an init file if
there is none OR of Emacs messing with an existing init file.

Sounds like a choice between being burned in the frying
pan or burned in the fire.  I don't see your proposal as a
"middle way", for users.  (It may be a middle way of sorts
for designers of the package system; dunno.)

> I'd like to fix the brokenness of the current situation before
> trying to move towards the best possible solution. If we accept
> this proposal, we could later think about things like removing
> (package-initialize) from the template init-file (although I
> doubt this will happen anytime soon, if ever, based on the
> feedback I got).

Doesn't sound like a great plan, to me.  But again, mine is
just one opinion, from someone who is not very familar with
the package system.

Naively, I'd suggest that package.el should do what custom.el
and bookmark.el and lots of other libraries do: Use a separate
Lisp file for persisting info the library needs - a file whose
name/location the user can set using an option.

If that would mean that a user might need to load that file
at an appropriate place in an init file, so be it.  The
package system might well require some understanding of it
before using it.

Customize is a bit like that: If you use a `custom-file' then
it is up to you to load it at an appropriate place in your init
file.  If you don't use a `custom-file' then you don't need to
worry about finding the most appropriate such place.  (But then
you need to worry about crosstalk between manual and automatic
edits - not worth it.)

(FWIW, I am also not in favor of Customize writing to your
init file.  I would prefer that we not only encourage use of
`custom-file' but we make it mandatory.  bookmark.el has it
right: require the use of a separate file.)

If it is difficult for users to find an appropriate place in
their init file to load whatever state/settings file package.el
might require, or to call `package-initialize' or whatever,
then perhaps the package system could be improved to help with
that somehow.

I say "naively" because I know next to nothing about package.el,
and I'm not trying to design the package system.  I have little
to contribute here, other than a wish, as one user, not to have
the package system mess with me or complicate things for users
generally.  I have faith that there is a way to do things sanely
for package.el, even if I don't have such a sane design to offer.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> So I don't quite follow you there.

Let me clarify. I see the following spectrum in this debate:

    People who want package.el to be enabled by default in spite of
    the resulting UX problems (too much magic, automatic modification
    of init-file).

    ^
    | (a) Have Emacs write (package-initialize) into the existing
    |     init-file at startup.
    |
    | (b) Have Emacs create a template init-file with
    |     (package-initialize) at startup, if one doesn't exist
    |     already.
    |
    | (c) Don't create (package-initialize) calls anywhere, but still
    |     call package-initialize in startup.el.
    |
    | (d) Don't call (package-initialize) anywhere; the user must call
    |     it manually in their init-file.
    v

    People who want package.el to be disabled by default in spite of
    the resulting UX problems (package management system doesn't work
    out of the box).

You and I are both at point (d). However, this is a rather extreme
position, so I am not arguing for point (d) or even point (c), but
rather point (b), because I view point (b) as an improvement over
point (a), which is the current situation.

Most people on this thread view any solution that doesn't involve
package.el working out of the box as a non-starter. That is why I am
not proposing such a solution. Instead, I am proposing a solution
which has much less potential for annoyance, while still having
package.el work out of the box.

Does this clear up the confusion?

> > I agree wholeheartedly with everything you have said. However, a
> > significant number of people on this list disagree,
>
> Hm.  I haven't read this thread very carefully.  But I also
> haven't gotten the impression that a significant number of
> people here have argued for generating init files.

You're right. I meant that you're arguing that Emacs shouldn't go
through contortions to make package.el work out of the box, and people
disagree with that. Meanwhile I'm just trying to make the contortions
less horrifying, as an immediate improvement to the current situation.

> So I can't say whether your proposal is better or is a compromise of
> some sort.

The part of my proposal that is relevant to you is the part where
Emacs doesn't try to write (package-initialize) into your init-file
automatically. Do you consider that an improvement?

> I don't like the idea of Emacs generating an init file if there is
> none OR of Emacs messing with an existing init file.

Same here. Unfortunately, the technical challenges in making
package.el work out of the box pretty much guarantee we have to do one
of these. Now, whether or not it's worth all this trouble to make
package.el work out of the box is another debate, but that's a much
harder debate than this one, which is why I'm not arguing it (at least
for now).

> I don't see your proposal as a "middle way", for users.

It's a middle way because creating a default init-file is less
intrusive than modifying an existing one, but still more intrusive
than doing nothing at all.

> Naively, I'd suggest that package.el should do what custom.el
> and bookmark.el and lots of other libraries do: Use a separate
> Lisp file for persisting info the library needs - a file whose
> name/location the user can set using an option.
>
> If that would mean that a user might need to load that file
> at an appropriate place in an init file, so be it.  The
> package system might well require some understanding of it
> before using it.

Again, agreed -- but emacs-devel does not want the user to have to
have any understanding of the package system before using it, so I'm
not making that argument.

(By the way, you're spot on with the separate Lisp file idea -- in
fact, package.el already does this, and the separate Lisp file
[directory] is ~/.emacs.d/elpa. This debate is over how to force users
to use package.el by activating it automatically. Again, while neither
of us thinks such a thing should be done, I think it's more realistic
to try to get it done in the least bad way possible, at least for
now.)

> FWIW, I am also not in favor of Customize writing to your init file.

Same.

> If it is difficult for users to find an appropriate place in
> their init file to load whatever state/settings file package.el
> might require, or to call `package-initialize' or whatever,
> then perhaps the package system could be improved to help with
> that somehow.

The solution here is improving the documentation of package.el, which
I've already suggested.

> I say "naively" because I know next to nothing about package.el, and
> I'm not trying to design the package system.

As someone who has in fact already written an Emacs package manager
from scratch (straight.el), I think all your points are spot on.
Thanks for your comments.

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> Let me clarify. I see the following spectrum in this debate:
> 
>     People who want package.el to be enabled by default in spite of
>     the resulting UX problems (too much magic, automatic modification
>     of init-file).
> 
>     ^
>     | (a) Have Emacs write (package-initialize) into the existing
>     |     init-file at startup.
>     |
>     | (b) Have Emacs create a template init-file with
>     |     (package-initialize) at startup, if one doesn't exist
>     |     already.
>     |
>     | (c) Don't create (package-initialize) calls anywhere, but still
>     |     call package-initialize in startup.el.
>     |
>     | (d) Don't call (package-initialize) anywhere; the user must call
>     |     it manually in their init-file.
>     v
> 
>     People who want package.el to be disabled by default in spite of
>     the resulting UX problems (package management system doesn't work
>     out of the box).
> 
> You and I are both at point (d). However, this is a rather extreme
> position, so I am not arguing for point (d) or even point (c), but
> rather point (b), because I view point (b) as an improvement over
> point (a), which is the current situation.
> 
> Most people on this thread view any solution that doesn't involve
> package.el working out of the box as a non-starter. That is why I am
> not proposing such a solution. Instead, I am proposing a solution
> which has much less potential for annoyance, while still having
> package.el work out of the box.
> 
> Does this clear up the confusion?

I guess so, and I appreciate the time you spent spelling that out.

What I would actually prefer, IIUC, is both of these at the same time:

* package.el to be enabled  by default
* package.el to be disabled by default

;-)

Rather, as a user I don't want to have to prescribe whether
package.el should be enabled or disabled, by default or
otherwise.  I don't want to have to care about package.el.

I would like accessing packages to just happen, when/if I
ask to access them - e.g. if I do `M-x list-packages'.

And if I have updated whatever persistent info is needed for
it, I would like Emacs to automatically enable this or that
package when I start Emacs.  It should be easy for me to
tell Emacs that I prefer to automatically enable this or that
package on startup.

I would want something like what exists for bookmark.el.
If/when you try to access bookmarks then your bookmarks file
is loaded automatically.  It's on-demand, but your bookmarks
file is separate from your init file.

> > > I agree wholeheartedly with everything you have said. However, a
> > > significant number of people on this list disagree,
> >
> > Hm.  I haven't read this thread very carefully.  But I also
> > haven't gotten the impression that a significant number of
> > people here have argued for generating init files.
> 
> You're right. I meant that you're arguing that Emacs shouldn't
> go through contortions to make package.el work out of the box,

Not sure what you mean.  I don't care whether Emacs goes
through such contorsions.  I don't think users should go
through contorsions.

If by Emacs going through contortions you mean Emacs adding
stuff to user init files or creating new init files, then yes,
I don't think Emacs should do that.

If by Emacs going through contortions you mean that it does
some work under the covers then no, I don't care whether it
does that.

> and people disagree with that. Meanwhile I'm just trying
> to make the contortions less horrifying, as an immediate
> improvement to the current situation.
> 
> > So I can't say whether your proposal is better or is a
> > compromise of some sort.
> 
> The part of my proposal that is relevant to you is the part where
> Emacs doesn't try to write (package-initialize) into your init-file
> automatically. Do you consider that an improvement?

I've already said that I don't think Emacs should write to
init files.  (And I don't think it should have to do that.
Why is that considered a hard requirement?)

But I've also said that if the package system needs to
record some info persistently for a user then it should use
a different file from the init file.  That doesn't mean that
I disagree with Emacs writing such info locally.  It just
means hands-off init files, please.

> > I don't like the idea of Emacs generating an init file if there is
> > none OR of Emacs messing with an existing init file.
> 
> Same here. Unfortunately, the technical challenges in making
> package.el work out of the box pretty much guarantee we have
> to do one of these.

I don't see why.  But I don't pretend to be familiar with the
technical challenges.  If it can do what it needs to do by
writing to an init file then why can't it do what it needs to
do by using a separate file?

Is the problem the difficulty for a user to determine where
in the init file to load the separate file?  If so, then
Emacs can perhaps detect bad placement - not by examining
the init file but by the effects of this or that inappropriate
placement.  E.g., if it is loaded before XYZ has been defined
or initialized then tell the user to move it etc.

> Now, whether or not it's worth all this trouble to make
> package.el work out of the box is another debate, but that's a much
> harder debate than this one, which is why I'm not arguing it (at least
> for now).
> 
> > I don't see your proposal as a "middle way", for users.
> 
> It's a middle way because creating a default init-file is less
> intrusive than modifying an existing one, but still more intrusive
> than doing nothing at all.

Debatable whether it is less bothersome/intrusive for a user.
In any case, I don't argue that Emacs must do nothing at all.
I argue (admittedly naively) that Emacs should keep its hands
off init files; that's all.
 
> > Naively, I'd suggest that package.el should do what custom.el
> > and bookmark.el and lots of other libraries do: Use a separate
> > Lisp file for persisting info the library needs - a file whose
> > name/location the user can set using an option.
> >
> > If that would mean that a user might need to load that file
> > at an appropriate place in an init file, so be it.  The
> > package system might well require some understanding of it
> > before using it.
> 
> Again, agreed -- but emacs-devel does not want the user to have to
> have any understanding of the package system before using it, so I'm
       ^^^
> not making that argument.

I'll make the argument.  If Emacs cannot make it easy to use
packages without it writing to init files then it sounds like
a design flaw and time to get back to the drawing board.

> (By the way, you're spot on with the separate Lisp file idea -- in
> fact, package.el already does this, and the separate Lisp file
> [directory] is ~/.emacs.d/elpa. This debate is over how to force users
> to use package.el by activating it automatically. Again, while neither
> of us thinks such a thing should be done, I think it's more realistic
> to try to get it done in the least bad way possible, at least for
> now.)

I see.  Too bad.

> > FWIW, I am also not in favor of Customize writing to your
> > init file.
>
> Same.
> 
> > If it is difficult for users to find an appropriate place in
> > their init file to load whatever state/settings file package.el
> > might require, or to call `package-initialize' or whatever,
> > then perhaps the package system could be improved to help with
> > that somehow.
> 
> The solution here is improving the documentation of package.el,
> which I've already suggested.

Even without being familiar with that doc, I can pretty much
blindly guess "+1" to that suggestion.

But as someone will no doubt point out, lots of users
(especially nowadays) do not read doc.  Create a good video
presenting the same relevant information and that might be
all that's needed.

That plus making the package system perform some checks to
recognize problematic situations resulting from inappropriate
use - and then point to relevant videos/doc etc.

IOW, if this is about pilot error then (1) improve the pilot
doc and (2) add some indicator lights or announcements that
pop up telling a pilot about a mistake s?he made and how to
correct it.

> > I say "naively" because I know next to nothing about
> > package.el, and I'm not trying to design the package system.
> 
> As someone who has in fact already written an Emacs package manager
> from scratch (straight.el), I think all your points are spot on.
> Thanks for your comments.

Thank you for your explanations.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> Rather, as a user I don't want to have to prescribe whether
> package.el should be enabled or disabled, by default or
> otherwise.  I don't want to have to care about package.el.
>
> I would like accessing packages to just happen, when/if I
> ask to access them - e.g. if I do `M-x list-packages'.

I think this is how most users feel. And similarly to you, most users
aren't intimately familiar with the technical challenges of
integrating a package manager into Emacs. So I thought it might be
nice to briefly outline the implications of this goal.

[Well, I guess it's only brief by *my* standards. Sorry...]

1. Accessing most package manager operations on-demand is trivial. M-x
   package-list-packages is autoloaded, and information about
   available packages can be loaded when the function is invoked. This
   is not being debated.

2. Having your packages automatically be enabled is more complicated.
   This is where package.el diverges from other packages. For example,
   in the case of bookmark.el, your bookmarks are not loaded until you
   try to access bookmarks. In the case of packages which provide a
   passive effect, like electric-pair-mode, they will only be enabled
   automatically if you put some code to that effect in your
   init-file.

3. On the other hand, we want your installed packages to be available
   automatically without such code being present (or at least most
   people do; I personally don't). Practically, this means that
   `package-initialize' has to get called at some point during
   initialization; there is simply no other way for packages to be
   enabled.

4. We therefore have two options: require the user to call
   `package-initialize' in their init-file, or call it in startup.el.
   The former is more modular and I would prefer it, but if we work
   under the assumption that under all circumstances we want installed
   packages to be activated automatically, then we have to call
   `package-initialize' in startup.el.

5. It's a non-starter to call `package-initialize' before the
   init-file is loaded, since then it is impossible to customize
   package.el (unless you introduce a second init-file which is loaded
   even earlier). Thus `package-initialize' has to be called after
   loading the init-file.

6. This satisfies all the conditions you asked for: namely, that you
   don't have to care about package.el. However, there is one final
   problem: you will probably want to put some Lisp code in your
   init-file to customize your installed packages, and this can only
   be done after `package-initialize' is called.

7. The only way to handle this is, therefore, to call
   `package-initialize' in your init-file if you wish to customize
   your installed packages directly in the init-file. As a bookkeeping
   measure, of course, if `package-initialize' is called in the
   init-file, then it's not called again after loading the init-file.

8. Thus we are led inevitably to the conclusion that the natural state
   for the user's init-file is that it has `package-initialize' in it
   somewhere (i.e. after package.el configuration, if any, but before
   package configuration, if any). The question is how to make it easy
   for users to realize that this is the case.

9. A bad solution to this problem is to have Emacs write a
   `package-initialize' call into the init-file. This is bad because
   the call needs to go in a specific place, and possibly in a
   different file that is loaded from the init-file, and so the
   insertion location will usually be incorrect. It's also bad because
   Emacs should keep its hands off the init-file, as you've pointed
   out.

10. A better solution, though not ideal, is to have Emacs generate a
    template init-file with a `package-initialize' call if no
    init-file exists. That way, when users go to add Lisp code to
    customize their packages, they will see a comment saying "make
    sure to place Lisp code for customizing packages *after* this
    `package-initialize' call", and they will therefore do the right
    thing. It's been well-established that such "just-in-time nudges"
    are very effective at getting users to take the correct action.
    Much more effective than external documentation, for example.

11. Alternatively, we can just say that if you are writing Lisp code
    to customize your packages, it's your responsibility to initialize
    the package manager manually. I'd like to move to this point
    eventually, but for now I think the template init-file is a good
    middle ground.

> If by Emacs going through contortions you mean that it does
> some work under the covers then no, I don't care whether it
> does that.

That's reasonable. So you are probably fine with `package-initialize'
being called in startup.el.

> I've already said that I don't think Emacs should write to
> init files.  (And I don't think it should have to do that.
> Why is that considered a hard requirement?)

I don't mean that Emacs writing to init-files is a requirement. Like
you, I don't think Emacs should do that. What I meant was that having
a call to `package-initialize' in the init-file is a hard requirement
for being able to put customization of installed packages into the
init-file.

> But I've also said that if the package system needs to
> record some info persistently for a user then it should use
> a different file from the init file.  That doesn't mean that
> I disagree with Emacs writing such info locally.  It just
> means hands-off init files, please.

Yeah. The problem is that we're not talking about the persistent info,
which is already stored in ~/.emacs.d/elpa. We're talking about the
code that initializes package.el. This can be compared with the code
that enables a minor mode: it has to be in the user's init-file if
it's going to take effect. The issue is how to encourage users to set
up their init-file correctly to use package.el without having Emacs
modify the init-file.

> If it can do what it needs to do by writing to an init file then why
> can't it do what it needs to do by using a separate file?

To reiterate, that would just move the problem, since
`package-initialize' is, in essence, the function that loads this
separate file.

> Is the problem the difficulty for a user to determine where in the
> init file to load the separate file?

No, the problem is that users forget to put this loading call in the
init-file at all.

> If so, then Emacs can perhaps detect bad placement - not by
> examining the init file but by the effects of this or that
> inappropriate placement. E.g., if it is loaded before XYZ has been
> defined or initialized then tell the user to move it etc.

That's a wonderful idea, and I think that regardless of any results of
my proposal, we should implement this error checking as well.

> Debatable whether it is less bothersome/intrusive for a user.

Well, I suppose it is indeed debatable. But you can't argue that the
current behavior is more predictable than the proposed behavior, at
least, and certainly not that the current behavior is less dangerous.

We could eliminate the annoyance for existing users by allowing to
create a dotfile in ~/.emacs.d that tells Emacs not to create a
template init-file, even if none exists already. We could go even
farther and have the template init-file create that dotfile, so the
template init-file would really be a one-time thing.

I know I argued against magic dotfiles earlier, but I think it's
different in this case: the dotfile would now be customizing a part of
the startup process that is fundamentally impossible to put into the
init-file, rather than a single tiny part of Emacs' business logic,
i.e. one of its package managers.

> I'll make the argument.

And I'll join you just as soon as this proposal goes through.

> lots of users (especially nowadays) do not read doc. Create a good
> video presenting the same relevant information and that might be all
> that's needed.

True. I think the template init-file would help with this by putting
the relevant information in the right place at the right time, in a
hard-to-ignore way that is nevertheless not annoying like a popup
would be.

> Create a good video presenting the same relevant information and
> that might be all that's needed.

Like, literally a video? Or did you mean an interactive Emacs-y
tutorial thing? If the former, I worry it might be difficult for
people to find.

> That plus making the package system perform some checks to
> recognize problematic situations resulting from inappropriate
> use - and then point to relevant videos/doc etc.

+1

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

> The `package--ensure-init-file' logic will be removed, and package.el
> will not ever directly modify the user's init-file.

I already argued why I think this is a non-starter.


        Stefan

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> > The `package--ensure-init-file' logic will be removed, and package.el
> > will not ever directly modify the user's init-file.
>
> I already argued why I think this is a non-starter.

Could you repeat the gist of that argument here, please? I just
skimmed the previous thread again, and the only thing I found that
sounds like an argument against removing `package--ensure-init-file'
is that it would disadvantage users who already have an init-file and
start using package.el. But I already addressed this argument in the
first bullet point of the "Disadvantages" section of my email
beginning this thread.

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-20 04:38, Radon Rosborough wrote:
> If you object to this proposal, please tell me what the concrete
> disadvantages to it would be, or explain why the advantages I have
> listed are not valid.

Hi Radon,

Thanks for putting this proposal together. Can you explain what advantages it holds over my previous proposal, which didn't require creating an init file if there was none, nor having an explicit package-initialize form?

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> Thanks for putting this proposal together. Can you explain what
> advantages it holds over my previous proposal, which didn't require
> creating an init file if there was none, nor having an explicit
> package-initialize form?

Sure. I think your proposal would work in principle, but it seems like
an unnecessarily complicated way to solve a simple problem. No other
package in Emacs requires this level of infrastructure just to
customize and use it: why does package.el need it? (I've asked this
question before, comparing package.el to other packages and package
managers, and have not yet received an answer as to why package.el has
to be so special.) That said, here are the concrete disadvantages I
see with your proposal:

* It is impossible to customize `package-load-list'. If
  `package-initialize' is called before loading the init-file, and
  then I call `package-set-load-list' in order to load only a subset
  of packages, it won't have any effect since all the packages were
  already loaded the first time `package-initialize' was called. Since
  `package-initialize' is called once, explicitly, in my proposal,
  there is no such problem.

* More generally, `package-initialize' will be called before my
  customizations of package.el. If I customize `package-user-dir',
  then I think I can reasonably expect that only packages in the new
  location, and not ones in ~/.emacs.d/elpa, will be activated. Not so
  with your proposal; packages in both locations will be activated.
  Again, this problem does not exist in my proposal since users simply
  do the obvious thing: put package.el customizations before calling
  (package-initialize).

* Every other variable in Emacs is customized by setting it to a
  value. It is confusing for package.el to be customized by adding a
  special dotfile to ~/.emacs.d. My proposal requires no such
  inconsistency and therefore does not violate the Principle of Least
  Surprise.

* I find it hard to believe that we won't see significant performance
  regressions with the current implementation of package.el. Thus your
  proposal can't be adopted until the implementation is fixed so that
  calling `package-initialize' multiple times takes negligible extra
  time. OTOH, my proposal can be adopted right now.

The first two points are the real problems here: fundamentally I think
it's a non-starter to call `package-initialize' before loading the
init-file. Unless, of course, you introduce a new init-file that's
called before `package-initialize', and that's a whole new can of
worms.

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Sat, 19 Aug 2017 19:38:05 -0700
> 
> If you object to this proposal, please tell me what the concrete
> disadvantages to it would be, or explain why the advantages I have
> listed are not valid. Otherwise, let's start the process of figuring
> out the best implementation.

My take on this, after reading the discussions kindly pointed to by
Mark, is that your proposal will replace one problematic situation
with another.  Whether the new situation will be better than the
current one is arguable, but in any case I don't think it will be
significantly better.  Already a few people said they were unhappy
with such a solution.  I myself cannot say I like the idea of Emacs
creating an init file in the user's home directory.

I think we should instead explore the possibility that
package-initialize will be called only in startup.el.  That solution
came up during the discussions, but AFAICT was dismissed almost
without any serious consideration.  The issues raised against it could
probably be solved by splitting the package initialization in two, one
part before, the other after the user's init file is read.  We already
do something similar with initializing the frame appearance and its
faces.  However, such a possibility was not considered, unless I
missed something.

Admittedly, I know too little about package.el and the related
problems, so perhaps what I propose is hard to make work in practice,
or it even might be unworkable.  However, a couple of people proposed
the same idea back when this was discussed, and those people
definitely know much more about package.el than I do.  So I think we
should give this possibility one more chance, and this time we
shouldn't reject it unless some serious disadvantages of it are
uncovered and described in detail here, so we could all agree they are
serious obstacles.

Thanks.

Re: Summary and next steps for (package-initialize)

[Alex]

Eli Zaretskii <eliz@gnu.org> writes:

> I myself cannot say I like the idea of Emacs
> creating an init file in the user's home directory.

If you're only uncomfortable with Emacs creating that init file
automatically, how about having an Emacs command that creates a template
init file for the user?

There could be a link (and perhaps keybinding) on the splash
screen/tutorial, where Emacs could ask the user a few questions and
create an init file for the user. Possible questions include:

Would you like this file in ~/ or ~/.emacs.d (.emacs vs init.el)?

Would you like to change some defaults? (This could help out with the
(IMO correct) notion that Emacs has some poor defaults while leaving
those alone for users that don't use this template.)

Would you like to use a separate file for `custom-file'?

Would you like to set up the package system?


I believe that this would be a step forward in accommodating new users.

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Alex <agrambot@gmail.com>
> Cc: Radon Rosborough <radon.neon@gmail.com>,  emacs-devel@gnu.org
> Date: Sun, 20 Aug 2017 10:37:59 -0600
> 
> There could be a link (and perhaps keybinding) on the splash
> screen/tutorial, where Emacs could ask the user a few questions and
> create an init file for the user. Possible questions include:
> 
> Would you like this file in ~/ or ~/.emacs.d (.emacs vs init.el)?
> 
> Would you like to change some defaults? (This could help out with the
> (IMO correct) notion that Emacs has some poor defaults while leaving
> those alone for users that don't use this template.)
> 
> Would you like to use a separate file for `custom-file'?
> 
> Would you like to set up the package system?
> 
> 
> I believe that this would be a step forward in accommodating new users.

That could be a separate issue, but if we add it, it's different: in
this case the user explicitly tells Emacs to create an init file.
It's conceptually the same as "C-x C-f ~/.emacs RET", just via
different gesture.

By contrast, installing a package doesn't hint on the user's desire to
create an init file in any way.  So we should try to avoid that, IMO.

Re: Summary and next steps for (package-initialize)

[Yuri Khan]

On Sun, Aug 20, 2017 at 11:37 PM, Alex <agrambot@gmail.com> wrote:

> There could be a link (and perhaps keybinding) on the splash
> screen/tutorial, where Emacs could ask the user a few questions and
> create an init file for the user. Possible questions include:
>
> Would you like this file in ~/ or ~/.emacs.d (.emacs vs init.el)?
>
> Would you like to change some defaults? (This could help out with the
> (IMO correct) notion that Emacs has some poor defaults while leaving
> those alone for users that don't use this template.)
>
> Would you like to use a separate file for `custom-file'?
>
> Would you like to set up the package system?
>
>
> I believe that this would be a step forward in accommodating new users.

This amounts to asking the new user to perform customization up front.
A new user will not have enough knowledge to answer these questions.
Further, he or she may feel intimidated and choose a different editor,
one that works out of the box.

It is better to have reasonably sane defaults and let the user
customize when he or she is ready.

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> > There could be a link (and perhaps keybinding) on the splash
> > screen/tutorial, where Emacs could ask the user a few questions and
> > create an init file for the user. Possible questions include:
> >
> > Would you like this file in ~/ or ~/.emacs.d (.emacs vs init.el)?
> >
> > Would you like to change some defaults? (This could help out with the
> > (IMO correct) notion that Emacs has some poor defaults while leaving
> > those alone for users that don't use this template.)
> >
> > Would you like to use a separate file for `custom-file'?
> >
> > Would you like to set up the package system?
> >
> > I believe that this would be a step forward in accommodating new users.
> 
> This amounts to asking the new user to perform customization up front.
> A new user will not have enough knowledge to answer these questions.
> Further, he or she may feel intimidated and choose a different editor,
> one that works out of the box.
> 
> It is better to have reasonably sane defaults and let the user
> customize when he or she is ready.

+1 - what Yuri said.

And this is not just about new users.  Such an interrogation
is not friendly for new or old users, even if old users might
have a better chance of understanding what is meant.

There must be a way to make package.el behavior failsafe,
simple, and without gotchas, for users.  That should be
the goal.

Whether that can be done in a backward-compatible way is,
IMHO, less important than making sure the new behavior is
solid.

Using the package system should be opt-in, and it should
be simple and unintimidating.

(Just one opinion.)

Re: Summary and next steps for (package-initialize)

[Colin Baxter]

>>>>> Drew Adams <drew.adams@oracle.com> writes:

    > +1 - what Yuri said.

    > And this is not just about new users.  Such an interrogation is
    > not friendly for new or old users, even if old users might have a
    > better chance of understanding what is meant.

    > There must be a way to make package.el behavior failsafe, simple,
    > and without gotchas, for users.  That should be the goal.

    > Whether that can be done in a backward-compatible way is, IMHO,
    > less important than making sure the new behavior is solid.

    > Using the package system should be opt-in, and it should be simple
    > and unintimidating.

    > (Just one opinion.)

Emacs code browser (ecb) copies the old init file to
~/.emacs.before_ecb_2.40 before inserting an option into the original
~/.emacs. The user is told all this before he is asked to ok it. It's
all straightforward and transparent, even to a newbie like me.

Could not something similar be used in the case of package.el?

-- 
--
Colin Baxter
m43cap@yandex.com
GnuPG fingerprint: 68A8 799C 0230 16E7 BF68  2A27 BBFA 2492 91F5 41C8

Re: Summary and next steps for (package-initialize)

[Alex]

Yuri Khan <yuri.v.khan@gmail.com> writes:

> On Sun, Aug 20, 2017 at 11:37 PM, Alex <agrambot@gmail.com> wrote:
>
>> There could be a link (and perhaps keybinding) on the splash
>> screen/tutorial, where Emacs could ask the user a few questions and
>> create an init file for the user. Possible questions include:
>>
>> Would you like this file in ~/ or ~/.emacs.d (.emacs vs init.el)?
>>
>> Would you like to change some defaults? (This could help out with the
>> (IMO correct) notion that Emacs has some poor defaults while leaving
>> those alone for users that don't use this template.)
>>
>> Would you like to use a separate file for `custom-file'?
>>
>> Would you like to set up the package system?
>>
>>
>> I believe that this would be a step forward in accommodating new users.
>
> This amounts to asking the new user to perform customization up front.
> A new user will not have enough knowledge to answer these questions.

That would depend on the questions asked. I don't think asking "would
you like your init file in your home directory or in a separate
directory" or "what is your preferred tab size" is asking too much. The
other questions might be more palatable if they deal at a higher level
of abstraction (e.g., don't mention Customize or package.el directly).

> Further, he or she may feel intimidated and choose a different editor,
> one that works out of the box.

This procedure would be completely opt-in. I agree that it shouldn't
offer too much customization (or they'd be there forever), at least by
default.

I don't think that such a procedure would intimidate users, at least not
more than the current built-in options available. There's already
something similar on the splash screen: a link to a Customize buffer
about "Initialization".

> It is better to have reasonably sane defaults and let the user
> customize when he or she is ready.

I agree, but I don't see how that necessarily contradicts with my
proposal. Even if the defaults are improved, I still believe having an
opt-in procedure to make a skeleton init file would make it easier for
new users to adopt Emacs.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> My take on this, after reading the discussions kindly pointed to by
> Mark, is that your proposal will replace one problematic situation
> with another.

Agreed. Drew has already explained why the new situation is also
problematic.

> Whether the new situation will be better than the current one is
> arguable,

I've already explained in my email starting this thread why I think
the new situation will be better than the current one. If you
disagree, please tell me why.

> I don't think it will be significantly better.

Can you provide justification for this? Such justification would take
the form of explaining why the advantages I listed are not valid, or
there are disadvantages that I missed.

> I myself cannot say I like the idea of Emacs creating an init file
> in the user's home directory.

Do you like the idea of Emacs modifying the user's init-file
automatically, even if it already exists? If not, then do you agree
that my proposal at least reduces the problem?

> Already a few people said they were unhappy with such a solution.

But did they say they were less unhappy with the current solution?
Drew certainly didn't.

> I think we should instead explore the possibility that
> package-initialize will be called only in startup.el. That solution
> came up during the discussions, but AFAICT was dismissed almost
> without any serious consideration. The issues raised against it
> could probably be solved by splitting the package initialization in
> two, one part before, the other after the user's init file is read.

Can you please elaborate on exactly how this would work, so that we
can make an informed comparison of the advantages and disadvantages of
the proposals?

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Sun, 20 Aug 2017 10:22:20 -0700
> Cc: emacs-devel@gnu.org
> 
> > Whether the new situation will be better than the current one is
> > arguable,
> 
> I've already explained in my email starting this thread why I think
> the new situation will be better than the current one. If you
> disagree, please tell me why.
> 
> > I don't think it will be significantly better.
> 
> Can you provide justification for this? Such justification would take
> the form of explaining why the advantages I listed are not valid, or
> there are disadvantages that I missed.

Both solutions are problematic and cause user annoyance.  I don't know
how else to explain that.

> > I myself cannot say I like the idea of Emacs creating an init file
> > in the user's home directory.
> 
> Do you like the idea of Emacs modifying the user's init-file
> automatically, even if it already exists? If not, then do you agree
> that my proposal at least reduces the problem?

I don't like either of these, but again, I see no significant
improvement, if at all.

> > I think we should instead explore the possibility that
> > package-initialize will be called only in startup.el. That solution
> > came up during the discussions, but AFAICT was dismissed almost
> > without any serious consideration. The issues raised against it
> > could probably be solved by splitting the package initialization in
> > two, one part before, the other after the user's init file is read.
> 
> Can you please elaborate on exactly how this would work, so that we
> can make an informed comparison of the advantages and disadvantages of
> the proposals?

Basically, leave the call to package-initialize in startup.el where it
is, and add another call at a proper place in startup.el to do
whatever needs to be done before the user init file is processed.
That other call could be to a new function, or even to the same
package-initialize, if it's feasible to do both parts in the same
function (after all, whether the user init file was or wasn't
processed is just a simple test away).

I realize that this is just a repetition of what I said above, but I
don't really understand what needs to be explained here.  If this is
still unclear, perhaps you should ask more specific questions.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> > > I don't think it will be significantly better.
> >
> > Can you provide justification for this? Such justification would take
> > the form of explaining why the advantages I listed are not valid, or
> > there are disadvantages that I missed.
>
> Both solutions are problematic and cause user annoyance.  I don't know
> how else to explain that.
>
> > > I myself cannot say I like the idea of Emacs creating an init file
> > > in the user's home directory.
> >
> > Do you like the idea of Emacs modifying the user's init-file
> > automatically, even if it already exists? If not, then do you agree
> > that my proposal at least reduces the problem?
>
> I don't like either of these, but again, I see no significant
> improvement, if at all.

It would be best to start with explaining in what situation my
proposal would cause user annoyance. Explaining specifically why the
automatic creation of an init-file is "problematic", apart from the
fact that you don't like it, would be a good next step.

Even better would be explaining why each of these four specific
reasons that my proposal improves on the current situation are
invalid:

  * Emacs does not automatically modify the user's init-file without
    asking. This eliminates a wide range of unfortunate and annoying
    side-effects, as you can imagine. Here are two:

      - People who don't want to use package.el don't get irrelevant and
        damaging (because of duplicate loading) code stuck in their
        config.

      - People who use package.el but call (package-initialize) in some
        file other than init.el will not get a superfluous call inserted
        (which might well break their config) if there happens to be an
        error during init.

  * In future, if we wish to improve the "out-of-the-box" user
    experience, we can do so without needing to break backwards
    compatibility, by simply modifying the template init-file.

  * It's consistent with standard best practices. All other programs
    which have a similar problem to package.el solve it by providing a
    template config file. The reason that all these other programs avoid
    modifying their config files is the same reason that package.el
    should avoid modifying the init-file as well.

  * It will never accidentally place (package-initialize) in the wrong
    place, which happens frequently with the current system and defeats
    the entire purpose of an aggressive hack to make things "just work".
    In fact, the current system *always* places (package-initialize) in
    the wrong place if the user happened to customize anything like
    `package-archives' in their init-file (which is extremely common,
    and I'd go so far as to say that *not* doing this is the uncommon
    case).

> I realize that this is just a repetition of what I said above, but I
> don't really understand what needs to be explained here.  If this is
> still unclear, perhaps you should ask more specific questions.

The current (package-initialize) loads the autoload files for all
installed packages (or just a subset, if `package-load-list' was
modified). It also adds those packages' directories to the load-path.
At least this is my understanding.

You suggested continuing to do these operations after loading the
init-file, but also to

> add another call at a proper place in startup.el to do whatever
> needs to be done before the user init file is processed.

Please explain what exactly needs to be done before the user init-file
is processed, and how doing this would allow both package.el
configuration and package configuration in the init-file without the
prescence of an explicit (package-initialize) call.

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Sun, 20 Aug 2017 10:54:30 -0700
> Cc: emacs-devel@gnu.org
> 
> The current (package-initialize) loads the autoload files for all
> installed packages (or just a subset, if `package-load-list' was
> modified). It also adds those packages' directories to the load-path.
> At least this is my understanding.
> 
> You suggested continuing to do these operations after loading the
> init-file, but also to
> 
> > add another call at a proper place in startup.el to do whatever
> > needs to be done before the user init file is processed.
> 
> Please explain what exactly needs to be done before the user init-file
> is processed, and how doing this would allow both package.el
> configuration and package configuration in the init-file without the
> prescence of an explicit (package-initialize) call.

AFAIU, packages need to be loaded before we run this fragment:

  ;; Re-evaluate predefined variables whose initial value depends on
  ;; the runtime context.
  (mapc 'custom-reevaluate-setting
        ;; Initialize them in the same order they were loaded, in case there
        ;; are dependencies between them.
        (prog1 (nreverse custom-delayed-init-variables)
          (setq custom-delayed-init-variables nil)))

This is so the defcustom's in the loaded packages are known to Emacs
when the init file is loaded.  So my proposal is to call some
package.el function that would do that before that.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> > Please explain what exactly needs to be done before the user init-file
> > is processed, and how doing this would allow both package.el
> > configuration and package configuration in the init-file without the
> > prescence of an explicit (package-initialize) call.
>
> AFAIU, packages need to be loaded before we run this fragment:
>
>   ;; Re-evaluate predefined variables whose initial value depends on
>   ;; the runtime context.
>   (mapc 'custom-reevaluate-setting
>         ;; Initialize them in the same order they were loaded, in case there
>         ;; are dependencies between them.
>         (prog1 (nreverse custom-delayed-init-variables)
>           (setq custom-delayed-init-variables nil)))
>
> This is so the defcustom's in the loaded packages are known to Emacs
> when the init file is loaded.  So my proposal is to call some
> package.el function that would do that before that.

AFAICT, customizing `package-load-list' and `package-user-dir' is no
longer possible with that approach. If such customization is put in
the init-file, it's too late; packages have already been loaded at the
point in startup.el that you mentioned.

How are you proposing to deal with that problem?

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Mon, 21 Aug 2017 09:43:31 -0700
> Cc: emacs-devel@gnu.org
> 
> > AFAIU, packages need to be loaded before we run this fragment:
> >
> >   ;; Re-evaluate predefined variables whose initial value depends on
> >   ;; the runtime context.
> >   (mapc 'custom-reevaluate-setting
> >         ;; Initialize them in the same order they were loaded, in case there
> >         ;; are dependencies between them.
> >         (prog1 (nreverse custom-delayed-init-variables)
> >           (setq custom-delayed-init-variables nil)))
> >
> > This is so the defcustom's in the loaded packages are known to Emacs
> > when the init file is loaded.  So my proposal is to call some
> > package.el function that would do that before that.
> 
> AFAICT, customizing `package-load-list' and `package-user-dir' is no
> longer possible with that approach. If such customization is put in
> the init-file, it's too late; packages have already been loaded at the
> point in startup.el that you mentioned.
> 
> How are you proposing to deal with that problem?

You are unfairly challenging me with stuff I know very little about,
and therefore might make suggestions that are laughable.  I asked
people who are familiar with package.el and its various use cases to
try looking for a solution along these lines; if I myself had such a
solution handy, I'd describe it, or maybe even implement it.

That being said, one possibility is to have a function in package.el
scan the user init file for the definition of package-load-list, and
process that line before loading the packages.  Or something similar.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> You are unfairly challenging me with stuff I know very little about
> and therefore might make suggestions that are laughable.

I'm sorry. That wasn't my intention at all. I fully expect anyone else
to jump in and point out flaws in my reasoning, or to propose
alternative solutions. I'm not trying to suggest that because you made
the suggestion, you also have to have all the answers.

The only reason I'm attacking the proposal of only calling
`package-initialize' in startup.el is that I believe it's not only
difficult but fundamentally impossible to find any solution. I
outlined the reasons for that in my other email:

    PROPOSITION: Calling `package-initialize' only in startup.el is
    impossible without one of the following consequences:

      1. It is impossible to customize `package-load-list' and
         `package-user-dir'.
      2. We must add a second init-file.
      3. The user cannot add package customizations to their init-file
         without using `after-init-hook'.
      4. Packages are not activated automatically.

    PROOF: [...]

And if somebody finds a flaw in that reasoning, then I'm all ears. But
otherwise, it seems futile to try to find a solution in this
direction.

> That being said, one possibility is to have a function in package.el
> scan the user init file for the definition of package-load-list, and
> process that line before loading the packages.

I really don't want to sound rude, but... *please* no. That would be a
horrible hack. It will fail in so many different circumstances:
package-load-list set in a different file, package-load-list set
dynamically, package-load-list set in an unconventional way,
package-load-list set by a macro, etc. etc. etc.

Emacs Lisp is Turing-complete. It is futile for Emacs to try to
understand it. In particular, Rice's theorem [1] says that all
nontrivial behavioral properties of Turing-complete code (such as
whether the user's init-file sets `package-load-list' or not) are
formally undecidable. So this approach is doomed to failure from the
very start.

[In fact, whether the user's init-file calls `package-initialize' is
also formally undecidable, so it will always be impossible to prevent
duplicate calls from being inserted on occasion...]

[1]: https://en.wikipedia.org/wiki/Rice%27s_theorem

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Mon, 21 Aug 2017 13:33:51 -0700
> Cc: emacs-devel@gnu.org
> 
> > That being said, one possibility is to have a function in package.el
> > scan the user init file for the definition of package-load-list, and
> > process that line before loading the packages.
> 
> I really don't want to sound rude, but... *please* no. That would be a
> horrible hack.

We do something very similar in other situations, like finding the
file-local variables.

> It will fail in so many different circumstances:
> package-load-list set in a different file, package-load-list set
> dynamically, package-load-list set in an unconventional way,
> package-load-list set by a macro, etc. etc. etc.

Nothing that cannot be handled.

> Emacs Lisp is Turing-complete. It is futile for Emacs to try to
> understand it.

That's a strange thing to say about a program which implements that
very language.

Anyway, the proposal still stays, and I just pointed out one way,
perhaps not the best one, out of the conundrum that you described as
insoluble.  I hope someone will come up with a better solution, or
implement what I proposed.  Otherwise, I predict that we will remain
at an impasse for some time more.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Mon, 21 Aug 2017 13:33:51 -0700
> Cc: emacs-devel@gnu.org
>
> > That being said, one possibility is to have a function in package.el
> > scan the user init file for the definition of package-load-list, and
> > process that line before loading the packages.
>
> I really don't want to sound rude, but... *please* no. That would be a
> horrible hack.

We do something very similar in other situations, like finding the
file-local variables.

> It will fail in so many different circumstances:
> package-load-list set in a different file, package-load-list set
> dynamically, package-load-list set in an unconventional way,
> package-load-list set by a macro, etc. etc. etc.

Nothing that cannot be handled.

> Emacs Lisp is Turing-complete. It is futile for Emacs to try to
> understand it.

That's a strange thing to say about a program which implements that
very language.

Anyway, the proposal still stays, and I just pointed out one way,
perhaps not the best one, out of the conundrum that you described as
insoluble.  I hope someone will come up with a better solution, or
implement what I proposed.  Otherwise, I predict that we will remain
at an impasse for some time more.

=====

> > > That being said, one possibility is to have a function in package.el
> > > scan the user init file for the definition of package-load-list, and
> > > process that line before loading the packages.
> >
> > I really don't want to sound rude, but... *please* no. That would be a
> > horrible hack.
>
> We do something very similar in other situations, like finding the
> file-local variables.

That is completely different. File-local variables conform to a fixed,
simple syntax which is -- in particular -- not Turing-complete, unlike
Emacs Lisp.

If you'd like to propose that `package-load-list' and friends only be
settable via file-local variables, then that would indeed solve the
problem. But that's a different proposal.

> > Emacs Lisp is Turing-complete. It is futile for Emacs to try to
> > understand it.
>
> That's a strange thing to say about a program which implements that
> very language.

I should have said: "Emacs Lisp is Turing-complete. It is futile for
Emacs to try to understand it in any way other than executing it."
That is what I meant. Static analysis is very different from just
running the program and seeing what happens.

> > It will fail in so many different circumstances:
> > package-load-list set in a different file, package-load-list set
> > dynamically, package-load-list set in an unconventional way,
> > package-load-list set by a macro, etc. etc. etc.
>
> Nothing that cannot be handled.

This is objectively false, and can be formally proven as such. I'll
try to explain better this time. (But the gist is, if you could
somehow "handle" this in general, you would also be able to solve the
halting problem.)

The issue is that determining anything nontrivial about the behavior
of an arbitrary piece of code is impossible in general, if the
language is Turing-complete. This is the content of Rice's theorem.
Stated in terms of Turing machines, it is as follows:

    Let S be a property of Turing machines. Assume that S is a
    nontrivial property, meaning that there exists at least one Turing
    machine with property S and at least one Turing machine without
    property S. Assume also that S is a functional property, meaning
    that it depends only on the inputs accepted and rejected by the
    Turing machine. Then, it is undecidable whether an arbitrary
    Turing machine has property S.

Stated in terms of package.el:

    Let S be a property of Emacs Lisp init-files. In particular, let S
    be the property that `package-load-list' is set to a given value T
    when the init-file is evaluated. It can be verified that the
    property S is nontrivial and functional. Thus, it is undecidable
    whether an arbitrary Emacs Lisp init-file sets `package-load-list'
    to any given value.

There is a loophole, however, since in the case of init-files, we are
willing to limit ourself to the case where running the code is
guaranteed to terminate (otherwise Emacs would not start up in the
first place).

This allows us to determine, in general, what an init-file does to
`package-load-list'. The catch is that we must actually evaluate the
init-file to do so. Why is this important?

It means that we can't correctly figure out the values of
`package-load-list' and `package-user-dir' before we have actually
loaded the init-file. It is only possible to do a `package-initialize'
operation correctly *after* loading the init-file, and any attempt to
the contrary will fail in general.

> Anyway, the proposal still stays

OK, but given that there is a formal proof that no general solution
can possibly exist, I don't think anybody will find one. If you're
proposing that we simply accept that package.el cannot be customized
in a dynamic way, then yes, that would solve the problem. But that's a
different proposal.

> I just pointed out one way, perhaps not the best one, out of the
> conundrum that you described as insoluble.

Your way relies on solving the halting problem. I don't think it
counts.

> I hope someone will come up with a better solution, or implement
> what I proposed. Otherwise, I predict that we will remain at an
> impasse for some time more.

I started this thread by proposing a specific, comprehensive solution
to the problem. Nobody has pointed out any flaws yet. Why is there an
impasse?

Best,
Radon

P.S. Sorry if I'm being a bit blunt here. But it's hard not to be
blunt in analyzing a problem when computability theory says there is
no solution.

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-22 06:52, Radon Rosborough wrote:
> Your way relies on solving the halting problem. I don't think it
> counts.

That's incorrect. We don't need to decide the halting problem — we just need to semi-decide it, which is trivial. Your email hints at this at one point, in fact.

Let me explain: you want to determine what loading init.el sets package-load-list to, and then you want to actually load Emacs.  Given this, if running init.el doesn't complete, then loading Emacs won't complete either.

Thus you don't need a decision procedure for the problem of determining whether init.el sets package-load-list, and what it sets it to. You just need a semi-decision procedure (one that either returns the value or enters an infinite loop).

This gives you an easy way out:

Start Emacs. Load init.el. Record the value of package-load-list. Exit Emacs.
Start Emacs again; set package-load-list to the value previously recorded. Run package-initialize. Load init.el.

There's a subtle recursivity issue if init.el depends on package-load-list, of course.  And this is a horrible approach :) But you don't need to solve the halting problem to determine what package-load-list gets set to and call package-intiialize with that.

Clément.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> That's incorrect. We don't need to decide the halting problem — we
> just need to semi-decide it, which is trivial.

You are of course correct. I didn't go through this in my last email
since I thought it was already too long and technical, but I will now.

> Start Emacs. Load init.el. Record the value of package-load-list. Exit
> Emacs.

Yep, this would work. Unfortunately, it's impossible in practice
without actually loading the init-file twice, which I hope everyone
agrees is a non-starter.

(In particular, the properties of the init-file may rely on arbitrary
filesystem and network calls, which cannot be analyzed without
actually performing them. Loading the init-file twice when it was
supposed to be loaded once is therefore technically incorrect, even if
there *probably* wouldn't be any correctness problems in practice.)

Because of this problem, the fact that the problem is actually
semi-decidable doesn't have any practical consequences, and so I
didn't mention it. I probably should have. Sorry about that. I
overgeneralized in my previous email, but I think that my general
point (this approach will not work in general) still stands.

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Radon Rosborough <radon.neon@gmail.com>
> Date: Mon, 21 Aug 2017 21:52:03 -0700
> Cc: emacs-devel@gnu.org
> 
> > We do something very similar in other situations, like finding the
> > file-local variables.
> 
> That is completely different. File-local variables conform to a fixed,
> simple syntax which is -- in particular -- not Turing-complete, unlike
> Emacs Lisp.

I think you forget that file-local variables support 'eval' forms.

> If you'd like to propose that `package-load-list' and friends only be
> settable via file-local variables, then that would indeed solve the
> problem. But that's a different proposal.

I proposed _an_idea_.  You forced me to go into details in an area
where my expertise leaves a lot to be desired, to say the least.  So
we shouldn't reject the idea because the details of the possible
implementation I proposed don't add up (which I'm still not sure they
don't), as I excused myself in advance.

Instead, we should look for a better implementation.  Perhaps we
should support a separate .package-exclude file under ~/.emacs.d,
where users would list packages they don't want to load.  After all,
other features have their files there, like abbrev_defs or .mailcap or
.authinfo.

Or maybe it'd be good enough to unload the packages not in the list,
after processing the user init file.

Or maybe people who set package-load-list should also be told to place
the call to package-initialize in their init files, but do it
manually, not automatically by Emacs.

Or something else.  There could be any number of possible solutions
for this issue, which IMO is minor compared to the larger issue of
leaving the user init file alone, letting the users manage its content
as they see fit.

> I should have said: "Emacs Lisp is Turing-complete. It is futile for
> Emacs to try to understand it in any way other than executing it."
> That is what I meant. Static analysis is very different from just
> running the program and seeing what happens.

And yet we do something similar in file-local variables and elsewhere,
like the .dir-locals.el files.  So there are solutions along these
lines that don't violate the basic principles, but still solve
practical problems in a way that is acceptable by the user community.

> > Anyway, the proposal still stays
> 
> OK, but given that there is a formal proof that no general solution
> can possibly exist, I don't think anybody will find one.

Only because you are trying to solve a larger problem than we need to
solve, and because you are reasoning in absolute terms, rather than in
practical engineering terms.

> I started this thread by proposing a specific, comprehensive solution
> to the problem. Nobody has pointed out any flaws yet. Why is there an
> impasse?

Because AFAICT no one likes your proposal.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> I think you forget that file-local variables support 'eval' forms.

This is different, since such `eval' forms can be processed just by
evaluating them. Whereas the implementation you proposed involved
determining the evaluation result of Lisp code without actually
evaluating it.

> I proposed _an_idea_. You forced me to go into details in an area
> where my expertise leaves a lot to be desired, to say the least.

I see now that your proposal was more general than I took it to be.
I'll try to refrain from overgeneralizing from now on. (I still may
point out flaws in particular implementation ideas, though.)

> Perhaps we should support a separate .package-exclude file under
> ~/.emacs.d, where users would list packages they don't want to load.

This is roughly equivalent to having a second init-file, which is
indeed a valid solution (although it has the disadvantage that users
will inevitably put package.el customizations in their init-files and
then get confused when they don't work).

> Or maybe it'd be good enough to unload the packages not in the list,
> after processing the user init file.

I think the resulting performance regressions make this impractical.
Not to mention that packages can, in general, run arbitrary Lisp code
in their autoloads which cannot be unloaded.

> Or maybe people who set package-load-list should also be told to place
> the call to package-initialize in their init files, but do it
> manually, not automatically by Emacs.

This wouldn't solve the problem of package customizations in the
init-file not working, which is the primary issue. (But just to be
clear, I'm not insisting that you be the one to provide a solution to
that problem just because you made the suggestion.)

> There could be any number of possible solutions for this issue,
> which IMO is minor compared to the larger issue of leaving the user
> init file alone, letting the users manage its content as they see
> fit.

Do you consider a one-time default init-file as violating the
principle of "leaving the user init file alone"? That's a perfectly
reasonable position; I just haven't heard you say that yet.

> And yet we do something similar in file-local variables and
> elsewhere, like the .dir-locals.el files.

I've already said that this is completely different, since file-local
variables can be evaluated directly, whereas anything in the init-file
depends on the runtime behavior of the entire rest of the init-file.

> So there are solutions along these lines that don't violate the
> basic principles, but still solve practical problems in a way that
> is acceptable by the user community.

Is "not allowing dynamic customization of package-load-list"
acceptable by the user community? Honest question. It wouldn't be
acceptable by me but then I don't use package.el.

> Only because you are trying to solve a larger problem than we need to
> solve, and because you are reasoning in absolute terms, rather than in
> practical engineering terms.

I'm just trying to point out that any ahead-of-time static analysis of
the init-file will necessarily fail in some circumstances. If we're OK
with that, then fine, but we have to be aware of it.

> > I started this thread by proposing a specific, comprehensive solution
> > to the problem. Nobody has pointed out any flaws yet. Why is there an
> > impasse?
>
> Because AFAICT no one likes your proposal.

Well, nobody (including you) has told me *why* they don't like my
proposal. If somebody would explain that, then I would stop bringing
it up.

Also, Mark Oteiza clearly likes my proposal, so I think "no one" is an
exaggeration:

    > Radon Rosborough <radon.neon@gmail.com> writes:
    > > Last week I posted an inquiry [1] about package.el's auto-insertion of
    > > code into the init-file. Six people weighed in, but no definitive
    > > conclusion was reached. I would like to propose concrete next steps,
    > > summarizing relevant parts of the discussion in the process.
    >
    > Thanks for pushing the issue.
    >
    > > ==> The proposal
    > >
    > > The `package--ensure-init-file' logic will be removed
    >
    > +1 for reverting.
    >
    > I will happily propose and write documentation for various use cases of
    > package.el, but I will not waste my time as long as the current solution
    > is in place.

In fact, I think Mark is the only person in this whole thread who has
made a specific comment directly on my proposal. (I still haven't
received replies from you or Stefan requesting a specific explanation
of your comments that my proposal is "a non-starter" or will not be
"significantly better".)

In the meantime, here is my analysis of the options we have:

* Require package.el customization to happen in file-local variables.
* Scan the init-file for customization of package.el, and accept that
  this will fail in many cases.
* Introduce a second init-file just for package.el.
* Don't allow package.el to be customized.
* Use a default init-file.
* Require users to understand that the package manager must be loaded
  before packages can be customized.
* Leave it as is, where Emacs modifies the init-file programmatically,
  and accept that this usually does the wrong thing.

Let me know if I missed anything.

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-22 20:09, Radon Rosborough wrote:
> Well, nobody (including you) has told me *why* they don't like my
> proposal. If somebody would explain that, then I would stop bringing
> it up.

I think I'm not a big fan of your proposal because the only problem I see with the status quo is that a call to package-initialize in the init file is required at all.  Given this, I don't see a change to the status quo that still requires a call to package-initialize in init.el as an improvement.

I think your proposal is well-designed and reasonable, and I think you made a good case for it, but it solves a problem that I don't see as a problem, and it doesn't solve the part that I'm bothered about.

I hope this makes sense!
Clément.

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

> I think we should instead explore the possibility that
> package-initialize will be called only in startup.el.

I think this is unlikely to work well.  I think a better long-term ideal
situation is either:

- call package-initialize before reading ~/.emacs (this will require
  finding some way to handle the case where the user wants to change
  package.el's configuration in his .emacs).
- call package-initialize automatically when needed, by having a more
  declarative way to configure packages in ~/.emacs.

The first approach looks easier from where I sit, but that's just
a gut feeling.

In the short term, we can tweak the current approach so that
package--ensure-init-file is called in fewer cases.  E.g. only call it
during package installation rather than at Emacs startup.


        Stefan

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 21 Aug 2017 05:04:58 -0400
> 
> > I think we should instead explore the possibility that
> > package-initialize will be called only in startup.el.
> 
> I think this is unlikely to work well.

As I said, we should explore this in more detail, and have a detailed
and agreed-upon list of reasons why it's not TRT.  Doing this kind of
stuff in startup.el is quite standard and natural.  If we don't
analyze this possibility and have the data and conclusions on file,
this possibility will continue to pop up, and we potentially could
miss an opportunity to provide a robust and much less annoying
solution than what we have now.

So if you can elaborate the "unlikely" part, please do.

> - call package-initialize before reading ~/.emacs (this will require
>   finding some way to handle the case where the user wants to change
>   package.el's configuration in his .emacs).
> - call package-initialize automatically when needed, by having a more
>   declarative way to configure packages in ~/.emacs.
> 
> The first approach looks easier from where I sit, but that's just
> a gut feeling.

This is hard to interpret without knowing "where you sit".

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> > > I think we should instead explore the possibility that
> > > package-initialize will be called only in startup.el.
> >
> > I think this is unlikely to work well.
>
> As I said, we should explore this in more detail, and have a detailed
> and agreed-upon list of reasons why it's not TRT.

PROPOSITION: Calling `package-initialize' only in startup.el is
impossible without one of the following consequences:

  1. It is impossible to customize `package-load-list' and
     `package-user-dir'.
  2. We must add a second init-file.
  3. The user cannot add package customizations to their init-file
     without using `after-init-hook'.
  4. Packages are not activated automatically.

PROOF:

  * Packages must be activated at some point, since otherwise
    consequence (4) holds. This activation must happen either before
    loading the init-file, during the init-file, or after loading the
    init-file.

  * Packages must be activated either before loading the init-file or
    during the init-file, since otherwise consequence (3) holds.

  * Packages cannot be activated during the init-file since this
    violates the premise that `package-initialize' is called only in
    startup.el.

  * Therefore, packages must be activated before loading the
    init-file.

  * Even though packages are activated before loading the init-file,
    it must still be possible for the user to customize
    `package-load-list' and `package-user-dir' before the activation
    takes place, since otherwise consequence (1) holds.

  * The user cannot put such customization in a second init-file,
    since otherwise consequence (2) holds.

  * There is no way for `package-load-list' and `package-user-dir' to
    be customized outside of an init-file. But we said such
    customization was necessary.

  * Reductio ad absurdum.

Can someone find a hole in this reasoning?

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-21 18:34, Radon Rosborough wrote:
>>>> I think we should instead explore the possibility that
>>>> package-initialize will be called only in startup.el.
>>>
>>> I think this is unlikely to work well.
>>
>> As I said, we should explore this in more detail, and have a detailed
>> and agreed-upon list of reasons why it's not TRT.
> 
> PROPOSITION: Calling `package-initialize' only in startup.el is
> impossible without one of the following consequences:
> 
>   1. It is impossible to customize `package-load-list' and
>      `package-user-dir'.
>   2. We must add a second init-file.
>   3. The user cannot add package customizations to their init-file
>      without using `after-init-hook'.
>   4. Packages are not activated automatically.
> 
> …
> Can someone find a hole in this reasoning?

I can't find an obvious flaw. But I wouldn't find it bad if customizing package-user-dir and package-load-list required an extra call to package-initialize — except for the fact that the first package-initialize call would have loaded everything already from the wrong user-dir, which is indeed an issue.

Re: Summary and next steps for (package-initialize)

[Timur Aydin]

On 8/21/2017 7:34 PM, Radon Rosborough wrote:
> PROPOSITION: Calling `package-initialize' only in startup.el is
> impossible without one of the following consequences:
> 
>    1. It is impossible to customize `package-load-list' and
>       `package-user-dir'.
>    2. We must add a second init-file.
>    3. The user cannot add package customizations to their init-file
>       without using `after-init-hook'.
>    4. Packages are not activated automatically.

On what ground was item 2. dismissed in previous discussions? If there 
is an init file specific to the package manager, it seems to me that all 
problems go away. The only disadvantage is that there is an extra init 
file now. I think this is a small price to pay to solve all of the nasty 
issues that come with trying to use the "one and only" emacs init file. 
Could somebody please summarize why it's such a fundamental rule that 
there can only be one emacs init file?

-- 
Timur

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> On what ground was item 2. dismissed in previous discussions?

I've been arguing against it on the grounds that if one Emacs package
out of several thousand needs an extra init-file to work right, while
all the others don't, it's probably a sign of bad design on the part
of the one package. In other words, it seems like needless complexity.

And besides, if package.el gets its own init-file to make its job
easier, why not Org? And Gnus? And Dired? And EPG? And Flymake? It
would be chaos.

> The only disadvantage is that there is an extra init file now.

Well, consider the confusion when people put customization of
package.el in their init-file and it doesn't work. I feel like we
might be trading a bunch of user confusion over package customizations
not working for a bunch of user confusion over package.el
customizations not working. And I feel like "put package-initialize in
your init-file" is an easier-to-understand resolution than "put
customizations of this specific list of variables into a second
init-file".

> Could somebody please summarize why it's such a fundamental
> rule that there can only be one emacs init file?

There's no rule; just a strong precedent that's been followed for
several decades (AFAIK).

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-22 18:58, Radon Rosborough wrote:
> I've been arguing against it on the grounds that if one Emacs package
> out of several thousand needs an extra init-file to work right, while
> all the others don't, it's probably a sign of bad design on the part
> of the one package. In other words, it seems like needless complexity.

I think I don't see package.el as one out of many, but rather as a core component of Emacs.  Given this, I'm inclined to be more tolerant of exceptions made just for that package.

I'd be fine, for example, with saving package.el's configuration in a separate file that gets loaded early.  That file doesn't need to be a full-fledged ELisp file.  It could be an alist of relevant keys and values, like a dir-local file.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 3339 bytes --]

> I think I'm not a big fan of your proposal because the only problem I
> see with the status quo is that a call to package-initialize in the
> init file is required at all.

Thank you for this explanation. It is really helpful, as it helps me
see why we have been talking at cross purposes.

> I think I don't see package.el as one out of many, but rather as a
> core component of Emacs. Given this, I'm inclined to be more
> tolerant of exceptions made just for that package.

This helps too. It made me realize that package.el really is core to
Emacs, which I previously failed to see just because I don't really
like package.el as a package manager.

> I'd be fine, for example, with saving package.el's configuration in
> a separate file that gets loaded early.

In light of what I've said above, I would be fine with this too.
Specifically, if we:

* removed all traces of `package--ensure-init-file'
* moved the call to `package-initialize' from after loading the
  init-file to before loading the init-file
* loaded an optional second init-file before `package-initialize'
* made absolutely sure that it's trivial to disable package.el
  permanently and unconditionally by setting
  `package-enable-at-startup' in this second init-file

then I would be perfectly happy and would not bother anyone about this
again. I do want to raise two possible points of concern, which I
personally am fine with but maybe other people aren't:

* doing this would instantly break the configuration of everyone who
  customizes package.el or uses it in a nonstandard way -- can anyone
  think of a way to maintain some backward compatibility so Drew can
  keep his configuration working from Emacs 20 to Emacs 26? ;)
* you will get a bunch of bug reports from people trying and failing
  to customize package.el in their normal init-file, I guarantee it --
  but this could at least be minimized by inserting big flashy
  warnings into the docstrings of all the relevant variables

> That file doesn't need to be a full-fledged ELisp file. It could be
> an alist of relevant keys and values, like a dir-local file.

Bad idea. In particular, this will make it impossible to customize
`package-load-list' so that packages are loaded from package.el by
default, but can be overridden from a local checkout, which is a
pattern I've seen very frequently in the wild.

Besides, if we're going to require it to be static, why not just say
you can only customize package.el via file-local variables in the
primary init-file? (I know, I know, at least you could line-wrap
things. But still.)

                              ~~~

Drew, Stefan, Eli -- would you be comfortable with this alternative
proposal, to add a second init-file as I've outlined above? It would
make package.el "just work" in all cases, without requiring any change
to the user's init-file, as long as users read docstrings before
customizing `package-load-list' etc., and as long as we're prepared to
sacrifice a little backwards compatibility (although maybe we can be
smart about it and make the change pretty painless).

Angelo -- ... sorry :P

          I still think that calling package-initialize in the
          init-file is "the right thing" from an architecture and
          modularity perspective, but having a second init-file may be
          a more pragmatic solution.

[-- Attachment #2: Type: text/html, Size: 4347 bytes --]

Re: Summary and next steps for (package-initialize)

[Angelo Graziosi]

Hi Radon,

> Il 23 agosto 2017 alle 7.17 Radon Rosborough <radon.neon@gmail.com> ha scritto:
> [...]
> Drew, Stefan, Eli -- would you be comfortable with this alternative
> proposal, to add a second init-file as I've outlined above? It would
> make package.el "just work" in all cases, without requiring any change
> to the user's init-file, as long as users read docstrings before
> customizing `package-load-list' etc., and as long as we're prepared to
> sacrifice a little backwards compatibility (although maybe we can be
> smart about it and make the change pretty painless).
> 
> Angelo -- ... sorry :P
> 
>           I still think that calling package-initialize in the
>           init-file is "the right thing" from an architecture and
>           modularity perspective, but having a second init-file may be
>           a more pragmatic solution.


I don't know which problem you are trying to solve and its details, but the idea to introduce an init file to fix it sound wrong to me.

Why not to introduce an init file to fix something in foo1.el or foo2.el or... f..?

No, what ever you are trying to fix is 'wrong'... :P

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 588 bytes --]

> I don't know which problem you are trying to solve and its details

Making the built-in package manager work without any user
intervention.

> Why not to introduce an init file to fix something in foo1.el or
> foo2.el or... f..?
>
> No, what ever you are trying to fix is 'wrong'... :P

I would agree in the case of any other package. But I've been
convinced that the built-in package manager deserves a special place
in the startup process.

Note, if you look earlier in this thread, I said literally exactly the
same thing as you, for the same reason. I have however changed my
mind.

[-- Attachment #2: Type: text/html, Size: 854 bytes --]

RE: Summary and next steps for (package-initialize)

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 2732 bytes --]

* doing this would instantly break the configuration of everyone who

  customizes package.el or uses it in a nonstandard way -- can anyone

  think of a way to maintain some backward compatibility so Drew can

  keep his configuration working from Emacs 20 to Emacs 26? ;)

 

To be clear: I do not use the package system (except rarely, if I need to check a problem reported by someone).

 

This is partly because I don't use packages, in general. And it is mostly because I use multiple versions of Emacs and multiple configurations, for testing, and I don't want to be bothered with telling package.el what to do and not do each time I use Emacs. My use case is, I think, different from someone who typically wants the same stuff available and turned on in each Emacs session. 

 

So I don't imagine that what you envision (which I haven't followed in detail) should affect me, personally. If I can easily use or not use the package system for any given Emacs session then it will likely be OK by me. (I would in fact like that to be even easier than now. "Installing" packages and enabling them seems too monolithic, to me. It seems to assume that a given user has only one set of preferences, which s?he uses all the time.)

 

My feedback in this thread is not particularly about how it affects me, personally. It is about how (I think) it affects Emacs users generally (where by "generally" I mean taking into account their diverse use patterns, not just "mostly" as in a majority of users).

 

Drew, Stefan, Eli -- would you be comfortable with this alternative

proposal, to add a second init-file as I've outlined above? It would

make package.el "just work" in all cases, without requiring any change

to the user's init-file, as long as users read docstrings before

customizing `package-load-list' etc., and as long as we're prepared to

sacrifice a little backwards compatibility (although maybe we can be

smart about it and make the change pretty painless).

 

I'm not sure I understand it, i.e., just what changes for users. And just what compatibility loss is there?

 

For example, you say:

 

made absolutely sure that it's trivial to disable package.el permanently and unconditionally by setting `package-enable-at-startup' in this second init-file

 

But in your proposal the first init file is apparently loaded after `package-initialize' is done. So it sounds like users will have `package-initialize' inflicted on them unconditionally. How does that jibe with choosing not to use the package system (ever, or for a particular Emacs session)? How is it trivial to tell package.el "Hands off", if `package-initialize' is always done?

[-- Attachment #2: Type: text/html, Size: 7205 bytes --]

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]

> To be clear: I do not use the package system

Yeah, neither do I. So we're on the same page here.

> This is partly because I don't use packages, in general.

Remind me to ask why, sometime.

> And it is mostly because I use multiple versions of Emacs and
> multiple configurations, for testing, and I don't want to be
> bothered with telling package.el what to do and not do each time I
> use Emacs.

I know for a fact that there are alternative package managers which
are designed for this sort of use case, unlike package.el.

> (I would in fact like that to be even easier than now. "Installing"
> packages and enabling them seems too monolithic, to me. It seems to
> assume that a given user has only one set of preferences, which s?he
> uses all the time.)

I doubt this will change soon, as it's a fundamental part of how
package.el works. But that's why we have alternative package managers.

> But in your proposal the first init file is apparently loaded after
> `package-initialize' is done.

The order is:

1. Load package.el init-file
2. Run package-initialize, unless package-enable-at-startup was un-set
   in (1)
3. Load regular init-file
4. Nothing else is done afterwards, unlike currently

> So it sounds like users will have `package-initialize' inflicted on
> them unconditionally.

Wrong. You just (setq package-enable-at-startup nil) in the secondary
init-file, and are never bothered by package.el again. This is much
better than the current situation, where you still have to (setq
package-enable-at-startup nil), but furthermore you have to put an
advice on an internal package.el function in order to avoid
(package-initialize) from being inserted under some circumstances.

[-- Attachment #2: Type: text/html, Size: 2296 bytes --]

RE: Summary and next steps for (package-initialize)

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 2088 bytes --]

The order is:

 

1. Load package.el init-file

2. Run package-initialize, unless package-enable-at-startup was un-set in (1)

3. Load regular init-file

4. Nothing else is done afterwards, unlike currently

 

You just (setq package-enable-at-startup nil) in the secondary init-file, and are never bothered by package.el again. 

 

Again? How about not being bothered the first time? It sounds like `package-initialize' is evaluated the first time, unconditionally.

 

And it sounds like, even if I don't use the package system, I will have to edit a secondary init file (the first one read), just to tell package.el "Hands off!" And if I want to try a package in a particular Emacs session then I need to once again edit that first ("secondary") init file, to enable the package system. Is that right?

 

It's hard to believe this has to be so convoluted. (But I understand that you are saying that what you propose is less convoluted than what we have now.)

 

It seems to me that use of package.el should be just like use of any other library. Users should explicitly opt in. But I understand that you've said that it has been decided to enable the package system by default for everyone, at the outset. 

 

That seems like a mistake, to me. And the only argument I have noticed in favor of it is that some users make mistakes and get confused. For that, I'd say start by improving the doc and making things as simple as possible for them - but without the automatic turn-it-all-on-for-everyone approach.

 

Users are not idiots. We all make mistakes and get confused. If the doc is poor and that is a source of confusion then that is the first thing to fix. If we need to simplify a few things, then it should be done. If it would help to add tutorials or videos then that could be done.

 

It should not be the end of the world for a user (apparently the typical Emacs user) to tell Emacs to activate the package system, i.e., opt in. It doesn't matter, IMO, if 99% of the users want to opt in; it should still be opt-in.

[-- Attachment #2: Type: text/html, Size: 6212 bytes --]

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 3396 bytes --]

> Again? How about not being bothered the first time? It sounds like
> `package-initialize' is evaluated the first time, unconditionally.

This is wrong. `package-initialize' is called only once, in step (2).
This step happens before loading the primary init-file, but after
loading the secondary init-file. If you set
`package-enable-at-startup' to nil in the secondary init-file, then
`package-initialize' will never be evaluated under any circumstances.

> And it sounds like, even if I don't use the package system, I will
> have to edit a secondary init file (the first one read), just to
> tell package.el "Hands off!"

You will have to do that only once in the entire history of your
dotfiles. No longer is `package-initialize' called after loading the
primary init-file, so a single `setq' is all that is needed to
permanently disable package.el.

> And if I want to try a package in a particular Emacs session then I
> need to once again edit that first ("secondary") init file, to
> enable the package system. Is that right?

No, it's wrong. To try a package, all you have to do is run M-x
package-initialize. In fact, if it's a new package you are installing,
M-x package-install will suffice, as package.el is initialized
automatically when you try to use it.

> It's hard to believe this has to be so convoluted.

It seems very simple to me. The package manager is initialized before
reading the init-file, unless you tell it not to do that. And in any
case, you can initialize it later. The only nonstandard part of this
is that the code to disable automatic initialization goes in a
separate file.

> (But I understand that you are saying that what you propose is less
> convoluted than what we have now.)

Oh heck yes. Even I don't really understand the full ramifications of
the current system.

> It seems to me that use of package.el should be just like use of any
> other library. Users should explicitly opt in. But I understand that
> you've said that it has been decided to enable the package system by
> default for everyone, at the outset.

The rationale for this decision was that we want to treat package.el
as a core part of Emacs rather than as an extension. If you want to
debate that, then fine (I might join you), but let's take it as
granted for a moment. If package.el is a core part of Emacs, then we
expect all of its features to work by default. And we also expect
libraries installed using package.el to work similarly to libraries
that are shipped with Emacs. That's why it's enabled by default.

Note that I'm not really complaining about this decision, even though
I don't like package.el and never use it. It seems reasonable to me
for the built-in package manager to act like this, even if I don't use
said package manager.

> It doesn't matter, IMO, if 99% of the users want to opt in; it
> should still be opt-in.

It really depends on whether you view package.el as a core part of
Emacs or not. After all, the menu and tool bars are turned on by
default; VC is turned on by default (even though lots of people turn
it off and use Magit); why shouldn't the package manager be turned on
by default?

I think it's fine for stuff to be turned on by default as long as it's
easy to turn it off again and swap in something else:

     (menu-bar-mode -1)
     (tool-bar-mode -1)
     (setq vc-handled-backends nil)
     (setq package-enable-at-startup nil)

[-- Attachment #2: Type: text/html, Size: 4430 bytes --]

RE: Summary and next steps for (package-initialize)

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 4123 bytes --]

> It seems to me that use of package.el should be just like use of any

> other library. Users should explicitly opt in. But I understand that

> you've said that it has been decided to enable the package system by

> default for everyone, at the outset.

 

The rationale for this decision was that we want to treat package.el

as a core part of Emacs rather than as an extension. 

 

Lots of things are a core part of Emacs and not extensions, but we don't necessarily turn them on by default. `cua-mode' is a core part of Emacs, but it is off by default (good). `delete-selection-mode' is a core part of Emacs, but is off by default (bad).

 

We're not talking about whether package.el is core or external (at least I'm not). And we're not talking about loading package.el by default. We're talking about activating the package system by default.

 

If the rationale for the decision was only that you want to treat package.el as a core part of Emacs rather than as an extension then that's a bad rationale. You don't need to turn something on just because it is a core part of Emacs.

 

If you want to debate that, then fine (I might join you), but let's take it as

granted for a moment. If package.el is a core part of Emacs, then we

expect all of its features to work by default. 

 

All of `cua-mode' works by default, but it is not turned on by default.

 

And we also expect libraries installed using package.el to work similarly to libraries that are shipped with Emacs. 

 

Dunno what that means. Work similarly how? How could they work differently?

 

That's why it's enabled by default.

 

So that's a second rationale. That one is even worse than the "because it's core" rationale. Nebulous.

 

Why does enabling it by default follow from wanting libraries installed by it to "work similarly" to libraries shipped with Emacs?

 

Note that I'm not really complaining about this decision, even though

I don't like package.el and never use it. It seems reasonable to me

for the built-in package manager to act like this, even if I don't use

said package manager.

 

I still haven't seen an argument why we shouldn't have users opt in to turn on use of the package system. Other than the simple observation that some users have gotten confused about how to appropriately turn it on.

 

Sounds like the decision amounts to throwing up one's hands and exclaiming that since some users don't know how to properly turn it on we should just go ahead and turn it on at the outset for all users. Copout?

 

> It doesn't matter, IMO, if 99% of the users want to opt in; it

> should still be opt-in.

 

It really depends on whether you view package.el as a core part of

Emacs or not. 

 

I don't think so. See above. What does "core part" have to do with it? (And what does "work similarly" have to do with it?)

 

After all, the menu and tool bars are turned on by default; VC is turned on by default (even though lots of people turn it off and use Magit); why shouldn't the package manager be turned on by default?

 

The question is not why shouldn't it. The question is why should it. No good answer, so far.

 

`cua-mode' is not turned on by default. Yet its behavior is used by 90% of users outside of Emacs. `delete-selection-mode' is not turned on by default (but it should be). `transient-mark-mode' was not turned on by default for decades (it finally was, thank goodness, but only after a lot of time and debate). And so on. 

 

I think it's fine for stuff to be turned on by default as long as it's

easy to turn it off again and swap in something else:

 

     (menu-bar-mode -1)

     (tool-bar-mode -1)

     (setq vc-handled-backends nil)

     (setq package-enable-at-startup nil)

 

I'll certainly do that. Along with (electric-indent-mode -1).

 

(But apparently I'll need to put the `package-enable-at-startup' setting in another, "secondary" init file, as an exception. Not a big deal. Kinda clunky though.)

[-- Attachment #2: Type: text/html, Size: 11732 bytes --]

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-23 22:27, Drew Adams wrote:
> I still haven't seen an /argument *why*/ we shouldn't have users /opt
> in/ to turn on use of the package system. Other than the simple
> observation that some users have gotten confused about how to
> appropriately turn it on.

I can try to explain my position :)

I've come to consider package managers as a core part of a flexible text editor like Emacs.  I understand some people might not want to use package.el, or even any package manager — that's totally fine, and Emacs must work great for them.

But on the other hand, Emacs has now grown a very large collection of external packages: many features and programming languages are not supported out of the box any more, and instead need to be installed separately.  I'd like this to be as easy as possible.  Enabling the package manager by default is one way to do this.

I'd like us to give more visibility to Emacs packages, because I commonly run into people who use package in Atom or Visual Studio Code, but not in Emacs (and that's not because Emacs provides the corresponding features without an extra packages)

(Also: having the package manager enabled by default allows us to split some functionality out of core and into separate packages)

> `delete-selection-mode' is not turned on by default (but it should be). `transient-mark-mode' was not turned on by default for decades (it finally was, thank goodness, but only after a lot of time and debate). And so on. 

I agree with this too, and I see package.el as another feature that should be enabled by default.

Clément.

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> > I still haven't seen an /argument *why*/ we shouldn't have users /opt
> > in/ to turn on use of the package system. Other than the simple
> > observation that some users have gotten confused about how to
> > appropriately turn it on.
>
> ... external packages ... need to be installed separately.
> I'd like this to be as easy as possible.

I've already said that I too think it should be easy.  I think it
should be easier than it is now, more obvious, and less error prone.

Why isn't the package-system doc improvement started, as the first
thing to do?  You might be surprised how much better doc might help
with user confusion.  Seems like that would be a good thing for those
who are familiar with the package mgr and are strong proponents of it
to work on.

> Enabling the package manager by default is one way to do this.

One way, sure.  Not necessary, just to make things easy, but it is
presumably one way.  Overkill, I'd suggest.

If you have an Emacs feature that is unclear or overly complicated
to use, and yet it is very popular, it seems like the first thing
to work on would *not* be just turning it on by default but making
it clearer (both the interface and the doc) and easier to use.

> I'd like us to give more visibility to Emacs packages, because I commonly
> run into people who use package in Atom or Visual Studio Code, but not in
> Emacs (and that's not because Emacs provides the corresponding features
> without an extra packages)

That is *not* a great reason to enable something by default.

That is a bit like the fellow who adds a new feature and wants it
turned on by default because it is a shiny new feature.  "Turn it
on so people are aware of it", we hear occasionally.  Not a good
argument.

The use of packages by Emacs users has taken off very quickly and
sees no sign of abating.  I see no argument that we need to turn
on the package manager by default just to let users know that there
is a package manager and there are packages.  Emacs users know both.
YAGNI.  That's not where they need help with packages.

> (Also: having the package manager enabled by default allows us to
> split some functionality out of core and into separate packages)

All you need is the package mgr for that.  You don't need it enabled
by default.

> > `delete-selection-mode' is not turned on by default (but it should be).
> `transient-mark-mode' was not turned on by default for decades (it finally
> was, thank goodness, but only after a lot of time and debate). And so on.
> 
> I agree with this too, and I see package.el as another feature that should
> be enabled by default.

The question (still) is why.

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-23 23:21, Drew Adams wrote:
> The question (still) is why.

Sorry. You asked for my reasons, and I gave them :) I'm sure others can chip in if they feel it's necessary.

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> > The question (still) is why.
> 
> Sorry. You asked for my reasons, and I gave them :) I'm sure others can chip
> in if they feel it's necessary.

OK, thanks.  (No reason to be sorry.)

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 4492 bytes --]

> > And we also expect libraries installed using package.el to work
> > similarly to libraries that are shipped with Emacs.
>
> Dunno what that means. Work similarly how? How could they work
> differently?

Libraries that are shipped with Emacs are made available automatically
without anything needing to be put in the init-file. And this happens
during startup, before the init-file is loaded.

OTOH, if package.el needs to be set up manually, as you propose, then
libraries that are installed using package.el would not be made
available automatically unless (package-initialize) was in the user's
init-file, and they would not be made available until part-way through
the loading of the init-file.

Now you may prefer the latter, but you can't argue that it's more
consistent behavior.

> Why does enabling it by default follow from wanting libraries
> installed by it to "work similarly" to libraries shipped with Emacs?

Is this clear now?

> I still haven't seen an argument why we shouldn't have users opt in
> to turn on use of the package system.

It's because most people want to use the package system (you admitted
this yourself). As you observed, we have a large assortment of tidbits
in Emacs core, and whether they are enabled by default depends on
whether most users want them enabled by default.

> `cua-mode' is not turned on by default. Yet its behavior is used by
> 90% of users outside of Emacs.

Irrelevant. Most people don't use it *inside* of Emacs, so it's
disabled by default.

> `delete-selection-mode' is not turned on by default (but it should
> be).

I'm not saying the decisions about defaults have been made flawlessly.
But it's pretty hard to argue that it's more common to want the
package manager disabled than it is to want it enabled. Defaults are
supposed to eliminate work for the user; what good are they if they
don't align with what most people want?

> > (setq package-enable-at-startup nil)
>
> I'll certainly do that. Along with (electric-indent-mode -1).

As I'm sure you're aware, you're very much in the minority in both of
these things. So you probably shouldn't be surprised that the defaults
are not suitable for you. I'm aware that I'm in the minority in using
a package manager other than package.el, so I have no qualms with
having to override the defaults.

> (But apparently I'll need to put the `package-enable-at-startup'
> setting in another, "secondary" init file, as an exception. Not a
> big deal. Kinda clunky though.)

Agreed, it's clunky. But also agreed, it's not a big deal: and
considering how many other problems we solve by doing it this way, I
think it's a pretty good tradeoff to make.

> Why isn't the package-system doc improvement started, as the first
> thing to do?

Because the current situation (Emacs modifies the user's init-file) is
a catastrophe, and fixing that is an ASAP priority. Also, people (e.g.
Mark Oteiza) are reluctant to document package.el when the current
behavior is clearly broken.

> Seems like that would be a good thing for those who are familiar
> with the package mgr and are strong proponents of it to work on.

I agree but the docs improvements are orthogonal to fixing the code.
Indeed, improving the documentation would be our top priority if we
were going to disable package.el by default, which is what you want.
But we're not going to do that (based on what I've heard from everyone
else), so the docs are not necessarily our top priority.

> > I'd like us to give more visibility to Emacs packages, because I
> > commonly run into people who use package in Atom or Visual Studio
> > Code, but not in Emacs (and that's not because Emacs provides the
> > corresponding features without an extra packages)
>
> That is *not* a great reason to enable something by default.

Actually, it's a great reason. You know how GitHub is kicking GNU's
tail in marketing Atom? Well, "built-in package manager" is literally
on the front page of their website, right next to "use productively
without ever touching a config file". I'd say this is an indication
that people care very much indeed about their text editor coming with
a package manager that works right out of the box.

> Emacs users know both.

No, they don't. At least not the sort of users we want to attract, who
don't know anything about Emacs. Yes, all the people on emacs-devel
know this stuff, but emacs-devel is about 1% of Emacs users, and about
0.01% of the Emacs users we could have if Emacs worked better out of
the box.

[-- Attachment #2: Type: text/html, Size: 5951 bytes --]

RE: Summary and next steps for (package-initialize)

[Drew Adams]

[-- Attachment #1: Type: text/plain, Size: 13049 bytes --]

> > And we also expect libraries installed using package.el to work

> > similarly to libraries that are shipped with Emacs.

> 

> Dunno what that means. Work similarly how? How could they work

> differently?

 

Libraries that are shipped with Emacs are made available automatically

without anything needing to be put in the init-file. And this happens

during startup, before the init-file is loaded.

 

Yes, just by putting them in (the default value of) `load-path' - no? That's not all that the pkg mgr does, IIUC. If it were then I guess we wouldn't be having this discussion.

 

OTOH, if package.el needs to be set up manually, as you propose, 

 

I'm not aware that I proposed anything particular. I asked a few questions, and expressed some doubt, about (what I understand of) some proposals.

 

then libraries that are installed using package.el would not be made

available automatically unless (package-initialize) was in the user's

init-file, and they would not be made available until part-way through

the loading of the init-file.

 

The way things worked before is that a user put libraries in `load-path'. Which is what emacs -Q does, no?

 

That's a user putting something in a particular location. And then possibly loading it. Lots of Emacs "core" stuff is in the same boat: in `load-path', waiting passively to be loaded. Possibly autoloaded. But in any case it is the user who loads it, directly or indirectly. (Some core stuff is preloaded.)

 

The difference is that for a non-default library the user decides where to put it - the user might have customized `load-path'.

 

All of that still seems a fair distance from pkg-mgr behavior.

 

Now you may prefer the latter, but you can't argue that it's more

consistent behavior.

 

I said nothing about preferring anything about how to use `package-initialize'. My naive preference is not to have a`package-initialize' and so not to have to worry about where it is placed or when it is evaluated.

 

> Why does enabling it by default follow from wanting libraries

> installed by it to "work similarly" to libraries shipped with Emacs?

 

Is this clear now?

 

Not at all. Libraries shipped with Emacs do not, so far, use the pkg mgr or anything like it, afaik. They instead "work similarly" to how users have long accessed libraries that are not shipped with Emacs: explicit load or autoload from `load-path' - no pkg mgr-like behavior.

 

> I still haven't seen an argument why we shouldn't have users opt in

> to turn on use of the package system.

 

It's because most people want to use the package system (you admitted

this yourself). 

 

Actually, I think I probably said that many users now use the package system. I don't recall saying anything about what most people want.

 

I haven't said that we should do away with the pkg system. I said I haven't seen a good argument why we shouldn't have users opt in, to turn it on.

 

As you observed, we have a large assortment of tidbits

in Emacs core, and whether they are enabled by default depends on

whether most users want them enabled by default.

 

Again, I think you might be putting words in my mouth. I don't think that I said that the tidbits that we have chosen to enable by default were enabled because most users want them enabled by default. In fact, in the past I've argued that sometimes Emacs dev has decided to enable some tidbits that I doubt most users wanted enabled by default.

 

But all of that is, I think, irrelevant, or at least I don't see the relation.

 

> `cua-mode' is not turned on by default. Yet its behavior is used by

> 90% of users outside of Emacs.

 

Irrelevant. Most people don't use it *inside* of Emacs, so it's

disabled by default.

 

I don't think that's the reason it was not enabled. At least that's not my recollection of discussions about whether to enable CUA mode.

 

More like chicken vs egg. If Emacs had decided to enable it by default 20 years ago then perhaps most users would use it inside Emacs - dunno.

 

But again, it's not very relevant to this discussion, I think, whether most users use CUA or would use it. I didn't claim that most users want the pkg system enabled by default. Perhaps you did; I don't think I did. I have no good idea about what most Emacs users want in that regard.

 

(FWIW, I haven't seen a poll of the users yet, I think, on any question, even though RMS has requested multiple times that Emacs dev "poll the users" about this and that.)

 

> `delete-selection-mode' is not turned on by default (but it should

> be).

 

I'm not saying the decisions about defaults have been made flawlessly.

But it's pretty hard to argue that it's more common to want the

package manager disabled than it is to want it enabled. 

 

Dunno why you say that, which I guess really means that you think it is easier to argue that it's more common for users to want it enabled. What evidence do you have for that? I never argued that most users want the pkg mgr disabled.

 

Defaults are supposed to eliminate work for the user; what good are they if they don't align with what most people want?

 

No one has shown that most users want wrt automatic pkg-mgt enabling, afaik.

 

> > (setq package-enable-at-startup nil)

> I'll certainly do that. Along with (electric-indent-mode -1).

 

As I'm sure you're aware, you're very much in the minority in both of

these things. So you probably shouldn't be surprised that the defaults

are not suitable for you. I'm aware that I'm in the minority in using

a package manager other than package.el, so I have no qualms with

having to override the defaults.

 

I too have no qualms with overriding defaults.

 

And I already said that my contribution here is not related to my own use or preference wrt pkgs. I am not asking that Emacs set its default pkg-mgr behavior to suit my use - not at all, as I think I've made clear.

 

> (But apparently I'll need to put the `package-enable-at-startup'

> setting in another, "secondary" init file, as an exception. Not a

> big deal. Kinda clunky though.)

 

Agreed, it's clunky. But also agreed, it's not a big deal: and

considering how many other problems we solve by doing it this way, I

think it's a pretty good tradeoff to make.

 

I can't judge whether you are right about how many other problems your proposal will solve. Maybe you are. But if the clamor of support for your proposal so far is any indication, it might not be so many.

 

> Why isn't the package-system doc improvement started, as the first

> thing to do?

 

Because the current situation (Emacs modifies the user's init-file) is

a catastrophe, and fixing that is an ASAP priority. 

 

Was it considered so before you proposed your proposal?

 

I too don't like the idea of Emacs modifying user init files. But we've lived with that for at least as long as we've had Customize - many, many years.

 

I advise against not using a `custom-file', but I wouldn't say that the current situation of Emacs writing to init files is "a catastrophe". That's a bit hyperbolic.

 

How dire is the situation wrt the pkg system, realistically? I see the occasional question here and there about packages not being enabled after "installing" them, with the inevitable answer being to just add `package-initalize' to your init file. I don't see such questions a lot, frankly, but I do see them.

 

I certainly do not see lots of questions about complicated situations where `package-initialize' was called at the wrong time etc. I'm sure that arises, but those questions are not the ones I see often. Maybe I don't hang out where all the complex use cases are reported as problems.

 

Again, don't get me wrong. I am not saying the pkg mgr needs no improvement.

 

I'm questioning whether a good dose of better doc etc. wouldn't go a long way, and wondering why there isn't a first emphasis on that: getting out the good word to those who are confused about the simple need to call `package-initialize' (somewhere, at least).

 

Take care of that confusion/misunderstanding and my guess is that you'll take care of most of the user questions about enabling pkgs. (There, I said it: "most". Just a guess, though.)

 

Also, people (e.g. Mark Oteiza) are reluctant to document package.el when the current behavior is clearly broken.

 

Too bad, truly.

 

But it seems to work well enough that zillions of users make good use of it everyday. As I say, I don't see zillions of questions about it.

 

I do see some, which often boil down to this answer: call `package-initialize'. And I also see lots of other questions, many of which are also pretty simple to answer - questions about quoting or variable behavior or whatever.

 

We have all kinds of users, and some have questions, some of which have simple answers. Do you not think that most user problems/confusion with the pkg system can be answered fairly simply? You talk about "most users". Do you really think that most pkg users who have problems have the more complex kinds of problems for which your proposal is intended?

 

I don't expect so, but I could be wrong. I think better doc would greatly help most pkg users. It's a shame that someone might take the point of view that because the pkg system has some problems they wouldn't want to bother to document it better. That smells a bit like a copout.

 

> Seems like that would be a good thing for those who are familiar

> with the package mgr and are strong proponents of it to work on.

 

I agree but the docs improvements are orthogonal to fixing the code.

Indeed, improving the documentation would be our top priority if we

were going to disable package.el by default, which is what you want.

 

It could reasonably be an immediate priority whether or not the pkg system is disabled by default. It is the users of the pkg system who have the difficulties with it, mostly minor I expect. It's not the non-users like I who most need better doc for it.

 

But we're not going to do that (based on what I've heard from everyone

else), so the docs are not necessarily our top priority.

 

> > I'd like us to give more visibility to Emacs packages, because I

> > commonly run into people who use package in Atom or Visual Studio

> > Code, but not in Emacs (and that's not because Emacs provides the

> > corresponding features without an extra packages)

> 

> That is *not* a great reason to enable something by default.

 

Actually, it's a great reason. You know how GitHub is kicking GNU's

tail in marketing Atom? 

 

Sorry, I'm not in such a race. Not one of my goals for Emacs. Never has been.

 

Well, "built-in package manager" is literally on the front page of their website, right next to "use productively without ever touching a config file". I'd say this is an indication that people care very much indeed about their text editor coming with a package manager that works right out of the box.

 

(Do ours care enough to improve the docs and usability?)

 

Anyway, there the question was not about caring whether the editor comes with a working pkg mgr. It was about whether we especially need to give more visibility to the fact that Emacs has packages. Not the same thing.

 

> The use of packages by Emacs users has taken off very quickly and

> sees no sign of abating.  I see no argument that we need to turn

> on the package manager by default just to let users know that there

> is a package manager and there are packages.  Emacs users know both.

> YAGNI.  That's not where they need help with packages.

 

No, they don't [know both]. At least not the sort of users we want to attract, who don't know anything about Emacs. 

 

Emacs users do know both that there is a package manager and that there are packages, I think. Do you really doubt that?

 

You are changing the question in order to disagree, I think, by saying that non-users - but people you want to become users - do not know both things.

 

Yes, all the people on emacs-devel know this stuff, but emacs-devel is about 1% of Emacs users, and about 0.01% of the Emacs users we could have if Emacs worked better out of the box.

 

Poll Emacs users to see if they know Emacs has 3rd-party packages. I see user questions, at all levels, all the time. From my vantage point Emacs users certainly are aware of packages.

 

(Oh, and my guess is that emacs-devel is much less than 1% of Emacs users - probably much less than .01%. Whatever the ratio, it will be interesting to see the expected 100-fold increase in the number of users from implementing your proposal. ;-))

[-- Attachment #2: Type: text/html, Size: 31736 bytes --]

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 6787 bytes --]

Drew,

Some of this is a little redundant because it's already been said by
other people. Sorry about that.

> > Libraries that are shipped with Emacs are made available automatically
> > without anything needing to be put in the init-file. And this happens
> > during startup, before the init-file is loaded.
>
> Yes, just by putting them in (the default value of) `load-path' -
> no? That's not all that the pkg mgr does, IIUC. If it were then I
> guess we wouldn't be having this discussion.

You are wrong. Libraries that are shipped with Emacs are put in the
load path and have their autoloads evaluated, and this is exactly what
the package manager does to activate a third-party package -- nothing
more, nothing less.

In particular, the package manager *certainly* does not actually load
any features; that would be unacceptable.

> My naive preference is not to have a `package-initialize' and so not
> to have to worry about where it is placed or when it is evaluated.

What? How would package code get added to the load path, then? Would
users do that manually? That would defeat the entire purpose of
package.el, namely to add package code to the load path for you.

Honestly, I think the package.el-config-file proposal gets the closest
to your desired situation, by making it so that `package-initialize'
need not be called from your code, and you do not have to think about
where to place it.

> No one has shown that most users want wrt automatic pkg-mgt
> enabling, afaik.

That's because most of us consider it common knowledge. I challenge
you to either (1) find somebody else here who thinks that this
statement is false, or (2) suggest a practical way we could test its
validity.

> > As you observed, we have a large assortment of tidbits in Emacs
> > core, and whether they are enabled by default depends on whether
> > most users want them enabled by default.
>
> [...]
>
> But all of that is, I think, irrelevant, or at least I don't see the
> relation.

Because most users want to use a package manager, it should be enabled
by default. That is my argument, and that is why what I said is
relevant. It is irrelevant whether the default enable/disable state of
other Emacs features has been decided correctly.

> I can't judge whether you are right about how many other problems
> your proposal will solve.

Eli, at least, has proposed exactly the same as what I'm suggesting,
and had this to say about the package.el-specific config file:

    the problem we are solving with this file is a much larger one,
    and affects many more users [than the problem of dealing with a
    new config file].

> > Because the current situation (Emacs modifies the user's
> > init-file) is a catastrophe, and fixing that is an ASAP priority.
>
> Was it considered so before you proposed your proposal?

Yes. See https://www.reddit.com/r/emacs/comments/56fvgd for example.

> I too don't like the idea of Emacs modifying user init files. But
> we've lived with that for at least as long as we've had Customize -
> many, many years.

Customize never modified the init-file automatically, without asking.

> How dire is the situation wrt the pkg system, realistically?

It's impossible to answer this objectively. But the current situation
reflects a horrible design flaw, and I think it's important that core
Emacs code reflects good design. I think everyone else here agrees
with me there, although we may all have different opinions on what
that good design should be.

> I'm questioning whether a good dose of better doc etc. wouldn't go a
> long way, and wondering why there isn't a first emphasis on that:
> getting out the good word to those who are confused about the simple
> need to call `package-initialize' (somewhere, at least).

Because we could make it so that `package-initialize' doesn't need to
be called at all, and then such documentation would be superfluous.
Two birds with one stone.

> But it seems to work well enough that zillions of users make good
> use of it everyday.

Same for Microsoft Windows XP, which currently runs on about 140
million computers worldwide. The size of the userbase is irrelevant to
design questions.

> Do you really think that most pkg users who have problems have the
> more complex kinds of problems for which your proposal is intended?

Why do you say my proposal is intended to solve "complex kinds of
problems"? It is intended to solve the problem of people forgetting to
put `package-initialize' in their init-files, which as you mentioned
is the #1 most common problem people have with package.el.

Sure, the current situation also solves that problem. But it goes
about it in the wrong way, and introduces a number of silly negative
side-effects.

> I don't see zillions of questions about it.
>
> I do see some, which often boil down to this answer: call
> `package-initialize'.

Well, we could solve all of those questions in one fell swoop with the
package.el-specific config file, by eliminating the need to call
`package-initialize'.

> It's a shame that someone might take the point of view that because
> the pkg system has some problems they wouldn't want to bother to
> document it better. That smells a bit like a copout.

Sorry, but I feel the same way as Mark (and I know for a fact that
there are a fair few other users who concur, but are too burnt out to
comment here; see the Reddit thread I linked). If I tried to write
documentation for package.el, it would go like:

    In your init-file, you must call `package-initialize', which
    shouldn't be necessary but is due to design flaws. As a hacky
    workaround to fixing the underlying problem, Emacs will add this
    function call to your init-file automatically if it is absent.
    However, note that it usually does so incorrectly, and you will
    have to fix it by hand.

> > I agree but the docs improvements are orthogonal to fixing the
> > code. Indeed, improving the documentation would be our top
> > priority if we were going to disable package.el by default, which
> > is what you want.
>
> It could reasonably be an immediate priority whether or not the pkg
> system is disabled by default.

Fine. But as I said, that discussion is orthogonal to this one. Better
documentation is not a substitute for good design, nor does the
converse hold.

> > I'd say this is an indication that people care very much indeed
> > about their text editor coming with a package manager that works
> > right out of the box.
>
> (Do ours care enough to improve the docs and usability?)

I've written more than 40 emails here in an attempt to improve the
usability. Because this effort is time-consuming, I'm focusing on one
and only one thing: improving the usability. Not improving the docs,
even though I also think the docs should be improved.

[-- Attachment #2: Type: text/html, Size: 9013 bytes --]

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> Date: Wed, 23 Aug 2017 13:27:33 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: emacs-devel@gnu.org
> 
> If the rationale for the decision was only that you want to treat package.el as a core part of Emacs rather than
> as an extension then that's a bad rationale. You don't need to turn something on just because it is a core part
> of Emacs.

I don't really understand what this argument is about.  The call to
package-initialize does nothing at all, unless you installed some
packages with package.el.  So it is still an opt-in feature, because
the user must decide to install packages for it to do anything.

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> > If the rationale for the decision was only that you want to treat
> > package.el as a core part of Emacs rather than as an extension
> > then that's a bad rationale. You don't need to turn something on
> > just because it is a core part of Emacs.
> 
> I don't really understand what this argument is about.  The call to
> package-initialize does nothing at all, unless you installed some
> packages with package.el.  So it is still an opt-in feature, because
> the user must decide to install packages for it to do anything.

Deciding to "install" a package is somewhat analogous to putting
a library in your `load-path'.  That doesn't mean it gets loaded.
Activating a package (`package-initialize') is somewhat analogous
to loading a library.

Just because a user has installed a package, it doesn't follow
that s?he wants to always and immediately initialize that package
for each Emacs session.

Your comment is similar to saying that (require 'foo nil t) does
nothing if you have not put foo.el in your `load-path', so it
wouldn't hurt to automatically load everything that Emacs can
find in your `load-path'.  Just by putting something in `load-path'
you have opted in.

To that I say no, and Emacs has always said no.  "Installing" a
library does not imply that you also want to load/activate it.
We separate the two actions on purpose - two levels of opt-in for
a library.  And we have the same two levels for packages, IIUC.

A user might have many libraries in her `load-path', without ever
loading some of them in a given Emacs session.  A user might have
many packages installed, without wanting to activate some of them
in a given session.

Radon said that the rationale for activating the package system
automatically was that "we want to treat package.el as a core part
of Emacs rather than an extension."

My reply to that was to point out that being part of the core does
not imply activating.  I said that "lots of things are a core part
of Emacs and not extensions, but we don't necessarily turn them on
eagerly and by default."

Just because something is available by default in Emacs, i.e.,
without having to "install" it, does not mean that it is (or should
be) automatically loaded/activated.

Automatic loading/activation at the outset, just because something
is available, has not, in the past, been the behavior of "core" Emacs.
Core stuff is made available and easy to activate (and yes, some
small core set of the core is activated by default), but much of what
Emacs makes available to you by default is not activated by default.

Radon was saying, I think, that for the package system Emacs dev has
decided differently: it has decided that if you have installed a
package then you want it activated in all your Emacs sessions (except
emacs -Q).  He said, I think, that this was decided because opt-in
activation was too confusing for some users.

But he also said that it was decided because installed packages were
to be handled the way Emacs core is handled.  It was to that statement
that I replied as above: all of Emacs core is not activated by default.
Much of it is just made available, for a user to turn on (opt in).

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-24 19:44, Drew Adams wrote:
> Deciding to "install" a package is somewhat analogous to putting
> a library in your `load-path'.  That doesn't mean it gets loaded.
> Activating a package (`package-initialize') is somewhat analogous
> to loading a library.

Just the autoloads, though, right? Not like `load' or `require'.

> My reply to that was to point out that being part of the core does
> not imply activating.  I said that "lots of things are a core part
> of Emacs and not extensions, but we don't necessarily turn them on
> eagerly and by default."

I'm confused. We load their autoloads from loaddefs.el, don't we? Does package-initialize do more?

Cheers,
Clément.

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> > Deciding to "install" a package is somewhat analogous to putting
> > a library in your `load-path'.  That doesn't mean it gets loaded.
> > Activating a package (`package-initialize') is somewhat analogous
> > to loading a library.
> 
> Just the autoloads, though, right? Not like `load' or `require'.

I'm not sure what you mean, or which part of what I wrote you are
referring to.

My point was that for the "old" system there are two actions or kinds
of opt-in, before a library is "activated": (1) put the code in your
`load-path' and (2) load it.

Just putting a library in your `load-path' does not load it.

If a library has autoloads, and if those are processed, and if the
user then (at some point) triggers an autoloaded function etc., then
the library is loaded.  So there are actually 3 steps there.

Does `package-initialize' only process autoloads and not also load
a package?  I guess so.  And I guess that was your point/question.

> > My reply to that was to point out that being part of the core does
> > not imply activating.  I said that "lots of things are a core part
> > of Emacs and not extensions, but we don't necessarily turn them on
> > eagerly and by default."
> 
> I'm confused. We load their autoloads from loaddefs.el, don't we? Does
> package-initialize do more?

Maybe not.  It's a good point.

So maybe `package-initialize' is more analogous to functions such as
`update-directory-autoloads' and `update-file-autoloads'.

That does modify much of what I've said.  If "activating the package
system" means just processing the autoloads for packages that have
been "installed", so that it is like doing `update-directory-autoloads'
in each part of `load-path', then I guess I don't have a problem with
such automatic "activation".

(There remains, apparently, the question of how/when/where Emacs or
a user should invoke `package-initialize'.)

Re: Summary and next steps for (package-initialize)

[Clément Pit-Claudel]

On 2017-08-24 20:29, Drew Adams wrote:
> That does modify much of what I've said.  If "activating the package
> system" means just processing the autoloads for packages that have
> been "installed", so that it is like doing `update-directory-autoloads'
> in each part of `load-path', then I guess I don't have a problem with
> such automatic "activation".

I just double checked to make sure :) package-initialize does this:

  …
  (unless no-activate
    (dolist (elt package-alist)
      (package-activate (car elt))))
  …

Then package-activate calls package-activate-1,
                which calls package--load-files-for-activation, 
                which calls package--activate-autoloads-and-load-path and calls `load' on all package files that were *already loaded*
                which is this:

  (defun package--activate-autoloads-and-load-path (pkg-desc)
    "Load the autoloads file and add package dir to `load-path'.
  PKG-DESC is a `package-desc' object."
  …)

SUmmarizing, package-initialize sets up the loadpath to include directories of all installed packages, and loads their autoload files.  Additionally, it reloads any already loaded (or `require'd) package file (it checks load-history for that):

    ;; Call `load' on all files in `package-desc-dir' already present in
    ;; `load-history'.  This is done so that macros in these files are updated
    ;; to their new definitions.  If another package is being installed which
    ;; depends on this new definition, not doing this update would cause
    ;; compilation errors and break the installation. 

Cheers,
Clément.

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> I just double checked to make sure :) package-initialize does this:
> 
>   …
>   (unless no-activate
>     (dolist (elt package-alist)
>       (package-activate (car elt))))
>   …
> 
> Then package-activate calls package-activate-1,
>                 which calls package--load-files-for-activation,
>                 which calls package--activate-autoloads-and-load-path and
> calls `load' on all package files that were *already loaded*
>                 which is this:
> 
>   (defun package--activate-autoloads-and-load-path (pkg-desc)
>     "Load the autoloads file and add package dir to `load-path'.
>   PKG-DESC is a `package-desc' object."
>   …)
> 
> SUmmarizing, package-initialize sets up the loadpath to include directories
> of all installed packages, and loads their autoload files.  Additionally, it
> reloads any already loaded (or `require'd) package file (it checks load-
> history for that):
> 
>     ;; Call `load' on all files in `package-desc-dir' already present in
>     ;; `load-history'.  This is done so that macros in these files are updated
>     ;; to their new definitions.  If another package is being installed which
>     ;; depends on this new definition, not doing this update would cause
>     ;; compilation errors and break the installation.

That sounds reasonable.  Thx.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 187 bytes --]

> So maybe `package-initialize' is more analogous to functions such as
> `update-directory-autoloads' and `update-file-autoloads'.

No. Such functions are analogous to `package-install'.

[-- Attachment #2: Type: text/html, Size: 291 bytes --]

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> Date: Thu, 24 Aug 2017 10:44:09 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: radon.neon@gmail.com, emacs-devel@gnu.org
> 
> > I don't really understand what this argument is about.  The call to
> > package-initialize does nothing at all, unless you installed some
> > packages with package.el.  So it is still an opt-in feature, because
> > the user must decide to install packages for it to do anything.
> 
> Deciding to "install" a package is somewhat analogous to putting
> a library in your `load-path'.

So that's the flaw in your reasoning: these two are NOT analogous.  A
user who installs a package using package.el does mean to use it.
That's what package.el installation does, that's what it is for.

If one only wants to have the package somewhere on load-path, they
shouldn't use package.el, but instead just download the files, or
clone the Git repository, and then add the directory to load-path.

> That doesn't mean it gets loaded.

Actually, it does.

> Activating a package (`package-initialize') is somewhat analogous
> to loading a library.

No, it isn't.

> Just because a user has installed a package, it doesn't follow
> that s?he wants to always and immediately initialize that package
> for each Emacs session.

Yes, it does mean exactly that.

There are ways to disable an installed package after the installation,
but the default is to make it available when Emacs starts.

So you have some very different use case in mind, not the one we are
talking about here.

RE: Summary and next steps for (package-initialize)

[Drew Adams]

> > > I don't really understand what this argument is about.  The call to
> > > package-initialize does nothing at all, unless you installed some
> > > packages with package.el.  So it is still an opt-in feature, because
> > > the user must decide to install packages for it to do anything.
> >
> > Deciding to "install" a package is somewhat analogous to putting
> > a library in your `load-path'.
> 
> So that's the flaw in your reasoning: these two are NOT analogous.  A
> user who installs a package using package.el does mean to use it.
> That's what package.el installation does, that's what it is for.
> 
> If one only wants to have the package somewhere on load-path, they
> shouldn't use package.el, but instead just download the files, or
> clone the Git repository, and then add the directory to load-path.
> 
> > That doesn't mean it gets loaded.
> 
> Actually, it does.

Does it load the files of the package, or just process its autoloads,
so that using an autoload function etc. then loads the package?

> > Activating a package (`package-initialize') is somewhat analogous
> > to loading a library.
> 
> No, it isn't.

How about analogous to processing autoload cookies?

> > Just because a user has installed a package, it doesn't follow
> > that s?he wants to always and immediately initialize that package
> > for each Emacs session.
> 
> Yes, it does mean exactly that.

Too bad, if so.  A user should, I think, be able to hit a key to
download a package and have it _available_ to be used, without
necessarily loading its files in each Emacs session.  S?he should
then be able to load it (whether explicitly or by using and
autoloaded function) or not.

> There are ways to disable an installed package after the installation,
> but the default is to make it available when Emacs starts.
> 
> So you have some very different use case in mind, not the one we are
> talking about here.

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> Date: Thu, 24 Aug 2017 11:36:10 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: radon.neon@gmail.com, emacs-devel@gnu.org
> 
> > > Just because a user has installed a package, it doesn't follow
> > > that s?he wants to always and immediately initialize that package
> > > for each Emacs session.
> > 
> > Yes, it does mean exactly that.
> 
> Too bad, if so.  A user should, I think, be able to hit a key to
> download a package and have it _available_ to be used, without
> necessarily loading its files in each Emacs session.

That's not how package.el works.  Think of it as the Windows .msi
installer: running it installs the package, and then activates it, so
invoking after that boils down to running a command.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

> > > Just because a user has installed a package, it doesn't follow
> > > that s?he wants to always and immediately initialize that
> > > package for each Emacs session.
> >
> > Yes, it does mean exactly that.
>
> Too bad, if so. A user should, I think, be able to hit a key to
> download a package and have it _available_ to be used, without
> necessarily loading its files in each Emacs session.

"Initialize" is a bad word, since what actually happens is the files
are put on the load path and the autoloads are evaluated.

[-- Attachment #2: Type: text/html, Size: 754 bytes --]

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

> Deciding to "install" a package is somewhat analogous to putting a
> library in your `load-path'.

Correct.

> That doesn't mean it gets loaded.

Also correct.

> Activating a package (`package-initialize') is somewhat analogous to
> loading a library.

Completely wrong. Activating a package does no such thing.

> "Installing" a library does not imply that you also want to
> load/activate it.

As has been pointed out, loading and activating are two entirely
separate operations.

> Radon was saying, I think, that for the package system Emacs dev has
> decided differently: it has decided that if you have installed a
> package then you want it activated in all your Emacs sessions
> (except emacs -Q). He said, I think, that this was decided because
> opt-in activation was too confusing for some users.

No, this was decided because we want installed packages to act the
same way as built-in packages: being added to the load path and having
their autoloads evaluated, but not having any of their features
required.

[-- Attachment #2: Type: text/html, Size: 1466 bytes --]

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

>> > I think we should instead explore the possibility that
>> > package-initialize will be called only in startup.el.
>> I think this is unlikely to work well.

Re-reading what you wrote, I realize that I wrongly interpreted "only in
startup.el" to mean "after loading the .emacs", whereas My suggestion to
"call package-initialize before reading ~/.emacs" would actually also
imply that package-initialize is only called in startup.el.

So we're probably in agreement.

> As I said, we should explore this in more detail, and have a detailed
> and agreed-upon list of reasons why it's not TRT.

Calling it in startup.el means either calling it before or after loading
~/.emacs, AFAICT.

Calling it after ~/.emacs (like we currently do) is too late since it
means that ~/.emacs can't use installed packages, and that several
configurations can't be done in the usual way.

Calling it before ~/.emacs means that package-initialize is done before
the user got a chance to configure package.el.  So we need'd to provide
some way to reproduce the behavior you can currently get by setting
various package.el vars before calling package-initialize.  E.g. we
could let a second call to package-initialize de-activate previously
activated packages and activate previously not activated packages.

Packages bundled with Emacs are activated (long) before loading
~/.emacs, so calling package-initialize before ~/.emacs seems like the
most natural behavior in that it makes ELPA packages behave similarly to
bundled packages.

But then we need some way to deal with users setting
package-directory-list or package-pinned-packages or package-user-dir
after package-initialize was called.

> This is hard to interpret without knowing "where you sit".

The other alternative will involve comping up with a configuration
scheme that is declarative enough to be able to automatically call
package-initialize when needed.  So it will involve changing the way
~/.emacs is written.  This will take many years, and I don't think
anyboy has really started this journey,


        Stefan

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Tue, 22 Aug 2017 23:15:28 -0400
> 
> Re-reading what you wrote, I realize that I wrongly interpreted "only in
> startup.el" to mean "after loading the .emacs", whereas My suggestion to
> "call package-initialize before reading ~/.emacs" would actually also
> imply that package-initialize is only called in startup.el.
> 
> So we're probably in agreement.

Yes, I think so.

> Calling it in startup.el means either calling it before or after loading
> ~/.emacs, AFAICT.
> 
> Calling it after ~/.emacs (like we currently do) is too late since it
> means that ~/.emacs can't use installed packages, and that several
> configurations can't be done in the usual way.
> 
> Calling it before ~/.emacs means that package-initialize is done before
> the user got a chance to configure package.el.  So we need'd to provide
> some way to reproduce the behavior you can currently get by setting
> various package.el vars before calling package-initialize.  E.g. we
> could let a second call to package-initialize de-activate previously
> activated packages and activate previously not activated packages.

Would a ~/.emacs.d/.package-initialize.el file, to be read by
package-initialize before it does anything else, be a better
alternative for configurations that must be done before
package-initialize is called?

Or maybe you have some other alternative ideas for how to make this
happen?

> Packages bundled with Emacs are activated (long) before loading
> ~/.emacs, so calling package-initialize before ~/.emacs seems like the
> most natural behavior in that it makes ELPA packages behave similarly to
> bundled packages.
> 
> But then we need some way to deal with users setting
> package-directory-list or package-pinned-packages or package-user-dir
> after package-initialize was called.

Right, which is why I suggested to have a separate file for that.

Thanks.

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

>> Calling it before ~/.emacs means that package-initialize is done before
>> the user got a chance to configure package.el.  So we need'd to provide
>> some way to reproduce the behavior you can currently get by setting
>> various package.el vars before calling package-initialize.  E.g. we
>> could let a second call to package-initialize de-activate previously
>> activated packages and activate previously not activated packages.
> Would a ~/.emacs.d/.package-initialize.el file, to be read by
> package-initialize before it does anything else, be a better
> alternative for configurations that must be done before
> package-initialize is called?

To me, it'd be a worst case fallback if we can't find a better solution.

> Or maybe you have some other alternative ideas for how to make this
> happen?

Nothing very concrete yet, no:

- Change package-initialize so it keeps track of what it activated and if
  called a second time, perform a diff between what was activated before
  and what should be activated now and based on this activate the new pkgs
  and deactivate the excess ones?

- Process ~/.emacs specially so it can start with a special construct
  (with-early-config ...)

I've had other ideas, but they generally suck one way or another.


        Stefan

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Thu, 24 Aug 2017 13:48:36 -0400
> 
> - Change package-initialize so it keeps track of what it activated and if
>   called a second time, perform a diff between what was activated before
>   and what should be activated now and based on this activate the new pkgs
>   and deactivate the excess ones?

This could prove tricky, if some package that shouldn't be loaded
affects another package that should.  Right?

> - Process ~/.emacs specially so it can start with a special construct
>   (with-early-config ...)

Yes, that could work.

Thanks.

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

>> - Change package-initialize so it keeps track of what it activated and if
>> called a second time, perform a diff between what was activated before
>> and what should be activated now and based on this activate the new pkgs
>> and deactivate the excess ones?
> This could prove tricky, if some package that shouldn't be loaded
> affects another package that should.  Right?

Not sure what "This" refers to.  I'll assume you're referring to
"deactivate", which is indeed the trickier part.  I was thinking of
doing it via unload-feature and declare that if it doesn't work
quite-right, it's a bug in the package.


        Stefan

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]

> I was thinking of doing it via unload-feature

Isn't this out of scope for unload-feature? After all,
package-initialize does not load features.

> if it doesn't work quite-right, it's a bug in the package.

I think if we take an approach like this, we'll end up uncovering
quite a few bugs, indeed. And I worry that this will result in people
thinking that package.el is buggy, since the bugs will only show up
when people use package.el, as opposed to any other package manager or
as opposed to loading the packages manually.

[-- Attachment #2: Type: text/html, Size: 722 bytes --]

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

>> I was thinking of doing it via unload-feature
> Isn't this out of scope for unload-feature? After all,
> package-initialize does not load features.

Indeed, the `unload-feature` function isn't designed quite for that, so
we'd want to use another function for it, but it would basically mean
extract the bulk of unload-feature into a new function and use it for
both unload-feature as well as the package.el deactivation.

>> if it doesn't work quite-right, it's a bug in the package.
> I think if we take an approach like this, we'll end up uncovering
> quite a few bugs, indeed. And I worry that this will result in people
> thinking that package.el is buggy, since the bugs will only show up
> when people use package.el, as opposed to any other package manager or
> as opposed to loading the packages manually.

Unloading a <pkg>-autoload.el should be a fairly unusual occurrence, so
most users should be unaffected.


        Stefan

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

> Unloading a <pkg>-autoload.el should be a fairly unusual occurrence,
> so most users should be unaffected.

True, but you're introducing a fragile code path that is only called
rarely and under possibly surprising circumstances. I'm imagining
things like an unexpected package left in the old package-user-dir
causing an oblique side effect through its autoloads that weren't
unloaded quite right -- not fun to debug at all, especially since
package.el will insist that the troublesome package isn't installed!

It seems like we should avoid fragility, even if said fragility
probably(TM) won't affect most users.

[-- Attachment #2: Type: text/html, Size: 825 bytes --]

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

> True, but you're introducing a fragile code path that is only called
> rarely and under possibly surprising circumstances.

As I said, there's no doubt that do+undo is a poor way to do nothing.


        Stefan

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Thu, 24 Aug 2017 23:52:19 -0400
> 
> >> - Change package-initialize so it keeps track of what it activated and if
> >> called a second time, perform a diff between what was activated before
> >> and what should be activated now and based on this activate the new pkgs
> >> and deactivate the excess ones?
> > This could prove tricky, if some package that shouldn't be loaded
> > affects another package that should.  Right?
> 
> Not sure what "This" refers to.  I'll assume you're referring to
> "deactivate"

Good assumption.

> I was thinking of doing it via unload-feature and declare that if it
> doesn't work quite-right, it's a bug in the package.

What about packages that include accommodations to, and dependencies
on, other packages?  Imagine the following situation:

 . package B modifies its behavior if package A is loaded
 . the user sets up package-load-list such that package B should be
   loaded, but package A should not be
 . package B is activated by package-initialize after package A was
   activated, so B modifies its behavior
 . package-initialize unloads A
 . the result is that B behaves as if A is loaded, contrary to what
   the user wanted, and will probably produce weird errors at some
   point, or subtly incorrect behavior

I don't think we can claim in this case that there's a bug in either
of these two packages, can we?  Or is there a way for the packages to
be prepared for such situations?

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

> What about packages that include accommodations to, and dependencies
> on, other packages?  Imagine the following situation:
>
>  . package B modifies its behavior if package A is loaded
>  . the user sets up package-load-list such that package B should be
>    loaded, but package A should not be
>  . package B is activated by package-initialize after package A was
>    activated, so B modifies its behavior
>  . package-initialize unloads A
>  . the result is that B behaves as if A is loaded, contrary to what
>    the user wanted, and will probably produce weird errors at some
>    point, or subtly incorrect behavior
>
> I don't think we can claim in this case that there's a bug in either
> of these two packages, can we?  Or is there a way for the packages to
> be prepared for such situations?

[ Note: in the above "package A" and "package B" really refer to
  the files A-autoload.el and B-autoload.el, which aren't really
  "packages".  I'll try to be more precise below.  ]

I guess B-autoload.el could add a function to a "A-autoload-unload-hook".

But things can get worse: B-autoload.el could not only behave differently
depending on whether A-autoload.el was loaded, but it could also cause
A.el to be loaded, in which case unloading A-autoload.el won't
be sufficient, unless we arrange for the unloading of A-autoload.el to
also unload A.el.

Again, here we're back to the obvious observation that do+undo is a poor
and complicated way to do nothing.


        Stefan

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

> > Would a ~/.emacs.d/.package-initialize.el file, to be read by
> > package-initialize before it does anything else, be a better
> > alternative for configurations that must be done before
> > package-initialize is called?
>
> To me, it'd be a worst case fallback if we can't find a better
> solution.

That's a reasonable perspective. I personally do think that there is
no better solution.

Also, I really like the naming and the idea that the file is loaded by
`package-initialize', not directly by startup.el.

> Change package-initialize so it keeps track of what it activated and
> if called a second time, perform a diff between what was activated
> before and what should be activated now and based on this activate
> the new pkgs and deactivate the excess ones?

This will introduce performance regressions. Besides, having one part
of the init process undo a bunch of stuff that a previous part did
seems like a sketchy design to me. Also, it's impossible for this to
work correctly in general since autoloads can run arbitrary Lisp code.

> Process ~/.emacs specially so it can start with a special construct
> (with-early-config ...)

That seems like a hacky way to try to combine two files into one file.
Wouldn't it be better to just use two files?

> I've had other ideas, but they generally suck one way or another.

Me too. For example, I now think that my original idea of generating a
template init-file is a less effective solution than
~/.emacs.d/.package-initialize.el.

[-- Attachment #2: Type: text/html, Size: 2007 bytes --]

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

>> Change package-initialize so it keeps track of what it activated and
>> if called a second time, perform a diff between what was activated
>> before and what should be activated now and based on this activate
>> the new pkgs and deactivate the excess ones?
> This will introduce performance regressions.

Could be.  But I think this should only come up in very unusual
situations (e.g. when you have lots of packages in ~/.emacs.d/elpa even
though you either don't use this directory (and hence set
package-user-dir to something else) or you only activate very few of
them (by setting package-load-list)).

> Besides, having one part of the init process undo a bunch of stuff
> that a previous part did seems like a sketchy design to me.

Can't disagree that "do+undo" is a poor way to do nothing.

> Also, it's impossible for this to work correctly in general since
> autoloads can run arbitrary Lisp code.

The ability to undo is useful in general.  And while it's true that
Emacs can't magically know how to deactivate a package, in general, the
package can provide extra code that explains how to deactivate itself
(e.g. via <pkg>-unload-function).

>> Process ~/.emacs specially so it can start with a special construct
>> (with-early-config ...)
> That seems like a hacky way to try to combine two files into one file.
> Wouldn't it be better to just use two files?

I like to keep my config in a single file, and if you want to use
a separate file you can do (with-early-config (load "my-other-file")),
so while it may be hackish, it's more flexible.

BTW, here's another idea:
- use a separate "early-config" file.  Now, you probably wonder why
  I say it's "another" idea, so here's the reason: this file would be
  loaded before the initial GUI frame is created (so it would solve
  another similar problem at the same time, which is "how do I turn OFF
  those GUI elements in my .emacs such that they never show up, not even
  temporarily while we process the .emacs").
With the expected variant:
- rather than a separate file, accept a special (with-early-config ...)
  form at the beginning of the ~/.emacs.  Oh wait I already suggested
  it, but now this should also be used for "config before the initial frame".


        Stefan

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 1809 bytes --]

> The ability to undo is useful in general.

It's true, but most packages don't provide this capability, so if
package.el relies on it, then usage of package.el will produce more
bugs.

> And while it's true that Emacs can't magically know how to
> deactivate a package, in general, the package can provide extra code
> that explains how to deactivate itself (e.g. via
> <pkg>-unload-function).

That tells how to unload the feature <pkg>, not how to undo the
effects of the autoloads for all features in the package. Sure, the
package can provide metadata that tells how to undo its autoloads, but
this will be an additional system separate from <pkg>-unload-function.
Unless we want to tell people they have to write their
<pkg>-unload-function so that it works even if unloading a feature
that hasn't been loaded yet...

> use a separate "early-config" file. Now, you probably wonder why I
> say it's "another" idea, so here's the reason: this file would be
> loaded before the initial GUI frame is created (so it would solve
> another similar problem at the same time, which is "how do I turn
> OFF those GUI elements in my .emacs such that they never show up,
> not even temporarily while we process the .emacs").

I think this is a really great idea, and would fit in well
conceptually. It would also resolve the concern that Nikolay raised
earlier about ~/.emacs.d/.package-initialize.el later becoming
extraneous if we refactor the feature loading system.

> rather than a separate file, accept a special (with-early-config
> ...) form at the beginning of the ~/.emacs.

I guess I would be fine with this, even though I think it's a little
hacky.

Best,
Radon

P.S. I notice you keep saying ~/.emacs. Is that out of
habit/convenience? I thought ~/.emacs was deprecated in favor of
~/.emacs.d/init.el.

[-- Attachment #2: Type: text/html, Size: 2460 bytes --]

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

> That tells how to unload the feature <pkg>, not how to undo the
> effects of the autoloads for all features in the package.  Sure, the
> package can provide metadata that tells how to undo its autoloads, but
> this will be an additional system separate from <pkg>-unload-function.

I didn't mean unload-feature and <pkg>-unload-function literally.

Just that we'd use the same kind of code as used in unload-feature
(except use it to unload <pkg>-autoload.el, which is a file rather than
a feature), and that <pkg>-autoload.el would have to provide some kind
of unload function in order to deal with the parts that unload-feature
doesn't do automatically.

> P.S. I notice you keep saying ~/.emacs. Is that out of
> habit/convenience?

Yes, habit as well as conciseness.

> I thought ~/.emacs was deprecated in favor of
> ~/.emacs.d/init.el.

I can't remember if we truly declared it as deprecated, but yes, it's
something like that.  BTW any new config file should not be named
~/.emacs.d/.foobar.el but ~/.emacs.d/foobar.el.


        Stefan

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

> - use a separate "early-config" file.  Now, you probably wonder why
>   I say it's "another" idea, so here's the reason: this file would be
>   loaded before the initial GUI frame is created (so it would solve
>   another similar problem at the same time, which is "how do I turn OFF
>   those GUI elements in my .emacs such that they never show up, not even
>   temporarily while we process the .emacs").
> With the expected variant:
> - rather than a separate file, accept a special (with-early-config ...)
>   form at the beginning of the ~/.emacs.  Oh wait I already suggested
>   it, but now this should also be used for "config before the initial frame".

BTW, another element to keep in mind when choosing the best path forward
is how to allow users to use `M-x customize` in order to modify thing
like package-load-list and package-user-dir.


        Stefan

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 501 bytes --]

> how to allow users to use `M-x customize` in order to modify thing
> like package-load-list and package-user-dir.

I think this will require some special handling on the side of Custom,
right? Since I would guess you can't use the same form for customizing
package.el and customizing packages without having that form call
package-initialize in the middle of it.

Alternatively, you could say that anything requiring early init is
considered an "advanced" feature that can't be accessed via Custom.

[-- Attachment #2: Type: text/html, Size: 682 bytes --]

Re: Summary and next steps for (package-initialize)

[Stefan Monnier]

>> how to allow users to use `M-x customize` in order to modify thing
>> like package-load-list and package-user-dir.
> I think this will require some special handling on the side of Custom,
> right?

Different from what we do now, yes, but "special" in the sense of being
taylor-made ad-hoc for package.el, ideally not.

Maybe simply the ability to spread the saved custom settings over
several files is all that's needed (with a default arrangement that
places all the package.el settings into ~/.emacs.d/early-init.el and
everything else in ~/.emacs.d/init.el).


        Stefan

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

> Maybe simply the ability to spread the saved custom settings over
> several files is all that's needed (with a default arrangement that
> places all the package.el settings into ~/.emacs.d/early-init.el and
> everything else in ~/.emacs.d/init.el).

That sounds like an elegant solution (and it is also an argument for
having two separate files rather than a special `with-early-config'
form).

We would also want to consider whether people might want to place
config at any additional places in startup.el. We'd like to avoid
having more than two init-files, of course, so I think it's important
to try to capture as many use cases as possible with this new
init-file.

Probably, that means loading early-init.el as early as is humanly
possible, so that people can customize other points in the init by
adding things to hook variables.

[-- Attachment #2: Type: text/html, Size: 1097 bytes --]

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> > I think we should instead explore the possibility that
> > package-initialize will be called only in startup.el.
>
> I think this is unlikely to work well.

Agreed. The only way I can see this possibly working is if we
introduce a second init-file, which IMO is way too much overkill for
supporting a single package out of the thousands in Emacs.

> I think a better long-term ideal situation is either:
>
> - call package-initialize before reading ~/.emacs (this will require
>   finding some way to handle the case where the user wants to change
>   package.el's configuration in his .emacs).
> - call package-initialize automatically when needed, by having a more
>   declarative way to configure packages in ~/.emacs.
>
> The first approach looks easier from where I sit, but that's just
> a gut feeling.

There's also my proposal. From where *I* sit, that proposal looks like
the easiest, both for the end user and implementation-wise. Let's make
sure to do a comparison before discarding it.

> In the short term, we can tweak the current approach so that
> package--ensure-init-file is called in fewer cases.  E.g. only call it
> during package installation rather than at Emacs startup.

If that can be done right away, please let's do that right away. But
we can't stop there; having Emacs modify the init-file in this way is
still fundamentally broken.

Re: Summary and next steps for (package-initialize)

[Mark Oteiza]

Radon Rosborough <radon.neon@gmail.com> writes:
> Last week I posted an inquiry [1] about package.el's auto-insertion of
> code into the init-file. Six people weighed in, but no definitive
> conclusion was reached. I would like to propose concrete next steps,
> summarizing relevant parts of the discussion in the process.

Thanks for pushing the issue.

> ==> The proposal
>
> The `package--ensure-init-file' logic will be removed

+1 for reverting.

I will happily propose and write documentation for various use cases of
package.el, but I will not waste my time as long as the current solution
is in place.

Re: Summary and next steps for (package-initialize)

[Nikolay Kudryavtsev]

[-- Attachment #1: Type: text/plain, Size: 1436 bytes --]

Hello.

I have another proposal that was not considered so far.

The main problem we're trying to solve here is when a new user calls 
something from package.el without having it initialized first. We also 
want the solution to touch only that group of users and no one else.

So I propose, we don't do anything until user either:

1. Requires something that's not available.

2. Calls a function that's not available.

In either of those cases we check whether (package-initialize) was 
already called. And, if not, we give user an interactive window along 
the lines of:

    /"//You tried to call X which is not available and Emacs package
    manager was not initialized. Press: //
    /

    /i to initialize and try again/

    /w to initialize, try again and don't ask again/

    /x to dismiss/

    /K to dissmiss and never ask again."/

w writes (package-initialize) to init)

K would set variable 
dear-emacs-i-totally-don-t-need-any-begginers-advice-thanks to t.

We also don't show this window when kill-package-el-and-burn-its-body is t.

I'm not entirely sure on the technical side of catching requires and 
void-functions, but this seems to be the best solution when all other 
things are considered.

Also similar approach is already used in Emacs, there's some key binding 
that's disabled until you confirm it in a similar way, sorry at the 
moment I don't remember which one.

-- 
Best Regards,
Nikolay Kudryavtsev


[-- Attachment #2: Type: text/html, Size: 1960 bytes --]

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 3012 bytes --]

> "You tried to call X which is not available and Emacs package
> manager was not initialized.

I think this proposal is worse than adding a second init-file. Here is
why:

* It is much more annoying to the user. The second init-file would
  mean that everything works in all cases with *no* user interaction.
  Making users deal with pop-up dialogs wastes their time.

* The user might well be calling a function that is provided by
  another package manager, or indeed a manually loaded Lisp library.
  Emacs can't assume that all undefined functions are the fault of not
  calling package-initialize; undefined function errors have a scope
  that is many orders of magnitude greater than the package system.

  Besides, what about when Emacs is run in batch mode? Then we're back
  to where we started, since there's no option to prompt for user
  input.

* > w writes (package-initialize) to init)

  This is bad. We don't want Emacs doing this under any circumstances.
  Not because it might be annoying (though it is), but because *Emacs
  usually does it wrong*! If you put `package-initialize' in the
  init-file, you *must* put it after any package.el configuration, but
  before any package configuration. This point is impossible to
  determine in general, especially as it might be elsewhere on the
  filesystem.

* > K would set variable
  > dear-emacs-i-totally-don-t-need-any-begginers-advice-thanks to t.

  How? By modifying the init-file? This whole discussion started
  because we don't want package.el doing that. (Admittedly, this
  wouldn't be as bad as trying to add `package-initialize' -- but
  still.)

* > I'm not entirely sure on the technical side of catching requires
  > and void-functions

  Seems pretty sketchy. You're going to mucking around with one of the
  most fundamental parts of the Elisp interpreter (namely, function
  calls) just for the sake of the package manager.

  Adding a second init-file doesn't require anything this hacky.

> Also similar approach is already used in Emacs, there's some key
> binding that's disabled until you confirm it in a similar way, sorry
> at the moment I don't remember which one.

You're thinking of disabled commands. But that's very different: the
popup only appears when a user explicitly tries to do something
dangerous, rather than during noninteractive execution of arbitrary
Lisp code. And furthermore, the disabled-command mechanism serves to
prevent users from shooting themselves in the foot; whereas, if Emacs
tries and fails to put `package-initialize' in the right place
automatically, it is shooting the user in the foot all by itself.

> but this seems to be the best solution when all other things are
> considered

To me, it seems to be worse than the second init-file proposal in
every way except that it doesn't require the addition of a second
init-file, and that it might have better backwards compatibility. But
I am sure there are things I am missing in my analysis; please let me
know if this is misguided.

[-- Attachment #2: Type: text/html, Size: 3930 bytes --]

Re: Summary and next steps for (package-initialize)

[Nikolay Kudryavtsev]

Of your two reasons:

A. Yes, it's going to be annoying for two fractions of users, the 
targeted fraction - that is the beginners who didn't set up package.el 
right and old timers during some rare moments of messing with their 
configs. For the later this annoyance would be much rarer than the 
current situation... And I too would have preferred just removing any 
additions, but my proposal seems like an OK compromise.

B. That other package manager should set the variable that suppresses 
the message either within itself or within its recommended configuration 
blob that you paste in your init.

And I don't care about batch mode completely. If you're using batch 
mode, than you know what you're doing.

As for writing package-initialize - well, in case you a)don't use 
package.el and b)don't use any other package manager and c)don't 
suppress the proposed message manually, it's pretty safe to assume that 
you don't have much of init anyway and you're our intended target.

> This whole discussion started  because we don't want package.el doing 
> that.
No, we don't want to write to init, but writing vars to custom is pretty 
normal and used by many packages. It's just that we can't solve 
package-initialize by writing to custom.

> You're going to mucking around with one of the
>   most fundamental parts of the Elisp interpreter (namely, function
>   calls) just for the sake of the package manager.
I'm just proposing to tweak error handling, that's it. While you're 
proposing to add another init file. And we're not talking about the sake 
of a package manager, we're talking about upgrading experience of a very 
limited subset of users(only those new users who haven't yet acquired 
the skill of copy-pasting magic spells into their init). So, while this 
is somewhat ambitious, adding another init just for that is even more so.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 2960 bytes --]

> Yes, it's going to be annoying for two fractions of users, the
> targeted fraction - that is the beginners who didn't set up
> package.el right and old timers during some rare moments of messing
> with their configs.

Yes, but the second init-file would result in no annoyance for anyone.
Why is a small amount of annoyance better than no annoyance?

> And I too would have preferred just removing any additions, but my
> proposal seems like an OK compromise.

If by "additions" you mean Emacs writing to the init-file, then the
second init-file proposal eliminates that behavior entirely. Why is
there the need for a compromise? Having Emacs write to the init-file
is an order of magnitude more complexity than having an optional
second init-file that doesn't need to be used by most people, so it
seems to me like any solution that involves Emacs writing to the
init-file is a step backwards.

> And I don't care about batch mode completely. If you're using batch
> mode, than you know what you're doing.

Yes, but nobody likes inconsistent behavior. I think the fewer
unexpected differences there are between Emacs' modes of operation,
the better.

> It's just that we can't solve package-initialize by writing to
> custom.

Correct. And since we can solve package-initialize without writing to
anything at all, why not do that?

> I'm just proposing to tweak error handling, that's it.

You're injecting business logic into an otherwise generic piece of
code. I don't think this is a sign of good design.

> While you're proposing to add another init file.

Can you explain why this is a big problem? In what circumstance would
this cause trouble?

Maybe it would be better to call it a "config file", because that's
basically all it is. And we've got plenty of those already. Dozens, in
fact, scattered all over ~/.emacs.d and used by various packages and
different parts of Emacs.

> we're talking about upgrading experience of a very limited subset of
> users

The second init-file would accomplish this as well with fewer
problems.

> (only those new users who haven't yet acquired the skill of
> copy-pasting magic spells into their init)

The second init-file would not require anybody to do this. In fact, it
would free them from such a need, because (package-initialize) no
longer needs to be in the init-file.

> So, while this is somewhat ambitious, adding another init just for
> that is even more so.

I think we need to see some concrete disadvantages of the second
init-file proposal, since the error-catching proposal has some
already.

                             ~~~~~

But really, this is all just an aside. The real problem is that having
Emacs add `package-initialize' automatically is *fundamentally
broken*. There is NO WAY to determine the correct place to put the
call. And having a system that sometimes "fixes" the problem in a way
that breaks the user's config can't possibly be better than a system
that always works. Right?

[-- Attachment #2: Type: text/html, Size: 4011 bytes --]

Re: Summary and next steps for (package-initialize)

[Nikolay Kudryavtsev]

The problem with second init is that we're introducing a major user 
configuration change to fix some shortcoming in Emacs internals. The 
moment the feature lookup API I proposed in the other letter gets pushed 
to master we can cut out those warnings(or writing in init for that 
matter). And with the current state of packaging I think such API is 
almost inevitable. But with second init, people would still be stuck 
with those files, even when raison d'etre for them disappears. And 20 
years down the line you may need to explain to people "Oh, no, you don't 
need config.el, you needed it like 15 years ago, but not anymore".

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 2181 bytes --]

> we're introducing a major user configuration change

Is it really that major to move the place that two options are
configured from one file to a different file? To me, this seems pretty
trivial.

> some shortcoming in Emacs internals

I think this assertion requires justification. It's not clear that
there is any shortcoming in Emacs internals. On the contrary, I would
say that the pattern of configuring a package manager, having it load
the packages, and then using the packages is exactly the right pattern
to use.

> the feature lookup API I proposed in the other letter

That proposal requires its own discussion before we decide if it's the
right way to go or not. Your argument is only valid if we decide that
we want this API, and we haven't decided that yet.

> And with the current state of packaging I think such API is almost
> inevitable.

Whether or not this statement is true will be determined by that
discussion.

> But with second init, people would still be stuck with those files,
> even when raison d'etre for them disappears.

"if" raison d'etre for them disappears. We don't know that yet.

> 20 years down the line

Five years ago, we had no package manager at all. And that's one heck
of a breaking change, compared to moving two `setq' declarations from
one file to another.

> you may need to explain to people "Oh, no, you don't need config.el,
> you needed it like 15 years ago, but not anymore"

Remember that the only people who are going to be using this secondary
init-file are advanced users who want to customize the process of
loading packages. IOW, 0.1% of Emacs users, and not the ones we should
be concerned about confusing.

                                ~~~~~

But you still haven't addressed the real problem, which is that when
Emacs inserts `package-initialize' into the init-file, it usually does
so incorrectly. Do you really think that we should adopt a solution
which objectively does the wrong thing just for the sake of
compatibility with a hypothetical new API whose implementation is
likely months or years in the future?

In my opinion, this issue is a blocker for your proposal. How are you
suggesting to deal with it?

[-- Attachment #2: Type: text/html, Size: 2990 bytes --]

Re: Summary and next steps for (package-initialize)

[Nikolay Kudryavtsev]

> It's not clear that there is any shortcoming in Emacs internals.
The whole discussion is a proof that there is. All the practical 
solutions here are bad. Including mine.

> But you still haven't addressed the real problem, which is that when
> Emacs inserts `package-initialize' into the init-file, it usually does
> so incorrectly. 
I have addressed this before - "in case you a)don't use package.el and 
b)don't use any other package manager and c)don't suppress the proposed 
message manually, it's pretty safe to assume that you don't have much of 
init anyway and you're our intended target". We can add: d)you choose to 
do that insertion. When we combine abcd, such insertion would almost 
always improve users init. Won't fix other possible problems, but would 
still be an improvement.

Also, please note that such optional insertion is quite secondary to my 
proposal, just warning user without any interactive options is fine with 
me too.

P. S. I think that instead of using the phrase "second init-file" you 
should use "separate package.el config". Talking about a second init 
creates more confusion and resistance from other people, than necessary. 
Though I still feel that having a separate package.el config would not 
have been warranted even if it was the only alternative to the current 
dumb (package-initialize) insertion.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 1496 bytes --]

> > But you still haven't addressed the real problem, which is that
> > when Emacs inserts `package-initialize' into the init-file, it
> > usually does so incorrectly.
>
> I have addressed this before - "in case you a)don't use package.el
> and b)don't use any other package manager and c)don't suppress the
> proposed message manually, it's pretty safe to assume that you don't
> have much of init anyway and you're our intended target".

So in other words, your solution still does the wrong thing, it's just
that this fact is unlikely to cause real trouble? I still think we
should be aiming for a solution which always does the right thing.

In any case, as Eli has pointed out, adding flashy interactive
messages should be avoided except as a last resort. Adding
~/.emacs.d/.package-initialize.el is not intrusive, and most users
need not know the file even exists. Adding a popup dialog that happens
to every user at least once is highly intrusive.

> P. S. I think that instead of using the phrase "second init-file"
> you should use "separate package.el config".

I shall. In fact, I am very much a fan of Eli's suggestion:

    a ~/.emacs.d/.package-initialize.el file, to be read by
    package-initialize before it does anything else

In any case, the point is that this solution has NO PRACTICAL
DISADVANTAGES. Whereas every other solution has the potential to annoy
the user in some situation. In that light, it seems to make most sense
to choose the package-initialize.el solution.

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Date: Wed, 23 Aug 2017 23:09:37 +0300
> Cc: emacs-devel@gnu.org
> 
> The problem with second init is that we're introducing a major user 
> configuration change to fix some shortcoming in Emacs internals.

If the package.el configuration file is used only for stuff that needs
to be set before package-initialize is called, then its contents will
be minimal, and for many users it doesn't even have to exist, because
the use cases to which it caters are quite special (packages that are
installed, but the user doesn't want to use them).

By contrast, the problem we are solving with this file is a much
larger one, and affects many more users.

Re: Summary and next steps for (package-initialize)

[Nathan Moreau]

Hello,

> So I propose,


I really like the proposal, except that I would prefer to force
the call to package-initialize if something goes wrong (with
a warning written in the *Messages* buffer) instead of a prompt.

The idea being that

- this seems like a reasonable (a setup with `(package-initialize)'
  as only package.el stuff is quite common)

- while remaining backward compatible with existing configurations

- and makes users that don't want package.el happy: they don't need to
  put anything related to it in their configuration code, and the its
  code is unloaded as long as their configuration loads cleanly.


Nathan

On 23 August 2017 at 17:57, Nikolay Kudryavtsev
<nikolay.kudryavtsev@gmail.com> wrote:
> Hello.
>
> I have another proposal that was not considered so far.
>
> The main problem we're trying to solve here is when a new user calls
> something from package.el without having it initialized first. We also want
> the solution to touch only that group of users and no one else.
>
> So I propose, we don't do anything until user either:
>
> 1. Requires something that's not available.
>
> 2. Calls a function that's not available.
>
> In either of those cases we check whether (package-initialize) was already
> called. And, if not, we give user an interactive window along the lines of:
>
> "You tried to call X which is not available and Emacs package manager was
> not initialized. Press:
>
> i to initialize and try again
>
> w to initialize, try again and don't ask again
>
> x to dismiss
>
> K to dissmiss and never ask again."
>
> w writes (package-initialize) to init)
>
> K would set variable
> dear-emacs-i-totally-don-t-need-any-begginers-advice-thanks to t.
>
> We also don't show this window when kill-package-el-and-burn-its-body is t.
>
> I'm not entirely sure on the technical side of catching requires and
> void-functions, but this seems to be the best solution when all other things
> are considered.
>
> Also similar approach is already used in Emacs, there's some key binding
> that's disabled until you confirm it in a similar way, sorry at the moment I
> don't remember which one.
>
> --
> Best Regards,
> Nikolay Kudryavtsev

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

> - and makes users that don't want package.el happy: they don't need
>   to put anything related to it in their configuration code, and the
>   its code is unloaded as long as their configuration loads cleanly.

Can you explain what you mean by "its code is unloaded"?

> as long as their configuration loads cleanly

I worry that having any error trigger lots of package.el code,
filesystem calls, and arbitrary third-party Lisp code downloaded from
the Internet will make debugging more... "fun".

Also, if we don't have a popup, then how would the user end up in the
state where they have `package-initialize' in their init-file? Would
we just keep on running `package-initialize' at the first undefined
function error on every init? That seems... slightly hacky, to say the
least.

[-- Attachment #2: Type: text/html, Size: 1064 bytes --]

Re: Summary and next steps for (package-initialize)

[Nathan Moreau]

> Can you explain what you mean by "its code is unloaded"?

I mean `unloaded' as `never loaded'.

On 24 August 2017 at 02:54, Radon Rosborough <radon.neon@gmail.com> wrote:
>> - and makes users that don't want package.el happy: they don't need
>>   to put anything related to it in their configuration code, and the
>>   its code is unloaded as long as their configuration loads cleanly.
>
> Can you explain what you mean by "its code is unloaded"?
>
>> as long as their configuration loads cleanly
>
> I worry that having any error trigger lots of package.el code,
> filesystem calls, and arbitrary third-party Lisp code downloaded from
> the Internet will make debugging more... "fun".
>
> Also, if we don't have a popup, then how would the user end up in the
> state where they have `package-initialize' in their init-file? Would
> we just keep on running `package-initialize' at the first undefined
> function error on every init? That seems... slightly hacky, to say the
> least.
>

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Date: Wed, 23 Aug 2017 18:57:30 +0300
> 
> So I propose, we don't do anything until user either:
> 
> 1. Requires something that's not available.
> 
> 2. Calls a function that's not available.
> 
> In either of those cases we check whether (package-initialize) was already called. And, if not, we give user an
> interactive window along the lines of:
> 
>  "You tried to call X which is not available and Emacs package manager was not initialized. Press: 
> 
>  i to initialize and try again
> 
>  w to initialize, try again and don't ask again
> 
>  x to dismiss
> 
>  K to dissmiss and never ask again."
> 
> w writes (package-initialize) to init)

IME, annoying users in this way is not welcome in this community.
Doing this once is borderline, doing this multiple times in the same
session is going to annoy too much, I think.

And anyway, I don't see how this solves the problem of packages being
loaded when the user said not to by customizing package-load-list.
Your proposal in effect will load packages after Emacs reads ~/.emacs,
which is what we do now, so how is it different?

Re: Summary and next steps for (package-initialize)

[Nikolay Kudryavtsev]

I'm not proposing we do it multiple times. Disabling that warning after 
it's displayed the first time would be the trivial part. Same with 
package-load-list, if it's customized, we assume that the user is a 
grown up and we should not bother him.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Summary and next steps for (package-initialize)

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Cc: radon.neon@gmail.com, emacs-devel@gnu.org
> Date: Thu, 24 Aug 2017 20:52:46 +0300
> 
> I'm not proposing we do it multiple times. Disabling that warning after 
> it's displayed the first time would be the trivial part.

How can we ask this only once, when the "missing" functions can belong
to different packages?  IOW, what do you suggest to do when the next
such function is found? assume the same answer as we got the first
time?  That could be not what the user wants, I think.

> Same with package-load-list, if it's customized, we assume that the
> user is a grown up and we should not bother him.

Not bother him and do what? always load the package?  What if that
package doesn't appear in package-load-list?  The user would expect to
get an error, since he doesn't expect the package to be loaded at all.

Re: Summary and next steps for (package-initialize)

[Nikolay Kudryavtsev]

> That could be not what the user wants, I think
> The user would expect to
> get an error, since he doesn't expect the package to be loaded at all.
I don't think we should try to do too much here. We're aiming at exactly 
one possible packaging problem faced by the new users. You just 
installed Emacs, learned of a cool mode, try to require it and BAM. Then 
you get the message about adding package-initalize and that's it. 
Multiple criteria already described in this thread would prevent such 
warning to be shown to experienced users in 99% of cases and when it 
shows the user would immediately know that it's a real problem with his 
init and time to kill that session.

P. S. Maybe we should cobble something out of doctor-mode and disabled 
commands, give it an avatar of some office tool, call nanny-mode and 
enable by default. ;-)

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Summary and next steps for (package-initialize)

[angelo.g0]

> > Do you like the idea of Emacs modifying the user's init-file
> > automatically, even if it already exists? If not, then do you agree
> > that my proposal at least reduces the problem?
>
> I don't like either of these, but again, I see no significant
> improvement, if at all.

As simple user, I hope you do not proliferate the number of initialization files.

If someone remember, I already flagged that the silently modification of init.el is not good but adding another file is even worse; in that discussion there was the suggestion to add this to init.el:

  (setq custom-file "~/.emacs.d/custom.el")
  (load custom-file)

and now everything is under control..

BTW, Emacs works also without the information saved in the init.el or custom.el files...

Ciao,
  Angelo.

Re: Summary and next steps for (package-initialize)

[Radon Rosborough]

> As simple user, I hope you do not proliferate the number of
> initialization files.

I'm just replying here because you appear to be quoting one of my
arguments, and I want to make it clear that I am firmly against adding
additional init-files.

Re: Summary and next steps for (package-initialize)

[Angelo Graziosi]

> Il 21 agosto 2017 alle 18.22 Radon Rosborough <radon.neon@gmail.com> ha scritto:
> 
> 
> > As simple user, I hope you do not proliferate the number of
> > initialization files.
> 
> I'm just replying here because you appear to be quoting one of my
> arguments, and I want to make it clear that I am firmly against adding
> additional init-files.

That is a good news.. Thanks!