Proposals for fixing package-initialize

Proposals for fixing package-initialize

[Radon Rosborough]

Hello all,

You may have seen slightly over 150 emails fly by in the last few
weeks talking about package-initialize. Hilariously, we still haven't
managed to reach a conclusion.

This email is an attempt to present all the proposals that have been
discussed, in an even-handed way, so that we can make a decision
without everyone needing to have read the aforementioned 150 emails.

I will *not* be providing any context on the issue; see [1] for that,
but note that the discussion branched out so much that everyone's
positions are now completely different :P

==> Proposal A: Leave it alone

Summary:
  - call package-initialize in startup.el after loading the init-file
  - when package-initialize is called, also insert a call to it into
    the init-file, if one was not found there
  - this is the current behavior

Advantages:
  - requires no work
  - people are used to the current behavior
  - backwards compatible

Disadvantages:
  - modifying the user's init-file is ugly
  - doing it automatically is annoying and possibly dangerous
  - if the user has customized package.el, then Emacs will put
    package-initialize in the wrong place
  - this type of behavior is not common practice

==> Proposal B: Add a second init-file

Summary:
  - add a second init-file, ~/.emacs.d/early-init.el
  - this file is sourced earlier than init.el, during startup.el
  - move the call to package-initialize in startup.el from its current
    location to between early-init.el and startup.el
  - don't have package.el modify the init-file ever

Advantages:
  - does not require modifying the user's init-file
  - package.el still works out of the box
  - the only people affected by the change are the ones who have
    customized package.el
  - foolproof, never does the wrong thing (except for advanced
    existing configs that customize package.el)
  - people can also use early-init.el to disable things like the tool
    bar before they are loaded in the first place: two birds with one
    stone
  - the only people who need to know about early-init.el are advanced
    users who want to customize package.el
  - Customize works both for customizing package.el and also for
    customizing packages
  - does not require package-initialize to be put in the user's
    init-file, not even manually
  - makes it easy to disable package.el permanently

Disadvantages:
  - adds another init-file, which is additional complexity
  - we must add a mechanism to Customize that allows to specify the
    file into which a particular variable should be saved, so
    package.el can still be customized

==> Proposal B': Add an early-init special form

Summary:
  - same as above, but instead of a separate file, allow init.el to
    start with a special (with-early-config ...) form
  - if with-early-config form is found in init-file, load it when
    early-init.el would be loaded

Advantages:
  - we still only have one init-file

Disadvantages:
  - the startup sequence is a little weird
  - it's not clear how this would work with Customize

==> Proposal C: Add a template init-file

Summary:
  - if Emacs is started without an init-file, generate one from a
    template
  - the template init-file has a call to package-initialize in it, and
    some comments explaining where to put package and package.el
    configuration
  - the template init-file is not generated in 'emacs -Q' mode
  - get rid of package.el modifying the init-file

Advantages:
  - does not require modifying the user's init-file
  - package.el still works out of the box for new users
  - we could use the template init-file for encouraging users towards
    better defaults without breaking backwards compatibility

Disadvantages:
  - having Emacs automatically create files is not ideal
  - if we decide against using the template init-file for better
    defaults, then it's kind of awkward to just use it for package.el
  - doesn't work for users with existing init-files
  - still reqiures a call to package-initialize in the user's
    init-file

==> Proposal D: Hook into error handlers

Summary:
  - call package-initialize whenever a void-function or require error
    is encountered
  - get rid of package.el modifying the init-file
  - possibly get rid of call to package-initialize in startup.el?

Advantages:
  - doesn't require modifying the user's init-file
  - probably package.el still works out of the box

Disadvantages:
  - implications (performance, correctness, implementation details)
    are completely unexplored
  - not very elegant
  - business logic from the package manager is intermixed with a
    fundamental, generic part of the Elisp interpreter
  - hard to tell when package-initialize gets called, so best practice
    would probably still be to have package-initialize in the
    init-file
  - the fact that a core Lisp error causes lots of magic to happen
    will make debugging more "fun"

==> Proposal E: Call package-initialize multiple times

Summary:
  - add a second call to package-initialize before init.el is loaded
  - if package-initialize is called again, and package-load-list or
    other variables have been customized since the last time it was
    called, unload any packages that were previously loaded that now
    should not be loaded

Advantages:
  - does not require modifying the user's init-file
  - backwards compatible, at least if it works

Disadvantages:
  - impossible to disable package.el without introducing a magic
    dotfile (~/.emacs.d/.nopackage) or similar
  - impossible to unload packages correctly, in general, so this
    approach would be buggy in many edge cases
  - severe performance regressions if many packages are disabled using
    package-load-list
  - "As I said, there's no doubt that do+undo is a poor way to do
    nothing." "Again, here we're back to the obvious observation that
    do+undo is a poor and complicated way to do nothing." "Can't
    disagree that "do+undo" is a poor way to do nothing." -- Stefan
  - requires extra work to make multiple calls to package-initialize
    faster, not to mention figuring out how unloading is supposed to
    work, or if that's even possible in principle

==> Proposal F: Use more declarative configuration

Summary:
  - make it so that packages can be configured in a way that causes
    package-initialize to be called automatically

Advantages:
  - would solve all the problems

Disadvantages:
  - nobody has any idea how to do this
  - and it would probably take several years anyway

==> Proposal G: Parse the init-file

Summary:
  - call package-initialize in startup.el before loading init.el
  - parse the init-file in order to find out the values of
    package-load-list, etc. before running package-initialize

Advantages:
  - in the cases that it works, it would allow package.el to work out
    of the box without the need for package-initialize in the user's
    init-file
  - Emacs would not need to modify the user's init-file

Disadvantages:
  - would require solving the halting problem to work in general
  - is a horrifying hack

==> Proposal H: Make the user do it manually

Summary:
  - make it so that package.el doesn't modify the init-file, but don't
    change anything else
  - this is how it worked before the first "solution" to this problem
    was implemented

Advantages:
  - Emacs does not modify the user's init-file

Disadvantages:
  - package.el does not work out of the box, since package
    customizations will fail when put into the init-file
  - consequently we get lots of superfluous bug reports

==> My position

I think that the above advantages and disadvantages speak for
themselves, but to be clear, I think Proposal B is the best. It has
basically no practical disadvantages, solves every problem that is
tangled up in this issue with zero migration necessary except for very
advanced users, is trivial to implement, and simultaneously solves a
large class of other problems related to disabling things that are
enabled early during startup. Every other proposal has significant
practical disadvantages.

==> Postscript

I'd like to say thank you to Eli for pushing the question of whether
it would be practical to call package-initialize only in startup.el,
Timur for questioning why a second init-file would be so bad, and
Clément for explaining to me what was wrong with my original proposal.
Their arguments resulted in the discussion of what I now think is a
better proposal than my original one.

And yes, I really did review all 150 of those emails to gather
information for this summary :o

Best regards,
Radon Rosborough

[1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html

RE: Proposals for fixing package-initialize

[Drew Adams]

Great work summarizing and pushing the question.  Thx.

Re: Proposals for fixing package-initialize

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 2205 bytes --]


() Radon Rosborough <radon.neon@gmail.com>
() Sun, 3 Sep 2017 22:31:57 -0700

   This email is an attempt to present all the proposals that
   have been discussed, in an even-handed way, so that we can
   make a decision without everyone needing to have read the
   aforementioned 150 emails.

Cool, thanks!  I now feel less guilty skimming through that
backlog.  I appreciate the clear presentation style, too.

   ==> Proposal F: Use more declarative configuration

   Summary:
     - make it so that packages can be configured in a way that
       causes package-initialize to be called automatically

   Advantages:
     - would solve all the problems

   Disadvantages:
     - nobody has any idea how to do this
     - and it would probably take several years anyway

This strikes me as the best long-term fix (i.e., defining a data
format, thus giving space to various package manglers to bloom,
wither and die around it, improving it as time goes on).

It's hyperbole to say nobody has any idea, however -- there are
several general text-oriented serialization formats that people
have already defined (e.g., YAML, JSON, sexps).  Any of these
can be easily adapted to the package-metadata domain.  Re time,
granted, settling on a suitable filename extension might take
years (i'll throw out ".epdef" for "Emacs package definition"),
but i think a few weeks (months) of nailing down .epdef by one
person (serial) would inspire very rapid (re-)development of
programs afterwards (parallel) resulting in overall wall-time of
much less than several years.

Sez Perlis:

 We kid ourselves if we think that the ratio of procedure to
 data in an active data-base system can be made arbitrarily
 small or even kept small.

so the trick for Someone will be to define ONLY a data format,
and refrain from stepping into "an active data-base system" (as
much as possible -- w/ Emacs, there be hooks a-lurking 'neath).

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: Proposals for fixing package-initialize

[Radon Rosborough]

> It's hyperbole to say nobody has any idea, however -- there are
> several general text-oriented serialization formats that people have
> already defined (e.g., YAML, JSON, sexps). Any of these can be
> easily adapted to the package-metadata domain.

Oh, I certainly agree. The difficulty I see is that many people want
to do their package (and package manager) configuration dynamically.
For example, set package-load-list so that only packages are loaded
which aren't in some folder used for active development. Or only
interface with the package manager at arbitrary times to effect
deferred installation.

So a purely declarative format will never work, because of all the
interesting use cases it would exclude; it has to be an imperative
format that nevertheless allows Emacs some more control than the
present. Or a declarative format that allows for as much
customizability as an imperative one.

Re: Proposals for fixing package-initialize

[Howard Melman]

FWIW, I'm a 30+ year user of emacs. I'm comfortable writing
elisp and reading emacs elisp source but have never dived
into the C source and am a little behind on the newer
trendier packages.

When I started using package.el I ran into the common
problem and had to figure out to add (package-initialize) to
my init and thought it was clunky. When I read the reason
for me to do this I immediately wondered why it didn't work
like Proposal B. I've never felt the need to customize
package.el so I haven't had to change anything since.

But I wanted to point out another advantage of Proposal B,
it's obvious to semi-advanced users.

-- 

Howard

Re: Proposals for fixing package-initialize

[Clément Pit-Claudel]

Incredible work. Thanks a lot!

On 2017-09-04 07:31, Radon Rosborough wrote:
> Hello all,
> 
> You may have seen slightly over 150 emails fly by in the last few
> weeks talking about package-initialize. Hilariously, we still haven't
> managed to reach a conclusion.
> 
> This email is an attempt to present all the proposals that have been
> discussed, in an even-handed way, so that we can make a decision
> without everyone needing to have read the aforementioned 150 emails.
> 
> I will *not* be providing any context on the issue; see [1] for that,
> but note that the discussion branched out so much that everyone's
> positions are now completely different :P
> 
> ==> Proposal A: Leave it alone
> 
> Summary:
>   - call package-initialize in startup.el after loading the init-file
>   - when package-initialize is called, also insert a call to it into
>     the init-file, if one was not found there
>   - this is the current behavior
> 
> Advantages:
>   - requires no work
>   - people are used to the current behavior
>   - backwards compatible
> 
> Disadvantages:
>   - modifying the user's init-file is ugly
>   - doing it automatically is annoying and possibly dangerous
>   - if the user has customized package.el, then Emacs will put
>     package-initialize in the wrong place
>   - this type of behavior is not common practice
> 
> ==> Proposal B: Add a second init-file
> 
> Summary:
>   - add a second init-file, ~/.emacs.d/early-init.el
>   - this file is sourced earlier than init.el, during startup.el
>   - move the call to package-initialize in startup.el from its current
>     location to between early-init.el and startup.el
>   - don't have package.el modify the init-file ever
> 
> Advantages:
>   - does not require modifying the user's init-file
>   - package.el still works out of the box
>   - the only people affected by the change are the ones who have
>     customized package.el
>   - foolproof, never does the wrong thing (except for advanced
>     existing configs that customize package.el)
>   - people can also use early-init.el to disable things like the tool
>     bar before they are loaded in the first place: two birds with one
>     stone
>   - the only people who need to know about early-init.el are advanced
>     users who want to customize package.el
>   - Customize works both for customizing package.el and also for
>     customizing packages
>   - does not require package-initialize to be put in the user's
>     init-file, not even manually
>   - makes it easy to disable package.el permanently
> 
> Disadvantages:
>   - adds another init-file, which is additional complexity
>   - we must add a mechanism to Customize that allows to specify the
>     file into which a particular variable should be saved, so
>     package.el can still be customized
> 
> ==> Proposal B': Add an early-init special form
> 
> Summary:
>   - same as above, but instead of a separate file, allow init.el to
>     start with a special (with-early-config ...) form
>   - if with-early-config form is found in init-file, load it when
>     early-init.el would be loaded
> 
> Advantages:
>   - we still only have one init-file
> 
> Disadvantages:
>   - the startup sequence is a little weird
>   - it's not clear how this would work with Customize
> 
> ==> Proposal C: Add a template init-file
> 
> Summary:
>   - if Emacs is started without an init-file, generate one from a
>     template
>   - the template init-file has a call to package-initialize in it, and
>     some comments explaining where to put package and package.el
>     configuration
>   - the template init-file is not generated in 'emacs -Q' mode
>   - get rid of package.el modifying the init-file
> 
> Advantages:
>   - does not require modifying the user's init-file
>   - package.el still works out of the box for new users
>   - we could use the template init-file for encouraging users towards
>     better defaults without breaking backwards compatibility
> 
> Disadvantages:
>   - having Emacs automatically create files is not ideal
>   - if we decide against using the template init-file for better
>     defaults, then it's kind of awkward to just use it for package.el
>   - doesn't work for users with existing init-files
>   - still reqiures a call to package-initialize in the user's
>     init-file
> 
> ==> Proposal D: Hook into error handlers
> 
> Summary:
>   - call package-initialize whenever a void-function or require error
>     is encountered
>   - get rid of package.el modifying the init-file
>   - possibly get rid of call to package-initialize in startup.el?
> 
> Advantages:
>   - doesn't require modifying the user's init-file
>   - probably package.el still works out of the box
> 
> Disadvantages:
>   - implications (performance, correctness, implementation details)
>     are completely unexplored
>   - not very elegant
>   - business logic from the package manager is intermixed with a
>     fundamental, generic part of the Elisp interpreter
>   - hard to tell when package-initialize gets called, so best practice
>     would probably still be to have package-initialize in the
>     init-file
>   - the fact that a core Lisp error causes lots of magic to happen
>     will make debugging more "fun"
> 
> ==> Proposal E: Call package-initialize multiple times
> 
> Summary:
>   - add a second call to package-initialize before init.el is loaded
>   - if package-initialize is called again, and package-load-list or
>     other variables have been customized since the last time it was
>     called, unload any packages that were previously loaded that now
>     should not be loaded
> 
> Advantages:
>   - does not require modifying the user's init-file
>   - backwards compatible, at least if it works
> 
> Disadvantages:
>   - impossible to disable package.el without introducing a magic
>     dotfile (~/.emacs.d/.nopackage) or similar
>   - impossible to unload packages correctly, in general, so this
>     approach would be buggy in many edge cases
>   - severe performance regressions if many packages are disabled using
>     package-load-list
>   - "As I said, there's no doubt that do+undo is a poor way to do
>     nothing." "Again, here we're back to the obvious observation that
>     do+undo is a poor and complicated way to do nothing." "Can't
>     disagree that "do+undo" is a poor way to do nothing." -- Stefan
>   - requires extra work to make multiple calls to package-initialize
>     faster, not to mention figuring out how unloading is supposed to
>     work, or if that's even possible in principle
> 
> ==> Proposal F: Use more declarative configuration
> 
> Summary:
>   - make it so that packages can be configured in a way that causes
>     package-initialize to be called automatically
> 
> Advantages:
>   - would solve all the problems
> 
> Disadvantages:
>   - nobody has any idea how to do this
>   - and it would probably take several years anyway
> 
> ==> Proposal G: Parse the init-file
> 
> Summary:
>   - call package-initialize in startup.el before loading init.el
>   - parse the init-file in order to find out the values of
>     package-load-list, etc. before running package-initialize
> 
> Advantages:
>   - in the cases that it works, it would allow package.el to work out
>     of the box without the need for package-initialize in the user's
>     init-file
>   - Emacs would not need to modify the user's init-file
> 
> Disadvantages:
>   - would require solving the halting problem to work in general
>   - is a horrifying hack
> 
> ==> Proposal H: Make the user do it manually
> 
> Summary:
>   - make it so that package.el doesn't modify the init-file, but don't
>     change anything else
>   - this is how it worked before the first "solution" to this problem
>     was implemented
> 
> Advantages:
>   - Emacs does not modify the user's init-file
> 
> Disadvantages:
>   - package.el does not work out of the box, since package
>     customizations will fail when put into the init-file
>   - consequently we get lots of superfluous bug reports
> 
> ==> My position
> 
> I think that the above advantages and disadvantages speak for
> themselves, but to be clear, I think Proposal B is the best. It has
> basically no practical disadvantages, solves every problem that is
> tangled up in this issue with zero migration necessary except for very
> advanced users, is trivial to implement, and simultaneously solves a
> large class of other problems related to disabling things that are
> enabled early during startup. Every other proposal has significant
> practical disadvantages.
> 
> ==> Postscript
> 
> I'd like to say thank you to Eli for pushing the question of whether
> it would be practical to call package-initialize only in startup.el,
> Timur for questioning why a second init-file would be so bad, and
> Clément for explaining to me what was wrong with my original proposal.
> Their arguments resulted in the discussion of what I now think is a
> better proposal than my original one.
> 
> And yes, I really did review all 150 of those emails to gather
> information for this summary :o
> 
> Best regards,
> Radon Rosborough
> 
> [1]: https://lists.gnu.org/archive/html/emacs-devel/2017-08/msg00154.html
> 
> 

Re: Proposals for fixing package-initialize

[Ted Zlatanov]

On Sun, 3 Sep 2017 22:31:57 -0700 Radon Rosborough <radon.neon@gmail.com> wrote: 

RR> You may have seen slightly over 150 emails fly by in the last few
RR> weeks talking about package-initialize. Hilariously, we still haven't
RR> managed to reach a conclusion.

RR> This email is an attempt to present all the proposals that have been
RR> discussed, in an even-handed way, so that we can make a decision
RR> without everyone needing to have read the aforementioned 150 emails.

Thank you!

FWIW I'm in favor of Proposal F: Use more declarative configuration

Ted

Re: Proposals for fixing package-initialize

[Radon Rosborough]

I'm currently in the process of working on a patch to implement
Proposal B, since it is clear how to do that whereas Proposal F is
probably a fair ways off still.

The main complexities are:
- updating Custom to allow saving variables to different files,
because package.el config now needs to go in the early init file
- moving some things around in startup.el (e.g. username validation),
because the early init file needs to be loaded early of course!

Re: Proposals for fixing package-initialize

[Stefan Monnier]

> The main complexities are:
> A- updating Custom to allow saving variables to different files,
> because package.el config now needs to go in the early init file
> B- moving some things around in startup.el (e.g. username validation),
> because the early init file needs to be loaded early of course!

Note that the A part can be done as a second step (it's not a new
problem, so the step B doesn't make the step A more urgent).


        Stefan