Re: Calling (package-initialize) sooner during initialization

Re: Calling (package-initialize) sooner during initialization

[Vasilij Schneidermann]

Hello,

I am surprised that option #2 is being favoured and has been met with a
fair amount of discussion considering it's the hackier solution (why
*add* text to the init file if one could merely *do* the package
initialization step in startup.el, completely bypassing issues like init
code possibly getting added multiple times to the file and the general
annoyance as this cluttering cannot be disabled at all and can happen at
every installation) and involves more work.

Since option #1 wasn't discussed in the prior discussion, I'd suggest a
third one, enabling packages in startup.el unless an Emacs-specific
environment variable is set (or Emacs is started up in a special mode of
operation).  Doing so works cross-platform, avoids cluttering anything
(be it an existing or new file) and is simple to get right.

Any thoughts on this proposal?

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

2015-04-12 0:10 GMT+01:00 Vasilij Schneidermann <v.schneidermann@gmail.com>:
> Hello,
>
> I am surprised that option #2 is being favoured and has been met with a
> fair amount of discussion considering it's the hackier solution (why
> *add* text to the init file if one could merely *do* the package
> initialization step in startup.el, completely bypassing issues like init
> code possibly getting added multiple times to the file and the general
> annoyance as this cluttering cannot be disabled at all and can happen at
> every installation) and involves more work.
>
> Since option #1 wasn't discussed in the prior discussion, I'd suggest a
> third one, enabling packages in startup.el unless an Emacs-specific
> environment variable is set (or Emacs is started up in a special mode of
> operation).  Doing so works cross-platform, avoids cluttering anything
> (be it an existing or new file) and is simple to get right.
>
> Any thoughts on this proposal?

I'm fine with this too. Like I said in the previous post, I'm just
glad this issue is up for the slaughter, I don't care whether we kill
it with a magnum or a tennis racket.

It's important to note that an extra envvar will probably need to be
documented in places I'm not aware of.

Re: Calling (package-initialize) sooner during initialization

[Mark Oteiza]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> I'm just glad this issue is up for the slaughter, I don't care whether
> we kill it with a magnum or a tennis racket.

Better documenting the relationship between custom.el and package.el
should have been considered over resorting to indiscriminately
clobbering init.el. (info "(emacs) Package Installation") talks about
use of (package-initialize) in a user's init file.  This could be
improved.  Alternatively, why bother having the documentation if Emacs
is going to be in the habit of forcing this behaviour?

I have never needed package-initialize in my init.el for things
installed by package.el, either from an archive or locally.  Why am I
subjected to having my init.el overwritten when I install anything with
package.el?

custom.el and package.el simply aren't made for each other, please do not
hack around it in my init file:

https://lists.gnu.org/archive/html/emacs-devel/2015-03/msg01010.html
Drew Adams wrote:
> The user asked, "What did I do wrong?"  The answer is apparently
> that Emacs is not yet ready for you to use Customize to configure
> your use of package.el

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

> > I'm just glad this issue is up for the slaughter, I don't care whether
> > we kill it with a magnum or a tennis racket.
>
> Better documenting the relationship between custom.el and package.el
> should have been considered over resorting to indiscriminately
> clobbering init.el.

This is not an issue with custom.el. The custom.el problem is one of
the consequences. Another consequence is that users can't paste
configurations into their init file in the way that 90% of the
(emacs-related) internet is telling them to.
By adding the chosen snippet to the user's init file we are preventing
this sort of problem for new users, instead of telling them it's their
fault for not understanding package.el.

The only cost associated with this solution is that the advanced users
who don't want this feature now need to keep “;(package-initialize)”
somewhere inside their init file. That's all.
We are imposing 21 characters on the few (me included), in exchanged
for many lines and much headache we're no longer imposing on the many.

> (info "(emacs) Package Installation") talks about
> use of (package-initialize) in a user's init file.  This could be
> improved.  Alternatively, why bother having the documentation if Emacs is going to be in the habit of forcing this behaviour?

Nobody is being forced. The added snippet clearly explains that you
can comment it out. The only thing that's being forced upon you is the
presence of 21 characters in your init file.

> I have never needed package-initialize in my init.el for things
> installed by package.el, either from an archive or locally.  Why am I
> subjected to having my init.el overwritten when I install anything with
> package.el?

You will have one sexp added to your init-file exactly once. You are
being “subjected” to that because a lot of other people were being
subjected things much worse (see the original thread).
You only need to have “;(package-initialize)” somewhere in your init
file and this feature will never ever affect your again (it will never
even visit your init file again).

I'm truly sorry if you feel that is too much.

> custom.el and package.el simply aren't made for each other, please do not
> hack around it in my init file:

As mentioned above, this is not about custom.

Re: Calling (package-initialize) sooner during initialization

[Mark Oteiza]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> > I'm just glad this issue is up for the slaughter, I don't care whether
>> > we kill it with a magnum or a tennis racket.
>>
>> Better documenting the relationship between custom.el and package.el
>> should have been considered over resorting to indiscriminately
>> clobbering init.el.
>
> This is not an issue with custom.el. The custom.el problem is one of
> the consequences. Another consequence is that users can't paste
> configurations into their init file in the way that 90% of the
> (emacs-related) internet is telling them to.
> By adding the chosen snippet to the user's init file we are preventing
> this sort of problem for new users, instead of telling them it's their
> fault for not understanding package.el.

Perhaps if the related documentation didn't suck, then 90% of the advice
on the (emacs-related) internet wouldn't be incorrect.

>> (info "(emacs) Package Installation") talks about
>> use of (package-initialize) in a user's init file.  This could be
>> improved.  Alternatively, why bother having the documentation if
>> Emacs is going to be in the habit of forcing this behaviour?
>
> Nobody is being forced. The added snippet clearly explains that you
> can comment it out. The only thing that's being forced upon you is the
> presence of 21 characters in your init file.

"Nobody is being forced."

"The only thing that's being forced […]"

Wow.

>> I have never needed package-initialize in my init.el for things
>> installed by package.el, either from an archive or locally.  Why am I
>> subjected to having my init.el overwritten when I install anything with
>> package.el?
>
> You will have one sexp added to your init-file exactly once. You are
> being “subjected” to that because a lot of other people were being
> subjected things much worse (see the original thread).
> You only need to have “;(package-initialize)” somewhere in your init
> file and this feature will never ever affect your again (it will never
> even visit your init file again).

Yes, it seems the internet is giving bad advice and Emacs will now
require a specific string in a file to solve issues stemming from
that.

I suppose it's a fine solution in that it's easy to patch out of Emacs.

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

>> Nobody is being forced. The added snippet clearly explains that you
>> can comment it out. The only thing that's being forced upon you is the
>> presence of 21 characters in your init file.
>
> "Nobody is being forced."
>
> "The only thing that's being forced […]"
>
> Wow.
>

Forgive me. I should have said “Nobody is being forced to call
package-initialize, the only thing that's being forced is the presence
of the comment.”

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> initialization step in startup.el, completely bypassing issues like init
> code possibly getting added multiple times to the file and the general
> annoyance as this cluttering cannot be disabled at all and can happen at
> every installation) and involves more work.

We discussed this already.  There's the issue of configuring package.el
before loading it and/or before calling package-initialize.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

His suggestion is to disable the loading+ initializing based on an
environment variable.
On Apr 12, 2015 4:56 AM, "Stefan Monnier" <monnier@iro.umontreal.ca> wrote:

> > initialization step in startup.el, completely bypassing issues like init
> > code possibly getting added multiple times to the file and the general
> > annoyance as this cluttering cannot be disabled at all and can happen at
> > every installation) and involves more work.
>
> We discussed this already.  There's the issue of configuring package.el
> before loading it and/or before calling package-initialize.
>
>
>         Stefan
>
>

[-- Attachment #2: Type: text/html, Size: 911 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> His suggestion is to disable the loading+ initializing based on an
> environment variable.

So every user who just wants to configure one of those package variables
would need to set this env-var consistently everywhere?  Yuck.


        Stefan "Can you say «my package initialization works from the shell
                but not when I start Emacs from the GUI»?"

Re: Calling (package-initialize) sooner during initialization

[Vasilij Schneidermann]

> We discussed this already.  There's the issue of configuring package.el
> before loading it and/or before calling package-initialize.

OK, I've checked again and sighted eight customization candidates in
package.el:

- package-enable-at-startup: Would be equivalent to the proposed
  environment variable.
- package-load-list: Needs to be configured before.
- package-archives: Needs to be configured before for
  non-interactive package installation.
- package-pinned-packages: Needs to be configured before for
  non-interactive package installation.
- package-user-dir: Needs to be configured before.
- package-directory-list: Needs to be configured before.
- package-check-signature and package-unsigned-archives: Can be
  customized afterwards

So, while setting the environment variable in my proposal would equal
the (setq package-enable-at-startup nil) trick in combination with the
necessary customization and (package-initialize) at a later point in the
init file, it would be clunkier than the status quo.

Another possible technical solution to this would be introducing an
extra configuration file containing the listed startup/package-related
options only that is read in before the init file.

Finally, to suggest a non-technical solution to this "problem" of new
users not knowing about (package-initialize) for the use of
functionality from packages they've installed from package archives, why
not improve existing documentation?

GNU ELPA's homepage is minimal.  Its documentation seems to be the
"Packages" section of the Emacs info manual.  Linking to it would be a
fair improvement opportunity.  MELPA's "Getting started" page explains
the need for and placement of (package-initialize) reasonably well,
perhaps an explanation of the gotchas involved would be useful.
Marmalade has an interactive walkthrough, yet doesn't mention
(package-initialize) at all.  Adding that part to its homepage would be
better than passing the responsibility to package developers getting
issues reported by users not aware of this.  I would generally be
interested in empirical evidence of the numbers and package archives
involved in such bug reports to see whether my suggestions are right on
the mark.

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

> GNU ELPA's homepage is minimal.

I agree Gelpa's frontpage needs a bit more information (not a lot,
though, it's good that it's not cluttered), regardless of the fate of
this discussion.

>  MELPA's "Getting started" page explains the need for and placement of (package-initialize) reasonably well,
> perhaps an explanation of the gotchas involved would be useful.
> Marmalade has an interactive walkthrough, yet doesn't mention
> (package-initialize) at all.

Both archives have Github repos (and very responsive maintainers)
where you can file suggestions.

Re: Calling (package-initialize) sooner during initialization

[Taylan Ulrich Bayırlı/Kammer]

Hi,

has it been considered yet to simply support an optional

  ~/.emacs.d/pre-package-init.el

file that always gets executed before packages are initialized (just
like the current init.el), then packages get initialized, and then
init.el or ~/.emacs is loaded?

So startup looks like:

  1. Load ~/.emacs.d/pre-package-init.el if it exists.
  2. Initialize packages.
  3. Load ~/.emacs or ~/.emacs.d/init.el if one exists.

Are there any disadvantages to this approach?

Taylan


P.S.: We can even support a setting which further delays package
initialization, i.e. change the order of steps 2 and 3, if really
desired.  It should be as simple as setting a variable in
pre-package-init.el.

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

> has it been considered yet to simply support an optional
>
>   ~/.emacs.d/pre-package-init.el

Yes. I suggested this on a previous thread (before starting this one) and
someone complained of yet another file affecting emacs behaviour. Then I
suggested the streamlined version of that in the first email of this thread
(having a file that delays initialization, but doesn't actually get
evaluated), and the few responses I got still preferred the other option.

[-- Attachment #2: Type: text/html, Size: 521 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Nic Ferrier]

bruce.connor.am@gmail.com writes:

>> has it been considered yet to simply support an optional
>>
>>   ~/.emacs.d/pre-package-init.el
>
> Yes. I suggested this on a previous thread (before starting this one) and
> someone complained of yet another file affecting emacs behaviour. Then I
> suggested the streamlined version of that in the first email of this thread
> (having a file that delays initialization, but doesn't actually get
> evaluated), and the few responses I got still preferred the other
> option.

My preference would be to add a post-package-init.el thereby not
changing the defined semantics of the current system at all.


Nic

Re: Calling (package-initialize) sooner during initialization

[Taylan Ulrich Bayırlı/Kammer]

Nic Ferrier <nferrier@ferrier.me.uk> writes:

> My preference would be to add a post-package-init.el thereby not
> changing the defined semantics of the current system at all.

That would be good for backwards compatibility, but won't solve the
problem of people asking why a function in their init.el is undefined
when they installed the package providing it.

A clearly with-solution documented breakage in a major release should be
an acceptable trade-off, I think.  WDYT?

(The solution being "if you have (package-initialize) anywhere in your
init.el, copy everything before it to ~/.emacs.d/pre-package-init.el,
then remove the (package-initialize) and everything prior to it."
Correct me if I'm missing something.)

Taylan

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 635 bytes --]

> Nic Ferrier <nferrier@ferrier.me.uk> writes:
>
> > My preference would be to add a post-package-init.el thereby not
> > changing the defined semantics of the current system at all.
>
> That would be good for backwards compatibility, but won't solve the
> problem of people asking why a function in their init.el is undefined
> when they installed the package providing it.

Precisely. Whichever solution is adopted, it must work out of the box for
people who do nothing but add (require 'some-package) to their init file.
The goal here is to find the smallest semantics change that affects the
fewest people, given that restriction.

[-- Attachment #2: Type: text/html, Size: 793 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Nic Ferrier]

bruce.connor.am@gmail.com writes:

>> Nic Ferrier <nferrier@ferrier.me.uk> writes:
>>
>> > My preference would be to add a post-package-init.el thereby not
>> > changing the defined semantics of the current system at all.
>>
>> That would be good for backwards compatibility, but won't solve the
>> problem of people asking why a function in their init.el is undefined
>> when they installed the package providing it.
>
> Precisely. Whichever solution is adopted, it must work out of the box for
> people who do nothing but add (require 'some-package) to their init file.
> The goal here is to find the smallest semantics change that affects the
> fewest people, given that restriction.


I am kind of throwing up my arms in frustration at this.

A way was picked a long time ago. Y'all now want to break that way in
the most breaking way possible instead of being incremental.

Are you sure that this will be least damaging to most people? Why are
you sure?


Nic

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

2015-04-18 19:25 GMT+01:00 Nic Ferrier <nferrier@ferrier.me.uk>:
> I am kind of throwing up my arms in frustration at this.
>
> A way was picked a long time ago. Y'all now want to break that way in
> the most breaking way possible instead of being incremental.

All I want is to fix a problem a large number of less-knowledgeable
users are experiencing. I don't think there is a way to do that
without imposing at least a minor inconvenience on a (hopefully small)
portion of the more experienced users. But I'd like to keep it just
that at most (a small inconvenience), which is what the currently
implemented solution was about. I thought that being forced to keep
";(package-initialize)" somewhere inside your init file would be small
inconvenience at most, but some people seem to disagree, and that's
how these more complicated discussions started showing up.

> Are you sure that this will be least damaging to most people? Why are
> you sure?

Which specific proposal are you referring to?
Taylan's suggestions seems to be very non-intrusive on the user,
"seems" being an important word here. If you think it'll damage a lot
of people, please let us know.

AFAICT, people who need to do some special configuration before
`package-initialize' would now have to move that stuff into another
file (the suggested "pre-package-init.el") but that's about it. In
exchange, it will make package configuration less headache inducing
for inexperienced users, and even fixes the compatibility between
package.el and custom.el (which the currently solution does not).

Re: Calling (package-initialize) sooner during initialization

[Taylan Ulrich Bayırlı/Kammer]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> I suggested this on a previous thread (before starting this one) and
> someone complained of yet another file affecting emacs behaviour.

What's so bad about having one more file in ~/.emacs.d that affects
Emacs's behavior?  It's the place for such files.

I don't see any problem at all.  Maybe I'm missing something; can anyone
think of any actual disadvantages?

Taylan

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

I think all those discussions are missing the point.
If we want to improve the system to the point of considering adding new
init files, then we should try and fix other problems at the same time.

So here's one of the larger problems:

How can we make the system work such that the user can:
- Use customize to set package-user-dir, package-pinned-packages, etc...
- Use customize to configure a variable which has a :require which loads
  a file that's only available after calling package-initialize.
The first requires package-initialize to be called late, the second
requires package-initialize to be called early.

Maybe a solution is to simply make customize-set-variables lazier, so
that variables with a :require see their setting delayed to after
package-initialize was called.  Or else, have package-initialize be
called by customize-set-variables.  Or...


        Stefan


>>>>> "Artur" == Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> has it been considered yet to simply support an optional
>> 
>> ~/.emacs.d/pre-package-init.el

> Yes. I suggested this on a previous thread (before starting this one) and
> someone complained of yet another file affecting emacs behaviour. Then I
> suggested the streamlined version of that in the first email of this thread
> (having a file that delays initialization, but doesn't actually get
> evaluated), and the few responses I got still preferred the other option.

Re: Calling (package-initialize) sooner during initialization

[Dmitry Gutov]

On 04/18/2015 08:16 PM, Stefan Monnier wrote:

> Or else, have package-initialize be called by customize-set-variables.

This would be the easiest to implement, I think.

Re: Calling (package-initialize) sooner during initialization

[Taylan Ulrich Bayırlı/Kammer]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 04/18/2015 08:16 PM, Stefan Monnier wrote:
>
>> Or else, have package-initialize be called by customize-set-variables.
>
> This would be the easiest to implement, I think.

It contradicts with the goal of running package-initialize before the
loading of init.el, since you would need to apply Customize settings
before loading init.el.

We probably don't want to apply Customize settings before (or at the
start of) loading init.el, do we?

So I would go with making Customize smarter and writing settings in the
package group to pre-package-init.el or `package-custom-file' (analogous
to `custom-file').


Given that

1. initialization of installed packages intuitively belongs before
   loading user configuration (since it's likely to cover configuration
   specific to some packages),
   
2. one might nevertheless want to configure the package system itself,

I don't see any clean solution other than to separate pre-package
configuration from normal configuration, be it for Customize or user
init files.

Does anyone see disadvantages with this design?  I think it's the most
sensible approach.

Taylan

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

2015-04-18 19:16 GMT+01:00 Taylan Ulrich Bayırlı/Kammer
<taylanbayirli@gmail.com>:
> We probably don't want to apply Customize settings before (or at the
> start of) loading init.el, do we?

That's yet another discussion.
I usually tell people to place their custom-set-variables at the top
of their init file. Otherwise, if you install a new theme and try to
`(load-theme 'theme-name)' in your init file, you'll get problems
because `custom-safe-themes' hasn't been set yet.
At the same time, it can't just be automatically loaded before the
init-file, because a lot of people might setq the `custom-file'
variable to something else.

> Does anyone see disadvantages with this design?  I think it's the most
> sensible approach.

The only problem I see is the added complexity, which I'm mostly ok
with. But I'd hopefully like to hear from more than 3 people on this.

RE: Calling (package-initialize) sooner during initialization

[Drew Adams]

> I usually tell people to place their custom-set-variables at the top
> of their init file. 

Sorry, but any such advice is misguided, IMO.  Depending on one's
init file (including what particular libraries are loaded, and
when), it can make sense to load `custom-file' (the equivalent of,
but wiser than, positioning `custom-set-variables') in any number
of places in the file.

There is simply no such one-size-fits-all rule wrt when Customize
settings should be made.  If those fiddling now with redesigning
package setup think there is, then I fear we are in for a world of
trouble.  Dunno whether this is the kind of thing Nic was thinking
of, but it's what I worry about.

My suggestion: Don't try to be too clever.  Keep it simple.
Which is another way of saying avoid simplistic assumptions.

> Otherwise, if you install a new theme and try to
> `(load-theme 'theme-name)' in your init file, you'll get problems
> because `custom-safe-themes' hasn't been set yet.

Welcome to the world of Customize.  That is one consideration.
There are others.  You will not find a simple rule such as you
are suggesting that works in general, precisely because what
else is in the init file matters.

RE: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

On Apr 19, 2015 1:24 AM, "Drew Adams" <drew.adams@oracle.com> wrote:
>
> > I usually tell people to place their custom-set-variables at the top
> > of their init file.
>
> Sorry, but any such advice is misguided, IMO.
>
> There is simply no such one-size-fits-all rule wrt when Customize
> settings should be made.

I agree 100%, there's no right order that will cover every case. I was just
saying that keeping it at the top avoids a common problem with themes. It's
not something I advocate religiously, it's just something I tell new users
when they run into this theme issue.

> > Otherwise, if you install a new theme and try to
> > `(load-theme 'theme-name)' in your init file, you'll get problems
> > because `custom-safe-themes' hasn't been set yet.
>
> Welcome to the world of Customize.  That is one consideration.
> There are others.  You will not find a simple rule such as you
> are suggesting that works in general, precisely because what
> else is in the init file matters.

Again, agreed.

[-- Attachment #2: Type: text/html, Size: 1316 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

2015-04-18 18:56 GMT+01:00 Dmitry Gutov <dgutov@yandex.ru>:
> On 04/18/2015 08:16 PM, Stefan Monnier wrote:
>
>> Or else, have package-initialize be called by customize-set-variables.
>
> This would be the easiest to implement, I think.

Yes, but it would only solve the custom.el problem, not the more
general package.el problem.

Re: Calling (package-initialize) sooner during initialization

[Dmitry Gutov]

On 04/18/2015 09:24 PM, Artur Malabarba wrote:

> Yes, but it would only solve the custom.el problem, not the more
> general package.el problem.

The "more general package.el problem" is still there unless (or until) 
the user interacts with the Customize interface, isn't it? Then it's not 
a complete solution either.

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

On Apr 18, 2015 7:32 PM, "Dmitry Gutov" <dgutov@yandex.ru> wrote:
>
> On 04/18/2015 09:24 PM, Artur Malabarba wrote:
>
>> Yes, but it would only solve the custom.el problem, not the more
>> general package.el problem.
>
>
> The "more general package.el problem" is still there unless (or until)
the user interacts with the Customize interface, isn't it? Then it's not a
complete solution either.

With Stefan's proposals, yes. With other solutions being discussed, no, the
more general problem would solved regardless of custom.el.

[-- Attachment #2: Type: text/html, Size: 708 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Taylan Ulrich Bayırlı/Kammer]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> I think all those discussions are missing the point.
> If we want to improve the system to the point of considering adding new
> init files, then we should try and fix other problems at the same time.
>
> So here's one of the larger problems:
>
> How can we make the system work such that the user can:
> - Use customize to set package-user-dir, package-pinned-packages, etc...
> - Use customize to configure a variable which has a :require which loads
>   a file that's only available after calling package-initialize.
> The first requires package-initialize to be called late, the second
> requires package-initialize to be called early.
>
> Maybe a solution is to simply make customize-set-variables lazier, so
> that variables with a :require see their setting delayed to after
> package-initialize was called.  Or else, have package-initialize be
> called by customize-set-variables.  Or...

I'm not sure if this isn't an orthogonal problem, but indeed as someone
who doesn't use Customize I didn't think about it at all, and know
little about it so excuses in advance for ignorance.

How about Customize being "smart" and separating package-related
configurations from other ones?  I don't know how hard it would be to
implement, but it feels like the right design; obviously package related
customization should be applied before package-initialize, but all other
customization should be applied after package-initialize, since it's a
configuration system, meaning it should be one layer above the package
system, going purely by intuition.

So startup would look like:

  1. pre-package-init.el
  2. Package related customization.
  3. package-initialize
  4. init.el
  5. All other customization.

Would that work?

Taylan

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

2015-04-18 19:04 GMT+01:00 Taylan Ulrich Bayırlı/Kammer
<taylanbayirli@gmail.com>:
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
>> Maybe a solution is to simply make customize-set-variables lazier, so
>> that variables with a :require see their setting delayed to after
>> package-initialize was called.  Or else, have package-initialize be
>> called by customize-set-variables.  Or...
>
> I'm not sure if this isn't an orthogonal problem, but indeed as someone
> who doesn't use Customize I didn't think about it at all, and know
> little about it so excuses in advance for ignorance.

Yes.
It's not completely orthogonal, as the problems are related. But
things like "have package-initialize be called by
customize-set-variables" will not solve the problem, because the
problem goes beyond custom.el.

> How about Customize being "smart" and separating package-related
> configurations from other ones?  I don't know how hard it would be to
> implement

custom.el would have to make a new defcustom keyword availabe
(something like `:before-package'), and then package.el (and any other
packages that need this) would simply set that keyword.

> obviously package related
> customization should be applied before package-initialize, but all other
> customization should be applied after package-initialize, since it's a
> configuration system, meaning it should be one layer above the package
> system, going purely by intuition.
>
> So startup would look like:
>
>   1. pre-package-init.el
>   2. Package related customization.
>   3. package-initialize
>   4. init.el
>   5. All other customization.

I think "4. init.el + All other customization." would be more
accurate, as the two are currently loaded together (and separating
them, if we want to, is another discussion entirely).

> Would that work?

AFAICS, yes.
I'm not in love with the complexity of this solution. However, I do
believe it solves both problems (the custom.el issue and the general
package.el issue), and I believe there's no way to do that with a
simpler solution.

So, I guess I'm saying I'm ok with this if nobody has a better idea.

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

>> How can we make the system work such that the user can:
>> - Use customize to set package-user-dir, package-pinned-packages, etc...
>> - Use customize to configure a variable which has a :require which loads
>> a file that's only available after calling package-initialize.

Bonus points if your system does not make special assumptions about
there being a special "package.el", so that when some new "package+.el"
comes around the scheme will naturally extend to that as well (even if
both package.el and package+.el end up being in use at the same time).


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]

Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com> schrieb am Sa., 18.
Apr. 2015 um 20:04 Uhr:

> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
> > I think all those discussions are missing the point.
> > If we want to improve the system to the point of considering adding new
> > init files, then we should try and fix other problems at the same time.
> >
> > So here's one of the larger problems:
> >
> > How can we make the system work such that the user can:
> > - Use customize to set package-user-dir, package-pinned-packages, etc...
> > - Use customize to configure a variable which has a :require which loads
> >   a file that's only available after calling package-initialize.
> > The first requires package-initialize to be called late, the second
> > requires package-initialize to be called early.
> >
> > Maybe a solution is to simply make customize-set-variables lazier, so
> > that variables with a :require see their setting delayed to after
> > package-initialize was called.  Or else, have package-initialize be
> > called by customize-set-variables.  Or...
>
> I'm not sure if this isn't an orthogonal problem, but indeed as someone
> who doesn't use Customize I didn't think about it at all, and know
> little about it so excuses in advance for ignorance.
>
> How about Customize being "smart" and separating package-related
> configurations from other ones?  I don't know how hard it would be to
> implement, but it feels like the right design; obviously package related
> customization should be applied before package-initialize, but all other
> customization should be applied after package-initialize, since it's a
> configuration system, meaning it should be one layer above the package
> system, going purely by intuition.
>
> So startup would look like:
>
>   1. pre-package-init.el
>   2. Package related customization.
>   3. package-initialize
>   4. init.el
>   5. All other customization.
>
> Would that work?
>

I think that would be a good solution, although I'd modify it a bit:

1. Contents of init.el before applying customizations.
2. Applying customizations, either inline in init.el or by loading
custom.el from init.el
  2.a. Package related customization
  2.b. package-initialize
  2.c. All other customization
3. Rest of init.el

That would be closer to the current behavior and eliminate the need for
pre-package-init.el.

[-- Attachment #2: Type: text/html, Size: 3040 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]

On Apr 19, 2015 7:44 AM, "Philipp Stephani" <p.stephani2@gmail.com> wrote:
>
>
>
> Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com> schrieb am Sa.,
18. Apr. 2015 um 20:04 Uhr:
>>
>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>>
>> > I think all those discussions are missing the point.
>> > If we want to improve the system to the point of considering adding new
>> > init files, then we should try and fix other problems at the same time.
>> >
>> > So here's one of the larger problems:
>> >
>> > How can we make the system work such that the user can:
>> > - Use customize to set package-user-dir, package-pinned-packages,
etc...
>> > - Use customize to configure a variable which has a :require which
loads
>> >   a file that's only available after calling package-initialize.
>> > The first requires package-initialize to be called late, the second
>> > requires package-initialize to be called early.
>> >
>> > Maybe a solution is to simply make customize-set-variables lazier, so
>> > that variables with a :require see their setting delayed to after
>> > package-initialize was called.  Or else, have package-initialize be
>> > called by customize-set-variables.  Or...
>>
>> I'm not sure if this isn't an orthogonal problem, but indeed as someone
>> who doesn't use Customize I didn't think about it at all, and know
>> little about it so excuses in advance for ignorance.
>>
>> How about Customize being "smart" and separating package-related
>> configurations from other ones?  I don't know how hard it would be to
>> implement, but it feels like the right design; obviously package related
>> customization should be applied before package-initialize, but all other
>> customization should be applied after package-initialize, since it's a
>> configuration system, meaning it should be one layer above the package
>> system, going purely by intuition.
>>
>> So startup would look like:
>>
>>   1. pre-package-init.el
>>   2. Package related customization.
>>   3. package-initialize
>>   4. init.el
>>   5. All other customization.
>>
>> Would that work?
>
>
> I think that would be a good solution, although I'd modify it a bit:
>
> 1. Contents of init.el before applying customizations.

I'm afraid this step is already a problem. If these initial contents
contain a (require 'some-package), the user will be given an error during
initialization.

That's why Taylan's solution has an extra file. It's annoying, but
necessary.

> 2. Applying customizations, either inline in init.el or by loading
custom.el from init.el
>   2.a. Package related customization
>   2.b. package-initialize
>   2.c. All other customization
> 3. Rest of init.el
>
> That would be closer to the current behavior and eliminate the need for
pre-package-init.el.

It would be so close to the current behaviour that the problem wouldn't be
solved. ;-)

The point here is that it needs to just work for the less knowledgeable
user. We can't expect them to move their call to (require 'some-package)
until after the custom.el snippet is loaded. Not to mention, they might not
even have such a snippet.

[-- Attachment #2: Type: text/html, Size: 4096 bytes --]

Re: Calling (package-initialize) sooner during initialization

[chad]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]


> On 18 Apr 2015, at 10:16, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:
> 
> I think all those discussions are missing the point.
> If we want to improve the system to the point of considering adding new
> init files, then we should try and fix other problems at the same time.
> […]
> Maybe a solution is to simply make customize-set-variables lazier, so
> that variables with a :require see their setting delayed to after
> package-initialize was called.  Or else, have package-initialize be
> called by customize-set-variables.  Or…

I havent tried it as a solution to these issues, since I long-ago
split customize out into a separate file that is loaded manually
at a specific place in my .emacs file, but it seems that John
Wiegleys use-package package might address this issue more generally,
by deferring the relevant config steps programmatically. This would
be new infrastructure that new users could use in place of the
existing snippets that they place in their init files (presumably
from the web, email, friends, etc).

The project right now is hosted on github, which (IIRC) can be read
without an account and without allowing javascript.

	https://github.com/jwiegley/use-package <https://github.com/jwiegley/use-package>

I recall that some people in this thread use use-package, so perhaps
they can shoot down this idea or suggest ways to make it useful.

~Chad

[-- Attachment #2: Type: text/html, Size: 2244 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 709 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Sa., 18. Apr. 2015 um
19:17 Uhr:

> Maybe a solution is to simply make customize-set-variables lazier, so
> that variables with a :require see their setting delayed to after
> package-initialize was called.  Or else, have package-initialize be
> called by customize-set-variables.  Or...
>
>
I would prefer one of these options over the other discussed in this
thread. package.el and custom.el are now both part of Emacs, so there's no
break in abstraction if they know about each other. It would make custom.el
more complicated for the benefit of the users (who can stop worrying about
the correct initialization order), which I think is a good tradeoff.

[-- Attachment #2: Type: text/html, Size: 988 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Apr 19, 2015 7:40 AM, "Philipp Stephani" <p.stephani2@gmail.com> wrote:
>
>
>
> Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Sa., 18. Apr. 2015
um 19:17 Uhr:
>>
>> Maybe a solution is to simply make customize-set-variables lazier, so
>> that variables with a :require see their setting delayed to after
>> package-initialize was called.  Or else, have package-initialize be
>> called by customize-set-variables.  Or...
>>
>
> I would prefer one of these options over the other discussed in this
thread. package.el and custom.el are now both part of Emacs, so there's no
break in abstraction if they know about each other. It would make custom.el
more complicated for the benefit of the users (who can stop worrying about
the correct initialization order), which I think is a good tradeoff.

As I keep saying, this won't solve the issue where the user can't `require'
packages in his init file until package initialize has been manually
called. That's what prompted this whole thread and it's explained in the
first email.

[-- Attachment #2: Type: text/html, Size: 1328 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> As I keep saying, this won't solve the issue where the user can't `require'
> packages in his init file until package initialize has been manually
> called.  That's what prompted this whole thread and it's explained in the
> first email.

Indeed, for this use-case, adding "(package-initialize)" explicitly in
the ~/.emacs seems like the best solution.

Of course, as mentioned, having `require' in one's ~/.emacs should be rare.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Taylan Ulrich Bayırlı/Kammer]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> As I keep saying, this won't solve the issue where the user can't `require'
>> packages in his init file until package initialize has been manually
>> called.  That's what prompted this whole thread and it's explained in the
>> first email.
>
> Indeed, for this use-case, adding "(package-initialize)" explicitly in
> the ~/.emacs seems like the best solution.

I thought the point of this discussion was to find a solution that
doesn't require that?

> Of course, as mentioned, having `require' in one's ~/.emacs should be
> rare.

I think many people use `require' in their .emacs to load libraries,
though in this case it doesn't matter whether they use `require' or
`load-library', or try to call a function or access a variable or
anything else from the package.

A user is likely to expect everything defined by a package to be
available during the execution of their init.el; the failure to meet
this expectation is why we get so many confused users asking for help,
receiving the answer "just put (package-initialize) at the top of your
.emacs".


If nobody sees any disadvantages, and nobody beats me to it, I might
start working on implementing the solution that separates pre-package
configuration from normal configuration, including Customize.

Taylan

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> I thought the point of this discussion was to find a solution that
> doesn't require that?

Right, but in the use-case presented by Artur, I can't see how to do better.

>> Of course, as mentioned, having `require' in one's ~/.emacs should be
>> rare.
> I think many people use `require' in their .emacs to load libraries,
> though in this case it doesn't matter whether they use `require' or
> `load-library', or try to call a function or access a variable or
> anything else from the package.

Oh, right, of course, the problem is not limited to `require' or `load'
(both of which *should* be rare), but also affect the case where the
user calls a function from that library (typically a global minor mode)
or needs to tweak a variable's value (not just `setq' it, but actually
take the previous value and change it).

> A user is likely to expect everything defined by a package to be
> available during the execution of their init.el; the failure to meet
> this expectation is why we get so many confused users asking for help,
> receiving the answer "just put (package-initialize) at the top of your
> .emacs".

But the current code in Emacs's "master" solves this problem by adding
"(package-initialize)" in the user's ~/.emacs.

> If nobody sees any disadvantages, and nobody beats me to it, I might
> start working on implementing the solution that separates pre-package
> configuration from normal configuration, including Customize.

I don't really know what solution you're referring to.  But so far
I haven't seen any solution proposed that's really better than
what we already have.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Taylan Ulrich Bayırlı/Kammer]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> A user is likely to expect everything defined by a package to be
>> available during the execution of their init.el; the failure to meet
>> this expectation is why we get so many confused users asking for help,
>> receiving the answer "just put (package-initialize) at the top of your
>> .emacs".
>
> But the current code in Emacs's "master" solves this problem by adding
> "(package-initialize)" in the user's ~/.emacs.

I had not realized this is already implemented.

I generally feel uncomfortable about a tool automatically editing a file
that I assume to have control over, but maybe that's fine.

A concrete problem I can think of is when one has some settings for
package.el in their init file, then Emacs ends up putting
(package-initialize) at the top, when it should be after the package
related settings.  Is this not a problem?

Other than that, it simply feels wrong to "solve" the problem with a
trick like this as if initializing packages after init.el is the normal
case, when the normal case is that init.el assumes presence of installed
packages.  That is, behavior is broken by default, but gets auto-fixed
with a trick like this.  WTF?  It might be simple to implement, but
conceptually it's rather confusing.

One might argue it doesn't get conceptually any simpler because there's
simply one init.el and it's agnostic towards package.el, but that's both
wrong (packages are automatically initialized after loading init.el if
it hasn't been done there) and a sign of the package system being
crudely bolted on top of existing machinery.  That's stinky engineering.

I hope I'm not coming off as religious.

>> If nobody sees any disadvantages, and nobody beats me to it, I might
>> start working on implementing the solution that separates pre-package
>> configuration from normal configuration, including Customize.
>
> I don't really know what solution you're referring to.  But so far
> I haven't seen any solution proposed that's really better than
> what we already have.

Splitting pre-package-initialization configuration from normal
configuration.  (Here "configuration" refers to the user's init file as
well as Customize.)  Making Emacs init look like:

    1. pre-package-init.el, & Customize settings in it (or loading
       `package-custom-file')
       
    2. package-initialize
    
    3. init.el, & Customize settings in it (or loading `custom-file')

This entails extending Customize functionality to offer something like a
:pre-package-init flag which tells the system to write customization for
the relevant defcustom into `package-custom-file' (falling back to
pre-package-init.el) instead of `custom-file' (falling back to init.el).

Yes, this is a more involved solution, but it brings some structure into
Emacs initialization with regard to the role of the package system, and
its interaction with Customize.


By the way, I just noticed our support for default.el and site-start.el.
I guess said separation of pre- and post-package-init configuration
would be applied to those as well if we were being idealistic, though I
think it should be fine to leave that be and only load default.el and
site-start.el after package initialization.  Or can anyone think of
valid use-cases for a site admin to hook in some Elisp before users'
package initialization?

Taylan

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 4507 bytes --]

On Apr 20, 2015 4:31 PM, "Taylan Ulrich Bayırlı/Kammer" <
taylanbayirli@gmail.com> wrote:
>
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
> >> A user is likely to expect everything defined by a package to be
> >> available during the execution of their init.el; the failure to meet
> >> this expectation is why we get so many confused users asking for help,
> >> receiving the answer "just put (package-initialize) at the top of your
> >> .emacs".
> >
> > But the current code in Emacs's "master" solves this problem by adding
> > "(package-initialize)" in the user's ~/.emacs.
>
> I had not realized this is already implemented.
>
> I generally feel uncomfortable about a tool automatically editing a file
> that I assume to have control over, but maybe that's fine.

You're not the only one. Other people have voiced indignation.

> A concrete problem I can think of is when one has some settings for
> package.el in their init file, then Emacs ends up putting
> (package-initialize) at the top, when it should be after the package
> related settings.  Is this not a problem?

Slightly, yes. A person who does that sort of thing knows enough to
understand they only need to comment out the added call to
package-initialize. It would be preferable to not impose this on them
though (which is why the whole discussion has been revived).

> Other than that, it simply feels wrong to "solve" the problem with a
> trick like this as if initializing packages after init.el is the normal
> case, when the normal case is that init.el assumes presence of installed
> packages.  That is, behavior is broken by default, but gets auto-fixed
> with a trick like this.  WTF?  It might be simple to implement, but
> conceptually it's rather confusing.
>
> One might argue it doesn't get conceptually any simpler because there's
> simply one init.el and it's agnostic towards package.el, but that's both
> wrong (packages are automatically initialized after loading init.el if
> it hasn't been done there) and a sign of the package system being
> crudely bolted on top of existing machinery.  That's stinky engineering.

I'm not against a better solution.

> I hope I'm not coming off as religious.
>
> >> If nobody sees any disadvantages, and nobody beats me to it, I might
> >> start working on implementing the solution that separates pre-package
> >> configuration from normal configuration, including Customize.
> >
> > I don't really know what solution you're referring to.  But so far
> > I haven't seen any solution proposed that's really better than
> > what we already have.
>
> Splitting pre-package-initialization configuration from normal
> configuration.  (Here "configuration" refers to the user's init file as
> well as Customize.)  Making Emacs init look like:
>
>     1. pre-package-init.el, & Customize settings in it (or loading
>        `package-custom-file')
>
>     2. package-initialize
>
>     3. init.el, & Customize settings in it (or loading `custom-file')
>
> This entails extending Customize functionality to offer something like a
> :pre-package-init flag which tells the system to write customization for
> the relevant defcustom into `package-custom-file' (falling back to
> pre-package-init.el) instead of `custom-file' (falling back to init.el).

I'm in favor of this. I would suggest you send a new email as a separate
thread summarising what will be done, before you go through all the work
(though that's up to you). Something similar to what you've just written
here, with just a little bit more detail (like saying where the changes
will be made) and written with a clear statement of the advantages
(reconciling package.el, user code, and custom).

> By the way, I just noticed our support for default.el and site-start.el.
> I guess said separation of pre- and post-package-init configuration
> would be applied to those as well if we were being idealistic, though I
> think it should be fine to leave that be and only load default.el and
> site-start.el after package initialization.  Or can anyone think of
> valid use-cases for a site admin to hook in some Elisp before users'
> package initialization?

I don't think separation needs to be applied to them. If they're currently
run before package-initialize, just leave it like that. I don't know why a
maintainer would want to run code after package-initialize, but if they
want to do that they can use an after init hook.

[-- Attachment #2: Type: text/html, Size: 5464 bytes --]

Re: Calling (package-initialize) sooner during initialization

[chad]

> On 20 Apr 2015, at 09:01, Artur Malabarba <bruce.connor.am@gmail.com> wrote:
> 
> On Apr 20, 2015 4:31 PM, "Taylan Ulrich Bayırlı/Kammer" <taylanbayirli@gmail.com> wrote:
>>  I generally feel uncomfortable about a tool automatically editing a file
>>  that I assume to have control over, but maybe that's fine.
>> 
> You're not the only one. Other people have voiced indignation.

I saw this also, and Ive seen it before in other contexts, but I
dont quite understand it here, since customize already edits
user-init-file (to add custom-set-{variables,faces}).

I do wonder if all of this can be avoided by teaching users to
insert `use-package lines/snippets/etc into their .emacs, instead
of `require. It would also address some slow-start issues at the
same time.

~Chad

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

> I do wonder if all of this can be avoided by teaching users to
> insert `use-package lines/snippets/etc into their .emacs, instead
> of `require. It would also address some slow-start issues at the
> same time.

I started this thread because I was tired of telling people to add
package-initialize to their init file, and you want to teach them to use
use-package? :-)

The point is that users shouldn't run into trouble for doing such basic
configurations. If they have to ask for help on this then I we've already
failed.

[-- Attachment #2: Type: text/html, Size: 609 bytes --]

Re: Calling (package-initialize) sooner during initialization

[chad]

[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]


> On 20 Apr 2015, at 11:38, Artur Malabarba <bruce.connor.am@gmail.com> wrote:
> 
> > I do wonder if all of this can be avoided by teaching users to
> > insert `use-package lines/snippets/etc into their .emacs, instead
> > of `require. It would also address some slow-start issues at the
> > same time.
> 
> I started this thread because I was tired of telling people to add package-initialize to their init file, and you want to teach them to use use-package? :-)
> 
Thats fair, but (in case its not clear), Im suggesting that we try
to move everyone away from ever saying Add this line to .emacs:
(require, and *instead* get them to say the very similar Add this
line to .emacs: (use-package.

The hope (unsupported by actual evidence, of course) is that its a
simple enough change that its not as difficult as telling people
how to find the right point in their init file to put package-initialize
w.r.t. the various `require lines, which move around. As an additional
benefit, emacs will start faster, and the users init file will
naturally tell them where to customize that package. (In a potential
future, we could even have customize or a replacement add the options
to the use-package statement directly - but thats down the road).

> The point is that users shouldn't run into trouble for doing such basic configurations. If they have to ask for help on this then I we've already failed.
> 

Agreed. My hope was to get people to suggest an equally-simple but
better alternative to `require in .emacs (which is already something
that wed like to avoid).

Hope that helps. Sorry to add to the noise if not. Thanks!
~Chad




[-- Attachment #2: Type: text/html, Size: 2667 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> I generally feel uncomfortable about a tool automatically editing a file
> that I assume to have control over, but maybe that's fine.

We all do.  Yet that's also what custom.el does.

> A concrete problem I can think of is when one has some settings for
> package.el in their init file, then Emacs ends up putting
> (package-initialize) at the top, when it should be after the package
> related settings.  Is this not a problem?

Oh yes.  And there can be all kinds of other interesting cases.

> with a trick like this.  WTF?  It might be simple to implement, but
> conceptually it's rather confusing.

Well, conceptually it's simple as well, if you look at it from the
implementation point of view ;-)

> wrong (packages are automatically initialized after loading init.el if
> it hasn't been done there)

Actually, with the new code that auto-adds "(package-initialize)" to
~/.emacs, we should be able to get rid of this hack that calls
package-initialize if the .emacs didn't do so already (tho we'll need
to wait a little, for the usual compatibility reasons).

> and a sign of the package system being crudely bolted on top of
> existing machinery.  That's stinky engineering.

Welcome to the world of long-lived programs that need to live with
backward compatibility.

BTW, having a separate pre-package init file would also be stinky
engineering imposed by hysterical raisins.  Good engineering would
probably require re-thinking the whole customize-set-variables issue and
maybe also the semantics of defcustom.  I'd be happy if someone were to
try and do the work of figuring out what a clean design would look like,
so we'd at least be able to assess whether we can hope to get there from
here without too much pain.

> I hope I'm not coming off as religious.

This is Emacs we're talking about.

> Yes, this is a more involved solution, but it brings some structure into
> Emacs initialization with regard to the role of the package system, and
> its interaction with Customize.

Right, and that's exactly what I don't want: I don't want to impose an
arbitrary structure that happens to suit the current needs.

Tricky dependencies between setting vars and loading their packages can
also exist within customize-set-variables without the existence of package.el.

> think it should be fine to leave that be and only load default.el and
> site-start.el after package initialization.  Or can anyone think of

The difference between site-start.el and default.el is that one is
loaded before ~/.emacs and the other after.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Mark Oteiza]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I thought the point of this discussion was to find a solution that
>> doesn't require that?
>
> Right, but in the use-case presented by Artur, I can't see how to do better.

How about documenting the use-case in the Emacs manual instead?

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> But the current code in Emacs's "master" solves this problem by adding
> "(package-initialize)" in the user's ~/.emacs.

This is a very bad idea, I don't want my init file modified.
Only the user should modify this file.
Why don't you write to the custom-file (and make the use of custom-file
the default instead of writing custom settings in .emacs by default) ?

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Taylan Ulrich Bayırlı/Kammer]

Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:

> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
>> But the current code in Emacs's "master" solves this problem by adding
>> "(package-initialize)" in the user's ~/.emacs.
>
> This is a very bad idea, I don't want my init file modified.
> Only the user should modify this file.
> Why don't you write to the custom-file (and make the use of custom-file
> the default instead of writing custom settings in .emacs by default) ?

The intent is to solve the problem for users who are confused on why the
function foobar, used in their .emacs, is undefined, when they installed
the foobar package.  The solution is to load packages before loading
.emacs (or at the start of it).  If a user is advanced enough to use a
separate `custom-file' then they probably don't face the problem in
first place, and otherwise Customize writes to the end of .emacs, not
the start.

Elsewhere in this thread it has been discussed in some detail how the
problem can be solved in an arguably clean way; I would like to work on
implementing that solution but have little time these days.  (And I will
likely have to do the copyright assignment stuff too.)  If someone else
feels like doing it, please go ahead.  (But maybe first sum up once
more, in a new thread, in detail, what exactly you will do, and get
approval before starting.)

Taylan

RE: Calling (package-initialize) sooner during initialization

[Drew Adams]

> > make the use of custom-file the default instead of writing
> > custom settings in .emacs by default) ?

+1.  A recurring suggestion.

> If a user is advanced enough to use a separate `custom-file'
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> then they probably don't face the problem in first place,

Huh?  Who told you that using `custom-file' requires being an
"advanced" user?  That's silly.

The fact that use of `custom-file' is not the default means,
in practice, that many, including most novice, users don't know
about it.  That it is the only sense in which someone needs to
be "advanced" to use `custom-file': it takes you a while to
learn about it.

Newbies are among those who benefit most from using
`custom-file'. They, most of all, deserve a separate space for
their hand-written init-file code, protected from stuff that
Customize writes and manages.

It makes little sense to mix hand-coding and automatic coding
in the same file.  For everyone, including newbies.

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

2015-04-27 10:52 GMT+01:00 Thierry Volpiatto <thierry.volpiatto@gmail.com>:
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
>> But the current code in Emacs's "master" solves this problem by adding
>> "(package-initialize)" in the user's ~/.emacs.
>
> This is a very bad idea, I don't want my init file modified.
> Only the user should modify this file.

It's not a great idea, but the previous situation was unnacceptable,
so something was done about it. We can certainly improve on it.

> Why don't you write to the custom-file (and make the use of custom-file
> the default instead of writing custom settings in .emacs by default) ?

Because the custom-file might not be loaded at the top of the init
file. And package-initialize needs to be placed at the very top
(otherwise it's not solving anything).
Of course the user can then go ahead and move package-initialize
anywhere else that he wants (or even comment it out). But the first
time it is inserted needs to be at the very top, so that we solve the
problem for those users who don't know any better.

RE: Calling (package-initialize) sooner during initialization

[Drew Adams]

> > Why don't you write to the custom-file (and make the use of
> > custom-file the default instead of writing custom settings
> > in .emacs bydefault) ?
> 
> Because the custom-file might not be loaded at the top of the
> init file.

It should not be presumed to be loaded at the top or bottom or
at any other particular place in the init file.  Its definitions
can interact with other code in the init file, including loading
of packages in a specific order.

> And package-initialize needs to be placed at the very top
> (otherwise it's not solving anything).

Don't stick it in the init file or the `custom-file'.  Find
another solution for it.

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

>> And package-initialize needs to be placed at the very top
>> (otherwise it's not solving anything).
>
> Don't stick it in the init file or the `custom-file'.  Find
> another solution for it.

Yes, that's what we're currently discussing in this thread.

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-04-27 10:52 GMT+01:00 Thierry Volpiatto <thierry.volpiatto@gmail.com>:
>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>>
>>> But the current code in Emacs's "master" solves this problem by adding
>>> "(package-initialize)" in the user's ~/.emacs.
>>
>> This is a very bad idea, I don't want my init file modified.
>> Only the user should modify this file.
>
> It's not a great idea, but the previous situation was unnacceptable,
> so something was done about it. We can certainly improve on it.
>
>> Why don't you write to the custom-file (and make the use of custom-file
>> the default instead of writing custom settings in .emacs by default) ?
>
> Because the custom-file might not be loaded at the top of the init
> file.

Why ?
If the default is using always the custom-file, this file could be
loaded by default before the init file.
And IMO it should be the default, writing custom setting at end of init
file is the first thing one want to prevent when writing its emacs init
file.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> Why don't you write to the custom-file (and make the use of custom-file
> the default instead of writing custom settings in .emacs by default) ?

When should the custom-file be loaded?  Before .emacs is too early.
After .emacs is too late.

Déjà-vu?


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Andy Moreton]

On Mon 27 Apr 2015, Stefan Monnier wrote:

>> Why don't you write to the custom-file (and make the use of custom-file
>> the default instead of writing custom settings in .emacs by default) ?
>
> When should the custom-file be loaded?  Before .emacs is too early.
> After .emacs is too late.

Never - because customise is horrible :-)

I agree that emacs should not modify the user init file: it was
misguided to allow customise to do so, and customise should always have
used a separate file.

The code in package--ensure-init-file to add (package-initialize) will
cause breakage if the user had previously specified lexical binding (or
other file locals) in the first line of user-init-file.

    AndyM

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> The code in package--ensure-init-file to add (package-initialize) will
> cause breakage if the user had previously specified lexical binding (or
> other file locals) in the first line of user-init-file.

Good point, that needs to be fixed.  More specifically, I think we
should insert that lines after any leading space & comments.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

>> The code in package--ensure-init-file to add (package-initialize) will
>> cause breakage if the user had previously specified lexical binding (or
>> other file locals) in the first line of user-init-file.
>
> Good point, that needs to be fixed.  More specifically, I think we
> should insert that lines after any leading space & comments.

Done.

Re: Calling (package-initialize) sooner during initialization

[Oleh Krehel]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Mon 27 Apr 2015, Stefan Monnier wrote:
>
>>> Why don't you write to the custom-file (and make the use of custom-file
>>> the default instead of writing custom settings in .emacs by default) ?
>>
>> When should the custom-file be loaded?  Before .emacs is too early.
>> After .emacs is too late.
>
> Never - because customise is horrible :-)
>
> I agree that emacs should not modify the user init file: it was
> misguided to allow customise to do so, and customise should always have
> used a separate file.

I super-agree on this point. I even switched to using this macro, just
to prevent customize from writing stuff to my .emacs (which normally is
a one-liner):

(defmacro csetq (variable value)
  `(funcall (or (get ',variable 'custom-set) 'set-default) ',variable ,value))
(csetq tool-bar-mode nil)
(csetq menu-bar-mode nil)
...

It works out pretty well, basically Customize isn't aware that I've
modified anything. So when something does get saved (I think recently
package.el saved something like a list of installed packages), I see
only that one thing, instead of a horrible blob of 100 variables. That's
a lot more manageable.

Oleh

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

> I super-agree on this point. I even switched to using this macro, just
> to prevent customize from writing stuff to my .emacs (which normally is
> a one-liner):
>
> (defmacro csetq (variable value)
>   `(funcall (or (get ',variable 'custom-set) 'set-default) ',variable
,value))
> (csetq tool-bar-mode nil)
> (csetq menu-bar-mode nil)
> ...
>
> It works out pretty well, basically Customize isn't aware that I've
> modified anything. So when something does get saved (I think recently
> package.el saved something like a list of installed packages), I see
> only that one thing, instead of a horrible blob of 100 variables. That's
> a lot more manageable.

I'm not aware of any situation where a plain setq would cause customize to
save something to your init file. Could you give an example where that
macro helped you?

[-- Attachment #2: Type: text/html, Size: 1007 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Oleh Krehel]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

>> I super-agree on this point. I even switched to using this macro,
> just
>> to prevent customize from writing stuff to my .emacs (which normally
> is
>> a one-liner):
>>
>> (defmacro csetq (variable value)
>> `(funcall (or (get ',variable 'custom-set) 'set-default) ',variable ,
> value))
>> (csetq tool-bar-mode nil)
>> (csetq menu-bar-mode nil)
>> ...
>>
>> It works out pretty well, basically Customize isn't aware that I've
>> modified anything. So when something does get saved (I think
> recently
>> package.el saved something like a list of installed packages), I see
>> only that one thing, instead of a horrible blob of 100 variables.
> That's
>> a lot more manageable.
>
> I'm not aware of any situation where a plain setq would cause
> customize to save something to your init file. Could you give an
> example where that macro helped you?

A plain setq doesn't get in the way, but it also doesn't call the :set
property of defcustom. For instance, this works fine:

    (csetq ediff-diff-options "-w")

but this doesn't work:

    (setq ediff-diff-options "-w")

Oleh

Re: Calling (package-initialize) sooner during initialization

[Wolfgang Jenkner]

On Tue, Apr 28 2015, Oleh Krehel wrote:

>>> (defmacro csetq (variable value)
>>> `(funcall (or (get ',variable 'custom-set) 'set-default) ',variable ,
>> value))
[...]
> A plain setq doesn't get in the way, but it also doesn't call the :set
> property of defcustom. For instance, this works fine:
>
>     (csetq ediff-diff-options "-w")
>
> but this doesn't work:
>
>     (setq ediff-diff-options "-w")

This example also shows that the variable's docstring doesn't always
mention that a custom setter is used :-(

So, I wonder if `describe-variable' shouldn't add this automatically,
like in the following (barely tested) patch:

-- >8 --
Subject: [PATCH] * lisp/help-fns.el (describe-variable): Mention a custom
 setter.

---
 lisp/help-fns.el | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index 7ecd271..3eb9271 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -910,6 +910,16 @@ file-local variable.\n")
 
 	    ;; Make a link to customize if this variable can be customized.
 	    (when (custom-variable-p variable)
+              (let ((misfeature (get variable 'custom-set)))
+                (when misfeature
+                  (terpri)
+                  (terpri)
+                  (princ "This variable's `custom-set' property has the value")
+                  (if (symbolp misfeature)
+                      (with-current-buffer standard-output
+                        (insert (format " `%s'." misfeature)))
+                    (terpri)
+                    (pp misfeature))))
 	      (let ((customize-label "customize"))
 		(terpri)
 		(terpri)
-- 
2.3.6

RE: Calling (package-initialize) sooner during initialization

[Drew Adams]

> This example also shows that the variable's docstring doesn't always
> mention that a custom setter is used :-( So, I wonder if
> `describe-variable' shouldn't add this automatically

Doesn't sound like a great idea, to me.  (Perhaps it was a joke?)
You propose to pretty-print the `custom-set' value, which is a
lambda form or a symbol, systematically for `C-h v'.

There is other useful information in a `defcustom', besides `:set'.
It makes more sense to make all of it available from the Customize
buffer than it does to show only some of it (`:set') for `C-h v'.

For one thing, that information is already available from `C-h v',
by clicking the link for the file where the option is defined.
That shows you not only the custom setter but the `:type' info etc.
- the complete `defcustom'.

However, when the user follows the `customize' link from `C-h v'
to the Customize buffer, it might be helpful if that buffer too
had a link to the defining source-file `defcustom', when available.
Users can always use `C-h v' from the Customize buffer, but turning
the option name into a link to the defining `defcustom' would be
helpful.

Currently if you try to use `C-h v' with point on the variable name
in Customize, there is no default value proposed, because of the
colon that follows it.  Better to make the name a link, so that `RET'
or a mouse click takes you to the definition.  (The name and colon
are already highlighted blue, which is close to looking like a link;
better that it should really be a useful link.) 

Re: Calling (package-initialize) sooner during initialization

[Wolfgang Jenkner]

On Tue, Apr 28 2015, Drew Adams wrote:

> Doesn't sound like a great idea, to me.  (Perhaps it was a joke?)

Not at all.

The purpose is to clearly indicate that setting or binding the variable
in the straightforward (and normal, if you will) way won't work.

> You propose to pretty-print the `custom-set' value, which is a
> lambda form or a symbol, systematically for `C-h v'.

Yes (if it's non-nil, of course).

> For one thing, that information is already available from `C-h v',
> by clicking the link for the file where the option is defined.
> That shows you not only the custom setter but the `:type' info etc.
> - the complete `defcustom'.

The docstring is supposed to (informally) indicate which values make
sense.  So, usually, it's not necessary to follow the link to the source
for this.

Wolfgang

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

2015-04-28 18:29 GMT+01:00 Wolfgang Jenkner <wjenkner@inode.at>:
> On Tue, Apr 28 2015, Drew Adams wrote:
>
>> Doesn't sound like a great idea, to me.  (Perhaps it was a joke?)
>
> Not at all.
>
> The purpose is to clearly indicate that setting or binding the variable
> in the straightforward (and normal, if you will) way won't work.

I wouldn't oppose having `describe-variable' add a short sentence in
the same place where it says things like "becomes buffer local" or "is
risky". The sentence could be as short as:

     this variable has a `custom-set' property;

perhaps followed by something else such as

     so setting it directly might not have any effect

But I don't think printing the entire custom-set property is desirable.

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> I wouldn't oppose having `describe-variable' add a short sentence in
> the same place where it says things like "becomes buffer local" or "is
> risky". The sentence could be as short as:
>
>      this variable has a `custom-set' property;
>
> perhaps followed by something else such as
>
>      so setting it directly might not have any effect
>
> But I don't think printing the entire custom-set property is desirable.

Agreed,


        Stefan

Re: Calling (package-initialize) sooner during initialization

[raman]

So would it make sense to include your csetq macro in Emacs, and update
custom to generate (csetq ...) lines -- rather than the single
s-expression that sets all value s-- it's that latter that makes custom
so painful to manage.
-- 

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> A plain setq doesn't get in the way, but it also doesn't call the :set
> property of defcustom. For instance, this works fine:
>
>     (csetq ediff-diff-options "-w")
>
> but this doesn't work:
>
>     (setq ediff-diff-options "-w")

What is the problem with customize-set-variable ?
(customize-set-variable 'ediff-diff-options "-w")

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[raman]

This is a useful trick. I've set the value of custom-file early in my
.emacs so custom writes its bits somewhere else.  I'm very uncomfortable
with the way custom works -- in the past couple of years, I've had to
restore  the .custom file from git because something in emacs corrupted
the file. I've never been able to track down the root cause -- so I find
anything that goes off to touch custom settings early on fairly disconcerting.
-- 

RE: Calling (package-initialize) sooner during initialization

[Drew Adams]

> > Why don't you write to the custom-file (and make the use of
> > custom-file the default instead of writing custom settings
> > in .emacs by default) ?
> 
> When should the custom-file be loaded?  Before .emacs is too
> early.  After .emacs is too late.

It should *not* be loaded automatically.  It should be loaded
by the user, explicitly (from the init file, typically).

Unless the init file is absent or empty.  In that case, it
could be added to a virgin init file.  Emacs could also provide
a command that creates such a starter init-file, which contains
only a sexp to load `custom-file'.

A bigger question is how `custom-file' gets customized.  Using
Customize for that is problematic, for obvious reasons.

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Why don't you write to the custom-file (and make the use of custom-file
>> the default instead of writing custom settings in .emacs by default) ?
>
> When should the custom-file be loaded?  Before .emacs is too early.

Why?

What is the difference with: 

- (load custom-file) + (load user-init-file)
- loading init file with (load custom-file) on top of it.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

> Why ?
> If the default is using always the custom-file, this file could be
> loaded by default before the init file.
> And IMO it should be the default, writing custom setting at end of init
> file is the first thing one want to prevent when writing its emacs init
> file.

I see. So you're talking about loading custom-file automatically,
before loading the init file.
You can't just do that, because that will cause failure for some init
files that currently work fine.

Some users out there may have some code in his init file that needs to
be run before the custom variables are loaded.

2015-04-27 15:46 GMT+01:00 Thierry Volpiatto <thierry.volpiatto@gmail.com>:
>
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
>>> Why don't you write to the custom-file (and make the use of custom-file
>>> the default instead of writing custom settings in .emacs by default) ?
>>
>> When should the custom-file be loaded?  Before .emacs is too early.
>
> Why?
>
> What is the difference with:
>
> - (load custom-file) + (load user-init-file)
> - loading init file with (load custom-file) on top of it.

None. Stefan's point was that we can't force the user to load his
custom file at the top of his init file (nor can we load it before the
init file), because there are situations where the user needs to run
some code before the custom variables are loaded.

Besides, other options have been suggested around here, none of which
break the system for users with special custom-file needs, so I'd
suggest you look into them. One is about adding the directories under
emacs.d/elpa to the load-path before loading the init file (without
calling full-on package-initialize). The other is about dividing
startup between pre-initialize and post-initialize, it is more
heavyweight but it has the advantage of also fixing some other issues
with custom.el.

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> What is the difference with: 
> - (load custom-file) + (load user-init-file)
> - loading init file with (load custom-file) on top of it.

The first case will crash-and-burn if some of the settings in
custom-file only work after some of the code in the user's
user-init-file.

The second case is completely different, because the user has control
over where the (load custom-file) is placed, so if it crashes-and-burns
when placed at the very top, he can move it elsewhere.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Stephen J. Turnbull]

Stefan Monnier writes:

 > > What is the difference with: 
 > > - (load custom-file) + (load user-init-file)
 > > - loading init file with (load custom-file) on top of it.
 > 
 > The first case will crash-and-burn if some of the settings in
 > custom-file only work after some of the code in the user's
 > user-init-file.

XEmacs did this for years (when XEmacs was very popular), and there
were no such problems.  I suppose settings in custom-file rarely
depend on init settings in "crash and burn" way.  The real issue is
supporting the occasional user who wants fine control over the
custom-loading process.  See my reply to Thierry for a use case.

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> What is the difference with: 
>> - (load custom-file) + (load user-init-file)
>> - loading init file with (load custom-file) on top of it.
>
> The first case will crash-and-burn if some of the settings in
> custom-file only work after some of the code in the user's
> user-init-file.

Hmm, I am not sure of this, I don't see yet what kind of code have to be
run before setting variables from custom-file. 

> The second case is completely different, because the user has control
> over where the (load custom-file) is placed, so if it crashes-and-burns
> when placed at the very top, he can move it elsewhere.

User should not have do take care of this IMHO.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> Hmm, I am not sure of this, I don't see yet what kind of code have to be
> run before setting variables from custom-file.

E.g. any variable which comes from a non-bundled package and which has
a :require needs some code to setup load-path beforehand.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Stephen J. Turnbull]

Thierry Volpiatto writes:

 > What is the difference with: 
 > 
 > - (load custom-file) + (load user-init-file)
 > - loading init file with (load custom-file) on top of it.

I don't recall the exact use case for loading custom-file late, I
think it was something about using different custom-files on different
hosts, but they resided on the same network file system.

So the big difference is that if you load before user-init-file, you
can't affect what custom-file does in user-init-file.  Whereas if you
(by default) load late, you can load when you like in user-init-file,
and then inhibit the automatic late loading.

Such "manual loading" is indeed "advanced" usage in current XEmacs, as
it requires knowing exactly how the initialization process works and
some of the internals of customize.  But the recipe is short.

XEmacs has never regretted using custom-file for customizations by
default, although the automatic migration process was fraught (since
it occurs during initialization, errors in the init file could cause
loss of custom code).

I do disagree with those who say "don't load custom-file by default".
If you think customize sucks, just don't use it, or (in the case of
late automatic loading) inhibit loading.  Very few XEmacs users ever
cared about the details of loading.

Few have reported issues due to late loading, but that is probably due
to the fact that XEmacs has a protocol ("auto-autoloads.el" and
"custom-loads.el") for early initialization of packages.  Packages we
distribute are strictly vetted for "proper" use of autoload cookies.
Specifically, only autoloads of package entry point functions and
setting of package flags with properly prefixed names are allowed.
This slows initialization in the current implementation (since the
autoloads and customization files are per-package), but could be sped
up by caching.

Re: Calling (package-initialize) sooner during initialization

[Jorgen Schäfer]

Hello.
I think a major part of the problem here is that package-initialize does
a lot more things than it needs to. All that has to happen before the
user init file is loaded is to load the autoload files from ~/.emacs.d/elpa/.
Everything else can happen on demand or after the init file is loaded,
because it is not necessary to make packages available.

If package.el provided some sort of (package-load-directory DIR), and
ran that on ~/.emacs.d/elpa/, that should fix most use cases already:

> - package-enable-at-startup: Would be equivalent to the proposed
>   environment variable.

That's usually set when the user runs package-initialize manually, but
if the user does not want to enable packages at startup, they can simply
not have anything in ~/.emacs.d/elpa/.

> - package-load-list: Needs to be configured before.

If the user does not want to enable certain packages, they can move
them out of ~/.emacs.d/elpa/. If necessary, they can move the packages
into, say, ~/.emacs.d/held/ and run
(package-load-directory "~/.emacs.d/held") after configuring this variable.

> - package-archives: Needs to be configured before for
>   non-interactive package installation.
> - package-pinned-packages: Needs to be configured before for
>   non-interactive package installation.
> - package-check-signature and package-unsigned-archives: Can be
>   customized afterwards
> - package-directory-list: Needs to be configured before.

These are all only relevant for package installation, not for package
initialization. The user can configure them before installing packages in
their init file, if desired.

> - package-user-dir: Needs to be configured before.

In the new approach, changing this only means the new directory is not
loaded before the user init file is loaded. The user can simply do this
manually using (package-load-directory NEW-DIR) when they need the
new packages, which more or less replicates the current setup of requiring
a package-initialize before packages become available, but arguable more
obvious as the user has to set the variable in their init file already.


That is, Emacs just needs to adjust the load-path and load auto-loads
for the default location before the user init file is run to solve the original
problem of user confusion. None of the config options are necessary for this,
so the race condition that caused the original desire of not running
package-initialize before the user init file is gone. The user can set the
config options as necessary in their init file while simply using packages as
expected.

Regards,
Jorgen

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

>> - package-load-list: Needs to be configured before.
>
> If the user does not want to enable certain packages, they can move
> them out of ~/.emacs.d/elpa/. If necessary, they can move the packages
> into, say, ~/.emacs.d/held/ and run
> (package-load-directory "~/.emacs.d/held") after configuring this variable.

This sounds very inconvenient for those who don't want to activate a
specific package at startup, but want to keep it up-to-date with
package.el. Not to mention the package won't be seen by package.el and
so won't be considered when dependency checking.

Re: Calling (package-initialize) sooner during initialization

[Jorgen Schäfer]

On Sun, Apr 12, 2015 at 4:26 PM, Artur Malabarba
<bruce.connor.am@gmail.com> wrote:
>>> - package-load-list: Needs to be configured before.
>>
>> If the user does not want to enable certain packages, they can move
>> them out of ~/.emacs.d/elpa/. If necessary, they can move the packages
>> into, say, ~/.emacs.d/held/ and run
>> (package-load-directory "~/.emacs.d/held") after configuring this variable.
>
> This sounds very inconvenient for those who don't want to activate a
> specific package at startup, but want to keep it up-to-date with
> package.el. Not to mention the package won't be seen by package.el and
> so won't be considered when dependency checking.

The trivial solution to this would be to set package-user-dir to
~/.emacs.d/packages/
in your user init file, configure package-load-list as before, and run
(package-load-directory package-user-dir). That's one line more (setting
package-user-dir) than it is now, and achieves the same result, so I would not
exactly call this "very inconvenient".

I have to admit that this seems like a pretty niche use case, though. Out of
curiosity, why would you not want the autoloads of an installed package be
loaded at startup? What workflow makes this useful?

Regards,
Jorgen

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> I have to admit that this seems like a pretty niche use case, though. Out of
> curiosity, why would you not want the autoloads of an installed package be
> loaded at startup? What workflow makes this useful?

A typical example is Auctex which by default overrides the default
tex-mode (tho it comes with a configuration variable you can set to
prevent this override, but there again: this config var needs to
be set before package-initialize).


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Aneesh Kumar K.V]

Jorgen Schäfer <jorgen.schaefer@gmail.com> writes:

> Hello.
> I think a major part of the problem here is that package-initialize does
> a lot more things than it needs to. All that has to happen before the
> user init file is loaded is to load the autoload files from ~/.emacs.d/elpa/.
> Everything else can happen on demand or after the init file is loaded,
> because it is not necessary to make packages available.
>
> If package.el provided some sort of (package-load-directory DIR), and
> ran that on ~/.emacs.d/elpa/, that should fix most use cases already:
>

I ran into an issue with using package.el along with use-package.el. I
had most of the packages installed via package.el . One of the main
benefit of using elpa is the easy upgrade of packages that elpa
offers. So it would be nice to have a facility that enables to add all
the installed packages to the load-path so that a (require 'use-package)
can work. This enables us to delay the package initialization of elpa
packages using use-package,which in-turn helps the emacs startup time.


-aneesh

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

> I ran into an issue with using package.el along with use-package.el. I
> had most of the packages installed via package.el . One of the main
> benefit of using elpa is the easy upgrade of packages that elpa
> offers. So it would be nice to have a facility that enables to add all
> the installed packages to the load-path

(dolist (it (directory-files package-user-dir 'full "^[^\\.].*-"))
  (when (file-directory-p it)
    (push it load-path)))

[-- Attachment #2: Type: text/html, Size: 561 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

>> - package-load-list: Needs to be configured before.
> If the user does not want to enable certain packages, they can move
> them out of ~/.emacs.d/elpa/.

Nope: package-directory-list can include system directories (with
system-wide installed packages), so the user may not have the option to
move them out of the directory.

>> - package-archives: Needs to be configured before for
>>   non-interactive package installation.
>> - package-pinned-packages: Needs to be configured before for
>>   non-interactive package installation.
>> - package-check-signature and package-unsigned-archives: Can be
>>   customized afterwards.
>> - package-directory-list: Needs to be configured before.
> These are all only relevant for package installation, not for package

True, except for package-directory-list which is very much used by
package-initialize.

>> - package-user-dir: Needs to be configured before.

This one is only used during installation, AFAIK.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Jorgen Schäfer]

On Sun, Apr 12, 2015 at 10:44 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>>> - package-load-list: Needs to be configured before.
>> If the user does not want to enable certain packages, they can move
>> them out of ~/.emacs.d/elpa/.
>
> Nope: package-directory-list can include system directories (with
> system-wide installed packages), so the user may not have the option to
> move them out of the directory.

My proposal does not talk about initializing anything in package-directory-list.

>>> - package-archives: Needs to be configured before for
>>>   non-interactive package installation.
>>> - package-pinned-packages: Needs to be configured before for
>>>   non-interactive package installation.
>>> - package-check-signature and package-unsigned-archives: Can be
>>>   customized afterwards.
>>> - package-directory-list: Needs to be configured before.
>> These are all only relevant for package installation, not for package
>
> True, except for package-directory-list which is very much used by
> package-initialize.

I was not talking about package-initialize.

To make myself a bit more clear, my proposal would mean adding these
lines of code to startup.el just before the user init file is loaded:

(dolist (dir (directory-files "~/.emacs.d/elpa" t "\\`[^.]"))
  (when (file-directory-p dir)
    (add-to-list 'load-path dir)
    (dolist (autoload (directory-files dir t "-autoloads.el\\'"))
      (load autoload nil t))))

This would solve the problem of users being confused because their
init file can not access packages they installed, allow all the heavy
package initialization to happen after the init file is processed, and
finally allow users to revert to the old behavior if they so choose by
setting package-user-dir to a different directory than ~/.emacs.d/elpa/.

Regards,
Jorgen

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> To make myself a bit more clear, my proposal would mean adding these
> lines of code to startup.el just before the user init file is loaded:

> (dolist (dir (directory-files "~/.emacs.d/elpa" t "\\`[^.]"))
>   (when (file-directory-p dir)
>     (add-to-list 'load-path dir)
>     (dolist (autoload (directory-files dir t "-autoloads.el\\'"))
>       (load autoload nil t))))

OK, so it does a minimalist initialization, and the main direct downside
is that users can't directly "disable" packages, they have to move them
out of ~/.emacs.d/elpa instead.  That's not a bad tradeoff, indeed.

One more issue is for things like AUCTeX's TeX-modes, which is usually
set before auctex-autoloads.el is loaded, but indeed this already needs
to handle the case where the var is modified later (which is handled via
a :set function), so it's OK.

There are a few other issues, tho.  For example, the current code first
looks at all the installed packages and then only allocates the latest
version of each (modulo pinned versions).  So you can have several
versions of a given package installed (in different directories or not)
and package-initialize will know which one to initialize.

If we load all of ~/.emacs.d/elpa this won't handle all the corresponding
cases anymore.  More specifically, we'll end up initializing a package
in ~/.emacs.d/elpa even if a more recent version might be installed in
some other directory that will be provided later.

The main use case would be when the other directory is a system
directory, which we could handle by letting sysadmins change the
site-start.el file so as to (package-load-descriptors DIR).  This way,
they're not yet initialized, but package.el knows about it so it could
refrain from initializing a less recent package from ~/.emacs.d/elpa.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1935 bytes --]

On Apr 13, 2015 12:22 AM, "Stefan Monnier" <monnier@iro.umontreal.ca> wrote:
>
> > To make myself a bit more clear, my proposal would mean adding these
> > lines of code to startup.el just before the user init file is loaded:
>
> > (dolist (dir (directory-files "~/.emacs.d/elpa" t "\\`[^.]"))
> >   (when (file-directory-p dir)
> >     (add-to-list 'load-path dir)
> >     (dolist (autoload (directory-files dir t "-autoloads.el\\'"))
> >       (load autoload nil t))))
>
> OK, so it does a minimalist initialization, and the main direct downside
> is that users can't directly "disable" packages, they have to move them
> out of ~/.emacs.d/elpa instead.  That's not a bad tradeoff, indeed.

I'm fine with this solution too (magnums or rackets), but I wouldn't mind
hearing the opinion of people who will be forced to do that.

> There are a few other issues, tho.  For example, the current code first
> looks at all the installed packages and then only allocates the latest
> version of each (modulo pinned versions).  So you can have several
> versions of a given package installed (in different directories or not)
> and package-initialize will know which one to initialize.
>
> If we load all of ~/.emacs.d/elpa this won't handle all the corresponding
> cases anymore.  More specifically, we'll end up initializing a package
> in ~/.emacs.d/elpa even if a more recent version might be installed in
> some other directory that will be provided later.
>
> The main use case would be when the other directory is a system
> directory,

IIUC, There's another situation where this will come up. If a mass package
upgrade fails for some reason, there will be lots of old package
directories hanging around in the elpa subdir.

So we can't just blindly load all those directories, we'll need to actually
parse the packages contained in all of them, version sort them into
something like package-alist, and load only the appropriate ones.

[-- Attachment #2: Type: text/html, Size: 2408 bytes --]

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 708 bytes --]

On Apr 12, 2015 10:02 PM, "Jorgen Schäfer"
> (dolist (dir (directory-files "~/.emacs.d/elpa" t "\\`[^.]"))
>   (when (file-directory-p dir)
>     (add-to-list 'load-path dir)
>     (dolist (autoload (directory-files dir t "-autoloads.el\\'"))
>       (load autoload nil t))))
>
> This would solve the problem of users being confused because their
> init file can not access packages they installed, allow all the heavy
> package initialization to happen after the init file is processed,

Slightly off topic, but I'm curious to know how heavy that snippet is. I
always thought that loading the autoload files corresponded to most of the
legwork in package-initialize. But I may well be wrong.

[-- Attachment #2: Type: text/html, Size: 869 bytes --]

Calling (package-initialize) sooner during initialization

[Artur Malabarba]

On the "Customizable modes..." thread I suggested we run
(package-initialize) sooner than the way it's currently done. Right
now, it's called after loading the init file. Which means any user who
tries to customize an installed package by pasting some code into his
init file will be confronted with errors.
This happens A LOT.

Stefan kindly explains why it can't just be done before loading the init file:
> [...] the user may need/want to run some Elisp
> code of his own choosing *before* package-initialize is called.
> E.g. [...] set package-load-list and package-directory-list

But I'm asking that we try a little harder to find a better solution.
This package.el-induced "cannot find load file" error is the most
predominant issue I see people run into in the wild. Some people file
issues for this stuff on github (and waste developer time), others go
to the relevant forums, and others (I can only imagine) probably just
give up on configuring packages (or give up Emacs!) entirely.

So I hope we can try to converge on an actual solution instead of just
resigning to something that harms the majority of the users. I propose
here a couple of suggestions which still preserve the use-case outline
by Stefan, and I'm perfectly open to other ideas.
Even if we can't find an ideal solution, try to keep in mind how
absurdly unideal the current situation is.

Option 1) Check if the file `~/.emacs,d/.package-delay-init' exists.
If it does, just do it the way we currently do. If it doesn't exist,
do package-initialize first and then load the init-file. This
`.package-delay-init' file is not loaded, Emacs would only check if it
exists.

Option 2) Instead of us manually telling users to add
`(package-initialize)' to their init-files, we have Emacs do that
automatically. Similar to how `custom.el' adds a call to
`custom-set-variables' the first time you save a variable; package.el
could add a call to `package-initialize' the first time you install a
package. This would be a one-time-thing (and would take some
intelligent coding to prevent annoyiances).

Thank you all,
Artur

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> Option 2) Instead of us manually telling users to add
> `(package-initialize)' to their init-files, we have Emacs do that
> automatically. Similar to how `custom.el' adds a call to
> `custom-set-variables' the first time you save a variable; package.el
> could add a call to `package-initialize' the first time you install a
> package.

I think that's the way to go, indeed.

> This would be a one-time-thing (and would take some intelligent coding
> to prevent annoyiances).

I think "only add it if that string can't be found in ~/.emacs yet"
should do it reliably enough.

There's still one hard problem to solve, I think:

  How to make it so that Customizing package variable `foo-bar' works
  right (which requires loading foo-autoloads beforehand, usually done
  by package-initialize), while making it so that Customizing
  package-load-list and package-directory-list also works correctly
  (which requires delays package-initialize to after those
  custom-settings are applied)?


-- Stefan

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

Something like this?

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 526c0b4..cf92557 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1628,6 +1628,28 @@ PACKAGES are satisfied, i.e. that PACKAGES is computed
 using `package-compute-transaction'."
   (mapc #'package-install-from-archive packages))

+(defun package--ensure-init-file ()
+  "Ensure that the user's init file calls `package-initialize'."
+  (let ((buffer (find-buffer-visiting user-init-file)))
+    (with-current-buffer (or buffer (find-file-noselect user-init-file))
+      (save-excursion
+        (save-restriction
+          (widen)
+          (goto-char (point-min))
+          (unless (search-forward "(package-initialize)" nil 'noerror)
+            (goto-char (point-min))
+            (insert
+             ";; Added by Package.el.  This must come before
configurations of\n"
+             ";; installed packages.  Don't delete this line.  If you
don't want it,\n"
+             ";; just comment it out by adding a semicolon to the
start of the line.\n"
+             "(package-initialize)\n")
+            (unless (looking-at-p "$")
+              (insert "\n"))
+            (let ((file-precious-flag t))
+              (save-buffer)))
+          (unless buffer
+            (kill-buffer (current-buffer))))))))
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
@@ -1656,6 +1678,7 @@ to install it but still mark it as selected."
                                   package-archive-contents))
                     nil t))
            nil)))
+  (package--ensure-init-file)
   (let ((name (if (package-desc-p pkg)
                   (package-desc-name pkg)
                 pkg)))
@@ -1699,6 +1722,7 @@ is derived from the main .el file in the directory.

 Downloads and installs required packages as needed."
   (interactive)
+  (package--ensure-init-file)
   (let* ((pkg-desc
           (cond
             ((derived-mode-p 'dired-mode)

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> Something like this?

`package--ensure-init-file' looks OK, yes (tho please make sure
it doesn't misbehave when Emacs was started with "-Q", or when
the init file doesn't exist yet).
Not sure where the calls to it should go, ideally, but the choice you
made looks acceptable.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Ted Zlatanov]

On Mon, 30 Mar 2015 16:44:04 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Option 2) Instead of us manually telling users to add
>> `(package-initialize)' to their init-files, we have Emacs do that
>> automatically. Similar to how `custom.el' adds a call to
>> `custom-set-variables' the first time you save a variable; package.el
>> could add a call to `package-initialize' the first time you install a
>> package.

SM> I think that's the way to go, indeed.

Ugh.  Sadly it seems to be the only reasonable way without changing what
`package-initialize' does fundamentally.

But I would approach this with command-line switches for Emacs, see below.

On Sun, 12 Apr 2015 12:07:24 +0200 Vasilij Schneidermann <v.schneidermann@gmail.com> wrote: 

>> We discussed this already.  There's the issue of configuring package.el
>> before loading it and/or before calling package-initialize.

VS> OK, I've checked again and sighted eight customization candidates in
VS> package.el:

VS> - package-enable-at-startup: Would be equivalent to the proposed
VS>   environment variable.
VS> - package-load-list: Needs to be configured before.
VS> - package-archives: Needs to be configured before for
VS>   non-interactive package installation.
VS> - package-pinned-packages: Needs to be configured before for
VS>   non-interactive package installation.
VS> - package-user-dir: Needs to be configured before.
VS> - package-directory-list: Needs to be configured before.
VS> - package-check-signature and package-unsigned-archives: Can be
VS>   customized afterwards

I think `package-check-signature' should also be configured before for
batch operation.

There aren't that many variables and they are all package-related. So
maybe support "emacs --package-xyz=foo" for all the above, for the
exceptional cases where the default setup of `package-initialize' is
wrong. Unlike --eval, the --package switches would know to notify the
internals. And unlike environment variables, the operation mode would be
explicitly encoded in the invocation.

Regardless of how it's implemented, I would make it easier for beginners
at the expense of making customizations a little harder for advanced
users and making batch operations complicated (because they usually
already are). So, for instance, environment variables for begginers are
not great because beginners are not going to write a wrapper shell
script.

I think, unfortunately, in the end it's easiest to keep the current
"just put this in your init file" situation. We could do worse :)

Ted

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Stefan kindly explains why it can't just be done before loading the init file:
>> [...] the user may need/want to run some Elisp
>> code of his own choosing *before* package-initialize is called.
>> E.g. [...] set package-load-list and package-directory-list

Why not adding a set function to these vars calling
`package-initialize', so if one set these functions AFTER a
`package-initialize' call `package-initialize' will be called again.
Of course these vars have to be set with customize, not setq.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

> Why not adding a set function to these vars calling
> `package-initialize', so if one set these functions AFTER a
> `package-initialize' call `package-initialize' will be called again.
> Of course these vars have to be set with customize, not setq.

I'd be ok with this too. But the repeated initializing could add a
couple extra seconds to init time.

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

>> Stefan kindly explains why it can't just be done before loading the init file:
>>> [...] the user may need/want to run some Elisp
>>> code of his own choosing *before* package-initialize is called.
>>> E.g. [...] set package-load-list and package-directory-list

> Why not adding a set function to these vars calling
> `package-initialize', so if one set these functions AFTER a
> `package-initialize' call `package-initialize' will be called again.
> Of course these vars have to be set with customize, not setq.

The default of package-load-list is to enable everything, so the main
use for package-load-list is to *prevent* initialization of
some packages.
IOW, the "setter" would have to *undo* some initializations, which is
can't be done reliably.


        Stefan

Re: Calling (package-initialize) sooner during initialization

[Dmitry Gutov]

On 03/31/2015 03:29 PM, Stefan Monnier wrote:

> The default of package-load-list is to enable everything, so the main
> use for package-load-list is to *prevent* initialization of
> some packages.
> IOW, the "setter" would have to *undo* some initializations, which is
> can't be done reliably.

Maybe, in the long run, a nicer approach would be to put the 
`custom-set-variables` form for the ELPA packages into a separate file, 
and load it after `package-initialize' was called.

I realize a variation of this has already been suggested, but this seems 
quite sane to me.

The downside would be that it won't help with existing entries in 
`custom-file', but if a user currently has package-related entries 
there, either their Emacs doesn't start, or the user already calls 
`package-initialize' earlier, or each entry doesn't really depend on 
respective package having been loaded.

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Stefan kindly explains why it can't just be done before loading the init file:
>>>> [...] the user may need/want to run some Elisp
>>>> code of his own choosing *before* package-initialize is called.
>>>> E.g. [...] set package-load-list and package-directory-list
>
>> Why not adding a set function to these vars calling
>> `package-initialize', so if one set these functions AFTER a
>> `package-initialize' call `package-initialize' will be called again.
>> Of course these vars have to be set with customize, not setq.
>
> The default of package-load-list is to enable everything, so the main
> use for package-load-list is to *prevent* initialization of
> some packages.
> IOW, the "setter" would have to *undo* some initializations, which is
> can't be done reliably.

IIUC `package-load-list' once set with 'all (the default) can't be
changed to something else (not all packages) after running
`package-initialize' (i.e it can't unload the already loaded packages)
unless restarting emacs.
There is something wrong here IMHO.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> Stefan kindly explains why it can't just be done before loading the init file:
>>>>> [...] the user may need/want to run some Elisp
>>>>> code of his own choosing *before* package-initialize is called.
>>>>> E.g. [...] set package-load-list and package-directory-list
>>
>>> Why not adding a set function to these vars calling
>>> `package-initialize', so if one set these functions AFTER a
>>> `package-initialize' call `package-initialize' will be called again.
>>> Of course these vars have to be set with customize, not setq.
>>
>> The default of package-load-list is to enable everything, so the main
>> use for package-load-list is to *prevent* initialization of
>> some packages.
>> IOW, the "setter" would have to *undo* some initializations, which is
>> can't be done reliably.
>
> IIUC `package-load-list' once set with 'all (the default) can't be
> changed to something else (not all packages) after running
> `package-initialize' (i.e it can't unload the already loaded packages)
> unless restarting emacs.
> There is something wrong here IMHO.

Maybe `package-load-list' should have a nil default value instead of all
and once package-initialize run, it could set it to 'all if it don't
find a customized value.
This with not help the real time setting of this var (no way to unload), 
but may solve the problem at startup.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:

> Thierry Volpiatto <thierry.volpiatto@gmail.com> writes:
>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>>>>> Stefan kindly explains why it can't just be done before loading the init file:
>>>>>> [...] the user may need/want to run some Elisp
>>>>>> code of his own choosing *before* package-initialize is called.
>>>>>> E.g. [...] set package-load-list and package-directory-list
>>>
>>>> Why not adding a set function to these vars calling
>>>> `package-initialize', so if one set these functions AFTER a
>>>> `package-initialize' call `package-initialize' will be called again.
>>>> Of course these vars have to be set with customize, not setq.
>>>
>>> The default of package-load-list is to enable everything, so the main
>>> use for package-load-list is to *prevent* initialization of
>>> some packages.
>>> IOW, the "setter" would have to *undo* some initializations, which is
>>> can't be done reliably.
>>
>> IIUC `package-load-list' once set with 'all (the default) can't be
>> changed to something else (not all packages) after running
>> `package-initialize' (i.e it can't unload the already loaded packages)
>> unless restarting emacs.
>> There is something wrong here IMHO.
>
> Maybe `package-load-list' should have a nil default value instead of all
> and once package-initialize run, it could set it to 'all if it don't
> find a customized value.
> This with not help the real time setting of this var (no way to unload), 
> but may solve the problem at startup.

Seems it is working fine with this:

1) default value of package-load-list => nil
2) customize yourself package-load-list e.g => ' (all)
3) Add at beginning of package-initialize someting like:
    (unless package-load-list
       (setq package-load-list '(all)))

When starting emacs the value of package-load-list before calling
package-initialize is '(all).

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

>> Maybe `package-load-list' should have a nil default value instead of all
>> and once package-initialize run, it could set it to 'all if it don't
>> find a customized value.
>> This with not help the real time setting of this var (no way to unload),
>> but may solve the problem at startup.
>
> Seems it is working fine with this:
>
> 1) default value of package-load-list => nil
> 2) customize yourself package-load-list e.g => ' (all)
> 3) Add at beginning of package-initialize someting like:
>     (unless package-load-list
>        (setq package-load-list '(all)))
>
> When starting emacs the value of package-load-list before calling
> package-initialize is '(all).

I don't understand how this will solve the "package-initialize is
called too late" issue. People who try to configure a package in their
init file will still run into errors until  "(package-initialize)" is
called.

Re: Calling (package-initialize) sooner during initialization

[Thierry Volpiatto]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> I don't understand how this will solve the "package-initialize is
> called too late" issue. People who try to configure a package in their
> init file will still run into errors until  "(package-initialize)" is
> called.

Forget it.

What about:

1) Stop writing by default the custom settings in init file and use and
create by default the custom-file.
2) Add at start of this file a call to a function that add elpa packages
to load-path according to package-load-list.
3) Call package-initialize at end of this file.

-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997 

Re: Calling (package-initialize) sooner during initialization

[Sebastian Wiesner]

2015-03-30 21:01 GMT+02:00 Artur Malabarba <bruce.connor.am@gmail.com>:
> On the "Customizable modes..." thread I suggested we run
> (package-initialize) sooner than the way it's currently done. Right
> now, it's called after loading the init file. Which means any user who
> tries to customize an installed package by pasting some code into his
> init file will be confronted with errors.
> This happens A LOT.
>
> Stefan kindly explains why it can't just be done before loading the init file:
>> [...] the user may need/want to run some Elisp
>> code of his own choosing *before* package-initialize is called.
>> E.g. [...] set package-load-list and package-directory-list
>
> But I'm asking that we try a little harder to find a better solution.
> This package.el-induced "cannot find load file" error is the most
> predominant issue I see people run into in the wild. Some people file
> issues for this stuff on github (and waste developer time), others go
> to the relevant forums, and others (I can only imagine) probably just
> give up on configuring packages (or give up Emacs!) entirely.
>
> So I hope we can try to converge on an actual solution instead of just
> resigning to something that harms the majority of the users. I propose
> here a couple of suggestions which still preserve the use-case outline
> by Stefan, and I'm perfectly open to other ideas.
> Even if we can't find an ideal solution, try to keep in mind how
> absurdly unideal the current situation is.
>
> Option 1) Check if the file `~/.emacs,d/.package-delay-init' exists.
> If it does, just do it the way we currently do. If it doesn't exist,
> do package-initialize first and then load the init-file. This
> `.package-delay-init' file is not loaded, Emacs would only check if it
> exists.
>
> Option 2) Instead of us manually telling users to add
> `(package-initialize)' to their init-files, we have Emacs do that
> automatically. Similar to how `custom.el' adds a call to
> `custom-set-variables' the first time you save a variable; package.el
> could add a call to `package-initialize' the first time you install a
> package. This would be a one-time-thing (and would take some
> intelligent coding to prevent annoyiances).
>
> Thank you all,
> Artur
>

I probably underestimate the complexity of this issue, so please
forgive if I say anything stupid, but… isn't this much too
complicated?

For most defcustom's it doesn't actually matter whether they are set
before or after the corresponding feature is loaded, does it?  So
there's only a very specific class of defcustom's which is affected at
all:  Namely, modes enabled through customize.

For these specific defcustom's the corresponding feature must be
loaded immediately anyway, since we must have the mode function
available to actually enable the mode, right?

So we could just require that a mode that is supposed to be enabled
through customize also has a ':require' keyword for the corresponding
feature.

And for :require features the whole problem looks pretty simple to me:
 Try to load the feature on 'set-custom-variables'.  If it fails
remember the corresponding form somewhere (i.e. 'custom-delayed-vars'
or whatever), and just try again after 'package-initialize'.  If it
still fails, report an error.

That sounds like a simple and straight-forward approach that's not too
invasive or too magic.  Would that not work?

Re: Calling (package-initialize) sooner during initialization

[Artur Malabarba]

> That sounds like a simple and straight-forward approach that's not too
> invasive or too magic.  Would that not work?

For the purpose of custom.el, yes, that would be enough.
The reason I started this discussion was to reach a more general
solution. And if we solve this more general issue, it should solve the
custom.el problem as well.

Pasting elisp snippets into your init file is a perfectly valid way of
customizing packages. Actually, nevermind “customizing”. Even the
installation instructions of most packages include a “add the
following to your init-file”.

I guess some might say that such people who paste snippets into their
init file might as well learn to paste “(package-initialize)” along
with it. But it's not fair to punish newbies for just following the
Emacs culture. I'd rather fix this issue for good, at its source, than
having to keep explaining this to people for years to come.

Re: Calling (package-initialize) sooner during initialization

[Stefan Monnier]

> So we could just require that a mode that is supposed to be enabled
> through customize also has a ':require' keyword for the corresponding
> feature.

The require will fail if executed before calling package-initialize.

> And for :require features the whole problem looks pretty simple to me:
>  Try to load the feature on 'set-custom-variables'.  If it fails
> remember the corresponding form somewhere (i.e. 'custom-delayed-vars'
> or whatever), and just try again after 'package-initialize'.  If it
> still fails, report an error.

Doesn't seem significantly simpler than adding (package-initialize) in
the ~/.emacs file.  Also the "lazy calling" of that function makes it
harder to figure out what's going on.

Also, the canonical way to handle minor modes is not to have them
use :require but to mark them as autoload.  In this case, Custom might
not even know that this var is a minor-mode until after
package-initialize is called, so if package-initialize is not called
early, then Custom may simply "do the wrong thing", oblivious to the
fact that it should have called a minor mode function instead of just
setting the variable.


        Stefan