bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Shitikanth]

Until emacs 27.2, package.el used to correctly pick `package-user-dir'
based on the value of `user-emacs-directory', but it now seems to ignore
it and defaults to "~/.emacs.d/elpa".

Demo:
    (setq user-emacs-directory "/tmp")
    (require 'package)
    (message "%s" package-user-dir)

Starting with "emacs -Q" this would output "~/.emacs.d/elpa".
Expected output: "/tmp/elpa" (as in emacs 27.2).

Based on git bisect, the offending commit seems to be 9973019764.

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Eli Zaretskii]

> From: Shitikanth <shitikanth1@gmail.com>
> Date: Thu, 26 May 2022 16:37:48 +0530
> 
> Until emacs 27.2, package.el used to correctly pick `package-user-dir'
> based on the value of `user-emacs-directory', but it now seems to ignore
> it and defaults to "~/.emacs.d/elpa".
> 
> Demo:
>     (setq user-emacs-directory "/tmp")
>     (require 'package)
>     (message "%s" package-user-dir)
> 
> Starting with "emacs -Q" this would output "~/.emacs.d/elpa".
> Expected output: "/tmp/elpa" (as in emacs 27.2).
> 
> Based on git bisect, the offending commit seems to be 9973019764.

Why do you set user-emacs-directory instead of setting
package-user-dir directly?  You are supposed to set the latter in your
early-init file.

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Shitikanth]

I think almost all emacs packages follow the convention of setting
the location of their cache/configuration files by
calling`locate-user-emacs-file'.
This enables the user to just set the user-emacs-directory instead of
configuring all of them individually. (This works great for the use case of
wanting to put your emacs config in a location other than ~/.emacs.d or
even having multiple emacs configs.)

> (defcustom package-user-dir (locate-user-emacs-file "elpa")

Looking at the definition of `package-user-dir`, it ostensibly tries to
follow the same convention as well. It has just stopped working for
some reason since 9973019764, which is why I consider this a bug.

Is it actually intended behavior that `package-user-dir' should not
by default have the same value as (locate-user-emacs-file "elpa")?

On Thu, 26 May 2022 at 17:19, Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Shitikanth <shitikanth1@gmail.com>
> > Date: Thu, 26 May 2022 16:37:48 +0530
> >
> > Until emacs 27.2, package.el used to correctly pick `package-user-dir'
> > based on the value of `user-emacs-directory', but it now seems to ignore
> > it and defaults to "~/.emacs.d/elpa".
> >
> > Demo:
> >     (setq user-emacs-directory "/tmp")
> >     (require 'package)
> >     (message "%s" package-user-dir)
> >
> > Starting with "emacs -Q" this would output "~/.emacs.d/elpa".
> > Expected output: "/tmp/elpa" (as in emacs 27.2).
> >
> > Based on git bisect, the offending commit seems to be 9973019764.
>
> Why do you set user-emacs-directory instead of setting
> package-user-dir directly?  You are supposed to set the latter in your
> early-init file.

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Eli Zaretskii]

> From: Shitikanth <shitikanth1@gmail.com>
> Date: Thu, 26 May 2022 18:49:53 +0530
> Cc: 55655@debbugs.gnu.org
> 
> I think almost all emacs packages follow the convention of setting
> the location of their cache/configuration files by
> calling`locate-user-emacs-file'.
> This enables the user to just set the user-emacs-directory instead of
> configuring all of them individually. (This works great for the use case of
> wanting to put your emacs config in a location other than ~/.emacs.d or
> even having multiple emacs configs.)
> 
> > (defcustom package-user-dir (locate-user-emacs-file "elpa")
> 
> Looking at the definition of `package-user-dir`, it ostensibly tries to
> follow the same convention as well. It has just stopped working for
> some reason since 9973019764, which is why I consider this a bug.
> 
> Is it actually intended behavior that `package-user-dir' should not
> by default have the same value as (locate-user-emacs-file "elpa")?

Yes, but user-emacs-directory isn't supposed to be set, it is supposed
to be computed by Emacs early at startup, and then used for the rest
of the startup processing.

Stefan, am I missing something here?

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Shitikanth [2022-05-26 16:37:48] wrote:
> Until emacs 27.2, package.el used to correctly pick `package-user-dir'
> based on the value of `user-emacs-directory', but it now seems to ignore
> it and defaults to "~/.emacs.d/elpa".
>
> Demo:
>     (setq user-emacs-directory "/tmp")
>     (require 'package)
>     (message "%s" package-user-dir)
>
> Starting with "emacs -Q" this would output "~/.emacs.d/elpa".
> Expected output: "/tmp/elpa" (as in emacs 27.2).

That expectation is wrong, sadly, because it assumes that the
initialization of `package-user-dir` takes place during (require
'package), which is not the case because that variable is now
initialized earlier.

I understand it's a change, but it might be difficult to make this
specific recipe work the way you want without breaking other use-cases.

Could you give some more details about the use case that actually
affects you (I hope it's slightly different from the simplified recipe
shown above)?


        Stefan

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Shitikanth]

>Could you give some more details about the use case that actually
affects you

I just have a couple of different configs and use them with
"emacs -Q --load $config_dir/init.el" and set the user-emacs-directory
early in "$config_dir/init.el". This was also the easiest way to try out
other people's configs without disturbing your own.

I guess I was always aware that this was an abuse of the init system,
so if this is an intended change, you can close the bug report.

On Thu, 26 May 2022 at 20:59, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> Shitikanth [2022-05-26 16:37:48] wrote:
> > Until emacs 27.2, package.el used to correctly pick `package-user-dir'
> > based on the value of `user-emacs-directory', but it now seems to ignore
> > it and defaults to "~/.emacs.d/elpa".
> >
> > Demo:
> >     (setq user-emacs-directory "/tmp")
> >     (require 'package)
> >     (message "%s" package-user-dir)
> >
> > Starting with "emacs -Q" this would output "~/.emacs.d/elpa".
> > Expected output: "/tmp/elpa" (as in emacs 27.2).
>
> That expectation is wrong, sadly, because it assumes that the
> initialization of `package-user-dir` takes place during (require
> 'package), which is not the case because that variable is now
> initialized earlier.
>
> I understand it's a change, but it might be difficult to make this
> specific recipe work the way you want without breaking other use-cases.
>
> Could you give some more details about the use case that actually
> affects you (I hope it's slightly different from the simplified recipe
> shown above)?
>
>
>         Stefan
>

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Visuwesh]

[வெள்ளி மே 27, 2022] Shitikanth wrote:

>>Could you give some more details about the use case that actually
> affects you
>
> I just have a couple of different configs and use them with
> "emacs -Q --load $config_dir/init.el" and set the user-emacs-directory
> early in "$config_dir/init.el". This was also the easiest way to try out
> other people's configs without disturbing your own.
>
> I guess I was always aware that this was an abuse of the init system,
> so if this is an intended change, you can close the bug report.
>

FWIW, Emacs 29 added a --init-directory flag for these purposes.  Here's
the NEWS entry of the change,

    ** Emacs now supports setting 'user-emacs-directory' via '--init-directory'.

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Shitikanth]

Great news! That's exactly what I needed

Thanks for pointing that out!

On Fri, 27 May 2022 at 11:09, Visuwesh <visuweshm@gmail.com> wrote:
>
> [வெள்ளி மே 27, 2022] Shitikanth wrote:
>
> >>Could you give some more details about the use case that actually
> > affects you
> >
> > I just have a couple of different configs and use them with
> > "emacs -Q --load $config_dir/init.el" and set the user-emacs-directory
> > early in "$config_dir/init.el". This was also the easiest way to try out
> > other people's configs without disturbing your own.
> >
> > I guess I was always aware that this was an abuse of the init system,
> > so if this is an intended change, you can close the bug report.
> >
>
> FWIW, Emacs 29 added a --init-directory flag for these purposes.  Here's
> the NEWS entry of the change,
>
>     ** Emacs now supports setting 'user-emacs-directory' via '--init-directory'.

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Stefan Kangas]

Shitikanth <shitikanth1@gmail.com> writes:

> Great news! That's exactly what I needed
>
> Thanks for pointing that out!

Great.  I'm therefore closing this bug report.

> On Fri, 27 May 2022 at 11:09, Visuwesh <visuweshm@gmail.com> wrote:
>>
>> [வெள்ளி மே 27, 2022] Shitikanth wrote:
>>
>> >>Could you give some more details about the use case that actually
>> > affects you
>> >
>> > I just have a couple of different configs and use them with
>> > "emacs -Q --load $config_dir/init.el" and set the user-emacs-directory
>> > early in "$config_dir/init.el". This was also the easiest way to try out
>> > other people's configs without disturbing your own.
>> >
>> > I guess I was always aware that this was an abuse of the init system,
>> > so if this is an intended change, you can close the bug report.
>> >
>>
>> FWIW, Emacs 29 added a --init-directory flag for these purposes.  Here's
>> the NEWS entry of the change,
>>
>>     ** Emacs now supports setting 'user-emacs-directory' via '--init-directory'.

bug#55655: 28.1; package.el no longers respects `user-emacs-directory'

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>Could you give some more details about the use case that actually
>> affects you
>
> I just have a couple of different configs and use them with
> "emacs -Q --load $config_dir/init.el" and set the user-emacs-directory
> early in "$config_dir/init.el". This was also the easiest way to try out
> other people's configs without disturbing your own.

I see... for this specific case `--init-directory` or `--user` might
work better.

> I guess I was always aware that this was an abuse of the init system,
> so if this is an intended change, you can close the bug report.

What you see is not exactly an "intended change" in the sense that it
would be good to make it behave the way you want to in your case.

But it's not straightforward: `package-user-directory` is now predefined
in the dumped Emacs.  Obviously it does need to be (re)computed at
startup (since your $HOME was not know when Emacs was dumped), but we do
that at the very beginning before processing `--eval` or loading
init files.  For your case to work properly, we'd need to delay the
(re)computation of `package-user-directory` to do it more lazily.

It doesn't seem impossible to do, but it might be tricky and risks
introducing other problems.

You can workaround the problem by explicitly calling

    (custom-reevaluate-setting 'package-user-directory)

after you've set `user-emacs-directory`.


        Stefan