bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Zachary Kanfer]

[-- Attachment #1: Type: text/plain, Size: 6194 bytes --]

------------------
High-level summary:
------------------

If Emacs is started with no ~/.emacs.d/elpa directory,
#'package-initialize is not called, and I can't programatically work
with packages.

----------
Motivation
----------

I like my init file to completely set up Emacs: I want to be able to
start with an empty ~/.emacs.d/ move my init file into it, and start up
Emacs. To do that, I've had (previously to Emacs 27), this as the
preamble to my init file:

    (require 'package)
    (add-to-list 'package-archives
                 '("melpa" . "https://melpa.org/packages/"))
    (package-initialize)

    (unless (package-installed-p 'use-package)
      (package-refresh-contents)
      (package-install 'use-package))

This sets up package.el with the ability to use melpa, then installs
use-package, for use in the rest of my init file. This has worked fine.

But I recently upgraded my Emacs to a newer version built from source,
using commit 766b057e41df7316808ec7658836fda75facda75. As part of that,
I've read up on the early init file. My understanding is that users
should no longer call #'package-initialize in their init files. This is
supported by, if I do have a call to #'package-initialize, starting
Emacs gives me a warning "Warning (package): Unnecessary call to
‘package-initialize’ in init file". So I figured I would remove it. But
I quickly ran into a problem, or at least an area for better
documentation.

-------------------------
Reproduction instructions
-------------------------

A minimal init file that shows my problem is here:

    (require 'package)
    (add-to-list 'package-archives
                 '("melpa" . "https://melpa.org/packages/"))

    (message "use-package is installed: %s" (package-installed-p
'use-package))

Here's how I delete the elpa directory, and then start up Emacs:

    $ rm -r ~/.emacs.d/elpa
    $ src/emacs

Upon doing so, here's the error I get:

    Warning (initialization): An error occurred while loading
‘/home/zck/.emacs.d/init.el’:

    error: package.el is not yet initialized!

    To ensure normal operation, you should investigate and remove the
    cause of the error in your initialization file.  Start Emacs with
    the ‘--debug-init’ option to view a complete error backtrace.

---------------------------
Investigation & speculation
---------------------------

I started looking around for documentation, and found this interesting
comment in simple.el's function #'command-line:

    ;; If any package directory exists, initialize the package system.

This seems to be the cause. I'm not sure why the package system
shouldn't be initialized if there is no package directory; I want to be
able to do package operations even if I'm starting with no package
directory. You have to start somewhere.

Thanks!

------------------
System information
------------------

In GNU Emacs 27.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.18.9)
 of 2018-05-07 built on zck-laptop
Repository revision: 766b057e41df7316808ec7658836fda75facda75
Windowing system distributor 'The X.Org Foundation', version 11.0.11804000
System Description: Ubuntu 16.04.4 LTS

Recent messages:
Checking for load-path shadows...done
Auto-saving...
Overwrite mode enabled in current buffer
<pause> is undefined
Undo! [3 times]
user-error: No further undo information
C-c k is undefined
Quit
C-x C-g is undefined
Mark set

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 THREADS LCMS2

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml mml-sec epa derived epg gnus-util rmail
rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils cl-extra pp cl-print package-x thingatpt
help-fns radix-tree help-mode time-date elec-pair warnings package
easymenu epg-config url-handlers url-parse auth-source cl-seq eieio
eieio-core cl-macs eieio-loaddefs password-cache json map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote dbusbind inotify lcms2 dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 114821 11707)
 (symbols 48 21397 0)
 (miscs 40 99 151)
 (strings 32 34316 1830)
 (string-bytes 1 888700)
 (vectors 16 17166)
 (vector-slots 8 524482 10750)
 (floats 8 54 84)
 (intervals 56 292 144)
 (buffers 992 15)
 (heap 1024 44657 1840))

[-- Attachment #2: Type: text/html, Size: 6959 bytes --]

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no

[Zachary Kanfer]

[-- Attachment #1: Type: text/plain, Size: 87 bytes --]

Anything I can do to help narrow this down? I'm not sure what to do to make
this work.

[-- Attachment #2: Type: text/html, Size: 123 bytes --]

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Noam Postavsky]

Zachary Kanfer <zkanfer@gmail.com> writes:

> I like my init file to completely set up Emacs: I want to be able to
> start with an empty ~/.emacs.d/ move my init file into it, and start up
> Emacs.
[...]
> I've read up on the early init file. My understanding is that users
> should no longer call #'package-initialize in their init files. This is
> supported by, if I do have a call to #'package-initialize, starting
> Emacs gives me a warning "Warning (package): Unnecessary call to
> ‘package-initialize’ in init file". So I figured I would remove it. But
> I quickly ran into a problem, or at least an area for better
> documentation.
[...]
> A minimal init file that shows my problem is here:
>
>     (require 'package)
>     (add-to-list 'package-archives
>                  '("melpa" . "https://melpa.org/packages/"))
>
>     (message "use-package is installed: %s" (package-installed-p 'use-package))
>
> Here's how I delete the elpa directory, and then start up Emacs:
>
>     $ rm -r ~/.emacs.d/elpa
>     $ src/emacs
>
> Upon doing so, here's the error I get:
>
>     Warning (initialization): An error occurred while loading
> ‘/home/zck/.emacs.d/init.el’:
>
>     error: package.el is not yet initialized!


> Anything I can do to help narrow this down? I'm not sure what to do to make this work.

Perhaps adding Radon to Cc will help, not sure if they're subscribed to
the bug list.

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Radon Rosborough]

[-- Attachment #1: Type: text/plain, Size: 578 bytes --]

Thanks for the cc. I wasn't aware of this bug report.

The behavior makes sense, but is definitely undesirable. There are at
least two solutions:

1. Make `package-installed-p' and other functions automatically
   initialize package.el if necessary.

2. Call `package-initialize' unconditionally during startup.

Solution (1) might make package.el functions a little slower. But
solution (2) would definitely make startup slower for people who don't
use package.el.

Solution (1) seems like the correct way to do this. Is there any
reason it wasn't done already?

Any thoughts?

[-- Attachment #2: Type: text/html, Size: 853 bytes --]

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Noam Postavsky]

Radon Rosborough <radon.neon@gmail.com> writes:

> Thanks for the cc. I wasn't aware of this bug report.
>
> The behavior makes sense, but is definitely undesirable. There are at
> least two solutions:
>
> 1. Make `package-installed-p' and other functions automatically
>    initialize package.el if necessary.
>
> 2. Call `package-initialize' unconditionally during startup.
>
> Solution (1) might make package.el functions a little slower.

I don't think speed is an issue here.  For package-installed-p in
particular, it would have no effect on speed since that function already
checks package--initialized (and checking a single variable is pretty
negligible already).

> Solution (1) seems like the correct way to do this. Is there any
> reason it wasn't done already?

Not sure, I might guess that a predicate like package-installed-p
shouldn't have side-effects, but package-install doesn't auto-initialize
either when called non-interactively.

Stefan, do you have any insight on this?

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Stefan Monnier]

>> 1. Make `package-installed-p' and other functions automatically
>>    initialize package.el if necessary.

Yes, when I changed it recently I hesitated to do that and decided to
refrain from doing so in order to minimize the changes, but it makes
a lot of sense.


        Stefan


diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 94d98178c4..56be5ef70c 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -1889,8 +1889,8 @@ package-installed-p
     ;; We used the quickstart: make it possible to use package-installed-p
     ;; even before package is fully initialized.
     (memq package package-activated-list))
-   ((not package--initialized) (error "package.el is not yet initialized!"))
    (t
+    (unless package--initialized (package-initialize 'no-activate))
     (or
      (let ((pkg-descs (cdr (assq package package-alist))))
        (and pkg-descs

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Zachary Kanfer]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

I'm circling back around to this; I think that that change makes a lot of
sense. It seems better to me -- if you're calling one of these package
functions, you probably want to work with the package system, which is only
properly set up when you call package-initialize.

What other functions would need to call package-initialize, if it hasn't
been called? I notice that package-install already does that. Some other
functions for installing don't
(package-install-from-archive, package-install-from-buffer), but I'm not
sure whether they need to.

On Tue, May 29, 2018 at 6:58 PM, Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> >> 1. Make `package-installed-p' and other functions automatically
> >>    initialize package.el if necessary.
>
> Yes, when I changed it recently I hesitated to do that and decided to
> refrain from doing so in order to minimize the changes, but it makes
> a lot of sense.
>
>
>         Stefan
>
>
> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index 94d98178c4..56be5ef70c 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -1889,8 +1889,8 @@ package-installed-p
>      ;; We used the quickstart: make it possible to use package-installed-p
>      ;; even before package is fully initialized.
>      (memq package package-activated-list))
> -   ((not package--initialized) (error "package.el is not yet
> initialized!"))
>     (t
> +    (unless package--initialized (package-initialize 'no-activate))
>      (or
>       (let ((pkg-descs (cdr (assq package package-alist))))
>         (and pkg-descs
>

[-- Attachment #2: Type: text/html, Size: 2094 bytes --]

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Stefan Monnier]

> I'm circling back around to this; I think that that change makes a lot of
> sense.

I installed an alternative patch which doesn't fully initialize package,
but does make package-installed-p work even if package-initialize hasn't
been called yet.

> What other functions would need to call package-initialize, if it hasn't
> been called?

Good question.  I hope the answer is none, and I think if we find one,
we should change it so it initializes the relevant vars automatically.


        Stefan

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Stefan Kangas]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> I'm circling back around to this; I think that that change makes a lot of
>> sense.
>
> I installed an alternative patch which doesn't fully initialize package,
> but does make package-installed-p work even if package-initialize hasn't
> been called yet.

Thanks.

>> What other functions would need to call package-initialize, if it hasn't
>> been called?
>
> Good question.  I hope the answer is none, and I think if we find one,
> we should change it so it initializes the relevant vars automatically.

I agree, and have had a cursory look without finding any obvious
candidates.  Perhaps it's best to do them on a case-by-case basis,
i.e. treat them as normal bugs if/when they are found.

IOW, I'm not sure it makes sense if it makes sense to keep this bug
open just for the eventuality that there are more functions with
similar bugs.

Best regards,
Stefan Kangas

bug#31397: 27.0.50; Emacs doesn't call package-initialize if there's no elpa directory

[Stefan Monnier]

> I agree, and have had a cursory look without finding any obvious
> candidates.  Perhaps it's best to do them on a case-by-case basis,
> i.e. treat them as normal bugs if/when they are found.
>
> IOW, I'm not sure it makes sense if it makes sense to keep this bug
> open just for the eventuality that there are more functions with
> similar bugs.

Agreed, closing,


        Stefan