bug#56877: 29.0.50; Built-in packages are not always package-installed-p

bug#56877: 29.0.50; Built-in packages are not always package-installed-p

[Matt Armstrong]

Before `package--initialized' is set non-nil a call to
(package-installed-p PACKAGE) returns nil for symbols naming built-in
packages. After `package--initialized' is non-nil the same call returns
non-nil.

I discovered this by accident when I figured out this use-package form
caused Emacs to contact GNU ELPA during startup:

(use-package project
  :ensure t
  ;; stuff)

For the above use-package calls (package-installed-p 'project), which
returns nil.  Use-package then attempts to install it using package.el,
which causes network activity, etc., and then package.el decides that it
is already installed.  This repeats each time I start Emacs.

There is code in `package-installed-p' that consults
`package-activated-list' when `package--initalized' is non-nil.  This is
the code path I am exercising.  We also have this FIXME:

(defvar package-activated-list nil
  ;; FIXME: This should implicitly include all builtin packages.
  "List of the names of currently activated packages.")

I have a fix prepared, which I will send once i have a bug number.


Configured using:
 'configure --with-native-compilation --with-pgtk'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP
NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS XIM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: ELisp/l

Minor modes in effect:
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  auto-revert-mode: t
  bug-reference-prog-mode: t
  msb-mode: t
  display-time-mode: t
  shell-dirtrack-mode: t
  auto-insert-mode: t
  keyfreq-autosave-mode: t
  keyfreq-mode: t
  savehist-mode: t
  icomplete-vertical-mode: t
  icomplete-mode: t
  editorconfig-mode: t
  which-key-mode: t
  electric-pair-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  auto-save-visited-mode: t

Load-path shadows:
~/env/elisp/ol-notmuch hides /home/matt/.config/emacs/elpa/ol-notmuch-20220428.1337/ol-notmuch
/home/matt/.config/emacs/elpa/transient-20220717.1713/transient hides /home/matt/git/emacs-build/lisp/transient

Features:
(ert ewoc debug backtrace shadow sort mail-extr magit-extras face-remap
magit-submodule magit-obsolete magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log which-func imenu
magit-diff git-commit log-edit magit-core magit-autorevert magit-margin
magit-transient magit-process with-editor magit-mode transient magit-git
magit-base magit-section crm dash compat-27 compat-26 compat make-mode
emacsbug smerge-mode diff add-log vc-annotate vc-hg vc-bzr vc-src
vc-sccs vc-svn vc-cvs vc-rcs log-view pcvs-util whitespace vc sh-script
smie executable autorevert cl-print package-x dabbrev misearch
multi-isearch pulse color xref vc-git diff-mode vc-dispatcher
bug-reference shortdoc benchmark help-fns radix-tree textsec uni-scripts
idna-mapping ucs-normalize uni-confusable textsec-check protbuf msb time
copyright editorconfig-core editorconfig-core-handle
editorconfig-fnmatch cus-edit pp cus-start cus-load org-element avl-tree
generator ol-w3m ol-rmail ol-mhe ol-irc ol-info org-habit org-agenda
org-refile ol-gnus nnselect gnus-art mm-uu mml2015 mm-view mml-smime
smime dig gnus-sum gnus-group gnus-undo gnus-start gnus-dbus dbus
gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo parse-time
gnus-spec gnus-int gnus-range gnus-win ol-eww eww xdg url-queue
thingatpt shr pixel-fill kinsoku url-file url-dired svg xml dom mm-url
gnus nnheader range wid-edit ol-doi org-link-doi ol-docview doc-view
filenotify jka-compr image-mode exif ol-bibtex ol-bbdb server dirtrack
ob-shell shell ob-ruby ob-python python ob-dot org-protocol org ob
ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote org-src
ob-comint org-pcomplete pcomplete comint ansi-color ring org-list
org-faces org-entities noutline outline org-version ob-emacs-lisp
ob-core ob-eval org-table oc-basic bibtex iso8601 org-keys oc
org-loaddefs find-func cal-menu calendar cal-loaddefs ol-notmuch ol
org-compat org-macs format-spec skeleton autoinsert advice keyfreq
project finder-inf mm-archive message sendmail yank-media dired
dired-loaddefs rfc822 mml mml-sec epa derived gnus-util
text-property-search time-date mailabbrev gmm-utils mailheader mm-decode
mm-bodies mm-encode mail-utils gnutls network-stream url-cache url-http
url-auth mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums
mail-prsvr url-gw nsm puny epg rfc6068 epg-config edmacro kmacro
savehist icomplete comp comp-cstr warnings icons rx editorconfig
modus-operandi-theme modus-themes which-key package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source eieio eieio-core password-cache json subr-x map byte-opt
url-vars cl-macs gv cl-extra help-mode cl-seq elec-pair use-package
use-package-ensure use-package-delight use-package-diminish
use-package-bind-key bind-key easy-mmode use-package-core cl-loaddefs
cl-lib bytecomp byte-compile cconv info bazel-autoloads
clang-format+-autoloads clang-format-autoloads cmake-mode-autoloads
d-mode-autoloads debbugs-autoloads editorconfig-autoloads
eglot-autoloads elpy-autoloads company-autoloads
exec-path-from-shell-autoloads flymake-ruby-autoloads
flymake-easy-autoloads flymake-yamllint-autoloads go-mode-autoloads
google-c-style-autoloads graphviz-dot-mode-autoloads
highlight-indentation-autoloads magit-autoloads git-commit-autoloads
magit-section-autoloads dash-autoloads markdown-mode-autoloads
meson-mode-autoloads modus-themes-autoloads nixpkgs-fmt-autoloads
ol-notmuch-autoloads notmuch-autoloads orderless-autoloads
org-drill-autoloads persist-autoloads pylint-autoloads pyvenv-autoloads
s-autoloads shfmt-autoloads reformatter-autoloads transient-autoloads
use-package-autoloads bind-key-autoloads vertico-autoloads
which-key-autoloads with-editor-autoloads compat-autoloads
yaml-mode-autoloads yasnippet-autoloads rmc iso-transl tooltip eldoc
paren electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
mwheel term/pgtk-win pgtk-win term/common-win pgtk-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
faces cus-face macroexp files window text-properties overlay sha1 md5
base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
lcms2 multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 1041704 138279)
 (symbols 48 41342 108)
 (strings 32 237540 12847)
 (string-bytes 1 6538892)
 (vectors 16 112331)
 (vector-slots 8 2502178 111733)
 (floats 8 749 1306)
 (intervals 56 26923 3648)
 (buffers 992 42))

bug#56877: 29.0.50; Built-in packages are not always package-installed-p

[Matt Armstrong]

[-- Attachment #1: Type: text/plain, Size: 126 bytes --]

Matt Armstrong <matt@rfc20.org> writes:

> I have a fix prepared, which I will send once i have a bug number.

Now attached:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Consider-built-in-packages-to-be-installed.patch --]
[-- Type: text/x-diff, Size: 2356 bytes --]

From b9431eee1bc141a036aa42541f68b0c07474bc40 Mon Sep 17 00:00:00 2001
From: Matt Armstrong <matt@rfc20.org>
Date: Mon, 1 Aug 2022 11:18:37 -0700
Subject: [PATCH] Consider built-in packages to be installed

With this change (package-installed-p PACKAGE nil) evaluates to the
same thing before and after package initialization.

* lisp/emacs-lisp/package.el (package-installed-p): Check for built-in
packages before initialization. (bug#56877)
---
 lisp/emacs-lisp/package.el            |  5 ++++-
 test/lisp/emacs-lisp/package-tests.el | 15 +++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index df70f908da..482de52f85 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2085,7 +2085,10 @@ package-installed-p
          package-activated-list)
     ;; We used the quickstart: make it possible to use package-installed-p
     ;; even before package is fully initialized.
-    (memq package package-activated-list))
+    (or
+     (memq package package-activated-list)
+     ;; Also check built-in packages.
+     (package-built-in-p package min-version)))
    (t
     (or
      (let ((pkg-descs (cdr (assq package (package--alist)))))
diff --git a/test/lisp/emacs-lisp/package-tests.el b/test/lisp/emacs-lisp/package-tests.el
index d7a55998c2..75c3f061ab 100644
--- a/test/lisp/emacs-lisp/package-tests.el
+++ b/test/lisp/emacs-lisp/package-tests.el
@@ -638,6 +638,21 @@ package-test-update-archives/ignore-nil-entry
       (package-refresh-contents)
       (should (equal (length package-archive-contents) 2)))))
 
+(ert-deftest package-test-package-installed-p ()
+  "Foo"
+  (with-package-test ()
+    ;; Verify that `package-installed-p' evaluates true for a built-in
+    ;; package, in this case `project', before package initialization.
+    (should (not package--initialized))
+    (should (package-installed-p 'project nil))
+    (should (not (package-installed-p 'imaginary-package nil)))
+
+    ;; The results don't change after package initialization.
+    (package-initialize)
+    (should package--initialized)
+    (should (package-installed-p 'project nil))
+    (should (not (package-installed-p 'imaginary-package nil)))))
+
 (ert-deftest package-test-describe-package ()
   "Test displaying help for a package."
 
-- 
2.35.1

bug#56877: 29.0.50; Built-in packages are not always package-installed-p

[Matt Armstrong]

[-- Attachment #1: Type: text/plain, Size: 226 bytes --]

Matt Armstrong <matt@rfc20.org> writes:

> Matt Armstrong <matt@rfc20.org> writes:
>
>> I have a fix prepared, which I will send once i have a bug number.
>
> Now attached:

Second try.  Replaced "Foo" with a real docstring!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Consider-built-in-packages-to-be-installed.patch --]
[-- Type: text/x-diff, Size: 2418 bytes --]

From f66a0e6bf84f10d2688b2892a08e2d090e5bd6c7 Mon Sep 17 00:00:00 2001
From: Matt Armstrong <matt@rfc20.org>
Date: Mon, 1 Aug 2022 11:18:37 -0700
Subject: [PATCH] Consider built-in packages to be installed

With this change (package-installed-p PACKAGE nil) evaluates to the
same thing before and after package initialization.

* lisp/emacs-lisp/package.el (package-installed-p): Check for built-in
packages before initialization. (bug#56877)
---
 lisp/emacs-lisp/package.el            |  5 ++++-
 test/lisp/emacs-lisp/package-tests.el | 15 +++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index df70f908da..482de52f85 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2085,7 +2085,10 @@ package-installed-p
          package-activated-list)
     ;; We used the quickstart: make it possible to use package-installed-p
     ;; even before package is fully initialized.
-    (memq package package-activated-list))
+    (or
+     (memq package package-activated-list)
+     ;; Also check built-in packages.
+     (package-built-in-p package min-version)))
    (t
     (or
      (let ((pkg-descs (cdr (assq package (package--alist)))))
diff --git a/test/lisp/emacs-lisp/package-tests.el b/test/lisp/emacs-lisp/package-tests.el
index d7a55998c2..b903cd781b 100644
--- a/test/lisp/emacs-lisp/package-tests.el
+++ b/test/lisp/emacs-lisp/package-tests.el
@@ -638,6 +638,21 @@ package-test-update-archives/ignore-nil-entry
       (package-refresh-contents)
       (should (equal (length package-archive-contents) 2)))))
 
+(ert-deftest package-test-package-installed-p ()
+  "Test package-installed-p before and after package initialization."
+  (with-package-test ()
+    ;; Verify that `package-installed-p' evaluates true for a built-in
+    ;; package, in this case `project', before package initialization.
+    (should (not package--initialized))
+    (should (package-installed-p 'project nil))
+    (should (not (package-installed-p 'imaginary-package nil)))
+
+    ;; The results don't change after package initialization.
+    (package-initialize)
+    (should package--initialized)
+    (should (package-installed-p 'project nil))
+    (should (not (package-installed-p 'imaginary-package nil)))))
+
 (ert-deftest package-test-describe-package ()
   "Test displaying help for a package."
 
-- 
2.35.1

bug#56877: 29.0.50; Built-in packages are not always package-installed-p

[Lars Ingebrigtsen]

Matt Armstrong <matt@rfc20.org> writes:

> Second try.  Replaced "Foo" with a real docstring!

Thanks; pushed to Emacs 29 now.