Issues when upgrading built in packages

["Przemysław Kryger" <pkryger@gmail.com>]

Hello,

I have been doing some work involving upgrading built in packages from
ELPA.  Some of examples include: seq for older Emacs, org, transient, eglot.

I noticed that sometimes, these upgrades were failing because some files
couldn't be found or macro definitions were not available.  See at the bottom
of this message for concrete examples.

I've done some investigation and if I understood correctly the issue is caused
because `load-path' is updated only after the package is reloaded for the first
time.  The gist is that `package-install` calls `package-unpack` which in turn
does:

(1) Calls `package-activate-1' with :reload
(https://github.com/emacs-mirror/emacs/blob/bef9eeb/lisp/emacs-lisp/package.el#L1029). That
causes a call to `package--reload-previously-loaded' be made before loading
autoloads. The latter seems to be updating the `load-path' to include the newly
installed package directory, see
https://github.com/emacs-mirror/emacs/blob/bef9eeb/lisp/emacs-lisp/package.el#L907-L910.

(2) Byte compiles the package. The `load-path` already contains the newly
installed package directory.

(3) Calls `package--reload-previously-loaded', see
https://github.com/emacs-mirror/emacs/blob/bef9eeb/lisp/emacs-lisp/package.el#L1038

While I cant find example that fails on the most recent snapshot, it doesn't
seem that code has changed that much since Emacs-29.1, and I verified with
Edebug that relatively recent snapshot may still be affected.  At least
`load-path` doesn't include newly installed package directory when first reload
happens.

To my understanding this may also cause the very same macro issue that the
first package reloading is supposed to protect (or at least this is suggested
by this comment
https://github.com/emacs-mirror/emacs/blob/bef9eeb/lisp/emacs-lisp/package.el#L1026-L1028).
For an example, say there's a file foo.el that defines a macro `foo-macro` that
uses a macro `bar-macro` defined in file bar.el.  Both files are not byte
compile-able.  If both of these files exist in the old version of the package,
both defining the same macros, and it happens that file foo.el is loaded first,
then it will use an old version of `bar-macro`.  Is that correct?

When I added expected new package directory to `load-path' the issue doesn't
happen. I.e., I am able to `package-install` built-in packages from ELPA.

Thank you,
PK

Examples of failures:

(1) On Emacs-29.1 the seq-2.24 was not able to find seq-25:

(let ((debug-on-error t)
      (package-install-upgrade-built-in t)
      (desc (cadr (assq 'seq package-archive-contents))))
  (package-install desc))

Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "seq-25")
  require(seq-25)
  (if (version< emacs-version "25") (require 'seq-24) (require 'seq-25))
  eval-buffer(#<buffer  *load*-92326> nil "/home/runner/.emacs.d/elpa-..." nil t)  ; Reading at buffer position 1779
  load-with-code-conversion("/home/runner/.emacs.d/elpa-..." "/home/runner/.emacs.d/elpa-..." nil t)
  load("/home/runner/.emacs.d/elpa-..." nil t)
  #f(compiled-function (c) #<bytecode -0x142f67cbaf31655a>)(("/home/runner/.emacs.d/elpa-..." . 76))
  mapc(#f(compiled-function (c) #<bytecode -0x142f67cbaf31655a>) (("/home/runner/.emacs.d/elpa-..." . 76)))
  package--reload-previously-loaded(#s(package-desc :name seq :version (2 24) :summary "Sequence manipulation functions" :reqs nil :kind nil :archive nil :dir "/home/runner/.emacs.d/elpa-..." :extras ((:keywords "sequences") (:maintainer nil . "emacs-devel@gnu.org") (:authors ("Nicolas Petton" . "nicolas@petton.fr")) (:url . "https://elpa.gnu.org/packages/seq.html") (:commit . "27a90793a13f149121180e864fa53d68b9eac0b3")) :signed nil))
  package-activate-1(#s(package-desc :name seq :version (2 24) :summary "Sequence manipulation functions" :reqs nil :kind nil :archive nil :dir "/home/runner/.emacs.d/elpa-..." :extras ((:keywords "sequences") (:maintainer nil . "emacs-devel@gnu.org") (:authors ("Nicolas Petton" . "nicolas@petton.fr")) (:url . "https://elpa.gnu.org/packages/seq.html") (:commit . "27a90793a13f149121180e864fa53d68b9eac0b3")) :signed nil) :reload :deps)
  package-unpack(#s(package-desc :name seq :version (2 24) :summary "Sequence manipulation functions" :reqs nil :kind tar :archive "gnu" :dir nil :extras ((:keywords "sequences") (:maintainer nil . "emacs-devel@gnu.org") (:authors ("Nicolas Petton" . "nicolas@petton.fr")) (:url . "https://elpa.gnu.org/packages/seq.html") (:commit . "27a90793a13f149121180e864fa53d68b9eac0b3")) :signed nil))


(2) On Emacs-29.4 the org-9.7.14 was not able to find org-element-ast. This is
not happening every time, I guess due to order in which files are reloaded.

(let ((debug-on-error t)
      (package-install-upgrade-built-in t)
      (desc (cadr (assq 'org package-archive-contents))))
  (package-install desc))

Debugger entered--Lisp error: (error "Eager macro-expansion failure: (file-missing \"Cann...")
  signal(error ("Eager macro-expansion failure: (file-missing \"Cann..."))
  error("Eager macro-expansion failure: %S" (file-missing "Cannot open load file" "No such file or directory" "org-element-ast"))
  internal-macroexpand-for-load((defalias 'org-fold-core-region-folded-p #'(lambda (beg end &optional spec-or-alias) "Non-nil if the region between BEG and END is folde..." (org-with-point-at beg (catch :visible (while (< ... end) (unless ... ...) (goto-char ...)) t)))) t)
  eval-buffer(#<buffer  *load*-929646> nil "/home/runner/.emacs.d/elpa-..." nil t)  ; Reading at buffer position 38825
  load-with-code-conversion("/home/runner/.emacs.d/elpa-..." "/home/runner/.emacs.d/elpa-..." nil t)
  load("/home/runner/.emacs.d/elpa-..." nil t)
  #f(compiled-function (c) #<bytecode -0x142f67cbaf31655a>)(("/home/runner/.emacs.d/elpa-..." . 278))
  mapc(#f(compiled-function (c) #<bytecode -0x142f67cbaf31655a>) (("/home/runner/.emacs.d/elpa-..." . 273) ("/home/runner/.emacs.d/elpa-..." . 274) ("/home/runner/.emacs.d/elpa-..." . 277) ("/home/runner/.emacs.d/elpa-..." . 278) ("/home/runner/.emacs.d/elpa-..." . 279) ("/home/runner/.emacs.d/elpa-..." . 280) ("/home/runner/.emacs.d/elpa-..." . 285) ("/home/runner/.emacs.d/elpa-..." . 286) ("/home/runner/.emacs.d/elpa-..." . 287) ("/home/runner/.emacs.d/elpa-..." . 288) ("/home/runner/.emacs.d/elpa-..." . 289) ("/home/runner/.emacs.d/elpa-..." . 290) ("/home/runner/.emacs.d/elpa-..." . 291) ("/home/runner/.emacs.d/elpa-..." . 292) ("/home/runner/.emacs.d/elpa-..." . 294) ("/home/runner/.emacs.d/elpa-..." . 295) ("/home/runner/.emacs.d/elpa-..." . 296) ("/home/runner/.emacs.d/elpa-..." . 297) ("/home/runner/.emacs.d/elpa-..." . 299) ("/home/runner/.emacs.d/elpa-..." . 300) ("/home/runner/.emacs.d/elpa-..." . 301) ("/home/runner/.emacs.d/elpa-..." . 302) ("/home/runner/.emacs.d/elpa-..." . 303) ("/home/runner/.emacs.d/elpa-..." . 304) ("/home/runner/.emacs.d/elpa-..." . 305) ("/home/runner/.emacs.d/elpa-..." . 306) ("/home/runner/.emacs.d/elpa-..." . 307) ("/home/runner/.emacs.d/elpa-..." . 308) ("/home/runner/.emacs.d/elpa-..." . 309) ("/home/runner/.emacs.d/elpa-..." . 310) ("/home/runner/.emacs.d/elpa-..." . 311) ("/home/runner/.emacs.d/elpa-..." . 313) ("/home/runner/.emacs.d/elpa-..." . 314)))
  package--reload-previously-loaded(#s(package-desc :name org :version (9 7 14) :summary "Outline-based notes management and organizer" :reqs ((emacs (26 1))) :kind nil :archive nil :dir "/home/runner/.emacs.d/elpa-..." :extras ((:url . "https://orgmode.org") (:keywords "outlines" "hypermedia" "calendar" "text") (:maintainer "Bastien Guerry" . "bzg@gnu.org") (:authors ("Carsten Dominik" . "carsten.dominik@gmail.com")) (:commit . "49e0c4a74ad6c22b98702cfbcefff92a95af1146")) :signed nil))
  package-activate-1(#s(package-desc :name org :version (9 7 14) :summary "Outline-based notes management and organizer" :reqs ((emacs (26 1))) :kind nil :archive nil :dir "/home/runner/.emacs.d/elpa-..." :extras ((:url . "https://orgmode.org") (:keywords "outlines" "hypermedia" "calendar" "text") (:maintainer "Bastien Guerry" . "bzg@gnu.org") (:authors ("Carsten Dominik" . "carsten.dominik@gmail.com")) (:commit . "49e0c4a74ad6c22b98702cfbcefff92a95af1146")) :signed nil) :reload :deps)
  package-unpack(#s(package-desc :name org :version (9 7 14) :summary "Outline-based notes management and organizer" :reqs ((emacs (26 1))) :kind tar :archive "gnu" :dir nil :extras ((:url . "https://orgmode.org") (:keywords "outlines" "hypermedia" "calendar" "text") (:maintainer "Bastien Guerry" . "bzg@gnu.org") (:authors ("Carsten Dominik" . "carsten.dominik@gmail.com")) (:commit . "49e0c4a74ad6c22b98702cfbcefff92a95af1146")) :signed nil))