bug#72453: 29.4; "use-package" macro does not load all expected related files

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 03 Aug 2024 14:58:08 -0700
> From:  Derek Upham via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> 
> This problem exists for the current emacs-30 branch HEAD.
> 
> The following "init.el" file contents provide a test case:
> 
>   (defun use-package-ensure-trace (name args state)
>     (warn "use-package-ensure-trace %s %s %s" name args state)
>     t)
> 
>   ;; (require 'use-package)
> 
>   (setq use-package-ensure-function 'use-package-ensure-trace)
>   (setq use-package-always-ensure t)
> 
>   (use-package faces)
>   (use-package ansi-color)
>   (use-package files :ensure t)
>   (use-package rx)
>   (use-package avoid)
> 
> Place it in a scratch directory "use-package-ensure-test" and invoke
> Emacs with, for example:
> 
>   emacs --init-directory use-package-ensure-test

Is this step necessary, i.e. does this problem has anything to do with
the non-default --init-directory setting?

> The init file sets use-package-always-ensure to true, and we expect
> trace warnings to show up for each use-package invocation.  In fact, we
> only see them for the third and beyond.
> 
>  Warning (emacs): use-package-ensure-trace files (t) nil
>  Warning (emacs): use-package-ensure-trace rx (t) nil
>  Warning (emacs): use-package-ensure-trace avoid (t) nil
> 
> Uncomment the "require" line in the init file and we see all of the
> traces that we expect:
> 
>  Warning (emacs): use-package-ensure-trace faces (t) nil
>  Warning (emacs): use-package-ensure-trace ansi-color (t) nil
>  Warning (emacs): use-package-ensure-trace files (t) nil
>  Warning (emacs): use-package-ensure-trace rx (t) nil
>  Warning (emacs): use-package-ensure-trace avoid (t) nil
> 
> The root cause is the structure of the "use-package/use-package*.el"
> libraries.  The "use-package" macro exists in "use-package-core.el", and
> has an autoload.  When Emacs evalutes a use-package form, Emacs loads
> "use-package-core.el" only.  It does not load any of the other
> "use-package*.el" components.  That means that Emacs doesn't evaluate
> any of the `add-to-list' forms at the bottom of "use-package-ensure.el".
> As a result, the use-package-defaults mechanism doesn't start out with
> the expected :ensure or :pin behaviors.
> 
> But the first time we refer to an :ensure property explicitly, that is
> enough to trigger the autoload of "use-package-handler/:ensure", which
> loads the file and sets up the defaults.  All later invocations run the
> :ensure checks as expected.
> 
> Moving the "use-package" macro into the top-level "use-package" file
> should fix the problem.  The "use-package" file explicitly loads the
> other files, which sets up all of the defaults.

John, any comments or suggestions?