bug#72453: 29.4; "use-package" macro does not load all expected related files

[Derek Upham via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

This problem exists for the current emacs-30 branch HEAD.

The following "init.el" file contents provide a test case:

  (defun use-package-ensure-trace (name args state)
    (warn "use-package-ensure-trace %s %s %s" name args state)
    t)

  ;; (require 'use-package)

  (setq use-package-ensure-function 'use-package-ensure-trace)
  (setq use-package-always-ensure t)

  (use-package faces)
  (use-package ansi-color)
  (use-package files :ensure t)
  (use-package rx)
  (use-package avoid)

Place it in a scratch directory "use-package-ensure-test" and invoke
Emacs with, for example:

  emacs --init-directory use-package-ensure-test

The init file sets use-package-always-ensure to true, and we expect
trace warnings to show up for each use-package invocation.  In fact, we
only see them for the third and beyond.

 Warning (emacs): use-package-ensure-trace files (t) nil
 Warning (emacs): use-package-ensure-trace rx (t) nil
 Warning (emacs): use-package-ensure-trace avoid (t) nil

Uncomment the "require" line in the init file and we see all of the
traces that we expect:

 Warning (emacs): use-package-ensure-trace faces (t) nil
 Warning (emacs): use-package-ensure-trace ansi-color (t) nil
 Warning (emacs): use-package-ensure-trace files (t) nil
 Warning (emacs): use-package-ensure-trace rx (t) nil
 Warning (emacs): use-package-ensure-trace avoid (t) nil

The root cause is the structure of the "use-package/use-package*.el"
libraries.  The "use-package" macro exists in "use-package-core.el", and
has an autoload.  When Emacs evalutes a use-package form, Emacs loads
"use-package-core.el" only.  It does not load any of the other
"use-package*.el" components.  That means that Emacs doesn't evaluate
any of the `add-to-list' forms at the bottom of "use-package-ensure.el".
As a result, the use-package-defaults mechanism doesn't start out with
the expected :ensure or :pin behaviors.

But the first time we refer to an :ensure property explicitly, that is
enough to trigger the autoload of "use-package-handler/:ensure", which
loads the file and sets up the defaults.  All later invocations run the
:ensure checks as expected.

Moving the "use-package" macro into the top-level "use-package" file
should fix the problem.  The "use-package" file explicitly loads the
other files, which sets up all of the defaults.



In GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.42,
 cairo version 1.18.0) of 2024-06-29, modified by Debian built on sbuild
Windowing system distributor 'The X.Org Foundation', version 11.0.12101011
System Description: Debian GNU/Linux trixie/sid

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --with-cairo --with-x=yes
 --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -Werror=implicit-function-declaration
 -ffile-prefix-map=/build/reproducible-path/emacs-29.4+1=. -fstack-protector-strong
 -fstack-clash-protection -Wformat -Werror=format-security
 -fcf-protection -Wall' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
 LDFLAGS=-Wl,-z,relro'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2
XPM GTK3 ZLIB

-- 
Derek Upham
derek_upham@mailfence.com