bug#55305: 28.0.50: With async nativecomp, package manager fails to load hyperbole-autoloads.el before compilation

[Eli Zaretskii <eliz@gnu.org>]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: akrl@sdf.org,  55305@debbugs.gnu.org,  rswgnu@gmail.com,  rsw@gnu.org
> Date: Mon, 16 May 2022 12:40:41 -0400
> 
> Eli Zaretskii [2022-05-16 05:31:01] wrote:
> > My worry is about packages that do NOT depend on such macros.  Calling
> > package-activate-all will activate all the packages on the user's
> > system, and there's no way of knowing what those packages do at
> > activation time.  They can change variables, advise functions,
> > redefine commands, etc.
> 
> Yup.   I agree it can be worrisome, but:
> - Those packages are presumably already activated in the current session
>   and hence similarly affected the generation of the `.elc` files if the
>   `.elc` files were generated by `package-install`.

Two wrongs don't make one right.

Besides, as long as the *.elc files are produced by batch-compilation,
this will not happen.  And I'm talking only about batch compilation,
not about interactive compilation that the user does from a live
session.

> - Preserving the `load-path` like we already do also exposes similar
>   problems (some files may shadow the ones we need).

The effect of load-path is a much smaller one, and not propagating
load-path would cause much graver problems.

> > We have no idea what will be the state of the session after
> > doing that.
> 
> After the files are compiled, the session is killed (we're talking about
> an Emacs sub-process performing native compilation asynchronously), so
> I don't think we need to worry too much about that.

My worry is about the effects on the produced *.elc/*.eln files, so
killing the session doesn't help.

> >> I'm not claiming that calling `package-activate-all` is right for
> >> reasons of principle.  We sadly never clearly defined what it is that
> >> a package can count on.
> > Then we should do that _before_ we propose solutions that rely on
> > what's there, knowing that what's there was never intended to solve
> > this particular issue.
> 
> I don't understand what you're saying, here.

I'm sating we should define what is it the package can count on before
suggesting solutions based on lack of such a definition.

> There is a de-facto definition of what a package can count on (which is
> that its autoloads file has been loaded), which has been used ever since
> the inception of `package.el`.

That's when the package is loaded, not when it's compiled.

> I don't know enough of what you mean by "this particular issue" to judge
> whether this design was made to solve this issue or not, but I do know
> that the autoloads file is loaded on purpose before compiling the `.elc`
> files so as to make it more convenient to write the code of a package,
> just like we happily rely on `loaddefs.el` being loaded when working on
> Emacs's bundled code.

Loading the loaddefs file by some (as yet nonexistent) mechanism is
something I'm prepared to discuss with the purpose of maybe finding
some reasonable solution (though it isn't yet clear to me how to do
that ion general, for the reasons I explained several messages
up-thread).  What I'm NOT interested in considering is a call to
package-activate-all as part of compiling every Lisp file.

> > This cure is worse than the disease.
> 
> My proposed patch is not a cure.  The cure would be to compile from the
> `.elc` file (since the root of the problem is that we need those
> autoloads and the `load-path` to macroexpand the code, and the only
> reason why the native-compiler needs to macroexpand the code is because
> it (re)starts from the `.el` file instead of using the `.elc` file
> where no macro-expansion need to take place).

I'm also okay with leaving this issue unsolved until such time as we
have native compilation based on .elc files, and revisiting this then.