bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov <dmitry@gutov.dev>]

On 20/04/2023 17:25, Eli Zaretskii wrote:

> What I miss in this description is something a bit higher-level: the
> list of all the ways/commands/methods people use to install and
> upgrade the packages.  We must have this list before our eyes to make
> sure we review all the behaviors related to these activities, if we
> want them all to eventually behave consistently.

Why list the commands people use to install packages if we're talking 
about upgrading them? Hopefully we'll decide that 'package-install' 
won't upgrade packages because it hasn't done that in the past. Or if 
it's still going to do that because we don't want to revert commits, 
we'll just avoid touching that.

> E.g., you don't mention any commands invoked from the package menu.

The command which people use to upgrade packages from the package menu 
is called package-menu-mark-upgrades (bound to 'U'), mentioned here a 
few times. I also said that I hope it's *the* main way people use to 
upgrade packages.

The others would be package-update and package-update-all (can we please 
do the rename now? I know they've been in for a year, but just as this 
very discussion has shown, most people weren't even aware of their 
existence).

>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
> 
> I'm okay with adding the same prefix argument to package-upgrade,
> which would then allow upgrading a built-in package.  IOW, a change
> similar to what we did in package-install -- provided that the change
> is safe enough to go into Emacs 29.

If we agree it's a bug, why don't we just fix it? 'package-update' is an 
interactive function which itself it called from only one place: 
'package-update-all', and since the plan is to improve both, we can make 
sure they only do what we ask of them: package-update will upgrade 
builtins when invoked directly, and package-update-all will upgrade them 
only when the builtin has been upgraded before (making it not a builtin 
anymore), or a new user option is set.

> But note that AFAIU there's a relatively easy workaround for this:
> install the later version of the package manually, just once.

It's indeed a workaround that exists, and if we were satisfied with it, 
we would have closed this bug as "wontfix". But it's not a very friendly 
way to do that (the user must hunt for the appropriate version in the 
packages menu), and it's very poorly scriptable (you won't be able to 
provide an easy and readable snippet for the user to evaluate or put in 
their init script).

>> - package-menu-mark-upgrades and package-update-all don't upgrade them
>> either. That's not necessarily a bug, but a problem nevertheless. A new
>> user option could help.
> 
> A very relevant question is: can this wait till Emacs 30?

If you ask my opinion, then I already wrote it. The only thing that 
looks potentially complex is fixing the pinning of packages to archives. 
The rest should be fairly trivial.

Should we leave bugfixing and easy upgrading of builtin packages to 
Emacs 30? I suppose we could. Let me know if it's your foregone 
conclusion, then I'll just walk away now. This issue doesn't solve any 
problems for me personally, to be clear.

>> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
>> Joao's scenario because use-package-ensure-elpa short-circuits when it
>> find that the package is installed ('package-installed-p' returns t). So
>> (use-package eglot :ensure t) does not upgrade Eglot even when
>> package-install-upgrade-built-in is t.
> 
> I don't think (use-package eglot :ensure t) should automatically
> upgrade built-in packages.

I don't think it should, either.

But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.

> We could make it do that if
> package-install-upgrade-built-in is non-nil -- again, if such a change
> could be safe enough.  If not, then the same workaround as for
> package-upgrade would do here, I suppose?

What workaround would that be? use-package is not invoked interactively 
-- there is no prefix argument to pass.

>> - package-install-upgrade-built-in is not nuanced: if we suggest the
>> users to set it to t, that can result in making _all_ builtin packages
>> upgradable with 'package-install'.
> 
> It should be set to t only by users who indeed want all built-in
> packages to be updated.  That's why the default is nil.

Sounds unnecessarily dangerous.

>> Whereas I think we originally only wanted that for Eglot and maybe
>> for use-package.
> 
> "We" never did want that.  João did, for obvious reasons, but that was
> never my intent.  The issue is indeed more general: what should
> package-install and package.el in general do with built-in packages
> for which a newer version is on ELPA?

It could continue doing what it's done before: when a package is already 
installed, abort. For upgrading, we should recommend commands with 
"upgrade" in their names.

> We don't yet know the answer to
> that, and no hope of obtaining one before Emacs 29.1 is released
> (barring any calamities).

I'm not sure how we're going to get the answer to that question after 
29.1 is released, too. Similar to that thing with treesit modes and 
auto-mode-alist.

>> For this and other minor reasons I would suggest reverting
>> 580d8278c5f48.
> 
> Not going to happen, not unless someone comes up with a better
> solution that is much better and still safe enough.  Personally, I
> don't believe such a solution exists, since we don't really know the
> answer to the above question.

Could you specify which problem it's currently solving? Some particular 
scenario.

Then I could confidently propose an alternative (or not).

>> But I suppose we could also try to make it more granular (e.g. turn
>> the boolean option into a list). I'm not sure it's a good direction
>> overall, however.
> 
> I don't think we should go in this direction, precisely because we
> don't know it's a good one.

If we made it more granular, its use could become a reasonable option, 
then we'd be able to get some feedback from people who end up using it. 
I don't know what kind of people that will be, though.

>> - According to Jim P., package pinning doesn't work for builtin packages
>> either. Which could be a decent solution as well, e.g. putting something
>> like this in the docs: (use-package eglot :ensure t :pin gnu) if the
>> users want the Eglot version from ELPA -- and this form might even be
>> compatible with Emacs 28. I'm not sure how difficult it would be to fix
>> package pinning, however.
> 
> As long as this is optional behavior (and AFAIU it is), I won't
> object, but again, provided that the addition of this will not touch
> any other code in unsafe ways.

Yes, the use of pinning is and will remain optional. I'm not sure what 
code it will touch and how much, however. There is not a single function 
called "pin that" which needs to be fixed, the fix(es) will be somewhere 
on the common path(s) of package installation and upgrades, I think.

>> The reasons for me to be hopeful are:
>>
>> - The functions are propose to fix are "leaves": those that are supposed
>> to be used interactively, without (many) known callers in-tree. E.g.
>> fixing package-update-all shouldn't affect some other part of Emacs as
>> much as changing package-install could.
> 
> Experience until now in this thread indicates otherwise: almost every
> suggested change touched low-level code with many callers.  It isn't
> an accident that arriving at what was finally installed took so many
> iterations for such a simple change.

You might be correct at that, but note that I'm suggesting a slightly 
different set of goals than what had been discussed here before. So the 
changes will likewise need to be different.