bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii <eliz@gnu.org>]

> Date: Thu, 20 Apr 2023 16:39:20 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > So before we discuss solutions, we need a full and detailed
> > description of the problems to solve.  If someone can do the footwork
> > of collecting this information and posting it here, please do, and
> > TIA.
> 
> I think I have made a fair attempt at this, though. Here's an update:

Thanks.

What I miss in this description is something a bit higher-level: the
list of all the ways/commands/methods people use to install and
upgrade the packages.  We must have this list before our eyes to make
sure we review all the behaviors related to these activities, if we
want them all to eventually behave consistently.

E.g., you don't mention any commands invoked from the package menu.

> - package-upgrade (nee package-update) doesn't upgrade builtin packages 
> that never been upgraded before. It's a bug. Hopefully not too hard to fix.

I'm okay with adding the same prefix argument to package-upgrade,
which would then allow upgrading a built-in package.  IOW, a change
similar to what we did in package-install -- provided that the change
is safe enough to go into Emacs 29.

But note that AFAIU there's a relatively easy workaround for this:
install the later version of the package manually, just once.

> - package-menu-mark-upgrades and package-update-all don't upgrade them 
> either. That's not necessarily a bug, but a problem nevertheless. A new 
> user option could help.

A very relevant question is: can this wait till Emacs 30?

> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to 
> Joao's scenario because use-package-ensure-elpa short-circuits when it 
> find that the package is installed ('package-installed-p' returns t). So 
> (use-package eglot :ensure t) does not upgrade Eglot even when 
> package-install-upgrade-built-in is t.

I don't think (use-package eglot :ensure t) should automatically
upgrade built-in packages.  We could make it do that if
package-install-upgrade-built-in is non-nil -- again, if such a change
could be safe enough.  If not, then the same workaround as for
package-upgrade would do here, I suppose?

> - package-install-upgrade-built-in is not nuanced: if we suggest the 
> users to set it to t, that can result in making _all_ builtin packages 
> upgradable with 'package-install'.

It should be set to t only by users who indeed want all built-in
packages to be updated.  That's why the default is nil.

> Whereas I think we originally only wanted that for Eglot and maybe
> for use-package.

"We" never did want that.  João did, for obvious reasons, but that was
never my intent.  The issue is indeed more general: what should
package-install and package.el in general do with built-in packages
for which a newer version is on ELPA?  We don't yet know the answer to
that, and no hope of obtaining one before Emacs 29.1 is released
(barring any calamities).

> For this and other minor reasons I would suggest reverting
> 580d8278c5f48.

Not going to happen, not unless someone comes up with a better
solution that is much better and still safe enough.  Personally, I
don't believe such a solution exists, since we don't really know the
answer to the above question.

> But I suppose we could also try to make it more granular (e.g. turn
> the boolean option into a list). I'm not sure it's a good direction
> overall, however.

I don't think we should go in this direction, precisely because we
don't know it's a good one.

> - According to Jim P., package pinning doesn't work for builtin packages 
> either. Which could be a decent solution as well, e.g. putting something 
> like this in the docs: (use-package eglot :ensure t :pin gnu) if the 
> users want the Eglot version from ELPA -- and this form might even be 
> compatible with Emacs 28. I'm not sure how difficult it would be to fix 
> package pinning, however.

As long as this is optional behavior (and AFAIU it is), I won't
object, but again, provided that the addition of this will not touch
any other code in unsafe ways.

> The reasons for me to be hopeful are:
> 
> - The functions are propose to fix are "leaves": those that are supposed 
> to be used interactively, without (many) known callers in-tree. E.g. 
> fixing package-update-all shouldn't affect some other part of Emacs as 
> much as changing package-install could.

Experience until now in this thread indicates otherwise: almost every
suggested change touched low-level code with many callers.  It isn't
an accident that arriving at what was finally installed took so many
iterations for such a simple change.