bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov <dmitry@gutov.dev>]

On 21/04/2023 09:37, Eli Zaretskii wrote:

>> Why list the commands people use to install packages if we're talking
>> about upgrading them?
> 
> I said "to install and upgrade", not just "install".

I listed the upgrade commands.

> And I do think we should also consider commands that install packages
> because they are part of this issue, and in particular, should be
> consistent with the upgrade commands.  Also, package-install was the
> original cause of this thread.

Okay, we also have package-menu-mark-install and package-menu-execute.

>> Hopefully we'll decide that 'package-install'
>> won't upgrade packages because it hasn't done that in the past.
> 
> But it does upgrade non-built-in packages, doesn't it?

Apparently not. I didn't remember whether it does, and I deduced that it 
does just from reading this discussion previously, but it does not.

E.g. just now pressing 'U' after 'M-x list-packages' showed me that I 
have available upgrades for a lot of packages. But still if I evaluate

   (package-install 'sml-mode)

where sml-mode is one of said packages, I just get the message:

   ‘sml-mode’ is already installed

> And at least
> João (and I think others as well) expected it to upgrade Eglot even
> though it is now built in.

I think he wants that because this way (package-install 'eglot) and 
(use-package eglot :ensure t) could match the behavior of Emacs 28 with 
an empty init directory. Backward compatibility and all that.

But I think that's questionable, semantically. Given that Eglot is 
already "installed". Though, of course, one could argue that a bundled 
package is not exactly installed, but then we should change what 
'package-installed-p' does as well. And think hard before doing that.

>> The others would be package-update and package-update-all (can we please
>> do the rename now? I know they've been in for a year, but just as this
>> very discussion has shown, most people weren't even aware of their
>> existence).
> 
> If the list you provided covers all of the relevant commands and
> functions, fine.  Still, for completeness' sake, we should have the
> higher-level description as well, as a framework against which to
> examine the proposed solutions.  And I'm still not convinced that we
> covered all the relevant package-menu commands.

It might be easier if you just ask additional question during review.

>>>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
>>>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
>>>
>>> I'm okay with adding the same prefix argument to package-upgrade,
>>> which would then allow upgrading a built-in package.  IOW, a change
>>> similar to what we did in package-install -- provided that the change
>>> is safe enough to go into Emacs 29.
>>
>> If we agree it's a bug, why don't we just fix it?
> 
> Precisely because, as with package-install, this is a bug for some and
> a feature for others, depending on whether people do or don't want the
> built-in packages to be upgraded by default.

I'm having a hard time imagining someone evaluating (package-upgrade 
'eglot) without intention to upgrade it to the latest version. Or 
invoking it interactively with same argument, expecting a different result.

I don't think anyone will put it straight this way into their init 
script either.

>> 'package-update' is an
>> interactive function which itself it called from only one place:
>> 'package-update-all', and since the plan is to improve both, we can make
>> sure they only do what we ask of them: package-update will upgrade
>> builtins when invoked directly, and package-update-all will upgrade them
>> only when the builtin has been upgraded before (making it not a builtin
>> anymore), or a new user option is set.
> 
> This is one possibility, and it might make sense to some users.  But I
> don't think we can be sure it will make sense to an overwhelming
> majority of the users.

Hence the user option?

But okay, this particular addition, though trivial, we could probably 
postpone until Emacs 30, or even avoid adding at all. It is indeed not 
obvious that people will really need it, although

   (setq package-upgradable-builtins '(eglot use-package))
   (package-upgrade-all)
   ;; or M-x package-upgrade-all
   ;; or 'U' in the list-packages menu

seems like a plausible scenario for a certain kind of user. Because 
package-upgrade does not have a menu entry, or a button anywhere, 
whereas package-upgradable-builtins can be altered from the Customize UI.

>>>> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
>>>> Joao's scenario because use-package-ensure-elpa short-circuits when it
>>>> find that the package is installed ('package-installed-p' returns t). So
>>>> (use-package eglot :ensure t) does not upgrade Eglot even when
>>>> package-install-upgrade-built-in is t.
>>>
>>> I don't think (use-package eglot :ensure t) should automatically
>>> upgrade built-in packages.
>>
>> I don't think it should, either.
>>
>> But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.
> 
> No, it was supposed to allow the user to invoke package-install in a
> way that upgrades built-in packages, something that doesn't happen by
> default.  IOW, it was a partial solution to the original problem which
> started this discussion, see
> 
>    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5

Fair enough.

>>> We could make it do that if
>>> package-install-upgrade-built-in is non-nil -- again, if such a change
>>> could be safe enough.  If not, then the same workaround as for
>>> package-upgrade would do here, I suppose?
>>
>> What workaround would that be? use-package is not invoked interactively
>> -- there is no prefix argument to pass.
> 
> The workaround is to manually install the package from ELPA, once,
> using "C-u M-x package-install RET".

That's not the use-package workflow.

>>>> Whereas I think we originally only wanted that for Eglot and maybe
>>>> for use-package.
>>>
>>> "We" never did want that.  João did, for obvious reasons, but that was
>>> never my intent.  The issue is indeed more general: what should
>>> package-install and package.el in general do with built-in packages
>>> for which a newer version is on ELPA?
>>
>> It could continue doing what it's done before: when a package is already
>> installed, abort. For upgrading, we should recommend commands with
>> "upgrade" in their names.
> 
> If people agree with that, I don't think I'll object.  But this is in
> a sense a breaking change: package-install will only install, and
> thereafter users will need to use package-upgrade.  Some might dislike
> such behavior changes.  And we will need to make sure that all the
> available methods of "installing" do not "upgrade", for consistency.

Yeah, apparently it won't be a breaking change, or a change at all.

>>>> For this and other minor reasons I would suggest reverting
>>>> 580d8278c5f48.
>>>
>>> Not going to happen, not unless someone comes up with a better
>>> solution that is much better and still safe enough.  Personally, I
>>> don't believe such a solution exists, since we don't really know the
>>> answer to the above question.
>>
>> Could you specify which problem it's currently solving? Some particular
>> scenario.
> 
> The scenario which started this bug report, see the message whose URL
> I mentioned above.  IOW, we now allow users to explicitly request that
> package-install includes built-in packages in the list of candidates,
> and will therefore allow to upgrade them.

After we fix 'package-upgrade', users will be able to 'M-x 
package-upgrade RET eglot RET'.