bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii <eliz@gnu.org>]

> Date: Sat, 22 Apr 2023 02:12:23 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 21/04/2023 14:05, Eli Zaretskii wrote:
> 
> >>> And at least
> >>> João (and I think others as well) expected it to upgrade Eglot even
> >>> though it is now built in.
> >>
> >> I think he wants that because this way (package-install 'eglot) and
> >> (use-package eglot :ensure t) could match the behavior of Emacs 28 with
> >> an empty init directory. Backward compatibility and all that.
> > 
> > But if, with older Emacsen, package-install would refuse to update to
> > a newer version of Eglot if _some_ older version of Eglot is already
> > installed, then where's the problem with the default behavior of
> > package-install? it behaves exactly like in previous versions of
> > Emacs.  And why is this a problem for users of Eglot, if they couldn't
> > use package-install more than once for Eglot anyway?
> > 
> > Something is amiss here.
> 
> The above scenarios (package-install or (use-package eglot :ensure t)) 
> when ~/.emacs.d/elpa is empty, result in the very latest Eglot being 
> installed when using Emacs 28. And will result in something different 
> with Emacs 29 (not the latest version). That's not nothing: the CI 
> scripts will have to be updated, and the user instructions for reporting 
> problems will have to be made more complicated as well. Some 
> possibilities will simply be gone (the user won't be able to upgrade 
> Eglot to the very latest by deleting it from ~/.emacs.d/elpa and calling 
> package-install, for example). This *is* a non-backward compatible 
> change from the perspective of Eglot's maintainer.

We were talking about users installing and updating packages.  The CI
scenario doesn't belong here.  It is also much less important one
(test suites are always required to chase the changes in development).

Let's not complicate an already complicated set of issues by bringing
up unimportant secondary use cases.

> I, personally, don't really buy this kind of argument, but I figured you 
> might. After all, it's rather in line with reasoning we've seen voiced 
> around these parts many times ("X has worked this way for Y years, let's 
> never change it from now on"). Classic https://xkcd.com/1172/.

If you allude to my reasoning, then it is never that simplistic, and
always considers each case separately, not "by analogy".

> >> But I think that's questionable, semantically. Given that Eglot is
> >> already "installed". Though, of course, one could argue that a bundled
> >> package is not exactly installed, but then we should change what
> >> 'package-installed-p' does as well. And think hard before doing that.
> > 
> > I'd question why we have two commands instead of just one, but that's
> > probably water under the bridge at this point.
> 
> Either way would be fine, IMO, as long as the behavior is logical and 
> matches documentation. But having a separate command to upgrade now lets 
> us fix it separately without worry of breaking something more globally.

Except that, based on what we have (see below) ,we don't really have
an "upgrade" operation, we only have "install" and "delete"
(i.e. "uninstall").  So maybe we should preserve that, to minimize
problems and user surprise/confusion.

> > In interactive invocation, package-upgrade calls completing-read with
> > its 4th argument non-nil, so you cannot select a package which is not
> > in the collection returned by package--updateable-packages.  What I
> > meant above is to allow that collection to include built-in packages
> > as optional behavior.  I just tried invoking package-update for ElDoc,
> > and I get "No match" after typing "eldoc" to its prompt, although
> > eldoc version 1.14.0 is in the list presented by list-packages as
> > "available".
> 
> That's what I imagined: adding a new optional argument to 
> package--updateable-packages which would include builtins in the result.
> 
> And only pass it when called from package-upgrade.
> 
> Hopefully that's the kind of optional that you meant.

Yes, something like that.  Presumably activated by the same new option
introduced for package-installed.

> >>>> 'package-update' is an
> >>>> interactive function which itself it called from only one place:
> >>>> 'package-update-all', and since the plan is to improve both, we can make
> >>>> sure they only do what we ask of them: package-update will upgrade
> >>>> builtins when invoked directly, and package-update-all will upgrade them
> >>>> only when the builtin has been upgraded before (making it not a builtin
> >>>> anymore), or a new user option is set.
> >>>
> >>> This is one possibility, and it might make sense to some users.  But I
> >>> don't think we can be sure it will make sense to an overwhelming
> >>> majority of the users.
> >>
> >> Hence the user option?
> > 
> > Which one?  Are you suggesting to add a new one?  If so, why not use
> > the one we already added, package-install-upgrade-built-in?
> 
> The user option I was thinking of would probably be called a little 
> shorter: package-upgrade-built-in. And it would only affect the upgrader 
> commands.

We could rename the existing option, if the name is the problem.
Otherwise, I don't see why we would need two separate options: they do
the same job and have the same meaning from the user's POV.

> >> Because
> >> package-upgrade does not have a menu entry, or a button anywhere,
> >> whereas package-upgradable-builtins can be altered from the Customize UI.
> > 
> > Maybe marking a package in the list for update could be interpreted as
> > "upgrade that, no questions asked", and we will need no user options?
> 
> There is no handy "upgrade that" binding in the packages menu. The only 
> command that's available there related to upgrading is 
> package-menu-mark-upgrades, which does that to all packages (except 
> builtins).
> 
> To manually execute an upgrade of one package, one needs to both mark 
> the new version for installation (after first scrolling down the list to 
> find it), and mark the current version for deletion. This is what 
> currently an upgrade consists of.

Yes.  So users could manually mark an new version for installation,
and if needed mark the old version for deletion, thus indicating that
they want this upgrade.  No separate option to indicate the same is
needed.

> All this is to say, the first step (upgrading Eglot to the version from 
> ELPA) will be less user-friendly compared to the other UIs we have. But 
> it's probably manageable, especially if documented well.

I don't see why it would be less user-friendly.

> My point here, however, is that commit 580d8278c5f48 improved the 
> situation to a lesser degree that some people here might have expected.

One again: commit 580d8278c5f48 solved precisely the problem which
opened this bug report, nothing more, nothing less.  The rest are late
additions, which were not on the table when the original bug was
discussed.  So the above complaint is at least unfair.

> >>> If people agree with that, I don't think I'll object.  But this is in
> >>> a sense a breaking change: package-install will only install, and
> >>> thereafter users will need to use package-upgrade.  Some might dislike
> >>> such behavior changes.  And we will need to make sure that all the
> >>> available methods of "installing" do not "upgrade", for consistency.
> >>
> >> Yeah, apparently it won't be a breaking change, or a change at all.
> > 
> > I'm not sure, see above.
> > 
> > Also, when you mark packages for update from the list presented by
> > list-packages, the menu entry says
> > 
> >    i    Mark for Install
> > 
> > and its help-echo says "Mark a package for installation and move to
> > the next line", so we already confuse "install" and "upgrade".
> 
> But that's the thing: we don't have a separate action in the packages 
> list to "upgrade" a single package.

We agree.  I'm saying that your suggestion to make package-upgrade a
more important command is by itself a breaking change: there's no
"upgrade" operation, per se, in the current code.

> But we can conclude that we do have two different installation actions:
> 
>   - package-install which will install the latest version of a package, 
> but only if it's not installed;
>   - package-menu-mark-install, which will mark a specific version for 
> installation, disregarding whether some earlier version is already 
> installed; the previous version will remain installed still.

Which is again a breaking behavior change, AFAIU.  Is this a good idea
so late in the development of Emacs 29?

> >>>> Could you specify which problem it's currently solving? Some particular
> >>>> scenario.
> >>>
> >>> The scenario which started this bug report, see the message whose URL
> >>> I mentioned above.  IOW, we now allow users to explicitly request that
> >>> package-install includes built-in packages in the list of candidates,
> >>> and will therefore allow to upgrade them.
> >>
> >> After we fix 'package-upgrade', users will be able to 'M-x
> >> package-upgrade RET eglot RET'.
> > 
> > This goes back to the issue of having two confusingly-similar but
> > different commands, as mentioned above.  I guess we should first make
> > up our minds what, if anything, we want to do about this.
> 
> Sure.
> 
> But even if we decide that we want to eliminate that split, doing *that* 
> would really be a breaking change. I don't have a reasonable plan to 
> present for doing that in Emacs 29, so far.

There's no "split".  What I wanted to point out is that we don't seem
to have a clear vision of these two commands, since they are confusing
intertwined.  In fact, one could argue that package-upgrade in its
current form is simply a convenience shortcut for "delete old and
install new".