bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov <dmitry@gutov.dev>]

On 20/04/2023 13:02, Eli Zaretskii wrote:
>> Date: Thu, 20 Apr 2023 01:06:10 +0300
>> Cc: Eli Zaretskii <eliz@gnu.org>, arne_bab@web.de, jporterbugs@gmail.com,
>>   emacs-devel <emacs-devel@gnu.org>
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> OK then, I think have to re-evaluate my position on this. Previously, I
>> guess, I made some hasty conclusions from how the discussion went on
>> without refreshing the exact details about package.el and use-package
>> (the latter I never knew to begin with). Apologies.
>>
>> Eli, let me know if we should take this back to the bug tracker instead.
> 
> I've moved this back to the bug-tracker.  Please post all further
> replies about this particular issue, i.e. updating of built-in
> packages with package.el, to this bug and not to emacs-devel.

Noted.

>> So I would suggest to focus on functions that don't work as intended.
>> Namely:
>>
>> - Add a user option for the list of builtin packages which would be
>> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x
>> package-upgrade-all' (nee package-update-all). Maybe make it nil by
>> default, or maybe add 'eglot' to it. I don't have a strong opinion.
>>
>> - Fix 'M-x package-upgrade' (nee package-update) to suggest Eglot as one
>> of the options and actually perform the upgrade. That shouldn't require
>> changes to 'package-install' because, as we already know, the user can
>> already install a newer version of Eglot using the 'list-packages' menu
>> (and picking the exact version manually). That execution path is going
>> to go through 'package-install' as well, so it must be suitable already.
>>
>> - Revert 580d8278c5f48 because it creates odd semantics (upgrading
>> certain packages that are already installed but not others) and it
>> doesn't solve the issue with (use-package 'eglot :ensure t) anyway. We
>> could keep it, but seems like a half-measure that didn't make anyone
>> happy anyway. OTOH, it could minimize the rewrites of CI scripts.
> 
> I don't think we are ready to make any new decisions in this matter.
> I think we don't even have a comprehensive and detailed picture of the
> problems with updating/upgrading built-in packages in Emacs 29.
> People are still discovering facts and subtleties of various
> package.el commands and features, and are still arguing what exactly
> happens in this or that scenario.

True.

> So before we discuss solutions, we need a full and detailed
> description of the problems to solve.  If someone can do the footwork
> of collecting this information and posting it here, please do, and
> TIA.

I think I have made a fair attempt at this, though. Here's an update:

- package-upgrade (nee package-update) doesn't upgrade builtin packages 
that never been upgraded before. It's a bug. Hopefully not too hard to fix.

- package-menu-mark-upgrades and package-update-all don't upgrade them 
either. That's not necessarily a bug, but a problem nevertheless. A new 
user option could help.

- Fixing package-update should also obviate the need for eglot-update. 
Though perhaps the latter could still be useful as a single entry point 
to recommend to both users of Emacs 28 and 29+.

- The current fix (commit 580d8278c5f48) is not comprehensive WRT to 
Joao's scenario because use-package-ensure-elpa short-circuits when it 
find that the package is installed ('package-installed-p' returns t). So 
(use-package eglot :ensure t) does not upgrade Eglot even when 
package-install-upgrade-built-in is t.

- package-install-upgrade-built-in is not nuanced: if we suggest the 
users to set it to t, that can result in making _all_ builtin packages 
upgradable with 'package-install'. Whereas I think we originally only 
wanted that for Eglot and maybe for use-package. For this and other 
minor reasons I would suggest reverting 580d8278c5f48. But I suppose we 
could also try to make it more granular (e.g. turn the boolean option 
into a list). I'm not sure it's a good direction overall, however.

- According to Jim P., package pinning doesn't work for builtin packages 
either. Which could be a decent solution as well, e.g. putting something 
like this in the docs: (use-package eglot :ensure t :pin gnu) if the 
users want the Eglot version from ELPA -- and this form might even be 
compatible with Emacs 28. I'm not sure how difficult it would be to fix 
package pinning, however.

I haven't spent as much time on this bug as some others here, though, so 
corrections and additions are welcome.

> I will say up front that, given what I already read here and in the
> related thread on emacs-devel, there seem to be too many
> inconsistencies and dark corners in this, in particular when built-in
> packages are involved.  We will probably be unable to solve them in
> time for Emacs 29.1, certainly not all of them.  So don't raise your
> expectations too high.

The reasons for me to be hopeful are:

- The functions are propose to fix are "leaves": those that are supposed 
to be used interactively, without (many) known callers in-tree. E.g. 
fixing package-update-all shouldn't affect some other part of Emacs as 
much as changing package-install could.

- I imagine the diffs will be rather short. I haven't tried to take a 
stab at this yet, though, and if somebody wants to take the initiative, 
they're very welcome.

The reasons for me to be less hopeful:

- We seem to be unable to come to any agreement here. :-D

Joao's approval for the above list would go a long way.