bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Hello,

I've been meaning to report this for some time.  I really think this
should be fixed in Emacs 29 before it ships.

src/emacs -Q # Emacs 29

M-x package-install RET eglot RET

Echoes "No match", even though eglot 1.14 is available from GNU ELPA.
That's because Eglot is a :core ELPA package and it's already in Emacs
29, so package.el thinks there is nothing to install.

Evaluating (package-install 'eglot) also doesn't work.  Errors with
"'eglot' is already installed"

In Emacs 28, both these alternatives work as expected (because Eglot
isn't a :core package there).

It's not impossible to upgrade in Emacs 29, of course.  The only way I
know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
it with 'i' and confirm installation with 'x'.  But it is very awkward.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> It's not impossible to upgrade in Emacs 29, of course.  The only way I
> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
> it with 'i' and confirm installation with 'x'.  But it is very awkward.

Maybe we should add a `package-upgrade` command?


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca
> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 07 Apr 2023 23:12:43 +0100
> 
> I've been meaning to report this for some time.  I really think this
> should be fixed in Emacs 29 before it ships.
> 
> src/emacs -Q # Emacs 29
> 
> M-x package-install RET eglot RET
> 
> Echoes "No match", even though eglot 1.14 is available from GNU ELPA.
> That's because Eglot is a :core ELPA package and it's already in Emacs
> 29, so package.el thinks there is nothing to install.

Is :core new in Emacs 29?  If not, then this problem is not new: it
existed earlier for every :core package in ELPA, it just didn't exist
for Eglot.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2132 bytes --]

On Sat, Apr 8, 2023, 08:09 Eli Zaretskii <eliz@gnu.org> wrote:

> > Cc: monnier@iro.umontreal.ca
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 07 Apr 2023 23:12:43 +0100
> >
> > I've been meaning to report this for some time.  I really think this
> > should be fixed in Emacs 29 before it ships.
> >
> > src/emacs -Q # Emacs 29
> >
> > M-x package-install RET eglot RET
> >
> > Echoes "No match", even though eglot 1.14 is available from GNU ELPA.
> > That's because Eglot is a :core ELPA package and it's already in Emacs
> > 29, so package.el thinks there is nothing to install.
>
> Is :core new in Emacs 29?  If not, then this problem is not new: it
> existed earlier for every :core package in ELPA, it just didn't exist
> for Eglot.
>

Yes, it existed before. It's a bug but not a regression. But many of these
:core packages were primarily used as libraries supporting other packages,
not do much as user-facing, command-providing, UI-enhancing packages.

From what I've been seeing in the Eglot bug tracker there is a very
significant number of people using third-party package managers to install
and upgrade Eglot (straight, elpaca, ...)

I always tell them to report problems sweet package-install instead of
these managers, as I've seen more than my fair share of problems created by
these other managers.

My experience and the internal traffic reports tem me Eglot is a popular
Emacs package and people are not satisfied with just any old release. If
this isn't fixed, it'll be heaps of text to tell people how to upgrade. Not
the end of the world, but really awkward.

There is certainly code in Emacs 29 that allows upgrading Eglot or other
core packages but it's buried deep in that odd workflow. If the code for a
M-x package-upgrade, as Stefan suggests, is simple, then i think it should
be added to emacs-29.

Alternatively, there could be some kind of 'package-bugfixes' non-:core
package in elpa.git that adds this new command. Asking people to install
that package first is still easier than guiding then through that odd
workflow.

João

>

[-- Attachment #2: Type: text/html, Size: 3151 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> It's not impossible to upgrade in Emacs 29, of course.  The only way I
>> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
>> it with 'i' and confirm installation with 'x'.  But it is very awkward.
>
> Maybe we should add a `package-upgrade` command?

Just for :core packages or how would this differ from package-update?

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 8, 2023 at 11:43 AM Philip Kaludercic <philipk@posteo.net> wrote:
>
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> >> It's not impossible to upgrade in Emacs 29, of course.  The only way I
> >> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
> >> it with 'i' and confirm installation with 'x'.  But it is very awkward.
> >
> > Maybe we should add a `package-upgrade` command?
>
> Just for :core packages or how would this differ from package-update?

Just a note that I forgot M-x package-update existed.  I was
hoping it wouldn't suffer the same bug as package-install, but
alas it does.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Sat, Apr 8, 2023 at 11:43 AM Philip Kaludercic <philipk@posteo.net> wrote:
>>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>> >> It's not impossible to upgrade in Emacs 29, of course.  The only way I
>> >> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
>> >> it with 'i' and confirm installation with 'x'.  But it is very awkward.
>> >
>> > Maybe we should add a `package-upgrade` command?
>>
>> Just for :core packages or how would this differ from package-update?
>
> Just a note that I forgot M-x package-update existed.  I was
> hoping it wouldn't suffer the same bug as package-install, but
> alas it does.

As package-update just defers to package-install, I don't find this
surprising.  What you want is that package.el always treats a built-in
package as upgradable to a package from ELPA.  It seems like this should
be possible by adjusting the interactive spec of package-install and
modifying package-compute-transaction or even package-built-in-p.

> João

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

> Alternatively, there could be some kind of 'package-bugfixes' non-:core
> package in elpa.git that adds this new command. Asking people to install
> that package first is still easier than guiding then through that odd
> workflow.

I think that adding this into compat.el could be also justified.
With compat.el being ELPA dependency.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 8, 2023 at 3:49 PM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Alternatively, there could be some kind of 'package-bugfixes' non-:core
> > package in elpa.git that adds this new command. Asking people to install
> > that package first is still easier than guiding then through that odd
> > workflow.
>
> I think that adding this into compat.el could be also justified.
> With compat.el being ELPA dependency.

But compat.el is not :core right?  So Eglot cannot depend on it.

Also I think that's a bit beyond the purpose of compat, though
I don't oppose it. I really think package.el, like other package
managers, should learn to update itself.  It should be a :core
ELPA package itself.  But then that also requires this bug to
be fixed.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 8, 2023 at 3:42 PM Philip Kaludercic <philipk@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > On Sat, Apr 8, 2023 at 11:43 AM Philip Kaludercic <philipk@posteo.net> wrote:
> >>
> >> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> >>
> >> >> It's not impossible to upgrade in Emacs 29, of course.  The only way I
> >> >> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
> >> >> it with 'i' and confirm installation with 'x'.  But it is very awkward.
> >> >
> >> > Maybe we should add a `package-upgrade` command?
> >>
> >> Just for :core packages or how would this differ from package-update?
> >
> > Just a note that I forgot M-x package-update existed.  I was
> > hoping it wouldn't suffer the same bug as package-install, but
> > alas it does.
>
> As package-update just defers to package-install, I don't find this
> surprising.  What you want is that package.el always treats a built-in
> package as upgradable to a package from ELPA.  It seems like this should
> be possible by adjusting the interactive spec of package-install and
> modifying package-compute-transaction or even package-built-in-p.

Sounds good.  Can you propose a patch so we have something to look at
and test?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> I think that adding this into compat.el could be also justified.
>> With compat.el being ELPA dependency.
>
> But compat.el is not :core right?  So Eglot cannot depend on it.

Several packages in :core do depend on compat. The idea is to use
(require 'compat nil 'noerror), which does nothing in core, but enables
forward-compatibility features on ELPA.

See https://elpa.gnu.org/packages/doc/compat.html#Usage

> Also I think that's a bit beyond the purpose of compat, though
> I don't oppose it. I really think package.el, like other package
> managers, should learn to update itself.  It should be a :core
> ELPA package itself.  But then that also requires this bug to
> be fixed.

I agree that it is slightly out of scope of compat. But this particular
problem appears to be important enough as exception.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> As package-update just defers to package-install, I don't find this
> surprising.  What you want is that package.el always treats a built-in
> package as upgradable to a package from ELPA.  It seems like this should
> be possible by adjusting the interactive spec of package-install and
> modifying package-compute-transaction or even package-built-in-p.

I agree, tho I don't understand why you say "built-in" above.  Isn't the
problem also present for already-installed-but-out-of-date ELPA packages?


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

On Sat, Apr 8, 2023, 16:28 Ihor Radchenko <yantar92@posteo.net> wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> >> I think that adding this into compat.el could be also justified.
> >> With compat.el being ELPA dependency.
> >
> > But compat.el is not :core right?  So Eglot cannot depend on it.
>
> Several packages in :core do depend on compat. The idea is to use
> (require 'compat nil 'noerror), which does nothing in core, but enables
> forward-compatibility features on ELPA.
>

Sure, that works, your right. And it's a good idea.

But it doesn't solve _this_ problem even if compat.el patches package.el,
because once you get Emacs 29, you're locked out of ELPA Eglot (unless you
take that very circuitous route).

> Also I think that's a bit beyond the purpose of compat, though
> > I don't oppose it. I really think package.el, like other package
> > managers, should learn to update itself.  It should be a :core
> > ELPA package itself.  But then that also requires this bug to
> > be fixed.
>
> I agree that it is slightly out of scope of compat. But this particular
> problem appears to be important enough as exception.
>

Yes, but more importantly, i don't think there's a fix to this bug that can
be done fully ELPA-side.

João

>

[-- Attachment #2: Type: text/html, Size: 2279 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 8, 2023 at 4:45 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > As package-update just defers to package-install, I don't find this
> > surprising.  What you want is that package.el always treats a built-in
> > package as upgradable to a package from ELPA.  It seems like this should
> > be possible by adjusting the interactive spec of package-install and
> > modifying package-compute-transaction or even package-built-in-p.
>
> I agree, tho I don't understand why you say "built-in" above.  Isn't the
> problem also present for already-installed-but-out-of-date ELPA packages?

I bit the bullet and went after this in package.el.  The answer is no,
it's not present for those packages.

But Philip's imagined fix seems to be more complicated than it needs
to be. Here's a simple patch that works well in my tests.  It makes
M-x package-update also update built-in-packages.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..f54b6f39e40 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2243,11 +2243,16 @@ package-update
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
-        (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
+         (nonbuiltin (assq package package-alist)))
+    (cond (nonbuiltin
+           (let ((desc (cadr nonbuiltin)))
+             (if (package-vc-p desc)
+                 (package-vc-update desc)
+               (package-delete desc 'force)
+               (package-install package 'dont-select))))
+          (t
+           (package-install
+            (cadr (assq package package-archive-contents)))))))

 (defun package--updateable-packages ()
   ;; Initialize the package system to get the list of package
@@ -2261,10 +2266,14 @@ package--updateable-packages
                  (assq (car elt) package-archive-contents)))
             (and available
                  (version-list-<
-                  (package-desc-version (cadr elt))
+                  (if (vectorp (cdr elt)) (aref (cdr elt) 0)
+                    (package-desc-version (cadr elt)))
                   (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+          (and (consp (cdr elt))
+               (package-desc-p (cadr elt))
+               (package-vc-p (cadr elt)))))
+    (seq-union package-alist package--builtins
+               (lambda (a b) (eq (car a) (car b)))))))

 ;;;###autoload
 (defun package-update-all (&optional query)

The only thing that's slightly inelegant/hard-to-follow is that
the format of package-alist is different than package--builtins.
Lots of consp, cdr-taking, vectorp and so on.  Besides that, it's
a question of taking the union of the two sets and operating on that.

This also relies on seq-union's undocumented behavior of keeping
the first of any duplicates.  Feel free to rewrite.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Sat, Apr 8, 2023 at 4:45 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>
>> > As package-update just defers to package-install, I don't find this
>> > surprising.  What you want is that package.el always treats a built-in
>> > package as upgradable to a package from ELPA.  It seems like this should
>> > be possible by adjusting the interactive spec of package-install and
>> > modifying package-compute-transaction or even package-built-in-p.
>>
>> I agree, tho I don't understand why you say "built-in" above.  Isn't the
>> problem also present for already-installed-but-out-of-date ELPA packages?
>
> I bit the bullet and went after this in package.el.  The answer is no,
> it's not present for those packages.
>
> But Philip's imagined fix seems to be more complicated than it needs
> to be. Here's a simple patch that works well in my tests.  It makes
> M-x package-update also update built-in-packages.
>
> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index f92afe56b76..f54b6f39e40 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -2243,11 +2243,16 @@ package-update
>    (let* ((package (if (symbolp name)
>                        name
>                      (intern name)))
> -         (pkg-desc (cadr (assq package package-alist))))
> -    (if (package-vc-p pkg-desc)
> -        (package-vc-update pkg-desc)
> -      (package-delete pkg-desc 'force)
> -      (package-install package 'dont-select))))
> +         (nonbuiltin (assq package package-alist)))
> +    (cond (nonbuiltin
> +           (let ((desc (cadr nonbuiltin)))
> +             (if (package-vc-p desc)
> +                 (package-vc-update desc)
> +               (package-delete desc 'force)
> +               (package-install package 'dont-select))))
> +          (t
> +           (package-install
> +            (cadr (assq package package-archive-contents)))))))
>
>  (defun package--updateable-packages ()
>    ;; Initialize the package system to get the list of package
> @@ -2261,10 +2266,14 @@ package--updateable-packages
>                   (assq (car elt) package-archive-contents)))
>              (and available
>                   (version-list-<
> -                  (package-desc-version (cadr elt))
> +                  (if (vectorp (cdr elt)) (aref (cdr elt) 0)
> +                    (package-desc-version (cadr elt)))
>                    (package-desc-version (cadr available)))))
> -          (package-vc-p (cadr (assq (car elt) package-alist)))))
> -    package-alist)))
> +          (and (consp (cdr elt))
> +               (package-desc-p (cadr elt))
> +               (package-vc-p (cadr elt)))))
> +    (seq-union package-alist package--builtins
> +               (lambda (a b) (eq (car a) (car b)))))))

Will this not affect `package-update-all'?  I don't if we want that the
command installs all packages from ELPA that it can find.

>  ;;;###autoload
>  (defun package-update-all (&optional query)
>
> The only thing that's slightly inelegant/hard-to-follow is that
> the format of package-alist is different than package--builtins.
> Lots of consp, cdr-taking, vectorp and so on.  Besides that, it's
> a question of taking the union of the two sets and operating on that.

That is necessary complexity, so I don't think there is any way around it.

> This also relies on seq-union's undocumented behavior of keeping
> the first of any duplicates.  Feel free to rewrite.
>
> João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2275 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Will this not affect `package-update-all'?  I don't if we want that the
> command installs all packages from ELPA that it can find.

Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
updates the built-in and the manually installed packages that can be
updated.  It _doesn't_ install any packages that weren't installed yet,
of course.

In this case, in a bare emacs-29/src/emacs -Q, it updates the Eglot
package, the Eldoc package and few others.  I think this is exactly what
M-x package-update-all is supposed to do (but see notes at the end of
this email).

So I think my change to the existing package-update and
package--updateable-packages fixes the bug cleanly.

>> Lots of consp, cdr-taking, vectorp and so on.  Besides that, it's
>> a question of taking the union of the two sets and operating on that.
> That is necessary complexity, so I don't think there is any way around it.

There is, but that's just an improvement, in this case to the
type-starved 'package--bi-desc' structure and the 'package--builtins'
built by finder.el.  A common structure format should be used.  Or,
better yet, CLOS.

Anyway, with the version of the patch I posted earlier, there are the 6
(six) packages updated currently from a "bare" emacs 29.

    (csharp-mode eglot eldoc jsonrpc transient verilog-mode)

Of these, csharp-mode and transient are mistakes.  But that's not my
patch's fault :-) 

- csharp-mode.el is merely missing version information, so package.el
  thinks that the ELPA version supersedes it (when in fact it doesn't:
  it's older). The patch adds version information to csharp-mode.el to
  fix that.

- transient.el has version information but in a header that finder.el
  doesn't recognize.  The patch has a minimal fix for that, too.

So the final patch that I'm proposing for emacs 29 is attached.  M-x
package-update-all fixes those cases and correctly finds and updates 4
packages to their newest released versions, exactly as it should.

   (eglot eldoc jsonrpc verilog-mode)

Eli, what do you think?  Who is package.el's main maintainer?  Everyone?
Lars added M-x package-update (for Emacs 29) so I'm pinging him as well.

João


[-- Attachment #2: 0001-Add-ability-to-update-built-in-packages-bug-62720.patch --]
[-- Type: text/x-patch, Size: 4009 bytes --]

From 65e811a0fcf9ffd1f12b8b2a2d9d8a0474543b36 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20T=C3=A1vora?= <joaotavora@gmail.com>
Date: Tue, 11 Apr 2023 11:52:46 +0100
Subject: [PATCH] Add ability to update built-in packages (bug#62720)

Previously package.el's M-x package-update command completely ignored
built-in packages.  With this patch in place, it updates them along
with any other manually installed non-built-in packages, as long as
the version available from ELPA is newer.

To prevent misupdates of the 'transient.el' and 'csharp-mode.el'
packages, which are built into emacs-29, version information is now
correctly collected from these two.

* lisp/finder.el (finder-compile-keywords): Be aware of
"Package-Version" header.

* lisp/emacs-lisp/package.el (package-update): Rework.
(package--updateable-packages): Rework.

* lisp/progmodes/csharp-mode.el: Add version information.
---
 lisp/emacs-lisp/package.el    | 26 ++++++++++++++++++--------
 lisp/finder.el                |  2 +-
 lisp/progmodes/csharp-mode.el |  1 +
 3 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..286583100c3 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2243,11 +2243,16 @@ package-update
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
-        (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
+         (nonbuiltin (assq package package-alist)))
+    (cond (nonbuiltin
+           (let ((desc (cadr nonbuiltin)))
+             (if (package-vc-p desc)
+                 (package-vc-update desc)
+               (package-delete desc 'force)
+               (package-install package 'dont-select))))
+          (t
+           (package-install
+            (cadr (assq package package-archive-contents)))))))
 
 (defun package--updateable-packages ()
   ;; Initialize the package system to get the list of package
@@ -2261,10 +2266,15 @@ package--updateable-packages
                  (assq (car elt) package-archive-contents)))
             (and available
                  (version-list-<
-                  (package-desc-version (cadr elt))
+                  (if (vectorp (cdr elt))
+                      (aref (cdr elt) 0)
+                    (package-desc-version (cadr elt)))
                   (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+          (and (consp (cdr elt))
+               (package-desc-p (cadr elt))
+               (package-vc-p (cadr elt)))))
+    (seq-union package-alist package--builtins
+               (lambda (a b) (eq (car a) (car b)))))))
 
 ;;;###autoload
 (defun package-update-all (&optional query)
diff --git a/lisp/finder.el b/lisp/finder.el
index 5aec0149b89..ddc6d6f03da 100644
--- a/lisp/finder.el
+++ b/lisp/finder.el
@@ -231,7 +231,7 @@ finder-compile-keywords
 		  summary  (or (cdr
 			        (assq package finder--builtins-descriptions))
 			       (lm-synopsis))
-		  version  (lm-header "version")))
+		  version  (or (lm-header "package-version") (lm-header "version"))))
 	  (when summary
 	    (setq version (or (ignore-errors (version-to-list version))
                               (alist-get package package--builtin-versions)))
diff --git a/lisp/progmodes/csharp-mode.el b/lisp/progmodes/csharp-mode.el
index 47cd13e7fdb..cd045cd14d1 100644
--- a/lisp/progmodes/csharp-mode.el
+++ b/lisp/progmodes/csharp-mode.el
@@ -8,6 +8,7 @@
 ;;              Jostein Kjønigsen <jostein@kjonigsen.net>
 ;; Created    : September 2022
 ;; Keywords   : c# languages oop
+;; Version    : 3.0.0
 
 ;; This file is part of GNU Emacs.
 
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: eliz@gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>,
>   62720@debbugs.gnu.org, Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 11 Apr 2023 12:02:48 +0100
> 
> So the final patch that I'm proposing for emacs 29 is attached.  M-x
> package-update-all fixes those cases and correctly finds and updates 4
> packages to their newest released versions, exactly as it should.
> 
>    (eglot eldoc jsonrpc verilog-mode)
> 
> Eli, what do you think?

I'd prefer it to go to master, not to emacs-29.  The problem is not
grave enough and OTOH the workaround is simple enough.  So changing
package.el in such non-trivial ways is not something I'd like to risk
now.

> Who is package.el's main maintainer?  Everyone?
> Lars added M-x package-update (for Emacs 29) so I'm pinging him as well.

I think it's mostly Philip and Stefan.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: eliz@gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>,
>>   62720@debbugs.gnu.org, Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 11 Apr 2023 12:02:48 +0100
>> 
>> So the final patch that I'm proposing for emacs 29 is attached.  M-x
>> package-update-all fixes those cases and correctly finds and updates 4
>> packages to their newest released versions, exactly as it should.
>> 
>>    (eglot eldoc jsonrpc verilog-mode)
>> 
>> Eli, what do you think?
>
> I'd prefer it to go to master, not to emacs-29.  The problem is not
> grave enough and OTOH the workaround is simple enough.  So changing
> package.el in such non-trivial ways is not something I'd like to risk
> now.

Please reconsider.  If we do this, than Emacs 29 users will be almost
locked out of upgrading Eglot and a lot of other built-in packages.
I'll have to teach people that workaround in the manual, where such
workarounds don't really belong.

Note that Eglot moved from ELPA to core, but it had (and has) many users
on Emacs 26, 27 and 28.  Eglot is getting regular new features in
master, the bundled Emacs 29 version is now already pretty "old".  When
migrating to Emacs 29, these users will expect to keep being able to
update to the latest version, and will likely be baffled that it doesn't
work as smoothly as it used to.

M-x package-update and M-x package-update-all are new in Emacs 29.
They're buggy, so why ship them buggy?  The change I'm proposing it not
really "non-trivial".  I can walk you or anybody through the code, or
write tests if that would improve the outlook.

>> Who is package.el's main maintainer?  Everyone?
>> Lars added M-x package-update (for Emacs 29) so I'm pinging him as well.
>
> I think it's mostly Philip and Stefan.

Let's hear from them, to see if there's some kind of subtlety I might
have missed.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Tue, 11 Apr 2023 13:52:36 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Eli, what do you think?
> >
> > I'd prefer it to go to master, not to emacs-29.  The problem is not
> > grave enough and OTOH the workaround is simple enough.  So changing
> > package.el in such non-trivial ways is not something I'd like to risk
> > now.
> 
> Please reconsider.  If we do this, than Emacs 29 users will be almost
> locked out of upgrading Eglot and a lot of other built-in packages.
> I'll have to teach people that workaround in the manual, where such
> workarounds don't really belong.

OK, I looked closer at the patch and the code involved in this, and
also re-read this discussion.  I cannot agree with installing your
patch, as submitted, on the emacs-29 branch, sorry.  It modifies code
that affects "normal" invocations of package-update, and also numerous
other functions in package.el (via the change in
package--updateable-packages), in ways that are very hard for me to
audit.  It is hard to audit because there are parts of it that read
like some kind of "black magic":

> +         (nonbuiltin (assq package package-alist)))

Why is the return value of assq the sign that the package is
"nonbuiltin"?

> +    (cond (nonbuiltin
> +           (let ((desc (cadr nonbuiltin)))
> +             (if (package-vc-p desc)
> +                 (package-vc-update desc)
> +               (package-delete desc 'force)
> +               (package-install package 'dont-select))))
> +          (t
> +           (package-install
> +            (cadr (assq package package-archive-contents)))))))

Why the different way of calling package-install for "built-in"
packages?

> -                  (package-desc-version (cadr elt))
> +                  (if (vectorp (cdr elt))
> +                      (aref (cdr elt) 0)
> +                    (package-desc-version (cadr elt)))

What is the significance of the (vectorp (cdr elt)) test?

> -          (package-vc-p (cadr (assq (car elt) package-alist)))))
> -    package-alist)))
> +          (and (consp (cdr elt))
> +               (package-desc-p (cadr elt))
> +               (package-vc-p (cadr elt)))))
> +    (seq-union package-alist package--builtins
> +               (lambda (a b) (eq (car a) (car b)))))))

What is the significance of the (consp (cdr elt)) test?  And why do we
need to add package--builtins to the list?

How am I supposed to assess the safety of this patch, given all this
semi-obfuscated code, and given that I'm not the every-day maintainer
of package.el and am not familiar with all the quirks of its code?
(It is quite possible that this obfuscated nature of the code is not
your fault, but is caused by how package.el is implemented.  In which
case I hope that we could clean up the code of package.el on master to
allow updating :core packages more seamlessly and with simpler code.)

OTOH, the workaround you described in

   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5

doesn't sound too awful to me, given that this problem exists for a
while and is not specific to Eglot.

However, if we still want to have a better solution that will be safe
enough to be installed on emacs-29, I can suggest two alternatives:

 . add a prefix argument to package-update, which would mean to update
   a package unconditionally, even if it is a built-in or of an older
   version, and make this special case be handled by code that is
   completely independent of the code we have in package-update now,
   so that the "normal" case is unaffected; or
 . add a new command, say, package-update-core-package, which will
   then be used only for :core packages

OK?

> M-x package-update and M-x package-update-all are new in Emacs 29.

They might be new, but package-update was virtually unmodified since a
year ago, during which time it was used by many people, and so its
current code can be considered to be well tested.  Your modifications
are by contrast completely new.

> The change I'm proposing it not really "non-trivial".  I can walk
> you or anybody through the code, or write tests if that would
> improve the outlook.

See above.  Given the problems I mentioned, I'm allowed to doubt that
you yourself understand the changes well enough to vouch for them.
And even if you did vouch, my gray hair won't believe you.  So I
prefer to go for much safer, if slightly less clean, changes.  I hope
one of the two alternatives I suggested will be acceptable.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Tue, Apr 11, 2023 at 6:55 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Please reconsider.  If we do this, than Emacs 29 users will be almost
> > locked out of upgrading Eglot and a lot of other built-in packages.
> > I'll have to teach people that workaround in the manual, where such
> > workarounds don't really belong.
>
> OK, I looked closer at the patch and the code involved in this, and
> also re-read this discussion.  I cannot agree with installing your
> patch, as submitted, on the emacs-29 branch, sorry.  It modifies code
> that affects "normal" invocations of package-update, and also numerous
> other functions in package.el (via the change in
> package--updateable-packages),

I don't understand.  package--updateable-packages is an internal helper
that only has two users, both of which I tested.  That's not "numerous".

> in ways that are very hard for me to
> audit.  It is hard to audit because there are parts of it that read
> like some kind of "black magic":
>
> > +         (nonbuiltin (assq package package-alist)))
>
> Why is the return value of assq the sign that the package is
> "nonbuiltin"?

Because package-alist only contains packages that were installed
by the user explicitly.

>
> > +    (cond (nonbuiltin
> > +           (let ((desc (cadr nonbuiltin)))
> > +             (if (package-vc-p desc)
> > +                 (package-vc-update desc)
> > +               (package-delete desc 'force)
> > +               (package-install package 'dont-select))))
> > +          (t
> > +           (package-install
> > +            (cadr (assq package package-archive-contents)))))))
>
> Why the different way of calling package-install for "built-in"
> packages?

1. Because built-in packages cannot be deleted.  2. Because built-in
packages aren't described the same way that explicitly installed packages.
The description of a built-in package is much poorer in information.
To make package-install work with a built-in package, you have to give it
the richer description of the package that you want to install, fresh
from package-archive-contents.

> > -                  (package-desc-version (cadr elt))
> > +                  (if (vectorp (cdr elt))
> > +                      (aref (cdr elt) 0)
> > +                    (package-desc-version (cadr elt)))
>
> What is the significance of the (vectorp (cdr elt)) test?

It tells if the current element being iterated has, in its cdr
an object of type package--bi-desc.  That struct, is implemented
via a vector, and so, unfortunately has no recognizer predicate.

> > -          (package-vc-p (cadr (assq (car elt) package-alist)))))
> > -    package-alist)))
> > +          (and (consp (cdr elt))
> > +               (package-desc-p (cadr elt))
> > +               (package-vc-p (cadr elt)))))
> > +    (seq-union package-alist package--builtins
> > +               (lambda (a b) (eq (car a) (car b)))))))
>
> What is the significance of the (consp (cdr elt)) test?  And why do we
> need to add package--builtins to the list?

package-alist's form is

   ((SYM PACKAGE-DESC)...)

while package--builtins is

   ((SYM . PACKAGE--BI-DESC) ...)

> How am I supposed to assess the safety of this patch, given all this
> semi-obfuscated code, and given that I'm not the every-day maintainer
> of package.el and am not familiar with all the quirks of its code?
> (It is quite possible that this obfuscated nature of the code is not
> your fault, but is caused by how package.el is implemented.  In which
> case I hope that we could clean up the code of package.el on master to
> allow updating :core packages more seamlessly and with simpler code.)

Yes, quite so.  That was my point to Philip earlier: this code is awful
to read, but when you read it, you'll notice that it's not really
rocket science going on there.  That's why I think this is simple enough
patch to go for emacs 29.

I do hope Stefan and Philip can chime in.

Do note that if this change goes to master and not to emacs-29, people
will only be effectively testing the new functionality when the emacs-30
branch is cut.

> OTOH, the workaround you described in
>
>    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5
>
> doesn't sound too awful to me, given that this problem exists for a
> while and is not specific to Eglot.

As I explained, I don't think there were ever :core packages
as popular as Eglot.  There is also the fact that many people are
using non-package.el package managers, which is a maintenance burden
for me.  I always recommend package.el, the official package manager,
since I don't have the resources to learn about those other package
managers (some of which have brought problems in the past).

That workaround is awful to use, BTW.  It's quite slow,
(M-x package-list-packages takes ages, like almost a minute here).
Then you have to C-s and find a million false positives eglot-something
packages and then you have to know the `i` and `x` shortcuts, which
aren't really something Emacs newcomers know about.

On the other hand, M-x package-update gives you a completion
list of the packages you have already.

> See above.  Given the problems I mentioned, I'm allowed to doubt that
> you yourself understand the changes well enough to vouch for them.
> And even if you did vouch, my gray hair won't believe you.  So I
> prefer to go for much safer, if slightly less clean, changes.  I hope
> one of the two alternatives I suggested will be acceptable.

If this change can't go into emacs-29, I think it's better to add
an M-x eglot-update to eglot.el.  That's discoverable, easy to remember
and the absolute safest, as package.el is absolutely unchanged.

(defun eglot-update () "Update Eglot to latest version."
  (interactive)
  (unless package-archive-contents (package-refresh-contents))
  (package-install (assq 'eglot package-archive-contents)))

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 11 Apr 2023 19:31:09 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> > See above.  Given the problems I mentioned, I'm allowed to doubt that
> > you yourself understand the changes well enough to vouch for them.
> > And even if you did vouch, my gray hair won't believe you.  So I
> > prefer to go for much safer, if slightly less clean, changes.  I hope
> > one of the two alternatives I suggested will be acceptable.
> 
> If this change can't go into emacs-29, I think it's better to add
> an M-x eglot-update to eglot.el.

That's the worst of all worlds.

What is the problem with the two possible solutions I suggested?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
Status "incompat"?  If I press RET on its name, I see

  Package eglot is incompatible.

       Status: Incompatible because it depends on uninstallable packages.

Why is that?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
> updates the built-in and the manually installed packages that can be
> updated.  It _doesn't_ install any packages that weren't installed yet,
> of course.

Hmm... it might make sense to treat builtins specially in this respect.
Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
with the idea that `M-x package-update-all` would update all the
`:core` packages.

I'm not sure why, admittedly, but I think it comes down to the fact that
the first upgrade of a `:core` package from GNU ELPA feels to me more
like an "install" than an "upgrade".


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

On Tue, Apr 11, 2023, 20:44 Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
> > updates the built-in and the manually installed packages that can be
> > updated.  It _doesn't_ install any packages that weren't installed yet,
> > of course.
>
> Hmm... it might make sense to treat builtins specially in this respect.
> Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
> with the idea that `M-x package-update-all` would update all the
> `:core` packages.
>
> I'm not sure why, admittedly, but I think it comes down to the fact that
> the first upgrade of a `:core` package from GNU ELPA feels to me more
> like an "install" than an "upgrade".


Can't argue against feelings :)

João

>
>

[-- Attachment #2: Type: text/html, Size: 1425 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Tue, Apr 11, 2023, 19:51 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Tue, 11 Apr 2023 19:31:09 +0100
> > Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org,
>
> >       larsi@gnus.org
> >
> > > See above.  Given the problems I mentioned, I'm allowed to doubt that
> > > you yourself understand the changes well enough to vouch for them.
> > > And even if you did vouch, my gray hair won't believe you.  So I
> > > prefer to go for much safer, if slightly less clean, changes.  I hope
> > > one of the two alternatives I suggested will be acceptable.
> >
> > If this change can't go into emacs-29, I think it's better to add
> > an M-x eglot-update to eglot.el.
>
> That's the worst of all worlds.
>

Why? It's the safest option. Absolutely no package.el regression possible,
and doesn't solve a problem where you don't think I've exists.

>
> What is the problem with the two possible solutions I suggested?
>

They are incongruent and not very user friendly IMO.

João

>

[-- Attachment #2: Type: text/html, Size: 2306 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

LOn Tue, Apr 11, 2023, 21:08 João Távora <joaotavora@gmail.com> wrote:

> .
>> >
>> > If this change can't go into emacs-29, I think it's better to add
>> > an M-x eglot-update to eglot.el.
>>
>> That's the worst of all worlds.
>>
>
> Why? It's the safest option. Absolutely no package.el regression possible,
> and doesn't solve a problem where you don't think I've exists.
>

Sorry, I meant to write "where you don't think one exists".

>
>> What is the problem with the two possible solutions I suggested?
>>
>
> They are incongruent and not very user friendly IMO.
>

Admittedly, eglot-update is also incongruent, of course, but at least that
problem is confined to a single package.

João

>

[-- Attachment #2: Type: text/html, Size: 2130 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Tue, Apr 11, 2023, 19:53 Eli Zaretskii <eliz@gnu.org> wrote:

> Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
> Status "incompat"?  If I press RET on its name, I see
>
>   Package eglot is incompatible.
>
>        Status: Incompatible because it depends on uninstallable packages.
>
> Why is that?
>

No idea. Is this with our without my patch?

João

>

[-- Attachment #2: Type: text/html, Size: 952 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 11/04/2023 14:02, João Távora wrote:
> Philip Kaludercic<philipk@posteo.net>  writes:
> 
>> Will this not affect `package-update-all'?  I don't if we want that the
>> command installs all packages from ELPA that it can find.
> Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
> updates the built-in and the manually installed packages that can be
> updated.  It_doesn't_  install any packages that weren't installed yet,
> of course.

On a related note, do you know whether we upgrade the built-in packages 
when the user presses 'U' in the list-packages buffer?

Using the command package-menu-mark-upgrades, that is.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I'm not sure why, admittedly, but I think it comes down to the fact that
>> the first upgrade of a `:core` package from GNU ELPA feels to me more
>> like an "install" than an "upgrade".
> Can't argue against feelings :)

What can I say: I just know I'm right!

More seriously, I agree it's not a very strong/compelling argument.
I just wanted to mention it, in case I'm not the only weirdo.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Philip Kaludercic <philipk@posteo.net>,  eliz@gnu.org,
>   62720@debbugs.gnu.org,  Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 11 Apr 2023 15:44:22 -0400
> 
> > Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
> > updates the built-in and the manually installed packages that can be
> > updated.  It _doesn't_ install any packages that weren't installed yet,
> > of course.
> 
> Hmm... it might make sense to treat builtins specially in this respect.
> Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
> with the idea that `M-x package-update-all` would update all the
> `:core` packages.
> 
> I'm not sure why, admittedly, but I think it comes down to the fact that
> the first upgrade of a `:core` package from GNU ELPA feels to me more
> like an "install" than an "upgrade".

Which means my proposal of adding a new command
package-update-core-package makes more and more sense: we will
probably need to handle such packages specially for any number of
reasons, more so as we go with our plan to have them only on ELPA and
"bundle" them when the release is tarred.  So having such a command
now will be a good investment for the future.

Philip, if this makes sense, would you please add such a command on
the emacs-29 branch?  If the exact purpose and effects of the command
are not clear yet, let's talk about it and finalize that.

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 11 Apr 2023 21:08:32 +0100
> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
>  > If this change can't go into emacs-29, I think it's better to add
>  > an M-x eglot-update to eglot.el.
> 
>  That's the worst of all worlds.
> 
> Why? It's the safest option. Absolutely no package.el regression possible, and doesn't solve a problem
> where you don't think I've exists.

From your POV of the Eglot maintainer, it might make sense.  But from
my POV, it doesn't: the problem here is general, and a solution is at
hand that will give you what you want and also support all the other
core packages.

>  What is the problem with the two possible solutions I suggested?
> 
> They are incongruent and not very user friendly IMO.

The one which proposed a new command is a natural generalization of
your eglot-update proposal, so I think it is as user-friendly as it
gets.  As for congruency, Stefan just expressed his intuition about
that, and I tend to agree with him: upgrades of core packages should
indeed be handled by separate command(s) with somewhat different rules
and perhaps also a different UI.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 11 Apr 2023 21:28:26 +0100
> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Lars Ingebrigtsen <larsi@gnus.org>, 62720@debbugs.gnu.org
> 
>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
>  Status "incompat"?  If I press RET on its name, I see
> 
>    Package eglot is incompatible.
> 
>         Status: Incompatible because it depends on uninstallable packages.
> 
>  Why is that?
> 
> No idea. Is this with our without my patch?

Without.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I'm not sure why, admittedly, but I think it comes down to the fact that
>>> the first upgrade of a `:core` package from GNU ELPA feels to me more
>>> like an "install" than an "upgrade".
>> Can't argue against feelings :)
>
> What can I say: I just know I'm right!
>
> More seriously, I agree it's not a very strong/compelling argument.
> I just wanted to mention it, in case I'm not the only weirdo.

I would agree as well, and my argument would be that by installing a
core package from ELPA, I am indicating that I want to follow the newest
version of a package as it is developed upstream, which might be
perceived as less stable/predictable than tracking the development
through Emacs updates.

>         Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Philip Kaludercic <philipk@posteo.net>,  eliz@gnu.org,
>>   62720@debbugs.gnu.org,  Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 11 Apr 2023 15:44:22 -0400
>> 
>> > Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
>> > updates the built-in and the manually installed packages that can be
>> > updated.  It _doesn't_ install any packages that weren't installed yet,
>> > of course.
>> 
>> Hmm... it might make sense to treat builtins specially in this respect.
>> Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
>> with the idea that `M-x package-update-all` would update all the
>> `:core` packages.
>> 
>> I'm not sure why, admittedly, but I think it comes down to the fact that
>> the first upgrade of a `:core` package from GNU ELPA feels to me more
>> like an "install" than an "upgrade".
>
> Which means my proposal of adding a new command
> package-update-core-package makes more and more sense: 

I am not sure that "core package" is necessarily a term or concept that
users would be familiar with.  Or at least I have seen users confused
about the concept online, not realising that a core package can be
updated via ELPA.

How about something along the lines of `package-update-from-built-in'?

>                                                        we will
> probably need to handle such packages specially for any number of
> reasons, more so as we go with our plan to have them only on ELPA and
> "bundle" them when the release is tarred.  

Could you elaborate on this plan.  Or perhaps I just lack the background
to see how these issues are related?

>                                            So having such a command
> now will be a good investment for the future.
>
> Philip, if this makes sense, would you please add such a command on
> the emacs-29 branch?  If the exact purpose and effects of the command
> are not clear yet, let's talk about it and finalize that.

Can do.  This would prompt the user for a core package that hasn't been
installed from ELPA yet, and would make sure the package instead of the
core code is loaded?  If there is no difference in version between the
ELPA and the core package, should we say anything?

> Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Tue, 11 Apr 2023 21:08:32 +0100
>> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org, 
>> 	Lars Ingebrigtsen <larsi@gnus.org>
>> 
>>  > If this change can't go into emacs-29, I think it's better to add
>>  > an M-x eglot-update to eglot.el.
>> 
>>  That's the worst of all worlds.
>> 
>> Why? It's the safest option. Absolutely no package.el regression possible, and doesn't solve a problem
>> where you don't think I've exists.
>
> From your POV of the Eglot maintainer, it might make sense.  But from
> my POV, it doesn't: the problem here is general, and a solution is at
> hand that will give you what you want and also support all the other
> core packages.

Your solution doesn't "give me what I want".  If I add 'eglot-update' it
will work as a single command on every Emacs version from Emacs 26.3
onwards.  This new command you're proposing is for Emacs 29 only (and
will presumably be deprecated soon).

So "what I want(ed)" is for `M-x package-install RET eglot` to keep
working smoothly as it did up to here and forever onwards.  That's want
many package managers do (the "install" verb upgrades, or offers to
upgrade).  That not being possible, I had settled for a decent working
`M-x package-update RET eglot` instead.

So this is "what I want": smooth user experience with no new commands or
at least simple ones that don't require understanding emacs dev
concepts.

>>  What is the problem with the two possible solutions I suggested?
>> 
>> They are incongruent and not very user friendly IMO.
>
> The one which proposed a new command is a natural generalization of
> your eglot-update proposal, so I think it is as user-friendly as it
> gets.  As for congruency, Stefan just expressed his intuition about
> that, and I tend to agree with him: upgrades of core packages should
> indeed be handled by separate command(s) with somewhat different rules
> and perhaps also a different UI.

I think 'package-update-core-package' is just unfortunate, because the
regular user doesn't care what the heck is core, and can you blame her?

I can't stop you from adding it, of course.  Have you thought how it
should behave when the package is no longer a core package, i.e. has
already updated?  I.e. should 'package-update-core-package' update a
non-core package in certain situations?  If so, then we're inches away
from 'M-x package-update-any-package', which is just a fixed `M-x
package-update` as I proposed.  If not, then it's really quite strange
to have to run one command for one update and another command for the
next one.

I have to ask (though I can guess the answer): may I add 'eglot-update'
anyway to emacs-29 as a no-brainer shortcut in the meantime?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  joaotavora@gmail.com,
>   62720@debbugs.gnu.org,  larsi@gnus.org
> Date: Wed, 12 Apr 2023 07:44:20 +0000
> 
> >> I'm not sure why, admittedly, but I think it comes down to the fact that
> >> the first upgrade of a `:core` package from GNU ELPA feels to me more
> >> like an "install" than an "upgrade".
> >
> > Which means my proposal of adding a new command
> > package-update-core-package makes more and more sense: 
> 
> I am not sure that "core package" is necessarily a term or concept that
> users would be familiar with.  Or at least I have seen users confused
> about the concept online, not realising that a core package can be
> updated via ELPA.

I'm okay with an alternative terminology, like "built-in".

> How about something along the lines of `package-update-from-built-in'?

Not sure about the "from" part.  How about package-update-built-in
instead? or maybe package-upgrade-builtin (since built-in packages
will only ever be upgraded, at least normally)?

> >                                                        we will
> > probably need to handle such packages specially for any number of
> > reasons, more so as we go with our plan to have them only on ELPA and
> > "bundle" them when the release is tarred.  
> 
> Could you elaborate on this plan.  Or perhaps I just lack the background
> to see how these issues are related?

The plan is to provide a command similar to package-update, but which
will be used only for built-in packages which are also on ELPA
(a.k.a. "core packages").  The internals will probably be different,
since package-update offers only non-built-in packages as completion
candidates, the data structures package.el uses for built-in and
non-built-in packages are different, and installing an updated package
should NOT delete the bundled one, just install the newer version so
that it is used in preference to the bundled one.  Other than that,
the UI is supposed to be the same as in package-update.

If something else is unclear, please ask.

> >                                            So having such a command
> > now will be a good investment for the future.
> >
> > Philip, if this makes sense, would you please add such a command on
> > the emacs-29 branch?  If the exact purpose and effects of the command
> > are not clear yet, let's talk about it and finalize that.
> 
> Can do.  This would prompt the user for a core package that hasn't been
> installed from ELPA yet, and would make sure the package instead of the
> core code is loaded?

Yes.  But when you say "hasn't been installed from ELPA yet", do you
mean that once it has been installed, users will have to use a
different command for updating it further?  That might be confusing;
I'd prefer that the same command is used for all the updates of
built-in packages.  For example, if we'd later want to support
downgrading back to the bundled version, the implementation will be
different for built-in packages, so having a separate command will
save us some maintenance headaches.

> If there is no difference in version between the ELPA and the core
> package, should we say anything?

Probably just user-error, I guess?  Or maybe optionally provide a way
of forcing the update, like if the user wants to reinstall the package
anyway for whatever reasons?

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Wed, 12 Apr 2023 08:58:23 +0100
> 
> > From your POV of the Eglot maintainer, it might make sense.  But from
> > my POV, it doesn't: the problem here is general, and a solution is at
> > hand that will give you what you want and also support all the other
> > core packages.
> 
> Your solution doesn't "give me what I want".  If I add 'eglot-update' it
> will work as a single command on every Emacs version from Emacs 26.3
> onwards.  This new command you're proposing is for Emacs 29 only (and
> will presumably be deprecated soon).

Since you started by proposing a patch to package.el, that solution
had the same issues with older Emacsen.  IOW, that is a separate
problem.  In particular, Eglot is not bundled in those older Emacsen,
so package.el should support updating Eglot just fine for those older
versions.

> So this is "what I want": smooth user experience with no new commands or
> at least simple ones that don't require understanding emacs dev
> concepts.

If we want the solution be general, rather than having a separate
update command for each core package, then there's a limit to what we
can do to give this smooth user experience to users of older Emacsen.
However, I don't think in this case there's a problem, see above.

> I think 'package-update-core-package' is just unfortunate, because the
> regular user doesn't care what the heck is core, and can you blame her?

I disagree with your assessment.  Moreover, I think such a command is
needed anyway, for reasons other than the ones which prompted this bug
report.

> I can't stop you from adding it, of course.  Have you thought how it
> should behave when the package is no longer a core package, i.e. has
> already updated?

Yes, see my other messages.

> I have to ask (though I can guess the answer): may I add 'eglot-update'
> anyway to emacs-29 as a no-brainer shortcut in the meantime?

I'd prefer not to have package-specific upgrade commands.  I hope we
will soon add to package.el on emacs-29 a new command that will allow
users to update core packages, including Eglot, and that will solve
the problem for users of Emacs 29 and later, where Eglot is bundled.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 9:19 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Your solution doesn't "give me what I want".  If I add 'eglot-update' it
> > will work as a single command on every Emacs version from Emacs 26.3
> > onwards.  This new command you're proposing is for Emacs 29 only (and
> > will presumably be deprecated soon).
>
> Since you started by proposing a patch to package.el, that solution
> had the same issues with older Emacsen.

In onler Emacsen, M-x package-install works.  I was going to also propose
that M-x package-install on Emacs 29 offers to upgrade.  But if even a
"fixed" (IMO, of course) M-x package-update isn't accepted, then I won't
even bother proposing fixing M-x package-install on Emacs 29.

> IOW, that is a separate
> problem.  In particular, Eglot is not bundled in those older Emacsen,
> so package.el should support updating Eglot just fine for those older
> versions.

Yes, but the command that two users using, say, Emacs 28 and Emacs 29
will be different.  And happen to think that's unfortunate.

> > So this is "what I want": smooth user experience with no new commands or
> > at least simple ones that don't require understanding emacs dev
> > concepts.
>
> If we want the solution be general, rather than having a separate
> update command for each core package, then there's a limit to what we
> can do to give this smooth user experience to users of older Emacsen.
> However, I don't think in this case there's a problem, see above.

I don't think this "limit" really exists.  As far as I understand, if
I had detected this problem earlier or your perception of the stability
of my patch for emacs-29 were different, then this problem would be well
on its way to non-existence.

> > I think 'package-update-core-package' is just unfortunate, because the
> > regular user doesn't care what the heck is core, and can you blame her?
>
> I disagree with your assessment.  Moreover, I think such a command is
> needed anyway, for reasons other than the ones which prompted this bug
> report.
>
> > I can't stop you from adding it, of course.  Have you thought how it
> > should behave when the package is no longer a core package, i.e. has
> > already updated?
>
> Yes, see my other messages.

I can't well understand what the behaviour will be, I'll wait to see the
final version.

> > I have to ask (though I can guess the answer): may I add 'eglot-update'
> > anyway to emacs-29 as a no-brainer shortcut in the meantime?
>
> I'd prefer not to have package-specific upgrade commands.  I hope we
> will soon add to package.el on emacs-29 a new command that will allow
> users to update core packages, including Eglot, and that will solve
> the problem for users of Emacs 29 and later, where Eglot is bundled.

FTR I would also 100% also "prefer not to have package-specific upgrade
commands" :-) That would be my preference too. But again, I really have
to ask :-), may I add it or would you revert the commit immediately?
Again, I'm just looking to have a congruent, simple way to tell users
how to ensure latest Eglot version without needing to know if they have
Emacs 29, 27 or 47.  This command, while also not being my first "preference",
fulfills that requirement.  Can we have it?  It's only 4 lines of code.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Tue, 11 Apr 2023 21:28:26 +0100
>> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 
>> 	Lars Ingebrigtsen <larsi@gnus.org>, 62720@debbugs.gnu.org
>> 
>>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
>>  Status "incompat"?  If I press RET on its name, I see
>> 
>>    Package eglot is incompatible.
>> 
>>         Status: Incompatible because it depends on uninstallable packages.
>> 
>>  Why is that?
>> 
>> No idea. Is this with our without my patch?
>
> Without.

I can't reproduce this in a recent emacs-29.  Here's what I start Emacs
with

HOME=`mktemp -d` && src/emacs -Q -f list-packages

I get:

     Status: Available from gnu -- [Install]

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 11/04/2023 14:02, João Távora wrote:
>> Philip Kaludercic<philipk@posteo.net>  writes:
>> 
>>> Will this not affect `package-update-all'?  I don't if we want that the
>>> command installs all packages from ELPA that it can find.
>> Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
>> updates the built-in and the manually installed packages that can be
>> updated.  It_doesn't_  install any packages that weren't installed yet,
>> of course.
>
> On a related note, do you know whether we upgrade the built-in
> packages when the user presses 'U' in the list-packages buffer?
>
> Using the command package-menu-mark-upgrades, that is.

Nope, doesn't work, doesn't do anything to those packages.  I wish it
did, of course.

I also don't understand why this is using separate, but repeated logic
from package-update.  What is the difference between "upgrade" and
"update", if any?  Is "upgrade" more powerful?

BTW, I also noticed that Eglot's version on Emacs 29 is garbled.  I had
wanted 1.12-emacs29 to somehow show that it is Emacs 29 specific.  But
version-to-list doesn't like it and the package shows up as version
"nil" in package--builtins.  Will just change it to 1.12.29, which is
less perceptible but works fine (none of this makes any difference to
this bug, of course).

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  larsi@gnus.org,
>   62720@debbugs.gnu.org
> Date: Wed, 12 Apr 2023 10:18:08 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
> >>  Status "incompat"?  If I press RET on its name, I see
> >> 
> >>    Package eglot is incompatible.
> >> 
> >>         Status: Incompatible because it depends on uninstallable packages.
> >> 
> >>  Why is that?
> >> 
> >> No idea. Is this with our without my patch?
> >
> > Without.
> 
> I can't reproduce this in a recent emacs-29.  Here's what I start Emacs
> with
> 
> HOME=`mktemp -d` && src/emacs -Q -f list-packages
> 
> I get:
> 
>      Status: Available from gnu -- [Install]

I did just

  emacs -Q
  M-x list-packages RET

and I still see what I described.

I get the same if I point HOME to a scratch empty directory.

Maybe it's a Windows thing? why are some dependencies marked as "not
available"?

  Package eglot is incompatible.

       Status: Incompatible because it depends on uninstallable packages.
      Archive: gnu
      Version: 1.14
       Commit: 8125d4cfc5605ead9102b7d823c4241029eb76cc
      Summary: The Emacs Client for LSP servers
     Requires: emacs-26.3, jsonrpc-1.0.16, flymake-1.2.1,
		 project-0.9.8 (not available), xref-1.6.2 (not available),
		 eldoc-1.14.0, seq-2.23 (not available),
		 external-completion-0.1 (not available)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 09:51:08 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> > > I have to ask (though I can guess the answer): may I add 'eglot-update'
> > > anyway to emacs-29 as a no-brainer shortcut in the meantime?
> >
> > I'd prefer not to have package-specific upgrade commands.  I hope we
> > will soon add to package.el on emacs-29 a new command that will allow
> > users to update core packages, including Eglot, and that will solve
> > the problem for users of Emacs 29 and later, where Eglot is bundled.
> 
> FTR I would also 100% also "prefer not to have package-specific upgrade
> commands" :-) That would be my preference too. But again, I really have
> to ask :-), may I add it or would you revert the commit immediately?

Why do you insist on threats of reverting being the only efficient
means of communications about these matters?  This is supposed to be a
collaborative effort, not a confrontation.

If we are going to have a special command for upgrading core packages
(and it currently looks like we are), then having an additional
Eglot-specific command for a similar purpose would be against our
common goal of having a UX that changes as little as possible, given
the restrictions.  What else can I say to express clearly that I would
like such a command to not be installed?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 11:23 AM Eli Zaretskii <eliz@gnu.org> wrote:

> Why do you insist on threats of reverting being the only efficient
> means of communications about these matters?  This is supposed to be a
> collaborative effort, not a confrontation.

Of course.  But we don't agree Eli, and that's a normal part of
working with someone.  No need to use the words "threads" or
"confrontation", or suggest this nasty ill will on my end. No one
is threatening anybody. I was just asking nicely if I can add the
command: you're the head maintainer, you have to make a decision at
some point.

> If we are going to have a special command for upgrading core packages
> (and it currently looks like we are), then having an additional
> Eglot-specific command for a similar purpose would be against our
> common goal of having a UX that changes as little as possible, given
> the restrictions.  What else can I say to express clearly that I would
> like such a command to not be installed?

I wouldn't want it either, but it seems the least bad of all solutions
here.  Package.el is in my honest opinion very inconsistent already.
M-x package-isntall does one thing, package-update does another, then
there's the "package upgrades" in the menu which use yet different
logic.  I am beginning to understand why people are flocking away to
straight.el and elpaca.el.  Given this poor situation, I think that
adding yet _more_ complexity to package.el UX is worse.

But I don't want to stop you from doing that if you really think it's
the correct way.  No confrontation here, Eli.  And in fact, if I'm wrong
and people love `package-update-core-package` and find it consistent,
then `eglot-update` will just fall into oblivion and be forgotten,
and we can deprecate it just like we did `eglot-manual`.  So please,
work with me and give me the benefit of the doubt that this 4-line
helper command in Eglot won't hurt anybody and might actually
help somebody.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> the restrictions.  What else can I say to express clearly that I would
> like such a command to not be installed?

Had another idea: what about this very tiny patch, then?  It makes `M-x
package-install` work for installing a :core package.  This also rhymes
exactly with Stefan's intution/feeling that :core packages need to be
"installed" to promote them to installable.  The current M-x
package-install recommendation could remain flawlessly and then you can
do whatever you think is best for M-x package-update & friends.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..134505a96af 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2164,8 +2164,10 @@ package-installed-p
        (and pkg-descs
             (version-list-<= min-version
                              (package-desc-version (car pkg-descs)))))
-     ;; Also check built-in packages.
-     (package-built-in-p package min-version)))))
+     ;; Also check built-in packages.  But if min-version is nil, just
+     ;; act as if a package-desc had been passed (bug#62720)
+     (and min-version (package-built-in-p package min-version))))))
 
 (defun package-download-transaction (packages)
   "Download and install all the packages in PACKAGES.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 11:38:35 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> On Wed, Apr 12, 2023 at 11:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Why do you insist on threats of reverting being the only efficient
> > means of communications about these matters?  This is supposed to be a
> > collaborative effort, not a confrontation.
> 
> Of course.  But we don't agree Eli, and that's a normal part of
> working with someone.

We are talking here about relative merits of two mostly equivalent
commands, one of which is more general than the other and will allow
users to perform what you want, among other things.  The disagreement
is about how serious is the problem of having the UI change for Emacs
users from Emacs 29 onwards.  (Btw, having a special Eglot command
also constitutes a change in the UI.)  When this kind of matters are
in disagreement (as opposed to design and implementation of package
which you develop and maintain), I expect my opinion to have a
slightly larger weight.

> But I don't want to stop you from doing that if you really think it's
> the correct way.  No confrontation here, Eli.  And in fact, if I'm wrong
> and people love `package-update-core-package` and find it consistent,
> then `eglot-update` will just fall into oblivion and be forgotten,
> and we can deprecate it just like we did `eglot-manual`.  So please,
> work with me and give me the benefit of the doubt that this 4-line
> helper command in Eglot won't hurt anybody and might actually
> help somebody.

I think adding such a command would be a mistake, so I'm asking you
not to add such a command to Emacs.  Please!

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Wed, 12 Apr 2023 12:00:04 +0100
> 
> Had another idea: what about this very tiny patch, then?  It makes `M-x
> package-install` work for installing a :core package.  This also rhymes
> exactly with Stefan's intution/feeling that :core packages need to be
> "installed" to promote them to installable.  The current M-x
> package-install recommendation could remain flawlessly and then you can
> do whatever you think is best for M-x package-update & friends.

This has the same problem: it modifies a function that is called in
too many places.  package-installed-p has half a dozen callers in
package.el alone.  The change is tiny, but what about its
implications on every use case where it is involved?

So I still think a special command for updating a core package will be
a better, safer solution, and the reasons for that are more than just
this one, as mentioned up-thread.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 12:07 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
> >   larsi@gnus.org
> > Date: Wed, 12 Apr 2023 12:00:04 +0100
> >
> > Had another idea: what about this very tiny patch, then?  It makes `M-x
> > package-install` work for installing a :core package.  This also rhymes
> > exactly with Stefan's intution/feeling that :core packages need to be
> > "installed" to promote them to installable.  The current M-x
> > package-install recommendation could remain flawlessly and then you can
> > do whatever you think is best for M-x package-update & friends.
>
> This has the same problem: it modifies a function that is called in
> too many places.  package-installed-p has half a dozen callers in
> package.el alone.  The change is tiny, but what about its
> implications on every use case where it is involved?

What about them?  I volunteer to help test whichever cases you (or
anyone else) think may be problematic.

Can we put this tiny patch on master then?  Maybe there's half a
chance for a backport until the Emacs 29.1 release, or a quarter
of a chance that some future Emacs 29.2 will also get it.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 12:15:11 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> > This has the same problem: it modifies a function that is called in
> > too many places.  package-installed-p has half a dozen callers in
> > package.el alone.  The change is tiny, but what about its
> > implications on every use case where it is involved?
> 
> What about them?  I volunteer to help test whichever cases you (or
> anyone else) think may be problematic.

We will never be able to reveal them all, let alone test them, before
Emacs 29.1 is released.

> Can we put this tiny patch on master then?

Yes, if Philip and Stefan don't object.  But, since there will be a
command for updating core packages, doesn't this go against your
desire not to change the UX?

> Maybe there's half a chance for a backport until the Emacs 29.1
> release, or a quarter of a chance that some future Emacs 29.2 will
> also get it.

I don't think there is such a chance, and I'm not sure we even want
that, given that we are leaning towards a new special command.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 12:21 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 12 Apr 2023 12:15:11 +0100
> > Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org,
> >       larsi@gnus.org
> >
> > > This has the same problem: it modifies a function that is called in
> > > too many places.  package-installed-p has half a dozen callers in
> > > package.el alone.  The change is tiny, but what about its
> > > implications on every use case where it is involved?
> >
> > What about them?  I volunteer to help test whichever cases you (or
> > anyone else) think may be problematic.
>
> We will never be able to reveal them all, let alone test them, before
> Emacs 29.1 is released.

Were there even :core packages back in 2015 when this package-install
safeguard was added in package-installed-p?  I haven't checked, but
I think not. I think this is just a plain oversight in an irregularly
maintained library, if I've ever seen one.  Why not just fix it?
The other use cases of package-installed-p you mention pass a
package-desc structure, and there this safeguard doesn't
even exist (see the docstring).

> > Can we put this tiny patch on master then?
>
> Yes, if Philip and Stefan don't object.  But, since there will be a
> command for updating core packages, doesn't this go against your
> desire not to change the UX?

IMO The user experience is bad/broken.  So at least I want to change it,
if minimally.  That's what any bug report is about, to change UX for the
better.  Here, to let people not to have to go through odd hoops big or
small to install packages or know if something is built-in/core or not.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 12:35:23 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> > > Can we put this tiny patch on master then?
> >
> > Yes, if Philip and Stefan don't object.  But, since there will be a
> > command for updating core packages, doesn't this go against your
> > desire not to change the UX?
> 
> IMO The user experience is bad/broken.

We don't yet have the other command, let alone any user feedback.  So
I don't think you can know this is bad/broken yet.  You may think so,
but I happen to think differently, so let's not behave as if this is a
fact already.

> So at least I want to change it, if minimally.  That's what any bug
> report is about, to change UX for the better.  Here, to let people
> not to have to go through odd hoops big or small to install packages
> or know if something is built-in/core or not.

Using a specialized command for a special job is nowhere near jumping
through hoops.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Wed, 12 Apr 2023 12:15:11 +0100
>> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
>> 	larsi@gnus.org
>> 
>> > This has the same problem: it modifies a function that is called in
>> > too many places.  package-installed-p has half a dozen callers in
>> > package.el alone.  The change is tiny, but what about its
>> > implications on every use case where it is involved?
>> 
>> What about them?  I volunteer to help test whichever cases you (or
>> anyone else) think may be problematic.
>
> We will never be able to reveal them all, let alone test them, before
> Emacs 29.1 is released.
>
>> Can we put this tiny patch on master then?
>
> Yes, if Philip and Stefan don't object.  But, since there will be a
> command for updating core packages, doesn't this go against your
> desire not to change the UX?

After thinking about this for a bit, I think that the right approach is
to use package-install instead of writing a separate command.  After
all, this will make the behaviour of package-install consistent with
that of the package menu.

It might work but it should be tested somewhat thoroughly before the
patch is applied.  In the meantime, I just finished a similar approach
that does not modify `package-installed-p', but just adds another
utility function:


[-- Attachment #2: Type: text/plain, Size: 1807 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..5a79c277f0d 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,14 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package-core-p (package)
+  "Return non-nil the built-in version of PACKAGE is loaded."
+  (let ((package (if (package-desc-p package)
+                     (package-desc-name package)
+                   package)))
+    (and (assq package (package--alist))
+         (package-built-in-p package))))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -1908,7 +1916,7 @@ package-compute-transaction
                  (package-version-join (package-desc-version already)))))
       (cond
        (already nil)
-       ((package-installed-p next-pkg next-version) nil)
+       ((package-core-p (car next-pkg))) ;done
 
        (t
         ;; A package is required, but not installed.  It might also be
@@ -2205,11 +2213,9 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (seq-filter
+                     (lambda (elt) (package-core-p (car elt)))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)

[-- Attachment #3: Type: text/plain, Size: 291 bytes --]


>> Maybe there's half a chance for a backport until the Emacs 29.1
>> release, or a quarter of a chance that some future Emacs 29.2 will
>> also get it.
>
> I don't think there is such a chance, and I'm not sure we even want
> that, given that we are leaning towards a new special command.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 12:46 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 12 Apr 2023 12:35:23 +0100
> > Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org,
> >       larsi@gnus.org
> >
> > > > Can we put this tiny patch on master then?
> > >
> > > Yes, if Philip and Stefan don't object.  But, since there will be a
> > > command for updating core packages, doesn't this go against your
> > > desire not to change the UX?
> >
> > IMO The user experience is bad/broken.
>
> We don't yet have the other command, let alone any user feedback.  So
> I don't think you can know this is bad/broken yet.  You may think so,
> but I happen to think differently, so let's not behave as if this is a
> fact already.

I was talking about the current state, obviously.  It's bad/broken,
IMO, as I described the original bug report, and noone really contested
that I think.  As to the new command you're proposing I think we'll both
agree that it's adding _new_ UX to the Emacs 29 pretest, something
which is at least infrequent.

> > So at least I want to change it, if minimally.  That's what any bug
> > report is about, to change UX for the better.  Here, to let people
> > not to have to go through odd hoops big or small to install packages
> > or know if something is built-in/core or not.
>
> Using a specialized command for a special job is nowhere near jumping
> through hoops.

Right.  But we fundamentally disagree that this is a "special job".
There's 0 reason why it should be.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Philip Kaludercic <philipk@posteo.net> writes:

>>> Can we put this tiny patch on master then?
>>
>> Yes, if Philip and Stefan don't object.  But, since there will be a
>> command for updating core packages, doesn't this go against your
>> desire not to change the UX?
>
> After thinking about this for a bit, I think that the right approach is
> to use package-install instead of writing a separate command.  After
> all, this will make the behaviour of package-install consistent with
> that of the package menu.

+1000

> It might work but it should be tested somewhat thoroughly before the
> patch is applied.  In the meantime, I just finished a similar approach
> that does not modify `package-installed-p', but just adds another
> utility function:

I've just applied it, re-made emacs, tested with Emacs -Q and now M-x
package-install doesn't offer _any_ completions to install, core or
non-core.  Is this what you intended?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]

João Távora <joaotavora@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>>> Can we put this tiny patch on master then?
>>>
>>> Yes, if Philip and Stefan don't object.  But, since there will be a
>>> command for updating core packages, doesn't this go against your
>>> desire not to change the UX?
>>
>> After thinking about this for a bit, I think that the right approach is
>> to use package-install instead of writing a separate command.  After
>> all, this will make the behaviour of package-install consistent with
>> that of the package menu.
>
> +1000
>
>> It might work but it should be tested somewhat thoroughly before the
>> patch is applied.  In the meantime, I just finished a similar approach
>> that does not modify `package-installed-p', but just adds another
>> utility function:
>
> I've just applied it, re-made emacs, tested with Emacs -Q and now M-x
> package-install doesn't offer _any_ completions to install, core or
> non-core.  Is this what you intended?

No, that was my mistake (I changed something last minute before applying
the patch).  This patch should behave properly:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages.patch --]
[-- Type: text/x-diff, Size: 3497 bytes --]

From 8c742056f2669a21de16488652e114cba8c8147a Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 12 Apr 2023 14:26:39 +0200
Subject: [PATCH] Allow upgrading built-in packages

* lisp/emacs-lisp/package.el (package-core-p): Add new utility
function.
(package-compute-transaction): Check if an installed package is
built-in while resolving dependencies and allow it to be installed.
(package-install): Suggest upgrading built-in packages in the
interactive specification.
---
 lisp/emacs-lisp/package.el | 36 ++++++++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 6 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..91fbbb11f68 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,20 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package-core-p (package)
+  "Return non-nil the built-in version of PACKAGE is loaded.
+This command differs from `package-built-in-p' in that it only
+returns a non-nil value if the user has not installed a more a
+more recent version of the package from a package archive."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -1908,7 +1922,16 @@ package-compute-transaction
                  (package-version-join (package-desc-version already)))))
       (cond
        (already nil)
-       ((package-installed-p next-pkg next-version) nil)
+       ;; If a package is installed, we don't need to continue.
+       ;; Built-in packages constitute an exception, because we want
+       ;; to allow the user to "upgrade" from a built-in version to a
+       ;; potentially newer version available on ELPA (bug#62720).
+       ((and (package-installed-p next-pkg next-version)
+             (not (package-core-p next-pkg))))
+       ;; The pseudo-package Emacs is always installed and built-in.
+       ;; It cannot be upgraded, so we make sure not to proceed beyond
+       ;; this point when resolving dependencies.
+       ((eq next-pkg 'emacs))
 
        (t
         ;; A package is required, but not installed.  It might also be
@@ -2205,11 +2228,12 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (not (package-installed-p (car elt)))
+                                (package-core-p (car elt)))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>   monnier@iro.umontreal.ca,
>   62720@debbugs.gnu.org,  larsi@gnus.org
> Date: Wed, 12 Apr 2023 12:00:09 +0000
> 
> > Yes, if Philip and Stefan don't object.  But, since there will be a
> > command for updating core packages, doesn't this go against your
> > desire not to change the UX?
> 
> After thinking about this for a bit, I think that the right approach is
> to use package-install instead of writing a separate command.  After
> all, this will make the behaviour of package-install consistent with
> that of the package menu.

Is this for master or for the release branch?

And I thought we all agreed built-in packages need special treatment
anyway, didn't we?  Then why having a separate command is not a
natural next step?

> It might work but it should be tested somewhat thoroughly before the
> patch is applied.  In the meantime, I just finished a similar approach
> that does not modify `package-installed-p', but just adds another
> utility function:

A new utility function is fine by me, even if this is e branch.  But I
don't quite understand how this is supposed to work in package-install
to allow updating built-in packages, and do that in a way that will
not touch the existing code for non-built-in packages in significant
ways (assuming you propose this from the release branch).  Can you
elaborate on that?

> +(defun package-core-p (package)
> +  "Return non-nil the built-in version of PACKAGE is loaded."

Didn't you say the "core" terminology was confusing people?

> +  (let ((package (if (package-desc-p package)
> +                     (package-desc-name package)
> +                   package)))
> +    (and (assq package (package--alist))
> +         (package-built-in-p package))))

It sounds like this doesn't check whether a package is "core", it
checks whether it's built-in and can be updated?  So maybe the name
should be changed to reflect that?  And the doc string as well (what
it means by "is loaded")?

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 10:53 AM Eli Zaretskii <eliz@gnu.org> wrote:

> I did just
>
>   emacs -Q
>   M-x list-packages RET
>
> and I still see what I described.
>
> I get the same if I point HOME to a scratch empty directory.
>
> Maybe it's a Windows thing? why are some dependencies marked as "not
> available"?

Can't reproduce on Windows, either.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 1:28 PM Philip Kaludercic <philipk@posteo.net> wrote:

> > I've just applied it, re-made emacs, tested with Emacs -Q and now M-x
> > package-install doesn't offer _any_ completions to install, core or
> > non-core.  Is this what you intended?
>
> No, that was my mistake (I changed something last minute before applying
> the patch).  This patch should behave properly:

Can confirm it does for M-x package-install.

I hope this patch goes far, but maybe rename package-core-p to
package--core-p and there's a duplicate "a more" in its docstring.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  larsi@gnus.org,
>>   62720@debbugs.gnu.org
>> Date: Wed, 12 Apr 2023 10:18:08 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
>> >>  Status "incompat"?  If I press RET on its name, I see
>> >> 
>> >>    Package eglot is incompatible.
>> >> 
>> >>         Status: Incompatible because it depends on uninstallable packages.
>> >> 
>> >>  Why is that?
>> >> 
>> >> No idea. Is this with our without my patch?
>> >
>> > Without.
>> 
>> I can't reproduce this in a recent emacs-29.  Here's what I start Emacs
>> with
>> 
>> HOME=`mktemp -d` && src/emacs -Q -f list-packages
>> 
>> I get:
>> 
>>      Status: Available from gnu -- [Install]
>
> I did just
>
>   emacs -Q
>   M-x list-packages RET
>
> and I still see what I described.
>
> I get the same if I point HOME to a scratch empty directory.
>
> Maybe it's a Windows thing? why are some dependencies marked as "not
> available"?
>
>   Package eglot is incompatible.
>
>        Status: Incompatible because it depends on uninstallable packages.
>       Archive: gnu
>       Version: 1.14
>        Commit: 8125d4cfc5605ead9102b7d823c4241029eb76cc
>       Summary: The Emacs Client for LSP servers
>      Requires: emacs-26.3, jsonrpc-1.0.16, flymake-1.2.1,
> 		 project-0.9.8 (not available), xref-1.6.2 (not available),
> 		 eldoc-1.14.0, seq-2.23 (not available),
> 		 external-completion-0.1 (not available)

On GNU/Linux: I have experienced a similar issue to this one in the
past, but found that it resolved itself without any action on my end
after a while.  Sadly I did not take the time to investigate what is
going on, but I believe that this is a subtle bug in package.el.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 3486 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: João Távora <joaotavora@gmail.com>,
>>   monnier@iro.umontreal.ca,
>>   62720@debbugs.gnu.org,  larsi@gnus.org
>> Date: Wed, 12 Apr 2023 12:00:09 +0000
>> 
>> > Yes, if Philip and Stefan don't object.  But, since there will be a
>> > command for updating core packages, doesn't this go against your
>> > desire not to change the UX?
>> 
>> After thinking about this for a bit, I think that the right approach is
>> to use package-install instead of writing a separate command.  After
>> all, this will make the behaviour of package-install consistent with
>> that of the package menu.
>
> Is this for master or for the release branch?

Personally I am indifferent, it should be compatible with both

> And I thought we all agreed built-in packages need special treatment
> anyway, didn't we?  Then why having a separate command is not a
> natural next step?

I don't necessarily agree that "special treatment" requires a separate
command.  I think it is wrong the assume that an built-in package should
automatically be updated to a ELPA package whenever possible.

It might be that I misunderstood something about your long-term plan
where package-install wouldn't suffice.  I'll go re-read the thread to
check.

>> It might work but it should be tested somewhat thoroughly before the
>> patch is applied.  In the meantime, I just finished a similar approach
>> that does not modify `package-installed-p', but just adds another
>> utility function:
>
> A new utility function is fine by me, even if this is e branch.  But I
> don't quite understand how this is supposed to work in package-install
> to allow updating built-in packages, and do that in a way that will
> not touch the existing code for non-built-in packages in significant
> ways (assuming you propose this from the release branch).  Can you
> elaborate on that?

The only reason we couldn't install built-in packages is that when
planning to install packages `package-compute-transaction' believes that
if a built-in package is provided, then everything is fine and we don't
need to proceed with installing any packages.  All I propose is to lift
this assumption, then this works fine.

One point that might be deliberated is that this means all built-in
dependencies are also installed, even if these are not strictly
necessary.  It shouldn't matter that much, since most users would
upgrade them eventually, but worth mentioning I guess? 

>> +(defun package-core-p (package)
>> +  "Return non-nil the built-in version of PACKAGE is loaded."
>
> Didn't you say the "core" terminology was confusing people?

TBH I am not really satisfied with the name (so any other suggestion is
just as fine for me), and as Joao said it would be better to make the
predicate as internal so that users are not expected to deal with it.

>> +  (let ((package (if (package-desc-p package)
>> +                     (package-desc-name package)
>> +                   package)))
>> +    (and (assq package (package--alist))
>> +         (package-built-in-p package))))
>
> It sounds like this doesn't check whether a package is "core", it
> checks whether it's built-in and can be updated?  So maybe the name
> should be changed to reflect that?  And the doc string as well (what
> it means by "is loaded")?

Right the "loaded" doesn't make sense.  How about this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages.patch --]
[-- Type: text/x-diff, Size: 3577 bytes --]

From 12e0b209992675a042112e790571d427a003c30d Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 12 Apr 2023 14:26:39 +0200
Subject: [PATCH] Allow upgrading built-in packages

* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
utility predicate.
(package-compute-transaction): Check if an installed package is
built-in while resolving dependencies and allow it to be installed.
(package-install): Suggest upgrading built-in packages in the
interactive specification.  Allow upgrading built-in packages
---
 lisp/emacs-lisp/package.el | 36 ++++++++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 6 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..2cf98290bba 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,20 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Check if a built-in PACKAGE can be upgraded.
+This command differs from `package-built-in-p' in that it only
+returns a non-nil value if the user has not installed a more
+recent version of the package from a package archive."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -1908,7 +1922,16 @@ package-compute-transaction
                  (package-version-join (package-desc-version already)))))
       (cond
        (already nil)
-       ((package-installed-p next-pkg next-version) nil)
+       ;; If a package is installed, we don't need to continue.
+       ;; Built-in packages constitute an exception, because we want
+       ;; to allow the user to "upgrade" from a built-in version to a
+       ;; potentially newer version available on ELPA (bug#62720).
+       ((and (not (package--upgradable-built-in-p next-pkg))
+             (package-installed-p next-pkg next-version)))
+       ;; The pseudo-package Emacs is always installed and built-in.
+       ;; It cannot be upgraded, so we make sure not to proceed beyond
+       ;; this point when resolving dependencies.
+       ((eq next-pkg 'emacs))
 
        (t
         ;; A package is required, but not installed.  It might also be
@@ -2205,11 +2228,12 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (package--upgradable-built-in-p (car elt))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> +(defun package-core-p (package)
>>> +  "Return non-nil the built-in version of PACKAGE is loaded."
>>
>> Didn't you say the "core" terminology was confusing people?
>
> TBH I am not really satisfied with the name (so any other suggestion is
> just as fine for me), and as Joao said it would be better to make the
> predicate as internal so that users are not expected to deal with it.

Beside "core" and "built-in" (with or without dash), another name that's
been used over the years is "bundled".  I don't have a strong preference
between those.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Hmm... it might make sense to treat builtins specially in this respect.
>> Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
>> with the idea that `M-x package-update-all` would update all the
>> `:core` packages.
>> 
>> I'm not sure why, admittedly, but I think it comes down to the fact that
>> the first upgrade of a `:core` package from GNU ELPA feels to me more
>> like an "install" than an "upgrade".
>
> Which means my proposal of adding a new command
> package-update-core-package makes more and more sense: we will

FWIW, I'm not sure it's needed: when the user asks to `package-update`
the Eglot package, then we should upgrade `eglot`, regardless if
it's builtin.

My comment was specifically for `M-x package-update-all`.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 3:11 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> >>> +(defun package-core-p (package)
> >>> +  "Return non-nil the built-in version of PACKAGE is loaded."
> >>
> >> Didn't you say the "core" terminology was confusing people?
> >
> > TBH I am not really satisfied with the name (so any other suggestion is
> > just as fine for me), and as Joao said it would be better to make the
> > predicate as internal so that users are not expected to deal with it.
>
> Beside "core" and "built-in" (with or without dash), another name that's
> been used over the years is "bundled".  I don't have a strong preference
> between those.

Isn't "bundled" better left for when non-core, non-builtin ELPA packages
are somehow thrown together with the Emacs release tarball or the
platform-specific
builds we make of them?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Isn't "bundled" better left for when non-core, non-builtin ELPA
> packages are somehow thrown together with the Emacs release tarball or
> the platform-specific builds we make of them?

I can't see any reason why that would make a difference here.

AFAICT this only matters in terms of which Git repository/branch holds
the source code and how it gets included into the Emacs distribution,
but after that it's an "internal detail".


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 3:17 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > Isn't "bundled" better left for when non-core, non-builtin ELPA
> > packages are somehow thrown together with the Emacs release tarball or
> > the platform-specific builds we make of them?
>
> I can't see any reason why that would make a difference here.
>
> AFAICT this only matters in terms of which Git repository/branch holds
> the source code and how it gets included into the Emacs distribution,
> but after that it's an "internal detail".

Fair enough, and lets hope updating them will be easy.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Wed, 12 Apr 2023 13:42:56 +0000
> 
> >> After thinking about this for a bit, I think that the right approach is
> >> to use package-install instead of writing a separate command.  After
> >> all, this will make the behaviour of package-install consistent with
> >> that of the package menu.
> >
> > Is this for master or for the release branch?
> 
> Personally I am indifferent, it should be compatible with both

If we want to install this on the release branch, the changes must be
very safe, which in this case basically means "do not touch any code
used when updating non-core packages".

If the patch you presented is all there is to it, then I'm afraid it
doesn't satisfy the above condition, because it does affect the use
case of updating an ELPA package that is not in core.  So I cannot
agree to installing this on emacs-29 in the form you posted, sorry.

> > And I thought we all agreed built-in packages need special treatment
> > anyway, didn't we?  Then why having a separate command is not a
> > natural next step?
> 
> I don't necessarily agree that "special treatment" requires a separate
> command.

Even if you don't agree with that in general, having a separate
command would allow us to install that command on emacs-29 without any
fears.  So that alone is a significant advantage, even if the rest are
not yet agreed upon.

> I think it is wrong the assume that an built-in package should
> automatically be updated to a ELPA package whenever possible.

This seems to be an argument in favor of a separate command?  Or what
did you mean by that, and how is it related to the issue at hand?

> >> It might work but it should be tested somewhat thoroughly before the
> >> patch is applied.  In the meantime, I just finished a similar approach
> >> that does not modify `package-installed-p', but just adds another
> >> utility function:
> >
> > A new utility function is fine by me, even if this is e branch.  But I
> > don't quite understand how this is supposed to work in package-install
> > to allow updating built-in packages, and do that in a way that will
> > not touch the existing code for non-built-in packages in significant
> > ways (assuming you propose this from the release branch).  Can you
> > elaborate on that?
> 
> The only reason we couldn't install built-in packages is that when
> planning to install packages `package-compute-transaction' believes that
> if a built-in package is provided, then everything is fine and we don't
> need to proceed with installing any packages.  All I propose is to lift
> this assumption, then this works fine.

My problem is _how_ to lift this assumption.  The way you propose
doing that affects updating non-core packages in ways that we will not
have enough time to test well enough, not compared to the year that
these commands exist in package.el and were used by many people in
many cases.  So we have the following alternatives for the way
forward:

 . install your changes on master only, and leave the problem of
   updating a core package unsolved in Emacs 29 (with the workaround
   mentioned in the beginning of this bug's discussion available to
   alleviate the problem to some extent)
 . come up with safer changes for package-install that could be
   installed on emacs-29
 . add a new command for updating a core package, which can then be
   safely installed on emacs-29

The last 2 alternatives can be for emacs-29 only, whereas on master we
install your proposed change (or something else).

For the 2nd alternative above to be acceptable, the added/modified
code must be completely separate from the code we have now, so that
any possibility of its destabilizing the current code could be
eliminated.  It could be a separate code, triggered by the prefix
argument, for example.

> One point that might be deliberated is that this means all built-in
> dependencies are also installed, even if these are not strictly
> necessary.  It shouldn't matter that much, since most users would
> upgrade them eventually, but worth mentioning I guess? 

That just confirms my fears that we are opening a Pandora's box.  We
have no idea what this will do, and no time to test the results.
Unintended consequences are abundant.  We must draw any such
consequences to the absolute minimum, at least the way the commands
work by default.  Even if the result is less than elegant.

> >> +(defun package-core-p (package)
> >> +  "Return non-nil the built-in version of PACKAGE is loaded."
> >
> > Didn't you say the "core" terminology was confusing people?
> 
> TBH I am not really satisfied with the name (so any other suggestion is
> just as fine for me), and as Joao said it would be better to make the
> predicate as internal so that users are not expected to deal with it.

The name should still be self-explanatory enough, because we the Emacs
maintainers will read this code and should be able to understand what
the function does without reading is source every time.

> 
> >> +  (let ((package (if (package-desc-p package)
> >> +                     (package-desc-name package)
> >> +                   package)))
> >> +    (and (assq package (package--alist))
> >> +         (package-built-in-p package))))
> >
> > It sounds like this doesn't check whether a package is "core", it
> > checks whether it's built-in and can be updated?  So maybe the name
> > should be changed to reflect that?  And the doc string as well (what
> > it means by "is loaded")?
> 
> Right the "loaded" doesn't make sense.  How about this:
> 
> +(defun package--upgradable-built-in-p (package)
> +  "Check if a built-in PACKAGE can be upgraded.
> +This command differs from `package-built-in-p' in that it only
> +returns a non-nil value if the user has not installed a more
> +recent version of the package from a package archive."

Note that what the doc string says is not what the name tells us.
"Upgradeable" and "user has not installed a more recent version" are
not the same.  What the doc string says calls for a name like
package--built-in-and-up-to-date or something.

> +  (and (not (assq (cond
> +                   ((package-desc-p package)
> +                    (package-desc-name package))
> +                   ((stringp package) (intern package))
> +                   ((symbolp package) package)
> +                   ((error "Unknown package format: %S" package)))
> +                  (package--alist)))
> +       (package-built-in-p package)))

Why do we need all these conditions, where we didn't need them in the
current code?

Also, package-alist is documented as "alist of all packages available
for activation", so it is not clear how the fact that assq returns nil
is evidence that "the user has not installed a more recent version".
Looking at what package--alist and package-load-all-descriptors do, it
looks like they just collect packages that were downloaded into the
relevant directories?  Is that enough to consider any package not in
the list to be "not installed"?  And what about the "more recent
version" condition -- this doesn't seem to be tested anywhere in
package--alist?

The above questions and undocumented subtleties is what scares me in
installing such changes at this late stage.  I'm not sure everyone
involved, yourself included, have a clear understanding of what the
modified code will do in each possible use case.  That is why I very
much prefer separate code, which will then free us from the need of
considering all these subtleties, as the last year of user's
experience with this code can vouch that it does its job correctly, by
and large.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 12/04/2023 12:34, João Távora wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> On 11/04/2023 14:02, João Távora wrote:
>>> Philip Kaludercic<philipk@posteo.net>  writes:
>>>
>>>> Will this not affect `package-update-all'?  I don't if we want that the
>>>> command installs all packages from ELPA that it can find.
>>> Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
>>> updates the built-in and the manually installed packages that can be
>>> updated.  It_doesn't_  install any packages that weren't installed yet,
>>> of course.
>>
>> On a related note, do you know whether we upgrade the built-in
>> packages when the user presses 'U' in the list-packages buffer?
>>
>> Using the command package-menu-mark-upgrades, that is.
> 
> Nope, doesn't work, doesn't do anything to those packages.  I wish it
> did, of course.

All right, then it's not just about package-install.

> I also don't understand why this is using separate, but repeated logic
> from package-update.  What is the difference between "upgrade" and
> "update", if any?  Is "upgrade" more powerful?

They're supposed to be the same. But that's bug#62750 which you've 
already read by now.

> BTW, I also noticed that Eglot's version on Emacs 29 is garbled.  I had
> wanted 1.12-emacs29 to somehow show that it is Emacs 29 specific.  But
> version-to-list doesn't like it and the package shows up as version
> "nil" in package--builtins.  Will just change it to 1.12.29, which is
> less perceptible but works fine (none of this makes any difference to
> this bug, of course).

Cool.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 12/04/2023 14:00, João Távora wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>> the restrictions.  What else can I say to express clearly that I would
>> like such a command to not be installed?
> Had another idea: what about this very tiny patch, then?  It makes `M-x
> package-install` work for installing a :core package.  This also rhymes
> exactly with Stefan's intution/feeling that :core packages need to be
> "installed" to promote them to installable.  The current M-x
> package-install recommendation could remain flawlessly and then you can
> do whatever you think is best for M-x package-update & friends.
> 
> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index f92afe56b76..134505a96af 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -2164,8 +2164,10 @@ package-installed-p
>          (and pkg-descs
>               (version-list-<= min-version
>                                (package-desc-version (car pkg-descs)))))
> -     ;; Also check built-in packages.
> -     (package-built-in-p package min-version)))))
> +     ;; Also check built-in packages.  But if min-version is nil, just
> +     ;; act as if a package-desc had been passed (bug#62720)
> +     (and min-version (package-built-in-p package min-version))))))
>   
>   (defun package-download-transaction (packages)
>     "Download and install all the packages in PACKAGES.

If the package becomes "mass-upgradable" afterwards, this seems like the 
best solution, behavior-wise (imminent 29 release aside, of course).

Does 'M-x package-update/upgrade' get fixed automatically with this, or 
does it need separate changes?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 12/04/2023 14:08, Eli Zaretskii wrote:
>> From: João Távora<joaotavora@gmail.com>
>> Cc:philipk@posteo.net,monnier@iro.umontreal.ca,62720@debbugs.gnu.org,
>>    larsi@gnus.org
>> Date: Wed, 12 Apr 2023 12:00:04 +0100
>>
>> Had another idea: what about this very tiny patch, then?  It makes `M-x
>> package-install` work for installing a :core package.  This also rhymes
>> exactly with Stefan's intution/feeling that :core packages need to be
>> "installed" to promote them to installable.  The current M-x
>> package-install recommendation could remain flawlessly and then you can
>> do whatever you think is best for M-x package-update & friends.
> This has the same problem: it modifies a function that is called in
> too many places.  package-installed-p has half a dozen callers in
> package.el alone.  The change is tiny, but what about its
> implications on every use case where it is involved?

What if we only fix 'package-upgrade' (nee package-update) in emacs-29?

It's a new function, after all. And semantically, the result would be 
somewhat sensible: since 'eglot' is already installed ("bundled" or 
whatever), having 'package-install' fail is not that big a deal, as long 
as 'M-x package-upgrade' works.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Wed, 12 Apr 2023 18:49:09 +0300
> Cc: larsi@gnus.org, 62720@debbugs.gnu.org, philipk@posteo.net,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 12/04/2023 14:08, Eli Zaretskii wrote:
> >> From: João Távora<joaotavora@gmail.com>
> >> Cc:philipk@posteo.net,monnier@iro.umontreal.ca,62720@debbugs.gnu.org,
> >>    larsi@gnus.org
> >> Date: Wed, 12 Apr 2023 12:00:04 +0100
> >>
> >> Had another idea: what about this very tiny patch, then?  It makes `M-x
> >> package-install` work for installing a :core package.  This also rhymes
> >> exactly with Stefan's intution/feeling that :core packages need to be
> >> "installed" to promote them to installable.  The current M-x
> >> package-install recommendation could remain flawlessly and then you can
> >> do whatever you think is best for M-x package-update & friends.
> > This has the same problem: it modifies a function that is called in
> > too many places.  package-installed-p has half a dozen callers in
> > package.el alone.  The change is tiny, but what about its
> > implications on every use case where it is involved?
> 
> What if we only fix 'package-upgrade' (nee package-update) in emacs-29?

I believe that's what João was proposing.

I don't mind, but changes in package-update will also need to be safe
enough, since this command, while new in Emacs 29, is with us for the
last year, so is relatively well tested, and I don't want to risk
breaking it by last-minute changes of a non-trivial nature.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 4:17 PM Eli Zaretskii <eliz@gnu.org> wrote:

> The above questions and undocumented subtleties is what scares me in
> installing such changes at this late stage.  I'm not sure everyone
> involved, yourself included, have a clear understanding of what the
> modified code will do in each possible use case.  That is why I very
> much prefer separate code, which will then free us from the need of
> considering all these subtleties, as the last year of user's
> experience with this code can vouch that it does its job correctly, by
> and large.

Alright, I've tried my hand at making this clean separation, so that
no logic of transaction or existing predicates is touched.  I tried to
make it as intelligible as possible perhaps overdoing the commentary
and the naming, but we can always trim it.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 685f983e285..51d633131d9 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2178,6 +2178,24 @@ package--archives-initialize
   (unless package-archive-contents
     (package-refresh-contents)))

+(defun package--vestal-builtins-available-for-installation ()
+  "Return built-in packages that can but have never been updated.
+The return value is a subset of `package-archive-contents', i.e.
+a list ((SYM PACKAGE-DESC) ...) where SYM names such a built-in
+package.  For bug#62720.
+
+Because we reject packages that are in `package--alist', this set
+is guaranteed to _not_ intersect with the subset of
+`package-archive-contents', which verifies `package-installed-p'.
+which, for historical reasons (and in suspicious conditions) says
+that built-in packages as installed."
+  (let (res)
+    (dolist (bi package--builtins)
+      (when-let ((probe (and (not (assq (car bi) (package--alist)))
+                             (assq (car bi) package-archive-contents))))
+        (push probe res)))
+    res))
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
@@ -2196,19 +2214,27 @@ package-install
 to install it but still mark it as selected."
   (interactive
    (progn
-     ;; Initialize the package system to get the list of package
-     ;; symbols for completion.
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (append
+                     (delq nil
+                           (mapcar (lambda (elt)
+                                     (unless (package-installed-p (car elt))
+                                       (symbol-name (car elt))))
+                                   package-archive-contents))
+                     (mapcar #'car
+
(package--vestal-builtins-available-for-installation)))
                     nil t))
            nil)))
   (package--archives-initialize)
+  ;; See docstring and code of
+  ;; `package--vestal-builtins-available-for-installation' to
+  ;; understand how this code can only kick in if indeed the user
+  ;; selected a such a package from the completion list. bug#62720.
+  (when-let ((name-and-desc
+              (assq pkg
(package--vestal-builtins-available-for-installation))))
+    (setq pkg (cadr name-and-desc)))
   (add-hook 'post-command-hook #'package-menu--post-refresh)
   (let ((name (if (package-desc-p pkg)
                   (package-desc-name pkg)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 5:13 PM João Távora <joaotavora@gmail.com> wrote:
> > modified code will do in each possible use case.  That is why I very
> > much prefer separate code, which will then free us from the need of
> Alright, I've tried my hand at making this clean separation, so that
> no logic of transaction or existing predicates is touched.  I tried to

>  (defun package-install (pkg &optional dont-select)
>    "Install the package PKG.
> @@ -2196,19 +2214,27 @@ package-install
>  to install it but still mark it as selected."
>    (interactive
>     (progn
> -     ;; Initialize the package system to get the list of package
> -     ;; symbols for completion.

Didn't mean to take out this comment, sorry, was an oversight.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 4:58 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > >> Had another idea: what about this very tiny patch, then?  It makes `M-x
> > >> package-install` work for installing a :core package.  This also rhymes
> > >> exactly with Stefan's intution/feeling that :core packages need to be
> > >> "installed" to promote them to installable.  The current M-x
> > >> package-install recommendation could remain flawlessly and then you can
> > >> do whatever you think is best for M-x package-update & friends.
> > > This has the same problem: it modifies a function that is called in
> > > too many places.  package-installed-p has half a dozen callers in
> > > package.el alone.  The change is tiny, but what about its
> > > implications on every use case where it is involved?
> >
> > What if we only fix 'package-upgrade' (nee package-update) in emacs-29?
>
> I believe that's what João was proposing.

AFAICT, Dmitry was asking only for package-update, not
package-update-all.  Stefan was also inclined for that.

In my changes, I changed both.  But it is not hard for me to touch
only package-update and to do it with the utmost care for
separation and stability.

For the moment, I'm focusing on M-x package-install, like Philip is.
There seems to be more consensus there that it should offer to update
builtin packages that have never been updated.

I do believe there is high demand for a "upgrade/update" mechanism
that just "updates whatever there is to update, don't care if core
or whatnot" and people looking at package-update-all and
package-menu--mark-upgrades (the "U" command Dmitry brought up)
will eventually be disappointed.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 17:13:27 +0100
> Cc: Philip Kaludercic <philipk@posteo.net>, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> On Wed, Apr 12, 2023 at 4:17 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > The above questions and undocumented subtleties is what scares me in
> > installing such changes at this late stage.  I'm not sure everyone
> > involved, yourself included, have a clear understanding of what the
> > modified code will do in each possible use case.  That is why I very
> > much prefer separate code, which will then free us from the need of
> > considering all these subtleties, as the last year of user's
> > experience with this code can vouch that it does its job correctly, by
> > and large.
> 
> Alright, I've tried my hand at making this clean separation, so that
> no logic of transaction or existing predicates is touched.  I tried to
> make it as intelligible as possible perhaps overdoing the commentary
> and the naming, but we can always trim it.

Thanks, but this is not separate code.  It adds 21 built-in packages
to the list of completion candidates that the user can choose in
package-install, and who knows what kind of confusion this could cause
when users see packages like Xref and Package and Tramp and seq and
Python and Org in that list, and what accidents that could cause if
users select one of those by mistake, because they never saw them
there before?

PLEASE show me a completely separate code or a separate command, then
I will agree to this last-minute addition.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>   monnier@iro.umontreal.ca,
>   larsi@gnus.org,  62720@debbugs.gnu.org
> Date: Wed, 12 Apr 2023 13:20:27 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: João Távora <joaotavora@gmail.com>
> >> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  larsi@gnus.org,
> >>   62720@debbugs.gnu.org
> >> Date: Wed, 12 Apr 2023 10:18:08 +0100
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
> >> >>  Status "incompat"?  If I press RET on its name, I see
> >> >> 
> >> >>    Package eglot is incompatible.
> >> >> 
> >> >>         Status: Incompatible because it depends on uninstallable packages.
> >> >> 
> >> >>  Why is that?
> >> >> 
> >> >> No idea. Is this with our without my patch?
> >> >
> >> > Without.
> >> 
> >> I can't reproduce this in a recent emacs-29.  Here's what I start Emacs
> >> with
> >> 
> >> HOME=`mktemp -d` && src/emacs -Q -f list-packages
> >> 
> >> I get:
> >> 
> >>      Status: Available from gnu -- [Install]
> >
> > I did just
> >
> >   emacs -Q
> >   M-x list-packages RET
> >
> > and I still see what I described.
> >
> > I get the same if I point HOME to a scratch empty directory.
> >
> > Maybe it's a Windows thing? why are some dependencies marked as "not
> > available"?
> >
> >   Package eglot is incompatible.
> >
> >        Status: Incompatible because it depends on uninstallable packages.
> >       Archive: gnu
> >       Version: 1.14
> >        Commit: 8125d4cfc5605ead9102b7d823c4241029eb76cc
> >       Summary: The Emacs Client for LSP servers
> >      Requires: emacs-26.3, jsonrpc-1.0.16, flymake-1.2.1,
> > 		 project-0.9.8 (not available), xref-1.6.2 (not available),
> > 		 eldoc-1.14.0, seq-2.23 (not available),
> > 		 external-completion-0.1 (not available)
> 
> On GNU/Linux: I have experienced a similar issue to this one in the
> past, but found that it resolved itself without any action on my end
> after a while.  Sadly I did not take the time to investigate what is
> going on, but I believe that this is a subtle bug in package.el.

I've now tried on GNU/Linux (one of gnu.org machines), and I see the
same there: eglot, project, and use-package are marked as "incompat".

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2286 bytes --]

On Wed, Apr 12, 2023, 17:52 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 12 Apr 2023 17:13:27 +0100
> > Cc: Philip Kaludercic <philipk@posteo.net>, monnier@iro.umontreal.ca,
> 62720@debbugs.gnu.org,
> >       larsi@gnus.org
> >
> > On Wed, Apr 12, 2023 at 4:17 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > The above questions and undocumented subtleties is what scares me in
> > > installing such changes at this late stage.  I'm not sure everyone
> > > involved, yourself included, have a clear understanding of what the
> > > modified code will do in each possible use case.  That is why I very
> > > much prefer separate code, which will then free us from the need of
> > > considering all these subtleties, as the last year of user's
> > > experience with this code can vouch that it does its job correctly, by
> > > and large.
> >
> > Alright, I've tried my hand at making this clean separation, so that
> > no logic of transaction or existing predicates is touched.  I tried to
> > make it as intelligible as possible perhaps overdoing the commentary
> > and the naming, but we can always trim it.
>
> Thanks, but this is not separate code.  It adds 21 built-in packages
> to the list of completion candidates that the user can choose in
> package-install, and who knows what kind of confusion this could cause
> when users see packages like Xref and Package and Tramp and seq and
> Python and Org in that list, and what accidents that could cause if
> users select one of those by mistake, because they never saw them
> there before?
>

The intended behavior, of course. To install a newer version of the
package, of course. This is also what Philip's patch did, only with
different code.

PLEASE show me a completely separate code or a separate command, then
> I will agree to this last-minute addition.
>

The point is to change the broken behavior of the existing command,
package-install. Anything else is a waste of time, and noone except you has
demonstrated support for this separate command idea. Please, in normal
non-shouting case, explain to me how you think that the behavior of an
existing command can be changed with "completely separate code".

João

>

[-- Attachment #2: Type: text/html, Size: 3663 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 18:14:28 +0100
> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
> Please, in normal non-shouting case, explain to me how you think that the behavior of an existing
> command can be changed with "completely separate code". 

I already did: either (1) add a prefix argument to an existing
command, which will then trigger the new behavior, or (2) add a
separate command.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 6:22 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 12 Apr 2023 18:14:28 +0100
> > Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org,
> >       Lars Ingebrigtsen <larsi@gnus.org>
> >
> > Please, in normal non-shouting case, explain to me how you think that the behavior of an existing
> > command can be changed with "completely separate code".
>
> I already did: either (1) add a prefix argument to an existing
> command, which will then trigger the new behavior, or (2) add a
> separate command.

We're just going in circles.  Those ideas don't solve this bug,
as hinted by the fact that noone is coming up with the patches
you want.
I guess people with `(package-install 'eglot)` in their configs
will just have to cope with whatever comes of this. As for me,
I've put too way too much time and effort into this already, so
good luck to all.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 18:43:35 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> As for me, I've put too way too much time and effort into this
> already, so good luck to all.

Thank you for your efforts.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Wed, 12 Apr 2023 18:14:28 +0100
>> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier
>> <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org,
>> 	Lars Ingebrigtsen <larsi@gnus.org>
>> 
>> Please, in normal non-shouting case, explain to me how you think
>> that the behavior of an existing
>> command can be changed with "completely separate code". 
>
> I already did: either (1) add a prefix argument to an existing
> command, which will then trigger the new behavior, or (2) add a
> separate command.

Here you have (1):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages.patch --]
[-- Type: text/x-diff, Size: 4179 bytes --]

From 09a9e769ed0c7024c33bc135c57894daff8764af Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 12 Apr 2023 14:26:39 +0200
Subject: [PATCH] Allow upgrading built-in packages

* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
utility predicate.
(package-compute-transaction): Check if an installed package is
built-in while resolving dependencies and allow it to be installed.
(package-install): Suggest upgrading built-in packages in the
interactive specification, when invoked with a prefix argument.  Allow
upgrading built-in packages
---
 lisp/emacs-lisp/package.el | 41 +++++++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 7 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..03e63ee7227 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,20 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Check if a built-in PACKAGE can be upgraded.
+This command differs from `package-built-in-p' in that it only
+returns a non-nil value if the user has not installed a more
+recent version of the package from a package archive."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -1908,7 +1922,16 @@ package-compute-transaction
                  (package-version-join (package-desc-version already)))))
       (cond
        (already nil)
-       ((package-installed-p next-pkg next-version) nil)
+       ;; If a package is installed, we don't need to continue.
+       ;; Built-in packages constitute an exception, because we want
+       ;; to allow the user to "upgrade" from a built-in version to a
+       ;; potentially newer version available on ELPA (bug#62720).
+       ((and (not (package--upgradable-built-in-p next-pkg))
+             (package-installed-p next-pkg next-version)))
+       ;; The pseudo-package Emacs is always installed and built-in.
+       ;; It cannot be upgraded, so we make sure not to proceed beyond
+       ;; this point when resolving dependencies.
+       ((eq next-pkg 'emacs))
 
        (t
         ;; A package is required, but not installed.  It might also be
@@ -2187,7 +2210,9 @@ package-install
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
 available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+called interactively, prompt for the package name.  When invoked
+with a prefix argument, the prompt will include built-in packages
+that can be upgraded via an archive.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2205,11 +2230,13 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and current-prefix-arg
+                                     (package--upgradable-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Wed, 12 Apr 2023 13:42:56 +0000
>> 
>> >> After thinking about this for a bit, I think that the right approach is
>> >> to use package-install instead of writing a separate command.  After
>> >> all, this will make the behaviour of package-install consistent with
>> >> that of the package menu.
>> >
>> > Is this for master or for the release branch?
>> 
>> Personally I am indifferent, it should be compatible with both
>
> If we want to install this on the release branch, the changes must be
> very safe, which in this case basically means "do not touch any code
> used when updating non-core packages".
>
> If the patch you presented is all there is to it, then I'm afraid it
> doesn't satisfy the above condition, because it does affect the use
> case of updating an ELPA package that is not in core.  So I cannot
> agree to installing this on emacs-29 in the form you posted, sorry.

I understand your concern, 

>> > And I thought we all agreed built-in packages need special treatment
>> > anyway, didn't we?  Then why having a separate command is not a
>> > natural next step?
>> 
>> I don't necessarily agree that "special treatment" requires a separate
>> command.
>
> Even if you don't agree with that in general, having a separate
> command would allow us to install that command on emacs-29 without any
> fears.  So that alone is a significant advantage, even if the rest are
> not yet agreed upon.

Would this be a permanent thing, or would the command be deprecated by
emacs-30?  I get the need under the circumstances, but it doesn't seem
like the best solution if we were to set aside release-concerns.

>> I think it is wrong the assume that an built-in package should
>> automatically be updated to a ELPA package whenever possible.
>
> This seems to be an argument in favor of a separate command?  Or what
> did you mean by that, and how is it related to the issue at hand?

I don't think package-update should switch from a built-in package to a
package that was installed from ELPA.  The user should at least once
commit to the switch (be it with a separate command or package-install).

>> >> It might work but it should be tested somewhat thoroughly before the
>> >> patch is applied.  In the meantime, I just finished a similar approach
>> >> that does not modify `package-installed-p', but just adds another
>> >> utility function:
>> >
>> > A new utility function is fine by me, even if this is e branch.  But I
>> > don't quite understand how this is supposed to work in package-install
>> > to allow updating built-in packages, and do that in a way that will
>> > not touch the existing code for non-built-in packages in significant
>> > ways (assuming you propose this from the release branch).  Can you
>> > elaborate on that?
>> 
>> The only reason we couldn't install built-in packages is that when
>> planning to install packages `package-compute-transaction' believes that
>> if a built-in package is provided, then everything is fine and we don't
>> need to proceed with installing any packages.  All I propose is to lift
>> this assumption, then this works fine.
>
> My problem is _how_ to lift this assumption.  The way you propose
> doing that affects updating non-core packages in ways that we will not
> have enough time to test well enough, not compared to the year that
> these commands exist in package.el and were used by many people in
> many cases.  So we have the following alternatives for the way
> forward:
>
>  . install your changes on master only, and leave the problem of
>    updating a core package unsolved in Emacs 29 (with the workaround
>    mentioned in the beginning of this bug's discussion available to
>    alleviate the problem to some extent)
>  . come up with safer changes for package-install that could be
>    installed on emacs-29
>  . add a new command for updating a core package, which can then be
>    safely installed on emacs-29
>
> The last 2 alternatives can be for emacs-29 only, whereas on master we
> install your proposed change (or something else).

I would like to investigate option 2, but if nothing is found we can
fall back to 3.  But even if there are issues in this case, I don't
consider the matter in general to be that drastic.  If all Joao wants is
to avoid confusion, we can also improve the error message that
`package-install' generates when it says that a package ins already
installed.

> For the 2nd alternative above to be acceptable, the added/modified
> code must be completely separate from the code we have now, so that
> any possibility of its destabilizing the current code could be
> eliminated.  It could be a separate code, triggered by the prefix
> argument, for example.

OK.

>> One point that might be deliberated is that this means all built-in
>> dependencies are also installed, even if these are not strictly
>> necessary.  It shouldn't matter that much, since most users would
>> upgrade them eventually, but worth mentioning I guess? 
>
> That just confirms my fears that we are opening a Pandora's box.  We
> have no idea what this will do, and no time to test the results.
> Unintended consequences are abundant.  We must draw any such
> consequences to the absolute minimum, at least the way the commands
> work by default.  Even if the result is less than elegant.

Intuitively I would want to argue that this change has an upper-bound to
how much harm it could do, but as I cannot prove it in any way I'll
rather not assert that the point.

>> >> +(defun package-core-p (package)
>> >> +  "Return non-nil the built-in version of PACKAGE is loaded."
>> >
>> > Didn't you say the "core" terminology was confusing people?
>> 
>> TBH I am not really satisfied with the name (so any other suggestion is
>> just as fine for me), and as Joao said it would be better to make the
>> predicate as internal so that users are not expected to deal with it.
>
> The name should still be self-explanatory enough, because we the Emacs
> maintainers will read this code and should be able to understand what
> the function does without reading is source every time.

OK.

>> 
>> >> +  (let ((package (if (package-desc-p package)
>> >> +                     (package-desc-name package)
>> >> +                   package)))
>> >> +    (and (assq package (package--alist))
>> >> +         (package-built-in-p package))))
>> >
>> > It sounds like this doesn't check whether a package is "core", it
>> > checks whether it's built-in and can be updated?  So maybe the name
>> > should be changed to reflect that?  And the doc string as well (what
>> > it means by "is loaded")?
>> 
>> Right the "loaded" doesn't make sense.  How about this:
>> 
>> +(defun package--upgradable-built-in-p (package)
>> +  "Check if a built-in PACKAGE can be upgraded.
>> +This command differs from `package-built-in-p' in that it only
>> +returns a non-nil value if the user has not installed a more
>> +recent version of the package from a package archive."
>
> Note that what the doc string says is not what the name tells us.
> "Upgradeable" and "user has not installed a more recent version" are
> not the same.  What the doc string says calls for a name like
> package--built-in-and-up-to-date or something.
>
>> +  (and (not (assq (cond
>> +                   ((package-desc-p package)
>> +                    (package-desc-name package))
>> +                   ((stringp package) (intern package))
>> +                   ((symbolp package) package)
>> +                   ((error "Unknown package format: %S" package)))
>> +                  (package--alist)))
>> +       (package-built-in-p package)))
>
> Why do we need all these conditions, where we didn't need them in the
> current code?

Practically speaking these conditions are not necessary, I just added it
in case there was the need to use the function elsewhere at some point.

> Also, package-alist is documented as "alist of all packages available
> for activation", so it is not clear how the fact that assq returns nil
> is evidence that "the user has not installed a more recent version".
> Looking at what package--alist and package-load-all-descriptors do, it
> looks like they just collect packages that were downloaded into the
> relevant directories?  Is that enough to consider any package not in
> the list to be "not installed"?  

The point is that package.el will add all packages it manages (ie. are
"available to activation") to package-alist.  Built-in/core packages are
not managed by package.el, but just "acknowledged" via
`package--builtins'.  So this looks like a safe assumption to me.

>                                  And what about the "more recent
> version" condition -- this doesn't seem to be tested anywhere in
> package--alist?

You are right, but this is a misphrasing.

> The above questions and undocumented subtleties is what scares me in
> installing such changes at this late stage.  I'm not sure everyone
> involved, yourself included, have a clear understanding of what the
> modified code will do in each possible use case.  That is why I very
> much prefer separate code, which will then free us from the need of
> considering all these subtleties, as the last year of user's
> experience with this code can vouch that it does its job correctly, by
> and large.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 12/04/2023 19:29, João Távora wrote:
> On Wed, Apr 12, 2023 at 4:58 PM Eli Zaretskii<eliz@gnu.org>  wrote:
> 
>>>>> Had another idea: what about this very tiny patch, then?  It makes `M-x
>>>>> package-install` work for installing a :core package.  This also rhymes
>>>>> exactly with Stefan's intution/feeling that :core packages need to be
>>>>> "installed" to promote them to installable.  The current M-x
>>>>> package-install recommendation could remain flawlessly and then you can
>>>>> do whatever you think is best for M-x package-update & friends.
>>>> This has the same problem: it modifies a function that is called in
>>>> too many places.  package-installed-p has half a dozen callers in
>>>> package.el alone.  The change is tiny, but what about its
>>>> implications on every use case where it is involved?
>>> What if we only fix 'package-upgrade' (nee package-update) in emacs-29?
>> I believe that's what João was proposing.
> AFAICT, Dmitry was asking only for package-update, not
> package-update-all.  Stefan was also inclined for that.
> 
> In my changes, I changed both.  But it is not hard for me to touch
> only package-update and to do it with the utmost care for
> separation and stability.

I think that would make sense. Like Philip phrased it, updating from a 
bundled version is like switching to a different package repository, 
with different stability expectations.

I also seem to recall some logic somewhere that made sure that the 
package is only updated from the source that it was installed from 
(unless explicitly instructed otherwise by the user). We could treat 
"builtins" as a separate source for that purpose, too.

> For the moment, I'm focusing on M-x package-install, like Philip is.
> There seems to be more consensus there that it should offer to update
> builtin packages that have never been updated.
> 
> I do believe there is high demand for a "upgrade/update" mechanism
> that just "updates whatever there is to update, don't care if core
> or whatnot" and people looking at package-update-all and
> package-menu--mark-upgrades (the "U" command Dmitry brought up)
> will eventually be disappointed.

I think "U" should only update the packages that are either not 
built-in, or the built-ins that have been at least updated once 
(meaning, some "external" version is already installed). For reasons of 
stability, mostly. But using 'M-x package-upgrade' would opt-in 
individual packages into that upgrading mechanism too.

That might not be everyone's cup of tea, so adding a user option like 
'package-upgrade-all-builtins' would work too, allowing the user to opt 
into the more risky behavior.

The semantics of 'package-install' are less clear to me. E.g. it 
wouldn't be out of the question to always error out when the package 
(some version) is already installed. So I could see it being "fixed" 
either way sometime in the future.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>   monnier@iro.umontreal.ca,
>   62720@debbugs.gnu.org,  larsi@gnus.org
> Date: Wed, 12 Apr 2023 19:39:20 +0000
> 
> >> Please, in normal non-shouting case, explain to me how you think
> >> that the behavior of an existing
> >> command can be changed with "completely separate code". 
> >
> > I already did: either (1) add a prefix argument to an existing
> > command, which will then trigger the new behavior, or (2) add a
> > separate command.
> 
> Here you have (1):

Thanks.  This is almost on-target, but it modifies
package-compute-transaction.  Is that necessary?

> +(defun package--upgradable-built-in-p (package)
> +  "Check if a built-in PACKAGE can be upgraded.
> +This command differs from `package-built-in-p' in that it only
   ^^^^^^^^^^^^
This is not a command, this is a function.

Also, the name has a problem I pointed out earlier in this discussion:
"upgradeable" does not tell well enough what the function tests.

> @@ -2187,7 +2210,9 @@ package-install
>    "Install the package PKG.
>  PKG can be a `package-desc' or a symbol naming one of the
>  available packages in an archive in `package-archives'.  When
> -called interactively, prompt for the package name.
> +called interactively, prompt for the package name.  When invoked
> +with a prefix argument, the prompt will include built-in packages
> +that can be upgraded via an archive.

I wonder whether an invocation with the prefix argument should include
_only_ built-in packages in the prompt?  This could be a useful
feature regardless, and so would allow us to keep this option for
future uses.

Finally, there's still discussion going on whether built-in packages
should be handled only by package-update, not by package-install,
since built-in packages are always "installed".  WDYT?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Wed, 12 Apr 2023 20:10:50 +0000
> 
> >> > And I thought we all agreed built-in packages need special treatment
> >> > anyway, didn't we?  Then why having a separate command is not a
> >> > natural next step?
> >> 
> >> I don't necessarily agree that "special treatment" requires a separate
> >> command.
> >
> > Even if you don't agree with that in general, having a separate
> > command would allow us to install that command on emacs-29 without any
> > fears.  So that alone is a significant advantage, even if the rest are
> > not yet agreed upon.
> 
> Would this be a permanent thing, or would the command be deprecated by
> emacs-30?  I get the need under the circumstances, but it doesn't seem
> like the best solution if we were to set aside release-concerns.

We could keep the separate command on master as well, at least until
it is obvious that a special command is not needed in any use case.
The time to revisit this is when we take some practical steps towards
removing from emacs.git packages that are on ELPA and figuring out how
to bundle them with a release tarball and how to let users upgrade
them after they install a released Emacs.  It could be that when we
implement all that, there will be some aspects of upgrading core
packages that will benefit from having a separate command.  If we find
this is not needed, we can deprecate the command at that time.

Mind you, the command I have in mind is one that will allow upgrading
_only_ built-in packages, i.e. it will not deal with the packages that
don't come with an Emacs tarball.

> >> I think it is wrong the assume that an built-in package should
> >> automatically be updated to a ELPA package whenever possible.
> >
> > This seems to be an argument in favor of a separate command?  Or what
> > did you mean by that, and how is it related to the issue at hand?
> 
> I don't think package-update should switch from a built-in package to a
> package that was installed from ELPA.  The user should at least once
> commit to the switch (be it with a separate command or package-install).

I tend to agree.  Although, once again, we should revisit this when
core packages are no longer in emacs.git.  Taking a perhaps somewhat
different analogy of updating my smartphone, there's no particular
difference between updating a "built-in" app, one that is part of the
smartphone's system components and came with it OOTB, and updating any
other app, including those I installed myself.  I think the same is
true for updating system components and applications from any modern
distro, isn't it?  So this is yet to be decided, but for now we can
use this more conservative policy.

> >  . install your changes on master only, and leave the problem of
> >    updating a core package unsolved in Emacs 29 (with the workaround
> >    mentioned in the beginning of this bug's discussion available to
> >    alleviate the problem to some extent)
> >  . come up with safer changes for package-install that could be
> >    installed on emacs-29
> >  . add a new command for updating a core package, which can then be
> >    safely installed on emacs-29
> >
> > The last 2 alternatives can be for emacs-29 only, whereas on master we
> > install your proposed change (or something else).
> 
> I would like to investigate option 2, but if nothing is found we can
> fall back to 3.  But even if there are issues in this case, I don't
> consider the matter in general to be that drastic.  If all Joao wants is
> to avoid confusion, we can also improve the error message that
> `package-install' generates when it says that a package ins already
> installed.

Yes, I think if we go with the modified package-install or
package-update, then invoking them in a way that doesn't update
built-in packages should reflect that in the error message, and also
tell how to invoke them to be able to upgrade a built-in package.

> >> One point that might be deliberated is that this means all built-in
> >> dependencies are also installed, even if these are not strictly
> >> necessary.  It shouldn't matter that much, since most users would
> >> upgrade them eventually, but worth mentioning I guess? 
> >
> > That just confirms my fears that we are opening a Pandora's box.  We
> > have no idea what this will do, and no time to test the results.
> > Unintended consequences are abundant.  We must draw any such
> > consequences to the absolute minimum, at least the way the commands
> > work by default.  Even if the result is less than elegant.
> 
> Intuitively I would want to argue that this change has an upper-bound to
> how much harm it could do, but as I cannot prove it in any way I'll
> rather not assert that the point.

There is, of course, an upper bound.  But I don't think we know how
high that is at this time.  Bitter experience has taught me that we
are not very good in predicting that.

> >> +  (and (not (assq (cond
> >> +                   ((package-desc-p package)
> >> +                    (package-desc-name package))
> >> +                   ((stringp package) (intern package))
> >> +                   ((symbolp package) package)
> >> +                   ((error "Unknown package format: %S" package)))
> >> +                  (package--alist)))
> >> +       (package-built-in-p package)))
> >
> > Why do we need all these conditions, where we didn't need them in the
> > current code?
> 
> Practically speaking these conditions are not necessary, I just added it
> in case there was the need to use the function elsewhere at some point.

Then maybe these conditions should be only on master.

> > Also, package-alist is documented as "alist of all packages available
> > for activation", so it is not clear how the fact that assq returns nil
> > is evidence that "the user has not installed a more recent version".
> > Looking at what package--alist and package-load-all-descriptors do, it
> > looks like they just collect packages that were downloaded into the
> > relevant directories?  Is that enough to consider any package not in
> > the list to be "not installed"?  
> 
> The point is that package.el will add all packages it manages (ie. are
> "available to activation") to package-alist.  Built-in/core packages are
> not managed by package.el, but just "acknowledged" via
> `package--builtins'.  So this looks like a safe assumption to me.

These subtleties should ideally be documented somewhere in package.el.
Without being aware of all that, it is hard to read the code of
package.el and understand what it does and why, and the names and doc
strings of the relevant symbols in most cases don't help, as they all
use vague and too-broad words to describe their purpose and effects.

> >                                  And what about the "more recent
> > version" condition -- this doesn't seem to be tested anywhere in
> > package--alist?
> 
> You are right, but this is a misphrasing.

How so? what should be said instead?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: João Távora <joaotavora@gmail.com>,
>>   monnier@iro.umontreal.ca,
>>   62720@debbugs.gnu.org,  larsi@gnus.org
>> Date: Wed, 12 Apr 2023 19:39:20 +0000
>> 
>> >> Please, in normal non-shouting case, explain to me how you think
>> >> that the behavior of an existing
>> >> command can be changed with "completely separate code". 
>> >
>> > I already did: either (1) add a prefix argument to an existing
>> > command, which will then trigger the new behavior, or (2) add a
>> > separate command.
>> 
>> Here you have (1):
>
> Thanks.  This is almost on-target, but it modifies
> package-compute-transaction.  Is that necessary?

I have found an alternative that doesn't change the way
`package-compute-transaction' works, but requires a small change in
`package-install':


[-- Attachment #2: Type: text/plain, Size: 3483 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..461e92f27d8 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2187,7 +2198,9 @@ package-install
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
 available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+called interactively, prompt for the package name.  When invoked
+with a prefix argument, the prompt will include built-in packages
+that can be upgraded via an archive.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2205,11 +2218,13 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and current-prefix-arg
+                                     (package--upgradable-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2221,11 +2236,16 @@ package-install
       (package--save-selected-packages
        (cons name package-selected-packages)))
     (if-let* ((transaction
-               (if (package-desc-p pkg)
-                   (unless (package-installed-p pkg)
-                     (package-compute-transaction (list pkg)
-                                                  (package-desc-reqs pkg)))
-                 (package-compute-transaction () (list (list pkg))))))
+               (cond
+                ((package--upgradable-built-in-p pkg)
+                 (let ((desc (cadr (assq name package-archive-contents))))
+                   (package-compute-transaction
+                    (list desc) (package-desc-reqs desc))))
+                ((package-desc-p pkg)
+                 (and (not (package-installed-p pkg))
+                      (package-compute-transaction
+                       (list pkg) (package-desc-reqs pkg))))
+                ((package-compute-transaction () (list (list pkg)))))))
         (progn
           (package-download-transaction transaction)
           (package--quickstart-maybe-refresh)

[-- Attachment #3: Type: text/plain, Size: 1893 bytes --]


The idea here is that if we detect that a package is built-in, we
"pre-compute" part of the transaction and resolve the rest.  This does
not install any unnecessary dependencies.

When `package-install' is invoked interactively without a prefix
argument, the "new" code cannot run, since none of the completion
candidates satisfy `package--upgradable-built-in-p' (or whatever we end
up calling the predicate).  Note that (package-install 'eglot) does
download code, but I believe that this is the correct approach and would
align with what João wanted.

>> +(defun package--upgradable-built-in-p (package)
>> +  "Check if a built-in PACKAGE can be upgraded.
>> +This command differs from `package-built-in-p' in that it only
>    ^^^^^^^^^^^^
> This is not a command, this is a function.

I will address these issues as soon as we have working code.

> Also, the name has a problem I pointed out earlier in this discussion:
> "upgradeable" does not tell well enough what the function tests.
>
>> @@ -2187,7 +2210,9 @@ package-install
>>    "Install the package PKG.
>>  PKG can be a `package-desc' or a symbol naming one of the
>>  available packages in an archive in `package-archives'.  When
>> -called interactively, prompt for the package name.
>> +called interactively, prompt for the package name.  When invoked
>> +with a prefix argument, the prompt will include built-in packages
>> +that can be upgraded via an archive.
>
> I wonder whether an invocation with the prefix argument should include
> _only_ built-in packages in the prompt?  This could be a useful
> feature regardless, and so would allow us to keep this option for
> future uses.
>
> Finally, there's still discussion going on whether built-in packages
> should be handled only by package-update, not by package-install,
> since built-in packages are always "installed".  WDYT?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 07:38:28 +0000
> 
> > Thanks.  This is almost on-target, but it modifies
> > package-compute-transaction.  Is that necessary?
> 
> I have found an alternative that doesn't change the way
> `package-compute-transaction' works, but requires a small change in
> `package-install':

Thanks.

> @@ -2205,11 +2218,13 @@ package-install
>       (package--archives-initialize)
>       (list (intern (completing-read
>                      "Install package: "
> -                    (delq nil
> -                          (mapcar (lambda (elt)
> -                                    (unless (package-installed-p (car elt))
> -                                      (symbol-name (car elt))))
> -                                  package-archive-contents))
> +                    (mapcan
> +                     (lambda (elt)
> +                       (and (or (and current-prefix-arg
> +                                     (package--upgradable-built-in-p (car elt)))
> +                                (not (package-installed-p (car elt))))
> +                            (list (car elt))))
> +                     package-archive-contents)

Why did the original code use symbol-name, but the new one doesn't?

> @@ -2221,11 +2236,16 @@ package-install
>        (package--save-selected-packages
>         (cons name package-selected-packages)))
>      (if-let* ((transaction
> -               (if (package-desc-p pkg)
> -                   (unless (package-installed-p pkg)
> -                     (package-compute-transaction (list pkg)
> -                                                  (package-desc-reqs pkg)))
> -                 (package-compute-transaction () (list (list pkg))))))
> +               (cond
> +                ((package--upgradable-built-in-p pkg)
> +                 (let ((desc (cadr (assq name package-archive-contents))))
> +                   (package-compute-transaction
> +                    (list desc) (package-desc-reqs desc))))
> +                ((package-desc-p pkg)
> +                 (and (not (package-installed-p pkg))
> +                      (package-compute-transaction
> +                       (list pkg) (package-desc-reqs pkg))))
> +                ((package-compute-transaction () (list (list pkg)))))))

I think the first condition of 'cond' should be

    ((and current-prefix-arg (package--upgradable-built-in-p pkg))

to make sure we don't affect the non-prefix-arg invocations in any
way.

> Note that (package-install 'eglot) does download code, but I believe
> that this is the correct approach and would align with what João
> wanted.

I'm not sure I follow: which code does the above download?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 07:38:28 +0000
>> 
>> > Thanks.  This is almost on-target, but it modifies
>> > package-compute-transaction.  Is that necessary?
>> 
>> I have found an alternative that doesn't change the way
>> `package-compute-transaction' works, but requires a small change in
>> `package-install':
>
> Thanks.
>
>> @@ -2205,11 +2218,13 @@ package-install
>>       (package--archives-initialize)
>>       (list (intern (completing-read
>>                      "Install package: "
>> -                    (delq nil
>> -                          (mapcar (lambda (elt)
>> -                                    (unless (package-installed-p (car elt))
>> -                                      (symbol-name (car elt))))
>> -                                  package-archive-contents))
>> +                    (mapcan
>> +                     (lambda (elt)
>> +                       (and (or (and current-prefix-arg
>> +                                     (package--upgradable-built-in-p (car elt)))
>> +                                (not (package-installed-p (car elt))))
>> +                            (list (car elt))))
>> +                     package-archive-contents)
>
> Why did the original code use symbol-name, but the new one doesn't?

To my knowledge, completing-read given a collection of symbols will use
the symbol names as candidates, or is this more complicated?

>> @@ -2221,11 +2236,16 @@ package-install
>>        (package--save-selected-packages
>>         (cons name package-selected-packages)))
>>      (if-let* ((transaction
>> -               (if (package-desc-p pkg)
>> -                   (unless (package-installed-p pkg)
>> -                     (package-compute-transaction (list pkg)
>> -                                                  (package-desc-reqs pkg)))
>> -                 (package-compute-transaction () (list (list pkg))))))
>> +               (cond
>> +                ((package--upgradable-built-in-p pkg)
>> +                 (let ((desc (cadr (assq name package-archive-contents))))
>> +                   (package-compute-transaction
>> +                    (list desc) (package-desc-reqs desc))))
>> +                ((package-desc-p pkg)
>> +                 (and (not (package-installed-p pkg))
>> +                      (package-compute-transaction
>> +                       (list pkg) (package-desc-reqs pkg))))
>> +                ((package-compute-transaction () (list (list pkg)))))))
>
> I think the first condition of 'cond' should be
>
>     ((and current-prefix-arg (package--upgradable-built-in-p pkg))
>
> to make sure we don't affect the non-prefix-arg invocations in any
> way.

The issue here is that this breaks the non-interactive invocations like
(package-install 'eglot), unless they invoke the function while binding
`current-prefix-arg', which I don't think is a common practice.

>> Note that (package-install 'eglot) does download code, but I believe
>> that this is the correct approach and would align with what João
>> wanted.
>
> I'm not sure I follow: which code does the above download?

I did not change any of the code that downloads anything, all this does
is prompt the user for built-in packages when invoked interactively with
a prefix argument and if package-install is invoked with a built-in
package, then it will switch to the ELPA version.  This will not happen
in interactive usage, since `completing-read' is called with
REQUIRE-MATCH.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 11:23:12 +0000
> 
> > Why did the original code use symbol-name, but the new one doesn't?
> 
> To my knowledge, completing-read given a collection of symbols will use
> the symbol names as candidates, or is this more complicated?

So you are saying that the symbol-name call in the original code was
simply redundant?

> >> @@ -2221,11 +2236,16 @@ package-install
> >>        (package--save-selected-packages
> >>         (cons name package-selected-packages)))
> >>      (if-let* ((transaction
> >> -               (if (package-desc-p pkg)
> >> -                   (unless (package-installed-p pkg)
> >> -                     (package-compute-transaction (list pkg)
> >> -                                                  (package-desc-reqs pkg)))
> >> -                 (package-compute-transaction () (list (list pkg))))))
> >> +               (cond
> >> +                ((package--upgradable-built-in-p pkg)
> >> +                 (let ((desc (cadr (assq name package-archive-contents))))
> >> +                   (package-compute-transaction
> >> +                    (list desc) (package-desc-reqs desc))))
> >> +                ((package-desc-p pkg)
> >> +                 (and (not (package-installed-p pkg))
> >> +                      (package-compute-transaction
> >> +                       (list pkg) (package-desc-reqs pkg))))
> >> +                ((package-compute-transaction () (list (list pkg)))))))
> >
> > I think the first condition of 'cond' should be
> >
> >     ((and current-prefix-arg (package--upgradable-built-in-p pkg))
> >
> > to make sure we don't affect the non-prefix-arg invocations in any
> > way.
> 
> The issue here is that this breaks the non-interactive invocations like
> (package-install 'eglot), unless they invoke the function while binding
> `current-prefix-arg', which I don't think is a common practice.

Then let's add another optional argument, and let prefix arg set it.
Would that resolve this issue?

> >> Note that (package-install 'eglot) does download code, but I believe
> >> that this is the correct approach and would align with what João
> >> wanted.
> >
> > I'm not sure I follow: which code does the above download?
> 
> I did not change any of the code that downloads anything, all this does
> is prompt the user for built-in packages when invoked interactively with
> a prefix argument and if package-install is invoked with a built-in
> package, then it will switch to the ELPA version.  This will not happen
> in interactive usage, since `completing-read' is called with
> REQUIRE-MATCH.

So you are saying that non-interactive calls to package-install could
install Eglot from ELPA even without the changes, is that right?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 11:23:12 +0000
>> 
>> > Why did the original code use symbol-name, but the new one doesn't?
>> 
>> To my knowledge, completing-read given a collection of symbols will use
>> the symbol names as candidates, or is this more complicated?
>
> So you are saying that the symbol-name call in the original code was
> simply redundant?

Yes, it appears so.

>> >> @@ -2221,11 +2236,16 @@ package-install
>> >>        (package--save-selected-packages
>> >>         (cons name package-selected-packages)))
>> >>      (if-let* ((transaction
>> >> -               (if (package-desc-p pkg)
>> >> -                   (unless (package-installed-p pkg)
>> >> -                     (package-compute-transaction (list pkg)
>> >> -                                                  (package-desc-reqs pkg)))
>> >> -                 (package-compute-transaction () (list (list pkg))))))
>> >> +               (cond
>> >> +                ((package--upgradable-built-in-p pkg)
>> >> +                 (let ((desc (cadr (assq name package-archive-contents))))
>> >> +                   (package-compute-transaction
>> >> +                    (list desc) (package-desc-reqs desc))))
>> >> +                ((package-desc-p pkg)
>> >> +                 (and (not (package-installed-p pkg))
>> >> +                      (package-compute-transaction
>> >> +                       (list pkg) (package-desc-reqs pkg))))
>> >> +                ((package-compute-transaction () (list (list pkg)))))))
>> >
>> > I think the first condition of 'cond' should be
>> >
>> >     ((and current-prefix-arg (package--upgradable-built-in-p pkg))
>> >
>> > to make sure we don't affect the non-prefix-arg invocations in any
>> > way.
>> 
>> The issue here is that this breaks the non-interactive invocations like
>> (package-install 'eglot), unless they invoke the function while binding
>> `current-prefix-arg', which I don't think is a common practice.
>
> Then let's add another optional argument, and let prefix arg set it.
> Would that resolve this issue?

That could solve it, but a user option might be more elegant.  We could
set it to nil for now, and change it to non-nil for the next release.

>> >> Note that (package-install 'eglot) does download code, but I believe
>> >> that this is the correct approach and would align with what João
>> >> wanted.
>> >
>> > I'm not sure I follow: which code does the above download?
>> 
>> I did not change any of the code that downloads anything, all this does
>> is prompt the user for built-in packages when invoked interactively with
>> a prefix argument and if package-install is invoked with a built-in
>> package, then it will switch to the ELPA version.  This will not happen
>> in interactive usage, since `completing-read' is called with
>> REQUIRE-MATCH.
>
> So you are saying that non-interactive calls to package-install could
> install Eglot from ELPA even without the changes, is that right?

No, my proposed diff changes what package decides to download (the
planning phase), but doesn't touch anything after that.  The current
state is that (package-install 'eglot) just prints

  ‘eglot’ is already installed

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Philip Kaludercic <philipk@posteo.net>
>>> Cc: João Távora <joaotavora@gmail.com>,
>>>   monnier@iro.umontreal.ca,
>>>   62720@debbugs.gnu.org,  larsi@gnus.org
>>> Date: Wed, 12 Apr 2023 19:39:20 +0000
>>> 
>>> >> Please, in normal non-shouting case, explain to me how you think
>>> >> that the behavior of an existing
>>> >> command can be changed with "completely separate code". 
>>> >
>>> > I already did: either (1) add a prefix argument to an existing
>>> > command, which will then trigger the new behavior, or (2) add a
>>> > separate command.
>>> 
>>> Here you have (1):
>>
>> Thanks.  This is almost on-target, but it modifies
>> package-compute-transaction.  Is that necessary?
>
> I have found an alternative that doesn't change the way
> `package-compute-transaction' works, but requires a small change in
> `package-install':

I have found a smaller but equivalent change that would also solve the
issue:


[-- Attachment #2: Type: text/plain, Size: 2782 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..842a475290d 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2187,7 +2198,9 @@ package-install
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
 available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+called interactively, prompt for the package name.  When invoked
+with a prefix argument, the prompt will include built-in packages
+that can be upgraded via an archive.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2205,11 +2218,13 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and current-prefix-arg
+                                     (package--upgradable-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2235,8 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (package--upgradable-built-in-p pkg)
+      (setq pkg (cadr (assq name package-archive-contents))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)

[-- Attachment #3: Type: text/plain, Size: 381 bytes --]


This relies on the fact that package-install handles package descriptor
objects in a different way than when you just call the function with a
package name.

This would also mean that an alternative solution to this issue would be
to tell users to evaluate

  (package-install (cadr (assq 'eglot package-archive-contents)))

but I don't know how user-friendly of an idea this is.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 15:10:01 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The issue here is that this breaks the non-interactive invocations like
> >> (package-install 'eglot), unless they invoke the function while binding
> >> `current-prefix-arg', which I don't think is a common practice.
> >
> > Then let's add another optional argument, and let prefix arg set it.
> > Would that resolve this issue?
> 
> That could solve it, but a user option might be more elegant.  We could
> set it to nil for now, and change it to non-nil for the next release.

Adding an option is fine by me, as long as its default preserves
previous behavior.

Just to be sure we are on the same page: you suggest _both_ prefix
argument and user option, where user option could be used to avoid the
need for prefix argument?

> >> >> Note that (package-install 'eglot) does download code, but I believe
> >> >> that this is the correct approach and would align with what João
> >> >> wanted.
> >> >
> >> > I'm not sure I follow: which code does the above download?
> >> 
> >> I did not change any of the code that downloads anything, all this does
> >> is prompt the user for built-in packages when invoked interactively with
> >> a prefix argument and if package-install is invoked with a built-in
> >> package, then it will switch to the ELPA version.  This will not happen
> >> in interactive usage, since `completing-read' is called with
> >> REQUIRE-MATCH.
> >
> > So you are saying that non-interactive calls to package-install could
> > install Eglot from ELPA even without the changes, is that right?
> 
> No, my proposed diff changes what package decides to download (the
> planning phase), but doesn't touch anything after that.  The current
> state is that (package-install 'eglot) just prints
> 
>   ‘eglot’ is already installed

And does nothing else?  You seem to be saying it still downloads
something, but what is that?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 15:14:51 +0000
> 
> > I have found an alternative that doesn't change the way
> > `package-compute-transaction' works, but requires a small change in
> > `package-install':
> 
> I have found a smaller but equivalent change that would also solve the
> issue:

Are we still sure we want to change package-install, not
package-upgrade?  AFAIU, there were several voices that preferred the
latter, with the rationale that a built-in package is always
"installed", so "installing" it makes little or no sense.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 13/04/2023 18:59, Eli Zaretskii wrote:
>> From: Philip Kaludercic<philipk@posteo.net>
>> Cc:joaotavora@gmail.com,monnier@iro.umontreal.ca,62720@debbugs.gnu.org,
>>    larsi@gnus.org
>> Date: Thu, 13 Apr 2023 15:14:51 +0000
>>
>>> I have found an alternative that doesn't change the way
>>> `package-compute-transaction' works, but requires a small change in
>>> `package-install':
>> I have found a smaller but equivalent change that would also solve the
>> issue:
> Are we still sure we want to change package-install, not
> package-upgrade?  AFAIU, there were several voices that preferred the
> latter, with the rationale that a built-in package is always
> "installed", so "installing" it makes little or no sense.

Note that this notion would probably go better together with changing 
package-install's behavior to never update, for any packages that are 
already installed. That would make it consistent, but it's also a 
breaking change.

So I really just mentioned it as a weak justification for not fixing 
package-install just now (so we'd make the decision about it later). 
Where the main cause for (not) doing that was your reticence for 
changing it this late in the cycle.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Why did the original code use symbol-name, but the new one doesn't?
> To my knowledge, completing-read given a collection of symbols will use
> the symbol names as candidates, or is this more complicated?

It mostly works.  Strictly speaking a completion table can be either:
- a list of strings
- an alist whose keys are either strings or symbols
and the code handles those two cases together so a list of symbols tend
to work just as well, but if your first symbol happens to be `lambda` or
`closure`, you're out of luck.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 15:10:01 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> The issue here is that this breaks the non-interactive invocations like
>> >> (package-install 'eglot), unless they invoke the function while binding
>> >> `current-prefix-arg', which I don't think is a common practice.
>> >
>> > Then let's add another optional argument, and let prefix arg set it.
>> > Would that resolve this issue?
>> 
>> That could solve it, but a user option might be more elegant.  We could
>> set it to nil for now, and change it to non-nil for the next release.
>
> Adding an option is fine by me, as long as its default preserves
> previous behavior.
>
> Just to be sure we are on the same page: you suggest _both_ prefix
> argument and user option, where user option could be used to avoid the
> need for prefix argument?

I was thinking about both, but I supposed that a user option would be
enough.

>> >> >> Note that (package-install 'eglot) does download code, but I believe
>> >> >> that this is the correct approach and would align with what João
>> >> >> wanted.
>> >> >
>> >> > I'm not sure I follow: which code does the above download?
>> >> 
>> >> I did not change any of the code that downloads anything, all this does
>> >> is prompt the user for built-in packages when invoked interactively with
>> >> a prefix argument and if package-install is invoked with a built-in
>> >> package, then it will switch to the ELPA version.  This will not happen
>> >> in interactive usage, since `completing-read' is called with
>> >> REQUIRE-MATCH.
>> >
>> > So you are saying that non-interactive calls to package-install could
>> > install Eglot from ELPA even without the changes, is that right?
>> 
>> No, my proposed diff changes what package decides to download (the
>> planning phase), but doesn't touch anything after that.  The current
>> state is that (package-install 'eglot) just prints
>> 
>>   ‘eglot’ is already installed
>
> And does nothing else?  You seem to be saying it still downloads
> something, but what is that?

No, it just prints that message but doesn't download anything.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 17:49:00 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Adding an option is fine by me, as long as its default preserves
> > previous behavior.
> >
> > Just to be sure we are on the same page: you suggest _both_ prefix
> > argument and user option, where user option could be used to avoid the
> > need for prefix argument?
> 
> I was thinking about both, but I supposed that a user option would be
> enough.

No, I think having both is better.  It is easier to say "C-u" than to
change the value of an option, so for one-off update of a single
package, the prefix argument is more convenient.

> >> No, my proposed diff changes what package decides to download (the
> >> planning phase), but doesn't touch anything after that.  The current
> >> state is that (package-install 'eglot) just prints
> >> 
> >>   ‘eglot’ is already installed
> >
> > And does nothing else?  You seem to be saying it still downloads
> > something, but what is that?
> 
> No, it just prints that message but doesn't download anything.

OK, then allowing to install such packages under the proposed changes
will improve that case as well.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 17:49:00 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Adding an option is fine by me, as long as its default preserves
>> > previous behavior.
>> >
>> > Just to be sure we are on the same page: you suggest _both_ prefix
>> > argument and user option, where user option could be used to avoid the
>> > need for prefix argument?
>> 
>> I was thinking about both, but I supposed that a user option would be
>> enough.
>
> No, I think having both is better.  It is easier to say "C-u" than to
> change the value of an option, so for one-off update of a single
> package, the prefix argument is more convenient.
>
>> >> No, my proposed diff changes what package decides to download (the
>> >> planning phase), but doesn't touch anything after that.  The current
>> >> state is that (package-install 'eglot) just prints
>> >> 
>> >>   ‘eglot’ is already installed
>> >
>> > And does nothing else?  You seem to be saying it still downloads
>> > something, but what is that?
>> 
>> No, it just prints that message but doesn't download anything.
>
> OK, then allowing to install such packages under the proposed changes
> will improve that case as well.

After having added the user option, I am not sure about the prefix
argument.  I see this as a temporary fix due to the time constraints of
releasing Emacs 29.  It is disabled for now, but can be enabled on
master to see if there are any problems.

But for now, this patch supports both the user option and the prefix
argument.  I am still not satisfied with the documentation, but cannot
come up with a better phrase than

  installing potentially newer versions of built-in packages from
  package archives

for explaining the issue without getting too technical.  Do you have any
ideas?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages-with-package-insta.patch --]
[-- Type: text/x-diff, Size: 4855 bytes --]

From 99e92d78560ef1aec7217d90ed3a8108bdf2924c Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 13 Apr 2023 20:13:59 +0200
Subject: [PATCH] Allow upgrading built-in packages with 'package-install'

* etc/NEWS: Mention the change
* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
predicate.
(package-install-upgrade-built-in): Add new user option to enable
feature.
(package-install): Respect new user option.
---
 etc/NEWS                   |  5 +++++
 lisp/emacs-lisp/package.el | 44 +++++++++++++++++++++++++++++++-------
 2 files changed, 41 insertions(+), 8 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 84dbb94a71a..a7834cd0d2b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1876,6 +1876,11 @@ package maintainers.
 By customizing this user option you can specify specific packages to
 install.
 
+---
+*** New user option 'package-install-upgrade-built-in'.
+When enabled, 'package-install' can be used to install potentially
+newer versions of built-in packages.
+
 ** Emacs Sessions (Desktop)
 
 +++
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..882db8db719 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--active-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2182,12 +2193,19 @@ package--archives-initialize
   (unless package-archive-contents
     (package-refresh-contents)))
 
+(defcustom package-install-upgrade-built-in nil
+  "Non-nil means that built-in packages can be upgraded via a package archive.
+If disabled, then `package-install' will not allow installing
+potentially newer versions of built-in packages from package
+archives."
+  :type 'boolean
+  :version "29.1")
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
-available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+available packages in an archive in `package-archives'.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2197,7 +2215,11 @@ package-install
 `package-selected-packages'.
 
 If PKG is a `package-desc' and it is already installed, don't try
-to install it but still mark it as selected."
+to install it but still mark it as selected.
+
+If the command is invoked with a prefix argument, the upgrading
+of built-in packages will be possible, as if
+`package-install-upgrade-built-in' had been enabled."
   (interactive
    (progn
      ;; Initialize the package system to get the list of package
@@ -2205,11 +2227,14 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and (or current-prefix-arg
+                                         package-install-upgrade-built-in)
+                                     (package--active-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (symbol-name (car elt)))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2245,9 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (and (or current-prefix-arg package-install-upgrade-built-in)
+               (package--active-built-in-p pkg))
+      (setq pkg (cadr (assq name package-archive-contents))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 15:14:51 +0000
>> 
>> > I have found an alternative that doesn't change the way
>> > `package-compute-transaction' works, but requires a small change in
>> > `package-install':
>> 
>> I have found a smaller but equivalent change that would also solve the
>> issue:
>
> Are we still sure we want to change package-install, not
> package-upgrade?  AFAIU, there were several voices that preferred the
> latter, with the rationale that a built-in package is always
> "installed", so "installing" it makes little or no sense.

If you want that, here is one proposal.  Should be the smallest so far:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages-with-package-insta.patch --]
[-- Type: text/x-diff, Size: 4855 bytes --]

From 99e92d78560ef1aec7217d90ed3a8108bdf2924c Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 13 Apr 2023 20:13:59 +0200
Subject: [PATCH] Allow upgrading built-in packages with 'package-install'

* etc/NEWS: Mention the change
* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
predicate.
(package-install-upgrade-built-in): Add new user option to enable
feature.
(package-install): Respect new user option.
---
 etc/NEWS                   |  5 +++++
 lisp/emacs-lisp/package.el | 44 +++++++++++++++++++++++++++++++-------
 2 files changed, 41 insertions(+), 8 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 84dbb94a71a..a7834cd0d2b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1876,6 +1876,11 @@ package maintainers.
 By customizing this user option you can specify specific packages to
 install.
 
+---
+*** New user option 'package-install-upgrade-built-in'.
+When enabled, 'package-install' can be used to install potentially
+newer versions of built-in packages.
+
 ** Emacs Sessions (Desktop)
 
 +++
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..882db8db719 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--active-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2182,12 +2193,19 @@ package--archives-initialize
   (unless package-archive-contents
     (package-refresh-contents)))
 
+(defcustom package-install-upgrade-built-in nil
+  "Non-nil means that built-in packages can be upgraded via a package archive.
+If disabled, then `package-install' will not allow installing
+potentially newer versions of built-in packages from package
+archives."
+  :type 'boolean
+  :version "29.1")
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
-available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+available packages in an archive in `package-archives'.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2197,7 +2215,11 @@ package-install
 `package-selected-packages'.
 
 If PKG is a `package-desc' and it is already installed, don't try
-to install it but still mark it as selected."
+to install it but still mark it as selected.
+
+If the command is invoked with a prefix argument, the upgrading
+of built-in packages will be possible, as if
+`package-install-upgrade-built-in' had been enabled."
   (interactive
    (progn
      ;; Initialize the package system to get the list of package
@@ -2205,11 +2227,14 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and (or current-prefix-arg
+                                         package-install-upgrade-built-in)
+                                     (package--active-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (symbol-name (car elt)))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2245,9 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (and (or current-prefix-arg package-install-upgrade-built-in)
+               (package--active-built-in-p pkg))
+      (setq pkg (cadr (assq name package-archive-contents))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 18:49:05 +0000
> 
> After having added the user option, I am not sure about the prefix
> argument.  I see this as a temporary fix due to the time constraints of
> releasing Emacs 29.  It is disabled for now, but can be enabled on
> master to see if there are any problems.

Yes, we can change the default to t on master.  But that would also
require to adjust a few doc strings, see below.

> But for now, this patch supports both the user option and the prefix
> argument.  I am still not satisfied with the documentation, but cannot
> come up with a better phrase than
> 
>   installing potentially newer versions of built-in packages from
>   package archives
> 
> for explaining the issue without getting too technical.  Do you have any
> ideas?

See below.

> +*** New user option 'package-install-upgrade-built-in'.
> +When enabled, 'package-install' can be used to install potentially
> +newer versions of built-in packages.

I suggest

  When enabled, 'package-install' will include in the list of
  upgradeable packages those built-in packages (like Eglot, for
  example) for which a newer version is available on GNU ELPA.  By
  default, this is disabled; however, if 'package-install' is invoked
  with prefix argument, it will also include built-in packages in the
  list of packages which could be upgraded.

> +(defcustom package-install-upgrade-built-in nil
> +  "Non-nil means that built-in packages can be upgraded via a package archive.
> +If disabled, then `package-install' will not allow installing
> +potentially newer versions of built-in packages from package
> +archives."

The last sentence should be qualified by "...unless invoked with a
prefix argument".

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 19:14:37 +0000
> 
> > Are we still sure we want to change package-install, not
> > package-upgrade?  AFAIU, there were several voices that preferred the
> > latter, with the rationale that a built-in package is always
> > "installed", so "installing" it makes little or no sense.
> 
> If you want that, here is one proposal.  Should be the smallest so far:

Hmm... looks identical to the previous patch you sent, which changes
package-install?  Or what am I missing?

As for the question I asked: I'm okay with changing package-install if
there are no objections from those who proposed to change
package-upgrade instead.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Fri, 14 Apr 2023 13:54:11 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Philip Kaludercic <philipk@posteo.net>
    >> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
    >> larsi@gnus.org
    >> Date: Thu, 13 Apr 2023 18:49:05 +0000
    >> 
    >> After having added the user option, I am not sure about the prefix
    >> argument.  I see this as a temporary fix due to the time constraints of
    >> releasing Emacs 29.  It is disabled for now, but can be enabled on
    >> master to see if there are any problems.

    Eli> Yes, we can change the default to t on master.  But that would also
    Eli> require to adjust a few doc strings, see below.

So on master if I upgrade all packages, ':core' packages would be
automatically upgraded as well? I strongly object to that as a
default; just because thereʼs a newer version on elpa of a :core
package doesnʼt mean emacs should upgrade to it unless *explicitly*
told to do so.

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 1:34 PM Robert Pluim <rpluim@gmail.com> wrote:

> So on master if I upgrade all packages, ':core' packages would be
> automatically upgraded as well?

By definition, all :core packages in master are already at their
newest version.

> I strongly object to that as a
> default; just because thereʼs a newer version on elpa of a :core

I really planned to sit this one out, but I'd like to make
sure people understand the implications of what they're asking for.

On Emacs 26, 27, 28 if the user has

  (package-install 'some-package-now-in-core)

in her configuration, it gets upgraded to the most recent version
there is.  In subsequent forms, the config can start doing stuff with
the variables and definitions in 'some-package-now-in-core', etc.
And the user can enjoy the newest features and bugfixes.

On Emacs 29 and later, the very same config will do nothing
and even probably/possibly break with an error.

Furthermore, the subtle problem will grow more serious and
bizarre as time goes on and "some-package-now-in-core" evolves.
It might not break for users who upgrade to 29 next month
and break for users who upgrade to 29 in 6 months' time, because
"some-package-now-in-core" will have evolved significantly.

> package doesnʼt mean emacs should upgrade to it unless *explicitly*
> told to do so.

I really don't understand why M-x package-install RET
<types-name-of-package> RET isn't explicit enough.  But I guess a
a confirmation prompt could be logical.  I haven't followed
all mails, maybe someone has proposed that?

As for non-interactive package-install, I guess that finding an
explicit `package-install` somewhere in the configuration is reason
enough to assume that the user meant for it to have the meaning
and effect it has always had before she upgraded to a version
where the same package happens to be in :core, and that meaning is
"upgrade to the newest".

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Philip Kaludercic <philipk@posteo.net>,  larsi@gnus.org,
>   62720@debbugs.gnu.org,  joaotavora@gmail.com,  monnier@iro.umontreal.ca
> Date: Fri, 14 Apr 2023 14:34:16 +0200
> 
> >>>>> On Fri, 14 Apr 2023 13:54:11 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> Yes, we can change the default to t on master.  But that would also
>     Eli> require to adjust a few doc strings, see below.
> 
> So on master if I upgrade all packages, ':core' packages would be
> automatically upgraded as well?

is that what the default of that option means?

> I strongly object to that as a default; just because thereʼs a newer
> version on elpa of a :core package doesnʼt mean emacs should upgrade
> to it unless *explicitly* told to do so.

I said we _can_ change the default; I didn't say we _must_.  If enough
people object to making that the default, it won't be changed.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Fri, 14 Apr 2023 13:56:05 +0100, João Távora <joaotavora@gmail.com> said:

    João> On Fri, Apr 14, 2023 at 1:34 PM Robert Pluim <rpluim@gmail.com> wrote:
    >> So on master if I upgrade all packages, ':core' packages would be
    >> automatically upgraded as well?

    João> By definition, all :core packages in master are already at their
    João> newest version.

    >> I strongly object to that as a
    >> default; just because thereʼs a newer version on elpa of a :core

    João> I really planned to sit this one out, but I'd like to make
    João> sure people understand the implications of what they're asking for.

    João> On Emacs 26, 27, 28 if the user has

    João>   (package-install 'some-package-now-in-core)

    João> in her configuration, it gets upgraded to the most recent version
    João> there is.  In subsequent forms, the config can start doing stuff with
    João> the variables and definitions in 'some-package-now-in-core', etc.
    João> And the user can enjoy the newest features and bugfixes.

    João> On Emacs 29 and later, the very same config will do nothing
    João> and even probably/possibly break with an error.

    João> Furthermore, the subtle problem will grow more serious and
    João> bizarre as time goes on and "some-package-now-in-core" evolves.
    João> It might not break for users who upgrade to 29 next month
    João> and break for users who upgrade to 29 in 6 months' time, because
    João> "some-package-now-in-core" will have evolved significantly.

    >> package doesnʼt mean emacs should upgrade to it unless *explicitly*
    >> told to do so.

I have no objection to catering to people who have already asked for
the installation of a package that is now :core. But one that wasnʼt
installed explicitly (ie itʼs only there because Emacs now ships it)
shouldnʼt be upgraded.

    João> I really don't understand why M-x package-install RET
    João> <types-name-of-package> RET isn't explicit enough.  But I guess a
    João> a confirmation prompt could be logical.  I haven't followed
    João> all mails, maybe someone has proposed that?

I donʼt know. Iʼd be fine with that.

    João> As for non-interactive package-install, I guess that finding an
    João> explicit `package-install` somewhere in the configuration is reason
    João> enough to assume that the user meant for it to have the meaning
    João> and effect it has always had before she upgraded to a version
    João> where the same package happens to be in :core, and that meaning is
    João> "upgrade to the newest".

I think weʼre pretty much in agreement :-)

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 2:52 PM Robert Pluim <rpluim@gmail.com> wrote:

> I have no objection to catering to people who have already asked for
> the installation of a package that is now :core. But one that wasnʼt
> installed explicitly (ie itʼs only there because Emacs now ships it)
> shouldnʼt be upgraded.

I think I don't fully understand.

Maybe you can answer this: if a user is setting up Emacs 28 in company
laptop machines regularly and does M-x package-install RET eglot RET there,
or has a script with (package-install 'eglot), should or shouldn't this
user, in your opinion, be allowed to do exactly the same, with the same
predictable results (in this case getting the latest Eglot), when she starts
doing the same in Emacs 29, which now contains Eglot as a built-in?

What about other builtin packages that, contrary to Eglot, already existed
in Emacs 28?  If your answer is different now, maybe we should record
that difference somewhere, so we know what packages came from ELPA into
:core and when, and can decide accordingly.  Who knows maybe just
whitelisting Eglot and any other packages that weren't built-in in 28
and are now built-in Emacs 29 is the answer to all this.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Maybe you can answer this: if a user is setting up Emacs 28 in company
> laptop machines regularly and does M-x package-install RET eglot RET there,
> or has a script with (package-install 'eglot), should or shouldn't this
> user, in your opinion, be allowed to do exactly the same, with the same
> predictable results (in this case getting the latest Eglot), when she starts
> doing the same in Emacs 29, which now contains Eglot as a built-in?

Good point.  I think the answer is that we should distinguish "install"
from "upgrade" (and offer a way to do both, of course).

And I think we do want to break backward compatibility here (arguably we
even can't not break compatibility), because the Emacs<29 semantics of
`package-install` is "broken", since it does "install&upgrade" for
non-builtin packages but not for builtin packages: either we keep that
semantics and compatibility is broken when packages move to/from
builtin, or we change that semantics and compatibility is broken by the
change in semantics :-)


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 16:40, Eli Zaretskii wrote:
>> So on master if I upgrade all packages, ':core' packages would be
>> automatically upgraded as well?
> is that what the default of that option means?
> 
>> I strongly object to that as a default; just because thereʼs a newer
>> version on elpa of a :core package doesnʼt mean emacs should upgrade
>> to it unless*explicitly*  told to do so.
> I said we_can_  change the default; I didn't say we_must_.  If enough
> people object to making that the default, it won't be changed.

We need to have a change in behavior that allows 'M-x package-install' 
to upgrade built-in packages. But that shouldn't automatically mean that 
package-menu-mark-upgrades marks all built-in packages for upgrade, or 
package-upgrade-all (nee package-update-all) does that either.

We could have another option that enables the latter to upgrade all 
built-ins too, of course.

Regarding the currently proposed user option, does it make sense to you 
to have such option that decides whether package-install upgrades 
built-ins? Whereas one can always upgrade a built-in package using 'i' 
(package-menu-mark-install) in the list-packages menu, no matter the 
value of that option. I get the backward-compatibility intent, but user 
options should also do something logical from a user's point of view.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 4:52 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > Maybe you can answer this: if a user is setting up Emacs 28 in company
> > laptop machines regularly and does M-x package-install RET eglot RET there,
> > or has a script with (package-install 'eglot), should or shouldn't this
> > user, in your opinion, be allowed to do exactly the same, with the same
> > predictable results (in this case getting the latest Eglot), when she starts
> > doing the same in Emacs 29, which now contains Eglot as a built-in?
>
> Good point.  I think the answer is that we should distinguish "install"
> from "upgrade" (and offer a way to do both, of course).
>
> And I think we do want to break backward compatibility here (arguably we
> even can't not break compatibility), because the Emacs<29 semantics of
> `package-install` is "broken", since it does "install&upgrade" for
> non-builtin packages but not for builtin packages: either we keep that
> semantics and compatibility is broken when packages move to/from
> builtin, or we change that semantics and compatibility is broken by the
> change in semantics :-)

I would think it's too late in the game to break compatibility.
Naming aside package-install has certain behaviour that for a certain
set of inputs used to produce predictable things.
Now, for the same inputs it does nothing on Emacs 29.

I think it should do the same thing, not only because it's
nicer for the unsuspecting user, but also because trying to
protect this user from "unintentional" upgrade of certain "unstable"
packages, as it seems to be the idea here, is a losing game
anyway, just because dependencies.  Presumably, the packaging
system where it takes less than a day to fix release a bug fix and
deliver this to the archives, should be the protection.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 18:34, João Távora wrote:
> On Fri, Apr 14, 2023 at 2:52 PM Robert Pluim<rpluim@gmail.com>  wrote:
> 
>> I have no objection to catering to people who have already asked for
>> the installation of a package that is now :core. But one that wasnʼt
>> installed explicitly (ie itʼs only there because Emacs now ships it)
>> shouldnʼt be upgraded.
> I think I don't fully understand.
> 
> Maybe you can answer this: if a user is setting up Emacs 28 in company
> laptop machines regularly and does M-x package-install RET eglot RET there,
> or has a script with (package-install 'eglot), should or shouldn't this
> user, in your opinion, be allowed to do exactly the same, with the same
> predictable results (in this case getting the latest Eglot), when she starts
> doing the same in Emacs 29, which now contains Eglot as a built-in?

I think Robert asked whether 'M-x package-menu-mark-upgrades' would mark 
Eglot for upgrade even if the user never marked it for upgrade 
explicitly before (e.g. with 'M-x package-install').

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I would think it's too late in the game to break compatibility.
> Naming aside package-install has certain behaviour that for a certain
> set of inputs used to produce predictable things.
> Now, for the same inputs it does nothing on Emacs 29.

The only way I can think of to preserve compatibility is to change the
behavior so it doesn't pay attention to "is builtin or not" but to "used
to be builtin before Emacs-29".  This would make a bad semantics even
worse, so I'd rather we fix the semantics to something clean.

> I think it should do the same thing, not only because it's
> nicer for the unsuspecting user, but also because trying to
> protect this user from "unintentional" upgrade of certain "unstable"
> packages, as it seems to be the idea here, is a losing game
> anyway, just because dependencies.

You may be right: maybe the distinction between "install only" and
"install&upgrade" isn't worth the trouble.

I think to get closer to a useful "install only" behavior we'd want that
command to prompt the user before upgrading dependencies (tho probably
only for those in `package-selected-packages`).

BTW, for me the reluctance to upgrade when asked to install isn't due to
the risk that the package is "unstable".  I'm not completely sure what
is the reason, admittedly, but it's closer to viewing it as a silent
"change of distribution", or maybe it's because I like to know when
behavior may change and merely installing a package shouldn't change
Emacs's behavior.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 5:12 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 14/04/2023 18:34, João Távora wrote:
> > On Fri, Apr 14, 2023 at 2:52 PM Robert Pluim<rpluim@gmail.com>  wrote:
> >
> >> I have no objection to catering to people who have already asked for
> >> the installation of a package that is now :core. But one that wasnʼt
> >> installed explicitly (ie itʼs only there because Emacs now ships it)
> >> shouldnʼt be upgraded.
> > I think I don't fully understand.
> >
> > Maybe you can answer this: if a user is setting up Emacs 28 in company
> > laptop machines regularly and does M-x package-install RET eglot RET there,
> > or has a script with (package-install 'eglot), should or shouldn't this
> > user, in your opinion, be allowed to do exactly the same, with the same
> > predictable results (in this case getting the latest Eglot), when she starts
> > doing the same in Emacs 29, which now contains Eglot as a built-in?
>
> I think Robert asked whether 'M-x package-menu-mark-upgrades' would mark
> Eglot for upgrade even if the user never marked it for upgrade
> explicitly before (e.g. with 'M-x package-install').

Sorry, I missed that.  This package-menu interface really seems like
its own thing, a separate package manager with its own nomenclature
and rules.

BTW use-package also calls package-install, and
(use-package eglot :ensure t) is the number one installation form
for Eglot says google, chatGPT and 4 out of 5 bugs I get.
Unless special steps are taken there, it will likely be broken,
too.  That form will no longer give users the latest Eglot starting
Emacs 29.  Fortunately, though the use-package manual "expects that
most users will find [package.el] capable enough", it also
explains how to switch away from package.el into a third party
package-manager. Given the state of this bug and for their sake,
I hope they do.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 19:28, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
>> I would think it's too late in the game to break compatibility.
>> Naming aside package-install has certain behaviour that for a certain
>> set of inputs used to produce predictable things.
>> Now, for the same inputs it does nothing on Emacs 29.
> 
> The only way I can think of to preserve compatibility is to change the
> behavior so it doesn't pay attention to "is builtin or not" but to "used
> to be builtin before Emacs-29".  This would make a bad semantics even
> worse, so I'd rather we fix the semantics to something clean.

We could add a new attribute/package header which would mean "eagerly 
upgradable [even when] built-in".

>> I think it should do the same thing, not only because it's
>> nicer for the unsuspecting user, but also because trying to
>> protect this user from "unintentional" upgrade of certain "unstable"
>> packages, as it seems to be the idea here, is a losing game
>> anyway, just because dependencies.
> 
> You may be right: maybe the distinction between "install only" and
> "install&upgrade" isn't worth the trouble.
> 
> I think to get closer to a useful "install only" behavior we'd want that
> command to prompt the user before upgrading dependencies (tho probably
> only for those in `package-selected-packages`).
> 
> BTW, for me the reluctance to upgrade when asked to install isn't due to
> the risk that the package is "unstable".  I'm not completely sure what
> is the reason, admittedly, but it's closer to viewing it as a silent
> "change of distribution",

That's my feeling too.

 > or maybe it's because I like to know when
behavior may change and merely installing a package shouldn't change
Emacs's behavior.

This is probably a different question: whether 'M-x package-install' 
should do upgrades at all, whether the package is built-in or not.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 19:14:37 +0000
>> 
>> > Are we still sure we want to change package-install, not
>> > package-upgrade?  AFAIU, there were several voices that preferred the
>> > latter, with the rationale that a built-in package is always
>> > "installed", so "installing" it makes little or no sense.
>> 
>> If you want that, here is one proposal.  Should be the smallest so far:
>
> Hmm... looks identical to the previous patch you sent, which changes
> package-install?  Or what am I missing?
>
> As for the question I asked: I'm okay with changing package-install if
> there are no objections from those who proposed to change
> package-upgrade instead.

Did I send the wrong patch.  I double-checked now and this one should be
using package-update:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-package-update-to-upgrade-built-in-packages.patch --]
[-- Type: text/x-diff, Size: 2283 bytes --]

From 7b3ca355785b7539dd417317442054647e66c045 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 13 Apr 2023 21:12:31 +0200
Subject: [PATCH] Allow 'package-update' to upgrade built-in packages

* lisp/emacs-lisp/package.el (package-update): Add support for
upgrading a built-in package to a version available from
ELPA.  (Bug#62720)
---
 lisp/emacs-lisp/package.el | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..8e7bc115a73 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2236,18 +2236,32 @@ package-install
 
 ;;;###autoload
 (defun package-update (name)
-  "Update package NAME if a newer version exists."
+  "Update package NAME if a newer version exists.
+When invoked with a prefix argument, the prompt will also include
+built-in packages that can be upgraded via a package archive."
   (interactive
    (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: "
+          (append
+           (and current-prefix-arg
+                (mapcan
+                 (lambda (elt)
+                   (and (not (assq (car elt) (package--alist)))
+                        (package-built-in-p (car elt))
+                        (list (symbol-name (car elt)))))
+                 package-archive-contents))
+           (package--updateable-packages))
+          nil t)))
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
-        (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
+         (old-desc (cadr (assq package package-alist)))
+         (new-desc (cadr (assq package package-archive-contents))))
+    (if (and old-desc (package-vc-p old-desc))
+        (package-vc-update old-desc)
+      (when old-desc
+        (package-delete old-desc 'force))
+      (package-install new-desc 'dont-select))))
 
 (defun package--updateable-packages ()
   ;; Initialize the package system to get the list of package
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 5:28 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> The only way I can think of to preserve compatibility is to change the
> behavior so it doesn't pay attention to "is builtin or not" but to "used
> to be builtin before Emacs-29".

Exactly, I suggested this via whiteslisting.  This would be blacklisting
and would be slightly cleaner.

> This would make a bad semantics even worse, so I'd rather we fix
> the semantics to something clean.

Bad semantics or not, at least it wouldn't break stuff, and that
blacklist would stay put.  Installing from it would require papal
benediction and that's it :-)

> > I think it should do the same thing, not only because it's
> > nicer for the unsuspecting user, but also because trying to
> > protect this user from "unintentional" upgrade of certain "unstable"
> > packages, as it seems to be the idea here, is a losing game
> > anyway, just because dependencies.
>
> You may be right: maybe the distinction between "install only" and
> "install&upgrade" isn't worth the trouble.
>
> I think to get closer to a useful "install only" behavior we'd want that
> command to prompt the user before upgrading dependencies (tho probably
> only for those in `package-selected-packages`).
>
> BTW, for me the reluctance to upgrade when asked to install isn't due to
> the risk that the package is "unstable".  I'm not completely sure what
> is the reason, admittedly, but it's closer to viewing it as a silent
> "change of distribution", or maybe it's because I like to know when
> behavior may change and merely installing a package shouldn't change
> Emacs's behavior.

I don't follow, they don't. Eexcept for major modes, that is, those autoload
changes into auto-mode-alist.  As for adding new commands, and
functionality to your existing modes, well you _did_ ask for the
package to be upgraded (yes I know the name is "install", but upgrading
is what it has always done).

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> BTW, for me the reluctance to upgrade when asked to install isn't due to
>> the risk that the package is "unstable".  I'm not completely sure what
>> is the reason, admittedly, but it's closer to viewing it as a silent
>> "change of distribution", or maybe it's because I like to know when
>> behavior may change and merely installing a package shouldn't change
>> Emacs's behavior.
[...]
> functionality to your existing modes, well you _did_ ask for the
> package to be upgraded

No, as the first line above shows, I'm talking about "asked to install",
not "asked to upgrade".


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

[...]

> Fortunately, though the use-package manual "expects that most users
> will find [package.el] capable enough", it also explains how to switch
> away from package.el into a third party package-manager. Given the
> state of this bug and for their sake, I hope they do.

As this issue only affects Eglot (a package that doesn't require that
much customisation to be used), I don't think the situation is as
drastic as you are portraying it to be.

If we all try to have a cooperative, non-alarmist attitude towards
solving the issue at hand, I am sure a suitable solution will be found.

Also keep in mind that I have proposed multiple patches that take
difference approaches.  Perhaps it would be of use to recapitulate them,
and you can explain which would be satisfying and which you think
wouldn't be as helpful:

- Use `package-install' to switch from a built-in package to the version
  from ELPA

  - Alternatively it has to be confirmed using a prefix argument
  - Alternatively it has to be enabled using a user option

- Use `package-update' (not `package-update-all') to switch from a
  built-in package to the version from ELPA

  (Same alternatives as above)

- Have both `package-update' and `package-update-all' switch potentially
  all packages from built-in versions to ELPA versions

  (Same alternatives as above)

- Provide a separate command to switch from a built-in version to a
  version from ELPA

That should be it?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 5:53 PM Philip Kaludercic <philipk@posteo.net> wrote:

> As this issue only affects Eglot

[Didn't other packages also make it into :core?]

> (a package that doesn't require that
> much customisation to be used), I don't think the situation is as
> drastic as you are portraying it to be.

> If we all try to have a cooperative, non-alarmist attitude towards
> solving the issue at hand, I am sure a suitable solution will be found.

It's true it doesn't have many customization now, so downright
erroring is probably rare.  But it's been advancing fast and
that may not always be the case. As I said Eglot 1.12 in Emacs
29 has lots of missing functionality and even bugs that are too
risky to fix in that version.

So I'm not being alarmist, I think.  With 5 years maintainership
of this package and well over 1000 issues, I think I make a fair
assessment of user's expectations.  As to being cooperative, I've
proposed 3 different patches already, answered every question about
them and proposed n other solutions.

But I like your optimism nonetheless :-)

> Also keep in mind that I have proposed multiple patches that take
> difference approaches.  Perhaps it would be of use to recapitulate them,
> and you can explain which would be satisfying and which you think
> wouldn't be as helpful:
>
> - Use `package-install' to switch from a built-in package to the version
>   from ELPA
>
>   - Alternatively it has to be confirmed using a prefix argument
>   - Alternatively it has to be enabled using a user option
>
> - Use `package-update' (not `package-update-all') to switch from a
>   built-in package to the version from ELPA
>
>   (Same alternatives as above)
>
> - Have both `package-update' and `package-update-all' switch potentially
>   all packages from built-in versions to ELPA versions
>
>   (Same alternatives as above)
>
> - Provide a separate command to switch from a built-in version to a
>   version from ELPA
>
> That should be it?

If it's not clear yet, I want(ed) something that users can use
_regardless_ of the version of Emacs to bring Eglot to the
latest version.  I want users to be able to do this easily
for obvious reasons.  The prime candidate for that "something"
is M-x package-install both in its command and non-interactive
form.  But if that isn't possible, the next best thing is
a 4 line eglot-update command in eglot.el which would eventually
boil down to the  same (because Emacs 26/27/28 users would eventually
also get it).  That idea was explicitly rejected, but I hope it
illustrates what I would prefer.

A command and function with enduring semantics, basically, the
kind you expect from a system like Emacs.

Any one of your or other's solutions that provide such a
command/function are most welcome.  Any other that doesn',
I'm indifferent to it.  Which doesn't mean "against" or "hostile".

And of course I thank you very much for your efforts in searching
for a solution.

Here's another solution you may want to consider.  package-install
in Emacs 29 updates built-ins non-interactively always.  Interactively
asks the user with a confirmation prompt.  This solution would
also work.

Here's yet another one. package-install like the above but, as has
been proposed, carefully vets the builtins that are subject to
from a controlled whitelist, which would include Eglot.  Stefan, Dmitry
and myself have proposed some variation of this.

Anyway, if my preference doesn't materialize, I'm going to recommend
in the manual:

M-: (package-install (alist-get 'eglot package-archive-contents) RET

which works interactively and non-interactively in every version
of Emacs.  It's not pretty but it's the best I have.  It's better
than to write "if you have emacs 29, do this, else do that".

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 14 Apr 2023 19:04:20 +0300
> Cc: 62720@debbugs.gnu.org, philipk@posteo.net, larsi@gnus.org,
>  monnier@iro.umontreal.ca, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/04/2023 16:40, Eli Zaretskii wrote:
> >> So on master if I upgrade all packages, ':core' packages would be
> >> automatically upgraded as well?
> > is that what the default of that option means?
> > 
> >> I strongly object to that as a default; just because thereʼs a newer
> >> version on elpa of a :core package doesnʼt mean emacs should upgrade
> >> to it unless*explicitly*  told to do so.
> > I said we_can_  change the default; I didn't say we_must_.  If enough
> > people object to making that the default, it won't be changed.
> 
> We need to have a change in behavior that allows 'M-x package-install' 
> to upgrade built-in packages. But that shouldn't automatically mean that 
> package-menu-mark-upgrades marks all built-in packages for upgrade, or 
> package-upgrade-all (nee package-update-all) does that either.
> 
> We could have another option that enables the latter to upgrade all 
> built-ins too, of course.
> 
> Regarding the currently proposed user option, does it make sense to you 
> to have such option that decides whether package-install upgrades 
> built-ins? Whereas one can always upgrade a built-in package using 'i' 
> (package-menu-mark-install) in the list-packages menu, no matter the 
> value of that option. I get the backward-compatibility intent, but user 
> options should also do something logical from a user's point of view.

All these questions should have been raised and discussed months ago.
Then we could have done the best for the users (what that is, is still
unclear even at this point).  Now it's too late, and the main factor
that will decide how Emacs 29.1 will behave in that regard is whether
the changes to implement that are safe enough to go into Emacs 29.1.

To answer your question more specifically: yes, it does make sense to
me.  The logic, so it seems, is in the eyes of the beholder, and my
eyes do see a certain logic there.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 20:43, Eli Zaretskii wrote:
>> Regarding the currently proposed user option, does it make sense to you
>> to have such option that decides whether package-install upgrades
>> built-ins? Whereas one can always upgrade a built-in package using 'i'
>> (package-menu-mark-install) in the list-packages menu, no matter the
>> value of that option. I get the backward-compatibility intent, but user
>> options should also do something logical from a user's point of view.
> All these questions should have been raised and discussed months ago.
> Then we could have done the best for the users (what that is, is still
> unclear even at this point).  Now it's too late, and the main factor
> that will decide how Emacs 29.1 will behave in that regard is whether
> the changes to implement that are safe enough to go into Emacs 29.1.

We certainly dropped the ball on this.

> To answer your question more specifically: yes, it does make sense to
> me.  The logic, so it seems, is in the eyes of the beholder, and my
> eyes do see a certain logic there.

Perhaps it does, and my concern is in vain.

But if we ultimately decide that it has weird semantics and adds 
undesirable cost to improving the package further, the 
obsoletion-deprecation cycle has always been a pain, so I really prefer 
to solve these issues without adding options, when possible.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 17:05:30 +0100
> Cc: Robert Pluim <rpluim@gmail.com>, 62720@debbugs.gnu.org, larsi@gnus.org, 
> 	Eli Zaretskii <eliz@gnu.org>, Philip Kaludercic <philipk@posteo.net>
> 
> On Fri, Apr 14, 2023 at 4:52 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > And I think we do want to break backward compatibility here (arguably we
> > even can't not break compatibility), because the Emacs<29 semantics of
> > `package-install` is "broken", since it does "install&upgrade" for
> > non-builtin packages but not for builtin packages: either we keep that
> > semantics and compatibility is broken when packages move to/from
> > builtin, or we change that semantics and compatibility is broken by the
> > change in semantics :-)
> 
> I would think it's too late in the game to break compatibility.
> Naming aside package-install has certain behaviour that for a certain
> set of inputs used to produce predictable things.
> Now, for the same inputs it does nothing on Emacs 29.

When you say "compatibility", you seem to have only one its aspect in
mind: that of Eglot.  But that is not the only aspect of the previous
behavior, and I, at least, must consider those other aspects as well.

That package-install doesn't upgrade core packages was how it behaved
in past versions.  In Emacs 29.1 I hope we will allow at least
overriding that by user-level means, so we will be closer to your
(Eglot-centric) ideal, without also breaking the other aspects of
previous behavior, since there are other core packages on ELPA besides
Eglot, and some of them were in that state before Emacs 29.

And that is all we can reasonably do at this time, guiven how close we
are to the release.

> I think it should do the same thing, not only because it's
> nicer for the unsuspecting user, but also because trying to
> protect this user from "unintentional" upgrade of certain "unstable"
> packages, as it seems to be the idea here, is a losing game
> anyway, just because dependencies.

"The same thing" for Eglot means "not the same thing" for other core
packages.  So you are in effect calling for breaking everyone else to
cater only to Eglot.  That is not going to happen, for more than one
reason.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 17:31:24 +0100
> Cc: Robert Pluim <rpluim@gmail.com>, Philip Kaludercic <philipk@posteo.net>, 62720@debbugs.gnu.org, 
> 	Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> BTW use-package also calls package-install, and
> (use-package eglot :ensure t) is the number one installation form
> for Eglot says google, chatGPT and 4 out of 5 bugs I get.
> Unless special steps are taken there, it will likely be broken,
> too.  That form will no longer give users the latest Eglot starting
> Emacs 29.

Is there any reason Eglot that will come with Emacs 29.1 will not be
the latest Eglot?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 14 Apr 2023 20:47:01 +0300
> Cc: rpluim@gmail.com, 62720@debbugs.gnu.org, philipk@posteo.net,
>  larsi@gnus.org, monnier@iro.umontreal.ca, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> I really prefer to solve these issues without adding options, when
> possible.

As a general principle, I cannot agree more, of course.  But we are
not talking in general here, and have some unique constraints as well.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Fri, Apr 14, 2023 at 5:53 PM Philip Kaludercic <philipk@posteo.net> wrote:
>
>> As this issue only affects Eglot
>
> [Didn't other packages also make it into :core?]

I made a mistake in assuming that the rest are non-user facing
libraries, but there also is use-package.  But since that is also a
slower package, I don't think this is an emergency.

>> (a package that doesn't require that
>> much customisation to be used), I don't think the situation is as
>> drastic as you are portraying it to be.
>
>> If we all try to have a cooperative, non-alarmist attitude towards
>> solving the issue at hand, I am sure a suitable solution will be found.
>
> It's true it doesn't have many customization now, so downright
> erroring is probably rare.  But it's been advancing fast and
> that may not always be the case. As I said Eglot 1.12 in Emacs
> 29 has lots of missing functionality and even bugs that are too
> risky to fix in that version.
>
> So I'm not being alarmist, I think.  With 5 years maintainership
> of this package and well over 1000 issues, I think I make a fair
> assessment of user's expectations.  As to being cooperative, I've
> proposed 3 different patches already, answered every question about
> them and proposed n other solutions.
>
> But I like your optimism nonetheless :-)

It follows from my experience.

>> Also keep in mind that I have proposed multiple patches that take
>> difference approaches.  Perhaps it would be of use to recapitulate them,
>> and you can explain which would be satisfying and which you think
>> wouldn't be as helpful:
>>
>> - Use `package-install' to switch from a built-in package to the version
>>   from ELPA
>>
>>   - Alternatively it has to be confirmed using a prefix argument
>>   - Alternatively it has to be enabled using a user option
>>
>> - Use `package-update' (not `package-update-all') to switch from a
>>   built-in package to the version from ELPA
>>
>>   (Same alternatives as above)
>>
>> - Have both `package-update' and `package-update-all' switch potentially
>>   all packages from built-in versions to ELPA versions
>>
>>   (Same alternatives as above)
>>
>> - Provide a separate command to switch from a built-in version to a
>>   version from ELPA
>>
>> That should be it?
>
> If it's not clear yet, I want(ed) something that users can use
> _regardless_ of the version of Emacs to bring Eglot to the
> latest version.  I want users to be able to do this easily
> for obvious reasons.  The prime candidate for that "something"
> is M-x package-install both in its command and non-interactive
> form.  But if that isn't possible, the next best thing is
> a 4 line eglot-update command in eglot.el which would eventually
> boil down to the  same (because Emacs 26/27/28 users would eventually
> also get it).  That idea was explicitly rejected, but I hope it
> illustrates what I would prefer.
>
> A command and function with enduring semantics, basically, the
> kind you expect from a system like Emacs.
>
> Any one of your or other's solutions that provide such a
> command/function are most welcome.  Any other that doesn',
> I'm indifferent to it.  Which doesn't mean "against" or "hostile".
>
> And of course I thank you very much for your efforts in searching
> for a solution.
>
> Here's another solution you may want to consider.  package-install
> in Emacs 29 updates built-ins non-interactively always.  Interactively
> asks the user with a confirmation prompt.  This solution would
> also work.

What would the confirmation prompt ask the user?

> Here's yet another one. package-install like the above but, as has
> been proposed, carefully vets the builtins that are subject to
> from a controlled whitelist, which would include Eglot.  Stefan, Dmitry
> and myself have proposed some variation of this.
>
> Anyway, if my preference doesn't materialize, I'm going to recommend
> in the manual:
>
> M-: (package-install (alist-get 'eglot package-archive-contents)) RET

Are you sure?  The `alist-get' call returns a list, since
`package-archive-contents' is a map of package names to a list of
package descriptors.

> which works interactively and non-interactively in every version
> of Emacs.  It's not pretty but it's the best I have.  It's better
> than to write "if you have emacs 29, do this, else do that".
>
> João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 6:49 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > I think it should do the same thing, not only because it's
> > nicer for the unsuspecting user, but also because trying to
> > protect this user from "unintentional" upgrade of certain "unstable"
> > packages, as it seems to be the idea here, is a losing game
> > anyway, just because dependencies.
>
> "The same thing" for Eglot means "not the same thing" for other core
> packages.  So you are in effect calling for breaking everyone else to
> cater only to Eglot.  That is not going to happen, for more than one
> reason.

> when you say "compatibility", you seem to have only one its aspect in
> mind: that of Eglot.  But that is not the only aspect of the previous
> behavior, and I, at least, must consider those other aspects as well.

You seem to be worried about "everyone else" typing, say, M-x package-install
RET seq RET and getting an updated 'seq' as a result.  OK, but who does
this and why, in your opinion?  And who has `(package-install seq)` in
their config and why?  And won't they get the same updated 'seq'
"accidentally" if they install anything else that depends on newer seq,
which is likely a lot of non-core packages and likely to grow?  I just
can't see these "other aspects", i.e. who or whose use cases you wish to
protect with multiple user knobs to get a razor thin level of protection.

But even if these people and use cases did exist, you're still
plainly misrepresenting my position by writing that I'm "calling
for breaking" them.  I even proposed making a simple whitelist
of packages that have migrated from outside core to core.  And
I've proposed confirmation prompts for the interactive calls.
And others have proposed blacklists.  These things are trivial
to implement in Elisp.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 7:26 PM Philip Kaludercic <philipk@posteo.net> wrote:

> > Here's another solution you may want to consider.  package-install
> > in Emacs 29 updates built-ins non-interactively always.  Interactively
> > asks the user with a confirmation prompt.  This solution would
> > also work.
>
> What would the confirmation prompt ask the user?

"Are you sure you want to install Foo which is [also|now] a built-in
package in Emacs 29?"

> Are you sure?  The `alist-get' call returns a list, since
> `package-archive-contents' is a map of package names to a list of
> package descriptors.

Yeah, probably needs a cdr or car there for good measure.  Great.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 6:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 14 Apr 2023 17:31:24 +0100
> > Cc: Robert Pluim <rpluim@gmail.com>, Philip Kaludercic <philipk@posteo.net>, 62720@debbugs.gnu.org,
> >       Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, monnier@iro.umontreal.ca
> >
> > BTW use-package also calls package-install, and
> > (use-package eglot :ensure t) is the number one installation form
> > for Eglot says google, chatGPT and 4 out of 5 bugs I get.
> > Unless special steps are taken there, it will likely be broken,
> > too.  That form will no longer give users the latest Eglot starting
> > Emacs 29.
>
> Is there any reason Eglot that will come with Emacs 29.1 will not be
> the latest Eglot?

Yes, it's not the latest Eglot already.  Eglot has been getting features
complex features and bugfixes in master that I haven't backported.
Eglot with Emacs 29.1 will be 1.12+simple bugfixes.

So if you're offline or don't like ELPA at all or are on sshing to a
machine, you can still have a decent LSP experience (well provided
you install a LSP server).


João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 19:32:40 +0100
> Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org, philipk@posteo.net
> 
> On Fri, Apr 14, 2023 at 6:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > when you say "compatibility", you seem to have only one its aspect in
> > mind: that of Eglot.  But that is not the only aspect of the previous
> > behavior, and I, at least, must consider those other aspects as well.
> 
> You seem to be worried about "everyone else" typing, say, M-x package-install
> RET seq RET and getting an updated 'seq' as a result.

That, too.  But also everything else.  Any incompatible change of
behavior can potentially break someone's workflow.

> OK, but who does this and why, in your opinion?  And who has
> `(package-install seq)` in their config and why?  And won't they get
> the same updated 'seq' "accidentally" if they install anything else
> that depends on newer seq, which is likely a lot of non-core
> packages and likely to grow?

I don't know.  But we did have core packages that were also on ELPA
before Emacs 29, and people did get along with that.  So much so that
I don't recall any complaints about this, definitely not complaints
that claimed package.el is as badly broken as you seem to represent.

> But even if these people and use cases did exist, you're still
> plainly misrepresenting my position by writing that I'm "calling
> for breaking" them.

I said "in effect calling for breaking them".  You need to realize
this might be the outcome of what you are requesting, even if your
intentions are benign.

> I even proposed making a simple whitelist of packages that have
> migrated from outside core to core.  And I've proposed confirmation
> prompts for the interactive calls.  And others have proposed
> blacklists.  These things are trivial to implement in Elisp.

They are not trivial enough to be considered for emacs-29.  On master,
sure, feel free to install such changes, if the others agree.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 19:44:35 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Fri, Apr 14, 2023 at 6:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Is there any reason Eglot that will come with Emacs 29.1 will not be
> > the latest Eglot?
> 
> Yes, it's not the latest Eglot already.  Eglot has been getting features
> complex features and bugfixes in master that I haven't backported.

Why not?

More generally, why does it make sense not to have in Emacs the latest
stable version of every core package at the time of the release?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 7:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 14 Apr 2023 19:32:40 +0100
> > Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, 62720@debbugs.gnu.org,
> >       larsi@gnus.org, philipk@posteo.net
> >
> > On Fri, Apr 14, 2023 at 6:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > when you say "compatibility", you seem to have only one its aspect in
> > > mind: that of Eglot.  But that is not the only aspect of the previous
> > > behavior, and I, at least, must consider those other aspects as well.
> >
> > You seem to be worried about "everyone else" typing, say, M-x package-install
> > RET seq RET and getting an updated 'seq' as a result.
>
> That, too.  But also everything else.  Any incompatible change of
> behavior can potentially break someone's workflow.

Of course.  But what is this "everything else" pandora's box that
the solution I'm proposing unleashes?  I just don't see the
connection.

> > OK, but who does this and why, in your opinion?  And who has
> > `(package-install seq)` in their config and why?  And won't they get
> > the same updated 'seq' "accidentally" if they install anything else
> > that depends on newer seq, which is likely a lot of non-core
> > packages and likely to grow?
>
> I don't know.  But we did have core packages that were also on ELPA
> before Emacs 29, and people did get along with that.  So much so that
> I don't recall any complaints about this, definitely not complaints
> that claimed package.el is as badly broken as you seem to represent.

I think it's simply and clearly because there weren't any non-core
packages that migrated to core before.  Were there?  Certainly
none with as fast a release cycle like Eglot's.

> > But even if these people and use cases did exist, you're still
> > plainly misrepresenting my position by writing that I'm "calling
> > for breaking" them.
>
> I said "in effect calling for breaking them".  You need to realize
> this might be the outcome of what you are requesting, even if your
> intentions are benign.

I _am_ realizing that, and even if I think the risks are minuscule,
I'm providing solutions that mitigate them.  And -- responsibly, I
think -- pointing to the fact that the risk surface is much larger
than you may think it is.

> > I even proposed making a simple whitelist of packages that have
> > migrated from outside core to core.  And I've proposed confirmation
> > prompts for the interactive calls.  And others have proposed
> > blacklists.  These things are trivial to implement in Elisp.
>
> They are not trivial enough to be considered for emacs-29.  On master,
> sure, feel free to install such changes, if the others agree.

Are you saying it's OK to propose this blacklist/whitelist idea?
I will work and prepare that patch if you promise to at least
look at the code, follow the logic with me and consider the
patch for backporting if it's as safe as relying on a simple
`(if (memq package package-safely-upgradeable-builtins) ...)`

If you don't want to do that, I'm not going to propose it, because
that in effect means I'm adding code that will only have any effect
in the Emacs 30 release, and the Emacs 29 will be a very odd
release in this aspect for several years to come.  And this is an even
worse outcome.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 21:51, Eli Zaretskii wrote:
> More generally, why does it make sense not to have in Emacs the latest
> stable version of every core package at the time of the release?

Same reason we're now arguing about adding a simple change to package.el 
(which is also a core package)? Stability?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 20:03:31 +0100
> Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org, philipk@posteo.net
> 
> > > OK, but who does this and why, in your opinion?  And who has
> > > `(package-install seq)` in their config and why?  And won't they get
> > > the same updated 'seq' "accidentally" if they install anything else
> > > that depends on newer seq, which is likely a lot of non-core
> > > packages and likely to grow?
> >
> > I don't know.  But we did have core packages that were also on ELPA
> > before Emacs 29, and people did get along with that.  So much so that
> > I don't recall any complaints about this, definitely not complaints
> > that claimed package.el is as badly broken as you seem to represent.
> 
> I think it's simply and clearly because there weren't any non-core
> packages that migrated to core before.  Were there?  Certainly
> none with as fast a release cycle like Eglot's.

You are missing the point.  My point is that users of all those other
core packages are used to the current situation, where package-install
doesn't update the built-ins.  They have workflows based on that.
Those are the workflows that I don't want to risk breaking.

Once the discussion with Philip runs to its completion, we _will_ have
in Emacs 29.1 a way of upgrading built-ins, only not by default, but
by an explicit user request.  Maybe this is not ideal, but we don't
have time to find out what is ideal for all those involved, and this
partial solution allows to get the job done without risking to break
anyone's workflow.  So this is a good compromise, given the time and
the constraints, at least from my POV.

> > I said "in effect calling for breaking them".  You need to realize
> > this might be the outcome of what you are requesting, even if your
> > intentions are benign.
> 
> I _am_ realizing that, and even if I think the risks are minuscule,
> I'm providing solutions that mitigate them.  And -- responsibly, I
> think -- pointing to the fact that the risk surface is much larger
> than you may think it is.

I respectfully disagree with your assessment of the risks.  And
eventually, it is my responsibility to do this job, so my assessment
of that is what counts more.  I've heard all of your arguments (and
also those of others), and considered all of them, and I still don't
think the risk is high enough to justify the potentially-breaking
changes that you propose, not on the release branch, not this late
into the release cycle.

> > > I even proposed making a simple whitelist of packages that have
> > > migrated from outside core to core.  And I've proposed confirmation
> > > prompts for the interactive calls.  And others have proposed
> > > blacklists.  These things are trivial to implement in Elisp.
> >
> > They are not trivial enough to be considered for emacs-29.  On master,
> > sure, feel free to install such changes, if the others agree.
> 
> Are you saying it's OK to propose this blacklist/whitelist idea?

For the release branch?  No.  For master?  Yes.

> I will work and prepare that patch if you promise to at least
> look at the code, follow the logic with me and consider the
> patch for backporting if it's as safe as relying on a simple
> `(if (memq package package-safely-upgradeable-builtins) ...)`

Backporting can be considered for Emacs 29.2, not before that.
Hopefully, by that time we will also have more data points regarding
the various uses of this, in particular with Eglot, so that our
decisions will be more informed.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 14 Apr 2023 22:14:44 +0300
> Cc: rpluim@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/04/2023 21:51, Eli Zaretskii wrote:
> > More generally, why does it make sense not to have in Emacs the latest
> > stable version of every core package at the time of the release?
> 
> Same reason we're now arguing about adding a simple change to package.el 
> (which is also a core package)? Stability?

So Eglot 1.14 is not stable?  Does Eglot has a stable branch on ELPA?
I meant to include the latest stable release, not an unstable one.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 7:51 PM Eli Zaretskii <eliz@gnu.org> wrote:

> Why not?
>
> More generally, why does it make sense not to have in Emacs the latest
> stable version of every core package at the time of the release?

Well, my answer to that would be: for the same reason it makes sense
for like new features not to go into Emacs release branches after
they have been cut.

The Eglot release 1.14 is not as stable as 1.12.29. It has new features.
Also a recent bugfix in commit a74403adda0 is quite complex and
I'm cautiously waiting for feedback.  Other simple bugfixes have
been backported.

So I'm not much different than most maintainers in that aspect, I suppose.
I also think stock Emacs releases should be very stable, despite
what you may think from my stance on this particular matter :-)

But if someone types M-x install-something they should get what
they ask for.  If they want to be 100% safe, they just shouldn't
invoke commands that download, compile and evaluate code.

Notwithstanding this personal opinion, I underline again that
it _is_ possible to craft a simple, emacs-29-safe modification,
to the package-install function that is even more cautious to
download certain types of things.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 22:19, Eli Zaretskii wrote:
> So Eglot 1.14 is not stable?  Does Eglot has a stable branch on ELPA?

All Eglot development is led on master.

> I meant to include the latest stable release, not an unstable one.

We consider tagged releases (ones that bump the version header) as 
"stable" ones, and all intermediate commits as "unstable". Those are not 
the same kind of "stable" as Emacs releases, obviously.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 20:20:20 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Fri, Apr 14, 2023 at 7:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> The Eglot release 1.14 is not as stable as 1.12.29. It has new features.
> Also a recent bugfix in commit a74403adda0 is quite complex and
> I'm cautiously waiting for feedback.  Other simple bugfixes have
> been backported.

So you don't recommend that users who want a stable Eglot upgrade to
1.14?  If so, why is it a problem that package-install by default
doesn't update built-in packages?  Users who want the cutting edge of
Eglot, and don't mind some instability, can always switch to the
master branch of Emacs, where we are free to change package-install to
upgrade core packages by default.

> But if someone types M-x install-something they should get what
> they ask for.  If they want to be 100% safe, they just shouldn't
> invoke commands that download, compile and evaluate code.

The logic should be consistent.  Emacs 29 is the stable branch of
Emacs, so it should come with the latest stable Eglot.  If that is
Eglot 1.12.29, then the fact that package-install won't upgrade it to
1.14 is consistent with the stability of Eglot's versions.  If, OTOH,
you think that it's imperative to allow _all_ users of Eglot with
Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
become available), then we should release Emacs 29 with 1.14.

> Notwithstanding this personal opinion, I underline again that
> it _is_ possible to craft a simple, emacs-29-safe modification,
> to the package-install function that is even more cautious to
> download certain types of things.

Philip presented such a safe modification, and we are in the final
stages of discussing its details, before it will be installed.  So
yes, it is possible.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 14 Apr 2023 22:21:41 +0300
> Cc: joaotavora@gmail.com, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/04/2023 22:19, Eli Zaretskii wrote:
> > So Eglot 1.14 is not stable?  Does Eglot has a stable branch on ELPA?
> 
> All Eglot development is led on master.
> 
> > I meant to include the latest stable release, not an unstable one.
> 
> We consider tagged releases (ones that bump the version header) as 
> "stable" ones, and all intermediate commits as "unstable". Those are not 
> the same kind of "stable" as Emacs releases, obviously.

So which version of Eglot is sufficiently "stable" to go with the
stable version of Emacs?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 8:18 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 14 Apr 2023 20:03:31 +0100
> > Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, 62720@debbugs.gnu.org,
> >       larsi@gnus.org, philipk@posteo.net
> >
> > > > OK, but who does this and why, in your opinion?  And who has
> > > > `(package-install seq)` in their config and why?  And won't they get
> > > > the same updated 'seq' "accidentally" if they install anything else
> > > > that depends on newer seq, which is likely a lot of non-core
> > > > packages and likely to grow?
> > >
> > > I don't know.  But we did have core packages that were also on ELPA
> > > before Emacs 29, and people did get along with that.  So much so that
> > > I don't recall any complaints about this, definitely not complaints
> > > that claimed package.el is as badly broken as you seem to represent.
> >
> > I think it's simply and clearly because there weren't any non-core
> > packages that migrated to core before.  Were there?  Certainly
> > none with as fast a release cycle like Eglot's.
>
> You are missing the point.  My point is that users of all those other
> core packages are used to the current situation, where package-install
> doesn't update the built-ins.  They have workflows based on that.
> Those are the workflows that I don't want to risk breaking.

The only workflows that traverse this use case I can think of:

1. they routinely invoke a useless M-x package-install RET seq RET
, it prints "already installed" and they breathe a daily sigh of relief.

2. they have `(package-install 'seq)` somewhere in their init file
which runs daily and does nothing.

> Once the discussion with Philip runs to its completion, we _will_ have
> in Emacs 29.1 a way of upgrading built-ins, only not by default, but
> by an explicit user request.  Maybe this is not ideal, but we don't
> have time to find out what is ideal for all those involved, and this
> partial solution allows to get the job done without risking to break
> anyone's workflow.  So this is a good compromise, given the time and
> the constraints, at least from my POV.

There is already this way, that was never in question.  Package menu,
M-: or anything else, it's not _that_ hard.  But it's a shame that
the canonical use-package recipe for Eglot (or for use-package itself)
it will break.

Looking at the patches, it seems, from my POV, that using a prefix
argument and an extra customization option is much, much more
complicated than a simple whitelist or blacklist that the user
needn't ever see.

> > I _am_ realizing that, and even if I think the risks are minuscule,
> > I'm providing solutions that mitigate them.  And -- responsibly, I
> > think -- pointing to the fact that the risk surface is much larger
> > than you may think it is.
> I respectfully disagree with your assessment of the risks.  And
> eventually, it is my responsibility to do this job, so my assessment
> of that is what counts more.  I've heard all of your arguments (and
> also those of others), and considered all of them, and I still don't
> think the risk is high enough to justify the potentially-breaking
> changes that you propose, not on the release branch, not this late
> into the release cycle.

And even though I also disagree, equally respectfully, with your
assessment of the risks, and still I'm providing a solution that
eliminates them totally.

> > I will work and prepare that patch if you promise to at least
> > look at the code, follow the logic with me and consider the
> > patch for backporting if it's as safe as relying on a simple
> > `(if (memq package package-safely-upgradeable-builtins) ...)`
>
> Backporting can be considered for Emacs 29.2, not before that.
> Hopefully, by that time we will also have more data points regarding
> the various uses of this, in particular with Eglot, so that our
> decisions will be more informed.

OK.  Fair enough. If there's half a chance for 29.2 I guess I can work on
it.  Not today though.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Fri, Apr 14, 2023 at 7:26 PM Philip Kaludercic <philipk@posteo.net> wrote:
>
>> > Here's another solution you may want to consider.  package-install
>> > in Emacs 29 updates built-ins non-interactively always.  Interactively
>> > asks the user with a confirmation prompt.  This solution would
>> > also work.
>>
>> What would the confirmation prompt ask the user?
>
> "Are you sure you want to install Foo which is [also|now] a built-in
> package in Emacs 29?"

I feel this is confusing, if a user is not familiar with the way
built-in packages can be upgraded and are maintained.  Perhaps instead
of a simple yes-or-no-p, we could use read-multiple-choice to add some
more background?

>> Are you sure?  The `alist-get' call returns a list, since
>> `package-archive-contents' is a map of package names to a list of
>> package descriptors.
>
> Yeah, probably needs a cdr or car there for good measure.  Great.

Specifically (cadr (assq 'eglot package-archive-contents))

> João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 8:21 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 14/04/2023 22:19, Eli Zaretskii wrote:
> > So Eglot 1.14 is not stable?  Does Eglot has a stable branch on ELPA?
>
> All Eglot development is led on master.
>
> > I meant to include the latest stable release, not an unstable one.
>
> We consider tagged releases (ones that bump the version header) as
> "stable" ones, and all intermediate commits as "unstable". Those are not
> the same kind of "stable" as Emacs releases, obviously.

Exactly this.  When I tag an Eglot release, I check that tests run.  I
run some manual tests and I come to a conclusion as to its stability.
But there is no pretest or "release candidate" or anything like that.
It's not feasible, I don't even know how that would work.
So I make the release, watch for feedback closely the next days or
weeks and fix bugs for the next release.  In the interim I give people
workarounds.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 8:28 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 14 Apr 2023 20:20:20 +0100
> > Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net,
> >       62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> >
> > On Fri, Apr 14, 2023 at 7:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > The Eglot release 1.14 is not as stable as 1.12.29. It has new features.
> > Also a recent bugfix in commit a74403adda0 is quite complex and
> > I'm cautiously waiting for feedback.  Other simple bugfixes have
> > been backported.
>
> So you don't recommend that users who want a stable Eglot upgrade to
> 1.14?

Depends on how "stable" they want it and how badly they want new
features.

> If so, why is it a problem that package-install by default
> doesn't update built-in packages?  Users who want the cutting edge of
> Eglot, and don't mind some instability, can always switch to the
> master branch of Emacs, where we are free to change package-install to
> upgrade core packages by default.

Users don't just switch to the master branch of Emacs.  Many just
can't because of enterprise complications, or just difficulty of compilation.
I've worked in companies using Emacs (some of them exclusively!!) for 20
years.  Most users are two, sometimes, three Emacs releases behind.
They are not remotely interested in updating.  "IT doesn't like it".
"It's not the official".  "This one is just fine".

But if a colleague goes to their workstation and shows them
M-x package-install RET very-fancy-nice-thing or sends them a
super-fancy init.el they will take it no problem, and buy you
coffee and rave about it. I can't be the only one who has
experienced this :-)

> > But if someone types M-x install-something they should get what
> > they ask for.  If they want to be 100% safe, they just shouldn't
> > invoke commands that download, compile and evaluate code.
>
> The logic should be consistent.  Emacs 29 is the stable branch of
> Emacs, so it should come with the latest stable Eglot.  If that is
> Eglot 1.12.29, then the fact that package-install won't upgrade it to
> 1.14 is consistent with the stability of Eglot's versions.  If, OTOH,
> you think that it's imperative to allow _all_ users of Eglot with
> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> become available), then we should release Emacs 29 with 1.14.

I think it's imperative to _allow_ -- as you say -- and also and
to _make easy_.  More importantly, and to the point, to _make it
as easy as it was in Emacs 26, 27 and 28_.  But I'd prefer not to force
Eglot 1.14 (or likely 1.15/1.16 by the time i expect the final RCs
to be around).

> > Notwithstanding this personal opinion, I underline again that
> > it _is_ possible to craft a simple, emacs-29-safe modification,
> > to the package-install function that is even more cautious to
> > download certain types of things.
>
> Philip presented such a safe modification, and we are in the final
> stages of discussing its details, before it will be installed.  So
> yes, it is possible.

As I've explained to Philip, the big drawback of that -- undoubtedly
safe -- modification is that it is not compatible to user's
configurations that have a (use-package 'eglot) or a
(package-install 'eglot) in them.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 8:33 PM Philip Kaludercic <philipk@posteo.net> wrote:

> >> > Here's another solution you may want to consider.  package-install
> >> > in Emacs 29 updates built-ins non-interactively always.  Interactively
> >> > asks the user with a confirmation prompt.  This solution would
> >> > also work.
> >>
> >> What would the confirmation prompt ask the user?
> >
> > "Are you sure you want to install Foo which is [also|now] a built-in
> > package in Emacs 29?"
>
> I feel this is confusing, if a user is not familiar with the way
> built-in packages can be upgraded and are maintained.  Perhaps instead
> of a simple yes-or-no-p, we could use read-multiple-choice to add some
> more background?

Sure, I actually don't know what that does, but whatever you think
is best, I hope the point has gotten across.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

>> > Notwithstanding this personal opinion, I underline again that
>> > it _is_ possible to craft a simple, emacs-29-safe modification,
>> > to the package-install function that is even more cautious to
>> > download certain types of things.
>>
>> Philip presented such a safe modification, and we are in the final
>> stages of discussing its details, before it will be installed.  So
>> yes, it is possible.

I might have missed a message, what was the last state here?

> As I've explained to Philip, the big drawback of that -- undoubtedly
> safe -- modification is that it is not compatible to user's
> configurations that have a (use-package 'eglot) or a
> (package-install 'eglot) in them.

Again: Are we sure about this?  After all, the package is installed
(which I think is the main thing), it just might not have the most
recent version.  Calling this "not compatible" seems excessive.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Philip Kaludercic <philipk@posteo.net> writes:

> João Távora <joaotavora@gmail.com> writes:
>>> Philip presented such a safe modification, and we are in the final
>>> stages of discussing its details, before it will be installed.  So
>>> yes, it is possible.
>
> I might have missed a message, what was the last state here?

Eli is talking about an interaction between you two, so you should be
able to figure out.

>> As I've explained to Philip, the big drawback of that -- undoubtedly
>> safe -- modification is that it is not compatible to user's
>> configurations that have a (use-package 'eglot) or a
>> (package-install 'eglot) in them.
>
> Again: Are we sure about this?  After all, the package is installed
> (which I think is the main thing), it just might not have the most
> recent version.  Calling this "not compatible" seems excessive.

I don't think we should look at this in legalese and disguise the fact
that the same form in Emacs 26, 27 and 28 has very different results.
In those versions (use-package 'eglot :ensure t) brings you always the
latest and greatest.  In Emacs 29, supposedly the more advanced version,
it will bring you a different, older, less powerful, and one that might
even be incompatible with the remainder of your configuration if you
aren't aware of this obscure quirk.

Furthermore the problem continues to worsen with time as packages
evolve.

This is incompatibility if there ever was such a thing.  So the "package
'eglot' is already installed" shown to the user, while technically
correct, is completely misleading.

The patch you're preparing, last I looked at it, doesn't solve this.  It
does makes Eglot slightly easier to upgrade, at least if the user is
aware of this quirky situation.  But it still doesn't fix the
use-package case, for example or the non-interactive config case.

Eli said something like this patch could be acceptable for master and
maybe 29.2.  

----------8<----------8<----------8<----------8<----------8<----------8<
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -652,6 +652,9 @@ package--builtins
 name (a symbol) and DESC is a `package--bi-desc' structure.")
 (put 'package--builtins 'risky-local-variable t)
 
+(defvar package--safely-upgradeable-builtins '(eglot use-package)
+  "See bug#62720 for longest docstring ever.")
+
 (defvar package-alist nil
   "Alist of all packages available for activation.
 Each element has the form (PKG . DESCS), where PKG is a package
@@ -2201,14 +2204,19 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
+                    (append
                      (delq nil
                            (mapcar (lambda (elt)
                                      (unless (package-installed-p (car elt))
                                        (symbol-name (car elt))))
                                    package-archive-contents))
+                     package--safely-upgradeable-builtins)
                     nil t))
            nil)))
   (package--archives-initialize)
+  (when-let ((desc (and (memq pkg package--safely-upgradeable-builtins)
+                        (cadr (assoc pkg package-archive-contents)))))
+    (setq pkg desc))
   (add-hook 'post-command-hook #'package-menu--post-refresh)
   (let ((name (if (package-desc-p pkg)
                   (package-desc-name pkg)
---------->8---------->8---------->8---------->8---------->8---------->8

1. Node code complexity.  It's 6 lines of trivial code.
2. Absolutely no risk of "silent installation of software that wasn't ever
   installed before"
3. Fixes all the aforementioned issues

Doesn't fix package-update, but that's minor in comparison, I didn't
want to add bloat this enormous patch with 2 more lines.

It is based on a hardcoded list.  Dmitry has suggested other ways to
whitelist.  There are infinite ways, of course.  Other have suggested
blacklists.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Fri, 14 Apr 2023 16:40:01 +0000
> 
> > Hmm... looks identical to the previous patch you sent, which changes
> > package-install?  Or what am I missing?
> >
> > As for the question I asked: I'm okay with changing package-install if
> > there are no objections from those who proposed to change
> > package-upgrade instead.
> 
> Did I send the wrong patch.  I double-checked now and this one should be
> using package-update:

Thanks.  It doesn't seem to be simpler than the change in
package-install, perhaps even more complex than that.

What is your preference? change package-install or package-update?  I
tend to the former.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 20:46:02 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Fri, Apr 14, 2023 at 8:28 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Date: Fri, 14 Apr 2023 20:20:20 +0100
> > > Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net,
> > >       62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> > >
> > > On Fri, Apr 14, 2023 at 7:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > The Eglot release 1.14 is not as stable as 1.12.29. It has new features.
> > > Also a recent bugfix in commit a74403adda0 is quite complex and
> > > I'm cautiously waiting for feedback.  Other simple bugfixes have
> > > been backported.
> >
> > So you don't recommend that users who want a stable Eglot upgrade to
> > 1.14?
> 
> Depends on how "stable" they want it and how badly they want new
> features.

You are dodging the question, which is unfortunate.  It is IMO an
important question, much more general and important than this
tempest-in-a-teapot issue we are discussing here.  We will need to
revisit this question seriously if we ever succeed in removing 'core'
packages from emacs.git and bundling them with release tarballs.  At
that time, we will need clear-cut criteria for how to select the
version of a package that is the most suitable one to go into the
tarball.  And that in turn will require the developers of each such
package to seriously address the questions I've asked: what are the
criteria for considering a package version "stable", and how is that
communicated via version numbers and/or repository branches.  For
example, Org has that encoded in its branches, so that the latest
changes from its stable branch are routinely merged into the stable
branch of Emacs, and the latest code from that branch will be in the
Emacs release tarball.

Based on your answer, it sounds like Eglot users are on their own in
this aspect: they have no real guidance from you which version is
currently considered "stable".

> > If so, why is it a problem that package-install by default
> > doesn't update built-in packages?  Users who want the cutting edge of
> > Eglot, and don't mind some instability, can always switch to the
> > master branch of Emacs, where we are free to change package-install to
> > upgrade core packages by default.
> 
> Users don't just switch to the master branch of Emacs.  Many just
> can't because of enterprise complications, or just difficulty of compilation.
> I've worked in companies using Emacs (some of them exclusively!!) for 20
> years.  Most users are two, sometimes, three Emacs releases behind.
> They are not remotely interested in updating.  "IT doesn't like it".
> "It's not the official".  "This one is just fine".
> 
> But if a colleague goes to their workstation and shows them
> M-x package-install RET very-fancy-nice-thing or sends them a
> super-fancy init.el they will take it no problem, and buy you
> coffee and rave about it. I can't be the only one who has
> experienced this :-)

I have no doubt there are such cases.  But I very much doubt they are
the majority.  Whatever we decide, it is clear that some use cases
will want a different behavior.  The issue discussed here is what
should be the default behavior, and whether the default behavior of
Emacs 29.1 would be wrong for most users of Eglot and of other core
packages.

To reiterate: I think each release of Emacs should ship with the
latest stable version of each core package.  If this is the case, the
need to upgrade core packages via package.el is not very important for
the majority of our users who prefer to use a stable Emacs.  Thus, the
arguments you present emphasize the needs of the minority, and
therefore I don't consider them strong enough to invalidate the
compromise solution to which we are converging.

> > > But if someone types M-x install-something they should get what
> > > they ask for.  If they want to be 100% safe, they just shouldn't
> > > invoke commands that download, compile and evaluate code.
> >
> > The logic should be consistent.  Emacs 29 is the stable branch of
> > Emacs, so it should come with the latest stable Eglot.  If that is
> > Eglot 1.12.29, then the fact that package-install won't upgrade it to
> > 1.14 is consistent with the stability of Eglot's versions.  If, OTOH,
> > you think that it's imperative to allow _all_ users of Eglot with
> > Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> > become available), then we should release Emacs 29 with 1.14.
> 
> I think it's imperative to _allow_ -- as you say -- and also and
> to _make easy_.

"C-u M-x package-install" is easy enough.

> More importantly, and to the point, to _make it as easy as it was in
> Emacs 26, 27 and 28_.

That's an impractical request, one that most probably prefers Eglot
users (and only part of them at that) to those of other core packages.
We must think about all the users, not just users of Eglot.  The price
of adaptation to the fact that Eglot is now a core package, while it
is not nil, is not high.  So once again, this solution is IMO a
reasonable compromise, given the constraints.

> > Philip presented such a safe modification, and we are in the final
> > stages of discussing its details, before it will be installed.  So
> > yes, it is possible.
> 
> As I've explained to Philip, the big drawback of that -- undoubtedly
> safe -- modification is that it is not compatible to user's
> configurations that have a (use-package 'eglot) or a
> (package-install 'eglot) in them.

That is not a problem serious enough to invalidate the solution.  We
can, for example, mention the change in NEWS, to alleviate some of the
costs.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> > So you don't recommend that users who want a stable Eglot upgrade to
>> > 1.14?
>> Depends on how "stable" they want it and how badly they want new
>> features.
> You are dodging the question, which is unfortunate.  

I'm not.  I don't have an answer.  I can't have one.  It depends on the
user's wishes.

Case 1: You use Emacs in a debian server VM you're setting up, perhaps
in a remote location, perhaps shared with other users.  Just 'sudo apt
install emacs clangd'.  No .emacs file at all and you can go edit little
C programs conveniently and quickly.  Then I recommend Eglot 1.12.29
bundled with emacs 29.

Case 2: For your "daily driver" home system, I recommend `M-x
package-install RET eglot' (at least I used to, that is ) since that
will bring you the latest nice features tested to a good degree, but not
as well tested as 1.12.29.

> important question, much more general and important than this
> tempest-in-a-teapot issue we are discussing here.  

If I were to use your rhetoric, I would say you are "shifting" the
question.  I've never had real problems about this.  But if you want to
"revisit this question seriously", I'll be there, of course.

> We will need to revisit this question seriously if we ever succeed in
> removing 'core'
...
> Based on your answer, it sounds like Eglot users are on their own in
> this aspect: they have no real guidance from you which version is
> currently considered "stable".

They have the NEWS file, the bug tracker database, and my best efforts.
I told you how I make releases.  I want them to be stable, but there is
no official pretest.  "Guidance"?  Are the two cases above I gave
guidance?  Just present some actual case and I'll give you my
recommendation.  Else it's like coming to a doctor and asking about
taking a drug: the doctor probably going to ask you questions.

>> But if a colleague goes to their workstation and shows them
>> M-x package-install RET very-fancy-nice-thing or sends them a
>> super-fancy init.el they will take it no problem, and buy you
>> coffee and rave about it. I can't be the only one who has
>> experienced this :-)
>
> I have no doubt there are such cases.  But I very much doubt they are
> the majority.  

I don't, whole '.emacs' get shared like crazy.  That's the number one
method of "getting a configuration". People see other's Emacs in
screenshots or over the shoulder and ask to see their init file.  Not
just Emacs of course, any util.  Then they copy over bits they think are
cool/useful.  What you think they won't copy "use-package" forms??

People don't read NEWS and they don't read the manual.  Most Emacs users
I knew don't even know NEWS exists, what with that strange all caps
name.  They probably think it's some major mode for journalists.

> To reiterate: I think each release of Emacs should ship with the
> latest stable version of each core package.  If this is the case, the
> need to upgrade core packages via package.el is not very important for
> the majority of our users who prefer to use a stable Emacs.  Thus, the
> arguments you present emphasize the needs of the minority, and
> therefore I don't consider them strong enough to invalidate the
> compromise solution to which we are converging.

Sorry, but IMO you come up with very complicated logic and premises
about stability just to arrive at where you want to arrive.

A simple, readily verifiable fact remains.  M-x package-install RET
eglot in Emacs 29 will bring you a older version than in Emacs 28.  If
not today, tomorrow, eventually it will.  And that's just bad in my
opinion.  And it will become worse.

If, in your opinion, this is somehow a good or indifferent thing, we can
stop the whole discussion right here.

Again: if you think it's a _good thing_ that, in Emacs 29

  (use-package eglot :ensure t) 

or

  (package-install 'eglot)

or 

  M-x package-install RET eglot

produces an older version of Eglot than the very same form in Emacs 26,
27 and 28, please do say so ASAP.

I was under the impression that you didn't prefer that, but I'm not sure
anymore after reading your complex last paragraph.

>> I think it's imperative to _allow_ -- as you say -- and also and
>> to _make easy_.
>
> "C-u M-x package-install" is easy enough.
>
>> More importantly, and to the point, to _make it as easy as it was in
>> Emacs 26, 27 and 28_.
>
> That's an impractical request, one that most probably prefers Eglot
> users (and only part of them at that) to those of other core packages.
> We must think about all the users, not just users of Eglot.  The price
> of adaptation to the fact that Eglot is now a core package, while it
> is not nil, is not high.  So once again, this solution is IMO a
> reasonable compromise, given the constraints.

It's _not_ impractical.  There doesn't have to be a "compromise
solution" at all.  There _can_ be a win-win solution (presuming I even
understand what counts as a win to you in this issue).

I just gave Philip a 7 line patch, my fourth in this discussion, that
does exactly I think answers all of your objections regarding
risk/stability.  Much simpler than any patch so far by _any_ measure.
Lines of code, cyclomatic complexity, number of user options, NEWS
changes.  Will you look at it?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: dmitry@gutov.dev,  rpluim@gmail.com,  philipk@posteo.net,
>   62720@debbugs.gnu.org,  larsi@gnus.org,  monnier@iro.umontreal.ca
> Date: Sat, 15 Apr 2023 11:24:41 +0100
> 
> A simple, readily verifiable fact remains.  M-x package-install RET
> eglot in Emacs 29 will bring you a older version than in Emacs 28.  If
> not today, tomorrow, eventually it will.  And that's just bad in my
> opinion.  And it will become worse.

It is not worse, it is better: they get Eglot integrated into Emacs.

> If, in your opinion, this is somehow a good or indifferent thing, we can
> stop the whole discussion right here.

Yes, let's stop.  There's no reason to keep it going.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> João Távora <joaotavora@gmail.com> writes:
>>>> Philip presented such a safe modification, and we are in the final
>>>> stages of discussing its details, before it will be installed.  So
>>>> yes, it is possible.
>>
>> I might have missed a message, what was the last state here?
>
> Eli is talking about an interaction between you two, so you should be
> able to figure out.

I know, I have just lost track.


[...]

> Eli said something like this patch could be acceptable for master and
> maybe 29.2.  
>
> ----------8<----------8<----------8<----------8<----------8<----------8<
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -652,6 +652,9 @@ package--builtins
>  name (a symbol) and DESC is a `package--bi-desc' structure.")
>  (put 'package--builtins 'risky-local-variable t)
>  
> +(defvar package--safely-upgradeable-builtins '(eglot use-package)
> +  "See bug#62720 for longest docstring ever.")
> +
>  (defvar package-alist nil
>    "Alist of all packages available for activation.
>  Each element has the form (PKG . DESCS), where PKG is a package
> @@ -2201,14 +2204,19 @@ package-install
>       (package--archives-initialize)
>       (list (intern (completing-read
>                      "Install package: "
> +                    (append
>                       (delq nil
>                             (mapcar (lambda (elt)
>                                       (unless (package-installed-p (car elt))
>                                         (symbol-name (car elt))))
>                                     package-archive-contents))
> +                     package--safely-upgradeable-builtins)
>                      nil t))
>             nil)))
>    (package--archives-initialize)
> +  (when-let ((desc (and (memq pkg package--safely-upgradeable-builtins)
> +                        (cadr (assoc pkg package-archive-contents)))))
> +    (setq pkg desc))
>    (add-hook 'post-command-hook #'package-menu--post-refresh)
>    (let ((name (if (package-desc-p pkg)
>                    (package-desc-name pkg)
> ---------->8---------->8---------->8---------->8---------->8---------->8
>
> 1. Node code complexity.  It's 6 lines of trivial code.
> 2. Absolutely no risk of "silent installation of software that wasn't ever
>    installed before"
> 3. Fixes all the aforementioned issues

Personally I think this would be a fine solution considering the
constraints.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Fri, 14 Apr 2023 16:40:01 +0000
>> 
>> > Hmm... looks identical to the previous patch you sent, which changes
>> > package-install?  Or what am I missing?
>> >
>> > As for the question I asked: I'm okay with changing package-install if
>> > there are no objections from those who proposed to change
>> > package-upgrade instead.
>> 
>> Did I send the wrong patch.  I double-checked now and this one should be
>> using package-update:
>
> Thanks.  It doesn't seem to be simpler than the change in
> package-install, perhaps even more complex than that.
>
> What is your preference? change package-install or package-update?  I
> tend to the former.

I have no preference either way.  If you think that package-install is
fine, and João has expressed interest in that route as well, we might as
well go that way.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 15, 2023 at 11:40 AM Philip Kaludercic <philipk@posteo.net> wrote:

> > Eli said something like this patch could be acceptable for master and
> > maybe 29.2.
> >
> > ----------8<----------8<----------8<----------8<----------8<----------8<
> > --- a/lisp/emacs-lisp/package.el
> > +++ b/lisp/emacs-lisp/package.el
> > @@ -652,6 +652,9 @@ package--builtins
> >  name (a symbol) and DESC is a `package--bi-desc' structure.")
> >  (put 'package--builtins 'risky-local-variable t)
> >
> > +(defvar package--safely-upgradeable-builtins '(eglot use-package)
> > +  "See bug#62720 for longest docstring ever.")
> > +
> >  (defvar package-alist nil
> >    "Alist of all packages available for activation.
> >  Each element has the form (PKG . DESCS), where PKG is a package
> > @@ -2201,14 +2204,19 @@ package-install
> >       (package--archives-initialize)
> >       (list (intern (completing-read
> >                      "Install package: "
> > +                    (append
> >                       (delq nil
> >                             (mapcar (lambda (elt)
> >                                       (unless (package-installed-p (car elt))
> >                                         (symbol-name (car elt))))
> >                                     package-archive-contents))
> > +                     package--safely-upgradeable-builtins)
> >                      nil t))
> >             nil)))
> >    (package--archives-initialize)
> > +  (when-let ((desc (and (memq pkg package--safely-upgradeable-builtins)
> > +                        (cadr (assoc pkg package-archive-contents)))))
> > +    (setq pkg desc))
> >    (add-hook 'post-command-hook #'package-menu--post-refresh)
> >    (let ((name (if (package-desc-p pkg)
> >                    (package-desc-name pkg)
> > ---------->8---------->8---------->8---------->8---------->8---------->8
> >
> > 1. Node code complexity.  It's 6 lines of trivial code.
> > 2. Absolutely no risk of "silent installation of software that wasn't ever
> >    installed before"
> > 3. Fixes all the aforementioned issues
>
> Personally I think this would be a fine solution considering the
> constraints.

Me too, of course.  Good luck :-)

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 10:41:52 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What is your preference? change package-install or package-update?  I
> > tend to the former.
> 
> I have no preference either way.  If you think that package-install is
> fine, and João has expressed interest in that route as well, we might as
> well go that way.

OK, then let's do that, and thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Kévin Le Gouguec]

João Távora <joaotavora@gmail.com> writes:

> A simple, readily verifiable fact remains.  M-x package-install RET
> eglot in Emacs 29 will bring you a older version than in Emacs 28.  If
> not today, tomorrow, eventually it will.  And that's just bad in my
> opinion.  And it will become worse.
>
> If, in your opinion, this is somehow a good or indifferent thing, we can
> stop the whole discussion right here.
>
> Again: if you think it's a _good thing_ that, in Emacs 29
>
>   (use-package eglot :ensure t) 
>
> or
>
>   (package-install 'eglot)
>
> or 
>
>   M-x package-install RET eglot
>
> produces an older version of Eglot than the very same form in Emacs 26,
> 27 and 28, please do say so ASAP.
>
> I was under the impression that you didn't prefer that, but I'm not sure
> anymore after reading your complex last paragraph.

If I may throw in my ¢2: in Emacs ≤28, users never had a choice between
a "installing the newest Eglot from GNU ELPA" and "requiring that Eglot
be available, possibly not the latest & greatest".  They could only
request the former.

It's anyone's guess which of those two things users who cargo-cult those
configuration lines would prefer, now that the question is up in the air
for Emacs 29.  FWIW, I'd lean toward the latter: IMHO, package-install,
resp. (use-package … :ensure t), merely suggest ensuring availability,
not proactively ugprading to the latest (unlike say "package-update").

So I wouldn't be shocked for package-install to be a no-op for :core
packages, the semantics being "make sure the package is present,
favoring any built-in version which presumably underwent lots of
validation & stability fixes on the release branch".

With that perspective, I don't think the change in behaviour users will
observe in Emacs 29 re. which version of Eglot they get with `M-x
package-install eglot' is necessarily "worse": it depends on how much
one values "a release branch's worth of stability fixes" vs "a
development branch's worth of new features".

(
  Although I can understand that, in the _specific_ context of an LSP
  client that is in active development & "competing" with a MELPA-only
  alternative, it is a bit of a bummer that M-x package-install 𝒫 will
  yield something that users might consider "inferior" feature-wise when
  𝒫=eglot.  A bit of a bummer, but not a deal-breaker IMO; as long as
  "M-x package-list U x" brings the latest & greatest, I still think
  package-install's behaviour change re. eglot in Emacs 29 is
  defensible.
)

(
  Sorry for butting in and adding more words to this lengthy discussion;
  just thought that hearing the perspective from one random user might
  help.  I must also confess that I might not have read the whole thing
  as attentively as I perhaps should have; the parallel subthread
  between Eli & Philip re. changing package-install or package-update
  makes me unsure what "U x" will actually do with eglot in Emacs 29, so
  my previous parenthesized digression might be moot.
)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 10:41:52 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > What is your preference? change package-install or package-update?  I
>> > tend to the former.
>> 
>> I have no preference either way.  If you think that package-install is
>> fine, and João has expressed interest in that route as well, we might as
>> well go that way.
>
> OK, then let's do that, and thanks.

Great, then just to that we are on the same page, what approach do we
finally want to decide on?

- User option to enable upgrading built-in packages
- Prefix argument to enable upgrading built-in packages
- Always upgrade built-in packages

I argue the last option should be safe.  Semantically it would also make
sense, since invoking the command can be taken to be take to be an
explicit request, and if it is not what a user wants (I assume that João
think this is not probable), then it is easy to revert.

If we decide that this is not acceptable, then we can fall back onto the
patch that uses a prefix argument or a user option.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 11:37:40 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I have no preference either way.  If you think that package-install is
> >> fine, and João has expressed interest in that route as well, we might as
> >> well go that way.
> >
> > OK, then let's do that, and thanks.
> 
> Great, then just to that we are on the same page, what approach do we
> finally want to decide on?
> 
> - User option to enable upgrading built-in packages
> - Prefix argument to enable upgrading built-in packages
> - Always upgrade built-in packages

The first two on emacs-29, the last one on master (if enough people
think it's a good idea; me, I think we should wait for a while before
deciding).

> I argue the last option should be safe.  Semantically it would also make
> sense, since invoking the command can be taken to be take to be an
> explicit request, and if it is not what a user wants (I assume that João
> think this is not probable), then it is easy to revert.
> 
> If we decide that this is not acceptable, then we can fall back onto the
> patch that uses a prefix argument or a user option.

Yes, that is what I think we should install on emacs-29.

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 15/04/2023 14:19, Kévin Le Gouguec wrote:

> If I may throw in my ¢2: in Emacs ≤28, users never had a choice between
> a "installing the newest Eglot from GNU ELPA" and "requiring that Eglot
> be available, possibly not the latest & greatest".  They could only
> request the former.
> 
> It's anyone's guess which of those two things users who cargo-cult those
> configuration lines would prefer, now that the question is up in the air
> for Emacs 29.  FWIW, I'd lean toward the latter: IMHO, package-install,
> resp. (use-package … :ensure t), merely suggest ensuring availability,
> not proactively ugprading to the latest (unlike say "package-update").
> 
> So I wouldn't be shocked for package-install to be a no-op for :core
> packages, the semantics being "make sure the package is present,
> favoring any built-in version which presumably underwent lots of
> validation & stability fixes on the release branch".

I can easily buy this argument (and made it myself in a different form, 
I think), but the problem is, there is no easily discoverable way for 
the user to force the upgrade. They'll need to hunt the docs for how to 
do that.

> (
>    Although I can understand that, in the _specific_ context of an LSP
>    client that is in active development & "competing" with a MELPA-only
>    alternative, it is a bit of a bummer that M-x package-install 𝒫 will
>    yield something that users might consider "inferior" feature-wise when
>    𝒫=eglot.  A bit of a bummer, but not a deal-breaker IMO; as long as
>    "M-x package-list U x" brings the latest & greatest, I still think
>    package-install's behaviour change re. eglot in Emacs 29 is
>    defensible.
> )

'M-x package-list U x' does not upgrade built-in Eglot either, AFAIU.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 15/04/2023 13:40, Philip Kaludercic wrote:
>> Eli said something like this patch could be acceptable for master and
>> maybe 29.2.
>>
>> ----------8<----------8<----------8<----------8<----------8<----------8<
>> --- a/lisp/emacs-lisp/package.el
>> +++ b/lisp/emacs-lisp/package.el
>> @@ -652,6 +652,9 @@ package--builtins
>>   name (a symbol) and DESC is a `package--bi-desc' structure.")
>>   (put 'package--builtins 'risky-local-variable t)
>>   
>> +(defvar package--safely-upgradeable-builtins '(eglot use-package)
>> +  "See bug#62720 for longest docstring ever.")
>> +
>>   (defvar package-alist nil
>>     "Alist of all packages available for activation.
>>   Each element has the form (PKG . DESCS), where PKG is a package
>> @@ -2201,14 +2204,19 @@ package-install
>>        (package--archives-initialize)
>>        (list (intern (completing-read
>>                       "Install package: "
>> +                    (append
>>                        (delq nil
>>                              (mapcar (lambda (elt)
>>                                        (unless (package-installed-p (car elt))
>>                                          (symbol-name (car elt))))
>>                                      package-archive-contents))
>> +                     package--safely-upgradeable-builtins)
>>                       nil t))
>>              nil)))
>>     (package--archives-initialize)
>> +  (when-let ((desc (and (memq pkg package--safely-upgradeable-builtins)
>> +                        (cadr (assoc pkg package-archive-contents)))))
>> +    (setq pkg desc))
>>     (add-hook 'post-command-hook #'package-menu--post-refresh)
>>     (let ((name (if (package-desc-p pkg)
>>                     (package-desc-name pkg)
>> ---------->8---------->8---------->8---------->8---------->8---------->8
>>
>> 1. Node code complexity.  It's 6 lines of trivial code.
>> 2. Absolutely no risk of "silent installation of software that wasn't ever
>>     installed before"
>> 3. Fixes all the aforementioned issues
> Personally I think this would be a fine solution considering the
> constraints.

+1

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 11:37:40 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> I have no preference either way.  If you think that package-install is
>> >> fine, and João has expressed interest in that route as well, we might as
>> >> well go that way.
>> >
>> > OK, then let's do that, and thanks.
>> 
>> Great, then just to that we are on the same page, what approach do we
>> finally want to decide on?
>> 
>> - User option to enable upgrading built-in packages
>> - Prefix argument to enable upgrading built-in packages
>> - Always upgrade built-in packages
>
> The first two on emacs-29, the last one on master (if enough people
> think it's a good idea; me, I think we should wait for a while before
> deciding).

OK, so let us use this change:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages-with-package-insta.patch --]
[-- Type: text/x-diff, Size: 2782 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..842a475290d 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2187,7 +2198,9 @@ package-install
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
 available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+called interactively, prompt for the package name.  When invoked
+with a prefix argument, the prompt will include built-in packages
+that can be upgraded via an archive.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2205,11 +2218,13 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and current-prefix-arg
+                                     (package--upgradable-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2235,8 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (package--upgradable-built-in-p pkg)
+      (setq pkg (cadr (assq name package-archive-contents))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)

[-- Attachment #3: Type: text/plain, Size: 749 bytes --]


(I intentionally picked this one without the user option since that
will probably become unnecessary with Emacs 30+).  I believe there was
an issue with the name `package--upgradable-built-in-p' and the
docstring?

>> I argue the last option should be safe.  Semantically it would also make
>> sense, since invoking the command can be taken to be take to be an
>> explicit request, and if it is not what a user wants (I assume that João
>> think this is not probable), then it is easy to revert.
>> 
>> If we decide that this is not acceptable, then we can fall back onto the
>> patch that uses a prefix argument or a user option.
>
> Yes, that is what I think we should install on emacs-29.
>
> Thanks.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

>   yield something that users might consider "inferior" feature-wise when
>   𝒫=eglot.  
> A bit of a bummer, but not a deal-breaker IMO; 

It's not the end of the world.  But certainly not good, especially if
those users have just upgraded to Emacs 29 on some new machine and
ported over their Emacs 28 config.  It could even break.  In fact
eventually it will simply break, if Eglot 1.16 adds, say, some
eglot-fancy-eldoc-function and the config has

  (package-install 'eglot)
  (add-hook 'eldoc-documentation-functions 'eglot-fancy-eldoc-function)

This will break sooner or later, and bizzarely so, IMO.

Just as a data point, I got a lot of confused users just because of
bug#62576 and the missing "project-name" function.  And, mind, you
there, the simplest of workarounds, restarting Emacs fixes the issue.
But it still confused a load of users so eventually I tooks steps to
avoid it.

Just see
https://github.com/joaotavora/eglot/search?q=project-name&type=discussions

>   Sorry for butting in and adding more words to this lengthy discussion;
>   just thought that hearing the perspective from one random user might
>   help.  

No problem.  I always like reading your feedback :-)

Let's see where the tip of this discussion is heading.  Philip proposes:

> > - User option to enable upgrading built-in packages
> > - Prefix argument to enable upgrading built-in packages
> > - Always upgrade built-in packages

And Eli replies:

> The first two on emacs-29, the last one on master 

Now, this means that things like (use-package eglot :ensure t) in Emacs
26, 27 and 28 will rev up Eglot to the very latest, Emacs 29 will stay at
1.12.29 and Emacs 30, 31, 32 again to the very latest.

This is almost the mathematical definition of instability.  And the
amplitude of the version cliff will just keep growing.  I just don't
think it's a good thing, but if Eli and you think it is, that's
perfectly legitimate.

The key thing here is that there was a package hitherto with given
 update semantics.  That package wasn't in core and now is.  It's a clear
clash of update semantics, and noone (including me) was able to foresee
it.

I do appreciate all the other arguments for stability.  Because of that
I proposed a very simple patch that addresses everyone's concern but
unfortunately I don't think is going anywhere -- who knows for what
reason.  Even though Philip, Dmitry and I himself think it is the
cleanest solution for emacs 29.  See the patch here: it's 6-7 lines of
code.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#467

>   between Eli & Philip re. changing package-install or package-update
>   makes me unsure what "U x" will actually do with eglot in Emacs 29, so
>   my previous parenthesized digression might be moot.

Alas U x in the package menu _also_ doesn't upgrade Eglot.  And neither
does M-x package-update-all.  I don't see any plans for doing so.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 13:21:53 +0000
> 
> >> - User option to enable upgrading built-in packages
> >> - Prefix argument to enable upgrading built-in packages
> >> - Always upgrade built-in packages
> >
> > The first two on emacs-29, the last one on master (if enough people
> > think it's a good idea; me, I think we should wait for a while before
> > deciding).
> 
> OK, so let us use this change:
> 
> (I intentionally picked this one without the user option since that
> will probably become unnecessary with Emacs 30+).

The user option allows those users who always want package-install to
upgrade core package to have what they want, easily.  So I think we
should keep it.  On master, the option could be t by default, or
become unnecessary if that's what happens (but I wouldn't bet on
that).

> I believe there was an issue with the name
> `package--upgradable-built-in-p' and the docstring?

Yes.  The doc string has a typo:

  "Return non-nil if PACKAGE if the built-in version is used."

See those two "if"s?  And even if I replace the second "if" with "is",
the sentence doesn't make sense.

As for the name, I think we can leave it at that.

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

>>   between Eli & Philip re. changing package-install or package-update
>>   makes me unsure what "U x" will actually do with eglot in Emacs 29, so
>>   my previous parenthesized digression might be moot.
>
> Alas U x in the package menu _also_ doesn't upgrade Eglot.  And neither
> does M-x package-update-all.  I don't see any plans for doing so.

But selecting the package with I and then installing it will "update"
it -- which is a good argument in favour of using `package-install'.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 13:21:53 +0000
>> 
>> >> - User option to enable upgrading built-in packages
>> >> - Prefix argument to enable upgrading built-in packages
>> >> - Always upgrade built-in packages
>> >
>> > The first two on emacs-29, the last one on master (if enough people
>> > think it's a good idea; me, I think we should wait for a while before
>> > deciding).
>> 
>> OK, so let us use this change:
>> 
>> (I intentionally picked this one without the user option since that
>> will probably become unnecessary with Emacs 30+).
>
> The user option allows those users who always want package-install to
> upgrade core package to have what they want, easily.  So I think we
> should keep it.  On master, the option could be t by default, or
> become unnecessary if that's what happens (but I wouldn't bet on
> that).

My argument against a user option is just that the whole deal is
something that will in practice at most affect two packages (if we
change the behaviour in Emacs 29).  Is it really worth adding a general
option for this very specific situation?

>> I believe there was an issue with the name
>> `package--upgradable-built-in-p' and the docstring?
>
> Yes.  The doc string has a typo:
>
>   "Return non-nil if PACKAGE if the built-in version is used."
>
> See those two "if"s?  And even if I replace the second "if" with "is",
> the sentence doesn't make sense.

Right, how does

  "Return non-nil if the built-in version of PACKAGE is used."

sound?

> As for the name, I think we can leave it at that.

Ok.

> Thanks.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 17:14:41 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The user option allows those users who always want package-install to
> > upgrade core package to have what they want, easily.  So I think we
> > should keep it.  On master, the option could be t by default, or
> > become unnecessary if that's what happens (but I wouldn't bet on
> > that).
> 
> My argument against a user option is just that the whole deal is
> something that will in practice at most affect two packages (if we
> change the behaviour in Emacs 29).  Is it really worth adding a general
> option for this very specific situation?

I think we should count users of those packages, not just the packages
themselves.  Yes, I think it's worth it, because we don't know how
many of the users will want the built-in packages to be included in an
update.

> >   "Return non-nil if PACKAGE if the built-in version is used."
> >
> > See those two "if"s?  And even if I replace the second "if" with "is",
> > the sentence doesn't make sense.
> 
> Right, how does
> 
>   "Return non-nil if the built-in version of PACKAGE is used."
> 
> sound?

I think we should explain what does "the built-in version of PACKAGE
is used" mean, in the context in which this predicate is used.  Maybe
say something like

    Return non-nil if the built-in version of PACKAGE is used.
  If the built-in version of PACKAGE is used and PACKAGE is
  also available for installation from an archive, it is an
  indication that PACKAGE was never upgraded to any newer
  version from the archive.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 17:14:41 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > The user option allows those users who always want package-install to
>> > upgrade core package to have what they want, easily.  So I think we
>> > should keep it.  On master, the option could be t by default, or
>> > become unnecessary if that's what happens (but I wouldn't bet on
>> > that).
>> 
>> My argument against a user option is just that the whole deal is
>> something that will in practice at most affect two packages (if we
>> change the behaviour in Emacs 29).  Is it really worth adding a general
>> option for this very specific situation?
>
> I think we should count users of those packages, not just the packages
> themselves.  Yes, I think it's worth it, because we don't know how
> many of the users will want the built-in packages to be included in an
> update.

OK, see below.

>> >   "Return non-nil if PACKAGE if the built-in version is used."
>> >
>> > See those two "if"s?  And even if I replace the second "if" with "is",
>> > the sentence doesn't make sense.
>> 
>> Right, how does
>> 
>>   "Return non-nil if the built-in version of PACKAGE is used."
>> 
>> sound?
>
> I think we should explain what does "the built-in version of PACKAGE
> is used" mean, in the context in which this predicate is used.  Maybe
> say something like
>
>     Return non-nil if the built-in version of PACKAGE is used.
>   If the built-in version of PACKAGE is used and PACKAGE is
>   also available for installation from an archive, it is an
>   indication that PACKAGE was never upgraded to any newer
>   version from the archive.

Sounds good to me.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages-with-package-insta.patch --]
[-- Type: text/x-diff, Size: 5042 bytes --]

From dae52770fa50bb68a4fdd0983de7811e427733a5 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 13 Apr 2023 20:13:59 +0200
Subject: [PATCH] Allow upgrading built-in packages with 'package-install'

* etc/NEWS: Mention the change
* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
predicate.
(package-install-upgrade-built-in): Add new user option to enable
feature.
(package-install): Respect new user option.
---
 etc/NEWS                   |  5 ++++
 lisp/emacs-lisp/package.el | 47 +++++++++++++++++++++++++++++++-------
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 84dbb94a71a..a7834cd0d2b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1876,6 +1876,11 @@ package maintainers.
 By customizing this user option you can specify specific packages to
 install.
 
+---
+*** New user option 'package-install-upgrade-built-in'.
+When enabled, 'package-install' can be used to install 
+newer versions of built-in packages.
+
 ** Emacs Sessions (Desktop)
 
 +++
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..c0cc7bebeb2 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,21 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--active-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used.
+If the built-in version of PACKAGE is used and PACKAGE is
+also available for installation from an archive, it is an
+indication that PACKAGE was never upgraded to any newer
+version from the archive."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2182,12 +2197,18 @@ package--archives-initialize
   (unless package-archive-contents
     (package-refresh-contents)))
 
+(defcustom package-install-upgrade-built-in nil
+  "Non-nil means that built-in packages can be upgraded via a package archive.
+If disabled, then `package-install' will not suggest to replace a
+built-in package with a version from a package archive."
+  :type 'boolean
+  :version "29.1")
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
-available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+available packages in an archive in `package-archives'.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2197,7 +2218,11 @@ package-install
 `package-selected-packages'.
 
 If PKG is a `package-desc' and it is already installed, don't try
-to install it but still mark it as selected."
+to install it but still mark it as selected.
+
+If the command is invoked with a prefix argument, the upgrading
+of built-in packages will be possible, as if
+`package-install-upgrade-built-in' had been enabled."
   (interactive
    (progn
      ;; Initialize the package system to get the list of package
@@ -2205,11 +2230,14 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and (or current-prefix-arg
+                                         package-install-upgrade-built-in)
+                                     (package--active-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (symbol-name (car elt)))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2248,9 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (and (or current-prefix-arg package-install-upgrade-built-in)
+               (package--active-built-in-p pkg))
+      (setq pkg (or (cadr (assq name package-archive-contents)) pkg)))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 23 bytes --]


-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 18:19:31 +0000
> 
> Sounds good to me.

The patch LGTM, thanks.  Please install it.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Kévin Le Gouguec]

João Távora <joaotavora@gmail.com> writes:

>>   between Eli & Philip re. changing package-install or package-update
>>   makes me unsure what "U x" will actually do with eglot in Emacs 29, so
>>   my previous parenthesized digression might be moot.
>
> Alas U x in the package menu _also_ doesn't upgrade Eglot.  And neither
> does M-x package-update-all.  I don't see any plans for doing so.

Interesting; thank you and Dmitry for confirming my impression.

This was perhaps the most surprising aspect of the equation IMO,
although now it does not sound so bad since, IIUC from reading
package.el, _once_ users manage to install eglot from ELPA (using
Philip's pending user option), _then_ either method should let them
fetch future upgrades transparently.

(
  Took me a couple of minutes to reach that conclusion; found it mildly
  confusing that package-update-all and package-menu-mark-upgrades each
  have their own heuristics for enumerating candidates.  IIUC the former
  iterates through package-alist; the latter looks at all *Packages*
  lines with status ∈ '("installed" "dependency" "unsigned" "external").

  So once eglot becomes "installed" under package-user-dir, both methods
  should allow users to stay on top of new versions… ?

  Hope that's not my optimism skewing my reading of the code.
)

(
  Neither here nor there, but this whole discussion reminds me of
  bug#59005, where another :core package (transient) gets silently
  updated from ELPA if it's a dependency of a package that the user asks
  to install (magit).  Assuming (a) I correctly understand what that
  other bug report is about (b) I correctly understand what Philip's
  patch does, ISTM that it will cause a behaviour change in that
  situation too?

  Not wholly sure about (a) nor (b) though, and even then, not wholly
  sure that the change would be for the worse, especially since there
  would now be a user option to make the desired behaviour explicit.
)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>>>   between Eli & Philip re. changing package-install or package-update
>>>   makes me unsure what "U x" will actually do with eglot in Emacs 29, so
>>>   my previous parenthesized digression might be moot.
>>
>> Alas U x in the package menu _also_ doesn't upgrade Eglot.  And neither
>> does M-x package-update-all.  I don't see any plans for doing so.
>
> Interesting; thank you and Dmitry for confirming my impression.
>
> This was perhaps the most surprising aspect of the equation IMO,
> although now it does not sound so bad since, IIUC from reading
> package.el, _once_ users manage to install eglot from ELPA (using
> Philip's pending user option), _then_ either method should let them
> fetch future upgrades transparently.

Yes, there are two sides to this question.

- User friendliness to interactive commands.  I think it's not very user
  friendly to do one behaviour in Emacs 28 and another in 29 and again
  another in 30.  But I've abandoned this battle.  When doing things
  interactively, users are more likely to ask themselves why and search
  for answers.  So the solution is not that bad.

- Non-interactive installations, such as the ones you get when trying
  someone's init files, setting up new systems after upgrade, CI system
  scripts etc.  Those will be broken, silently or violently, and the
  root cause will be much harder to find.  

  For example, a user enjoying, say, the bug fix to
  https://github.com/joaotavora/eglot/discussions/1206 in Emacs 28 with
  a simple (use-package eglot :ensure t) in her config, will suddenly
  see the bug pop up again in Emacs 29.  The same with a user enjoying
  the no-flooding-echo-area bugfix feature, which depended on changes to
  both Eglot and Eldoc.  The same for a user enjoying the new
  eglot-prefer-plaintext feature for bug#61373.

It's the second problem which is more serious IMO, and it's inevitable
it will keep geting worse.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Philip Kaludercic <philipk@posteo.net> writes:

> +---
> +*** New user option 'package-install-upgrade-built-in'.
> +When enabled, 'package-install' can be used to install 
> +newer versions of built-in packages.
> +

If this is the final version, at least please add to NEWS:

"Note that you must set this variable to t early in your configuration
if you expect to keep the existing behaviour of the Eglot and
'use-package' libraries after the upgrade to Emacs 29."

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 18:19:31 +0000
>> 
>> Sounds good to me.
>
> The patch LGTM, thanks.  Please install it.

Done.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Kévin Le Gouguec]

João Távora <joaotavora@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> +---
>> +*** New user option 'package-install-upgrade-built-in'.
>> +When enabled, 'package-install' can be used to install 
>> +newer versions of built-in packages.
>> +
>
> If this is the final version, at least please add to NEWS:
>
> "Note that you must set this variable to t early in your configuration
> if you expect to keep the existing behaviour of the Eglot and
> 'use-package' libraries after the upgrade to Emacs 29."

And/or something to the same effect in those packages's entries, which
might have more chances of getting noticed?


diff --git a/etc/NEWS b/etc/NEWS
index 0789fa49d75..a2160fa2a92 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -3249,6 +3249,12 @@ based on data provided by language servers using the Language Server
 Protocol (LSP).  See the new Info manual "(eglot) Top" for more.  Also
 see "etc/EGLOT-NEWS".
 
+Note that by default, Emacs's package manager does not fetch updates
+to built-in packages.  If you would like to upgrade to new versions of
+Eglot as they are released on GNU ELPA, enable the new user option
+'package-install-upgrade-built-in' and re-install Eglot with
+'package-install'.
+
 +++
 ** use-package: Declarative package configuration.
 use-package is now shipped with Emacs.  It provides the 'use-package'
@@ -3256,6 +3262,12 @@ macro, which allows you to isolate package configuration in your init
 file in a way that is declarative, tidy, and performance-oriented.
 See the new Info manual "(use-package) Top" for more.
 
+Note that by default, Emacs's package manager does not fetch updates
+to built-in packages.  If you would like to upgrade to new versions of
+use-package as they are released on GNU ELPA, enable the new user
+option 'package-install-upgrade-built-in' and re-install use-package
+with 'package-install'.
+
 ---
 ** New package 'wallpaper'.
 This package provides the command 'wallpaper-set', which sets the

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sun, 16 Apr 2023 13:45:24 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Philip Kaludercic <philipk@posteo.net>
> >> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> >> Date: Sat, 15 Apr 2023 18:19:31 +0000
> >> 
> >> Sounds good to me.
> >
> > The patch LGTM, thanks.  Please install it.
> 
> Done.

Thanks, I'm therefore closing this bug.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 22:28, Eli Zaretskii wrote:
> If, OTOH,
> you think that it's imperative to allow_all_  users of Eglot with
> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> become available), then we should release Emacs 29 with 1.14.

Was this question about stability only?

Because since we've decided in favor of stability of package.el, and 
against eglot's easy upgradability, I would suggest to backport Eglot 
1.14 to emacs-29. Together with the fix for bug#62816. The improvement 
in eldoc's behavior is pretty stark.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sun, Apr 16, 2023 at 9:46 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 14/04/2023 22:28, Eli Zaretskii wrote:
> > If, OTOH,
> > you think that it's imperative to allow_all_  users of Eglot with
> > Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> > become available), then we should release Emacs 29 with 1.14.
> Was this question about stability only?
>
> Because since we've decided in favor of stability of package.el, and
> against eglot's easy upgradability,

Needlessly, I might add, since the simplest patch that did both things
was completely ignored (except for you and Philip).

> I would suggest to backport Eglot
> 1.14 to emacs-29. Together with the fix for bug#62816. The improvement
> in eldoc's behavior is pretty stark.

I'm against that (though I could be convinced).  Note that there are many
things to backport, such as non-bugfix improvements in Eglot's dependencies,
which would _also_ have to be backported.  In this case ElDoc 1.14.  And
if you want the full "stark improvement" package for the echo-area flooding
business, Flymake would likely also need to be backported.

Besides, by the time Emacs 29 is released.  Eglot will probably be at
1.15 or even more with the "breadcrumb headline" feature request of
bug#58431, a class hierarchy browser and a few others.

We could cherry-pick the backports you think are most valuable though,
like bug#62816 (ask to reopen there and convince Eli that the fix
is sound) and maybe a few more.  I'd support that.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 23:46:37 +0300
> Cc: rpluim@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/04/2023 22:28, Eli Zaretskii wrote:
> > If, OTOH,
> > you think that it's imperative to allow_all_  users of Eglot with
> > Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> > become available), then we should release Emacs 29 with 1.14.
> 
> Was this question about stability only?

It was about the criteria for which versions of core packages to ship
with a release.

> Because since we've decided in favor of stability of package.el, and 
> against eglot's easy upgradability, I would suggest to backport Eglot 
> 1.14 to emacs-29.

I won't object.  In fact, I asked up front why not.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 16 Apr 2023 22:54:05 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> Besides, by the time Emacs 29 is released.  Eglot will probably be at
> 1.15 or even more with the "breadcrumb headline" feature request of
> bug#58431, a class hierarchy browser and a few others.

What would prevent us from upgrading Eglot on the release branch to
1.15, when that becomes available?  The Org folks already merge every
change on their stable branch to the Emacs release branch, right up to
the release date.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 17/04/2023 05:24, Eli Zaretskii wrote:
>> Date: Sun, 16 Apr 2023 23:46:37 +0300
>> Cc: rpluim@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 14/04/2023 22:28, Eli Zaretskii wrote:
>>> If, OTOH,
>>> you think that it's imperative to allow_all_  users of Eglot with
>>> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
>>> become available), then we should release Emacs 29 with 1.14.
>>
>> Was this question about stability only?
> 
> It was about the criteria for which versions of core packages to ship
> with a release.

I don't think we can get a single set of criteria across core packages.

E.g. Org is developed externally, has its own community of significant 
size, and does split off release branches (with additional testing, I',m 
guessing).

Eglot, OTOH, is developed only here, with no additional release workflow 
other than what MELPA/GNU ELPA historically provided: collect up some 
features/fixes, bump the Version header, and push a new release out to 
the users. The lack of extended testing period is made up for with the 
capability to push out a new fixed version overnight. That's why the 
difficulty in upgrading to the latest version (for Emacs 29 users) is 
going to hurt.

BTW, if you recall the threads before Eglot was added, I was against 
that, and one of the things I cited is an LSP client has inherently high 
development velocity. Maybe the LSP community will settle/mature/stop 
adding features one day, but it's not there yet.

>> Because since we've decided in favor of stability of package.el, and
>> against eglot's easy upgradability, I would suggest to backport Eglot
>> 1.14 to emacs-29.
> 
> I won't object.  In fact, I asked up front why not.

Note that that suggestion comes with a fix to eldoc which you so far 
have rejected for emacs-29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Tue, Apr 18, 2023 at 2:25 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > It was about the criteria for which versions of core packages to ship
> > with a release.
>
> I don't think we can get a single set of criteria across core packages.

Agree.  We should have categories and understand what is surprising
or workflow-breaking and what is not.

> BTW, if you recall the threads before Eglot was added, I was against
> that, and one of the things I cited is an LSP client has inherently high
> development velocity. Maybe the LSP community will settle/mature/stop
> adding features one day, but it's not there yet.

Very true, but the conclusion is only half-true.

It didn't have to be like this: Eglot _can_ grow rapidly in master
and and have its periodic stable releases.  And in the major Emacs
versions released to the public could have an even stabler release
(because it went through more testing).  This is just like any other
:core package until now.

The solution picked for this issue is bad in that it breaks some Eglot
users workflows and expectations when using very common configuration
recipes.  We should revert it and pick a fix that relies on recognizing
that there are different "sets of criteria" as you propose.  One
such fix remains uncriticized and unchallenged in this thread.

But if that doesn't happen, we shouldn't make a bad situation worse,
by backporting 100's of lines of code of Eglot and friends into Emacs 29.
That's the polar opposite in the pursuit of stability.  Hand-picked
bugfixes for problems manifesting themselves in Emacs 29, sure! But
wholesale changes are just asking for trouble and destroying the
value of the pretest and RC periods.  As bug#62907 shows, there are
certain edge-case bugs due to refactorings in upcoming Eglot 1.15
that are not in 1.12.29 bundled with Emacs 29.  Good! That's the way
it should be. Let's not ruin that.

> >> Because since we've decided in favor of stability of package.el, and
> >> against eglot's easy upgradability, I would suggest to backport Eglot
> >> 1.14 to emacs-29.
> >
> > I won't object.  In fact, I asked up front why not.
>
> Note that that suggestion comes with a fix to eldoc which you so far
> have rejected for emacs-29.

...to name but one change to the non-Eglot, already-there-in-Emacs-28,
libraries Eglot depends on (or will depend on).

I do think _that_ ElDoc fix should be just backported.  It's not
a complicated fix by any measure, it's easy to test, and it indeed
has value and safety.  Together with your similar fix for Company,
it'll make Emacs 29 users happier.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 18/04/2023 14:44, João Távora wrote:

>> BTW, if you recall the threads before Eglot was added, I was against
>> that, and one of the things I cited is an LSP client has inherently high
>> development velocity. Maybe the LSP community will settle/mature/stop
>> adding features one day, but it's not there yet.
> 
> Very true, but the conclusion is only half-true.
> 
> It didn't have to be like this: Eglot _can_ grow rapidly in master
> and and have its periodic stable releases.  And in the major Emacs
> versions released to the public could have an even stabler release
> (because it went through more testing).  This is just like any other
> :core package until now.

This particular one didn't have to, but it's a problem very 
characteristic of joining a strongly centralized project with ultimately 
one person having the last word in all major decisions. And it's not 
like Eli is being unreasonable: we do need a stability cutoff, and we're 
really long past it. These one-more-change kind of arguments repeat year 
over year, with reasonable, well-intentioned people on both sides.

I don't see any better solutions than better modularity (which delegates 
responsibility by its nature) and/or more frequent releases somehow.

> The solution picked for this issue is bad in that it breaks some Eglot
> users workflows and expectations when using very common configuration
> recipes.  We should revert it and pick a fix that relies on recognizing
> that there are different "sets of criteria" as you propose.  One
> such fix remains uncriticized and unchallenged in this thread.

Sure, and I agree, but I don't really see how to present that in terms 
Eli would feel suitable to accept. One "trick" that worked in the past 
was to somehow enumerate all potential execution flows (functions 
involved, etc) that would be affected by the change.

> But if that doesn't happen, we shouldn't make a bad situation worse,
> by backporting 100's of lines of code of Eglot and friends into Emacs 29.
> That's the polar opposite in the pursuit of stability.  Hand-picked
> bugfixes for problems manifesting themselves in Emacs 29, sure! But
> wholesale changes are just asking for trouble and destroying the
> value of the pretest and RC periods.  As bug#62907 shows, there are
> certain edge-case bugs due to refactorings in upcoming Eglot 1.15
> that are not in 1.12.29 bundled with Emacs 29.  Good! That's the way
> it should be. Let's not ruin that.

I don't insist, not at all. It was just my own impression of what would 
constitute a reasonable Eglot release that we could be satisfied with 
having a large number of people use without upgrading, for years. Issues 
like blinking eldoc messages, or eldoc messages that can take up half 
the height of the window seem like things that we wouldn't want in it.

Perhaps the second issue affects only a minority of servers, and I'm 
wrong to be worried. Because otherwise, I really don't understand why it 
hasn't been reported and fixed until recently. Not blaming you, just to 
be clear.

>>>> Because since we've decided in favor of stability of package.el, and
>>>> against eglot's easy upgradability, I would suggest to backport Eglot
>>>> 1.14 to emacs-29.
>>>
>>> I won't object.  In fact, I asked up front why not.
>>
>> Note that that suggestion comes with a fix to eldoc which you so far
>> have rejected for emacs-29.
> 
> ...to name but one change to the non-Eglot, already-there-in-Emacs-28,
> libraries Eglot depends on (or will depend on).
> 
> I do think _that_ ElDoc fix should be just backported.  It's not
> a complicated fix by any measure, it's easy to test, and it indeed
> has value and safety.  Together with your similar fix for Company,
> it'll make Emacs 29 users happier.

That's what I'm thinking, too.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Tue, Apr 18, 2023 at 9:38 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> This particular one didn't have to, but it's a problem very
> characteristic of joining a strongly centralized project with ultimately
> one person having the last word in all major decisions. And it's not
> like Eli is being unreasonable: we do need a stability cutoff, and we're
> really long past it. These one-more-change kind of arguments repeat year
> over year, with reasonable, well-intentioned people on both sides.

Yes. And here both people sides "one more change".  The change that _did_
make it in is more aggressive and more unstable than the one that didn't.

And I repeat it didn't (and doesn't) have to be like this for Eglot and
Emacs 29.  It could have been a minor decision, not a "major" one.

> Sure, and I agree, but I don't really see how to present that in terms
> Eli would feel suitable to accept. One "trick" that worked in the past
> was to somehow enumerate all potential execution flows (functions
> involved, etc) that would be affected by the change.

Right.  And IMO it's not a "trick", it's how it should be.  It's hard to prove a
negative, but at least it should be attempted.  Well, the patch I presented
(the one you +1'ed) makes it so that package-install keeps exactly its previous
behaviour unless its argument is one of (eglot use-package), which are arguments
that could not have ever been passed to that function as :core packages
in Emacs 28. M-x package-install RET seq or (package-install 'xref) keep EXACTLY
the same behaviour.  It's very clear to see from the minimal patch.

> I don't insist, not at all. It was just my own impression of what would
> constitute a reasonable Eglot release that we could be satisfied with
> having a large number of people use without upgrading, for years. Issues
> like blinking eldoc messages, or eldoc messages that can take up half
> the height of the window seem like things that we wouldn't want in it.

The issue has existed and has been worked around successfully for a long
period of time.  It's not actually a problem, it's a consequence of the
default values for eldoc-echo-area-use-multiline-p and max-mini-window-height,
both of which predate Eglot.

Of course I think the current behaviour is better.  But it's also different so
I don't think we should backport that particular one.  Even if so far the
only feedback we have had has been positive, it could well be that some
people _liked_ the half-the-window-height thing (after all their customization
options reflected that wish, even if by default).

And by the way, not really half-the-window.  I used this for a long time
without being much bothered by it.

> Perhaps the second issue affects only a minority of servers, and I'm
> wrong to be worried. Because otherwise, I really don't understand why it
> hasn't been reported and fixed until recently. Not blaming you, just to
> be clear.

It was reported a long time ago.  By you and others.  But there wasn't
the means -- or rather the energy from my part -- to fix it.  I couldn't
just have truncated that information.  So I enhanced ElDoc instead with
the :echo option.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 18/04/2023 23:56, João Távora wrote:
> On Tue, Apr 18, 2023 at 9:38 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> This particular one didn't have to, but it's a problem very
>> characteristic of joining a strongly centralized project with ultimately
>> one person having the last word in all major decisions. And it's not
>> like Eli is being unreasonable: we do need a stability cutoff, and we're
>> really long past it. These one-more-change kind of arguments repeat year
>> over year, with reasonable, well-intentioned people on both sides.
> 
> Yes. And here both people sides "one more change".  The change that _did_
> make it in is more aggressive and more unstable than the one that didn't.

Well, not really. It's by definition more conservative one. Problem is, 
it violates a practice established by third-party community outside of 
Emacs.

>> Sure, and I agree, but I don't really see how to present that in terms
>> Eli would feel suitable to accept. One "trick" that worked in the past
>> was to somehow enumerate all potential execution flows (functions
>> involved, etc) that would be affected by the change.
> 
> Right.  And IMO it's not a "trick", it's how it should be.  It's hard to prove a
> negative, but at least it should be attempted.  Well, the patch I presented
> (the one you +1'ed) makes it so that package-install keeps exactly its previous
> behaviour unless its argument is one of (eglot use-package), which are arguments
> that could not have ever been passed to that function as :core packages
> in Emacs 28. M-x package-install RET seq or (package-install 'xref) keep EXACTLY
> the same behaviour.  It's very clear to see from the minimal patch.

Perhaps a more structured/verbose outline of the same would help.

Although apparently the example I was thinking of 
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37774#479 and the 
surrounding thread) occurred when the corresponding pretest hadn't 
started yet.

>> I don't insist, not at all. It was just my own impression of what would
>> constitute a reasonable Eglot release that we could be satisfied with
>> having a large number of people use without upgrading, for years. Issues
>> like blinking eldoc messages, or eldoc messages that can take up half
>> the height of the window seem like things that we wouldn't want in it.
> 
> The issue has existed and has been worked around successfully for a long
> period of time.  It's not actually a problem, it's a consequence of the
> default values for eldoc-echo-area-use-multiline-p and max-mini-window-height,
> both of which predate Eglot.

So there is a workaround for it anyway, thanks for the reminder.

> Of course I think the current behaviour is better.  But it's also different so
> I don't think we should backport that particular one.  Even if so far the
> only feedback we have had has been positive, it could well be that some
> people _liked_ the half-the-window-height thing (after all their customization
> options reflected that wish, even if by default).
> 
> And by the way, not really half-the-window.  I used this for a long time
> without being much bothered by it.

I guess it depends on the number of overloads for the function around point.

>> Perhaps the second issue affects only a minority of servers, and I'm
>> wrong to be worried. Because otherwise, I really don't understand why it
>> hasn't been reported and fixed until recently. Not blaming you, just to
>> be clear.
> 
> It was reported a long time ago.  By you and others.  But there wasn't
> the means -- or rather the energy from my part -- to fix it.  I couldn't
> just have truncated that information.  So I enhanced ElDoc instead with
> the :echo option.

Aha, so the :echo thingy made it possible. Gotcha.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Tue, Apr 18, 2023 at 10:06 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 18/04/2023 23:56, João Távora wrote:
> > On Tue, Apr 18, 2023 at 9:38 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> >
> >> This particular one didn't have to, but it's a problem very
> >> characteristic of joining a strongly centralized project with ultimately
> >> one person having the last word in all major decisions. And it's not
> >> like Eli is being unreasonable: we do need a stability cutoff, and we're
> >> really long past it. These one-more-change kind of arguments repeat year
> >> over year, with reasonable, well-intentioned people on both sides.
> >
> > Yes. And here both people sides "one more change".  The change that _did_
> > make it in is more aggressive and more unstable than the one that didn't.
>
> Well, not really. It's by definition more conservative one. Problem is,
> it violates a practice established by third-party community outside of
> Emacs.

Hence, more unstable.  There aren't watertight borders here.  These users
in the "third-party" community are using nothing but Emacs and GNU ELPA
Eglot, those are the users that we're hurting while to protect the
non Eglot users.  But why on earth can't we protect the non-Eglot users and
not screw the Eglot users.  We can, with a simpler change.

> >> Sure, and I agree, but I don't really see how to present that in terms
> >> Eli would feel suitable to accept. One "trick" that worked in the past
> >> was to somehow enumerate all potential execution flows (functions
> >> involved, etc) that would be affected by the change.
> >
> > Right.  And IMO it's not a "trick", it's how it should be.  It's hard to prove a
> > negative, but at least it should be attempted.  Well, the patch I presented
> > (the one you +1'ed) makes it so that package-install keeps exactly its previous
> > behaviour unless its argument is one of (eglot use-package), which are arguments
> > that could not have ever been passed to that function as :core packages
> > in Emacs 28. M-x package-install RET seq or (package-install 'xref) keep EXACTLY
> > the same behaviour.  It's very clear to see from the minimal patch.
>
> Perhaps a more structured/verbose outline of the same would help.

It's 7 lines of code.  Elisp should be trivial to read.  Eli read
much more complicated code in this thread.

> > Of course I think the current behaviour is better.  But it's also different so
> > I don't think we should backport that particular one.  Even if so far the
> > only feedback we have had has been positive, it could well be that some
> > people _liked_ the half-the-window-height thing (after all their customization
> > options reflected that wish, even if by default).
> >
> > And by the way, not really half-the-window.  I used this for a long time
> > without being much bothered by it.
>
> I guess it depends on the number of overloads for the function around point.

I'm using C++ ;-)

> >> Perhaps the second issue affects only a minority of servers, and I'm
> >> wrong to be worried. Because otherwise, I really don't understand why it
> >> hasn't been reported and fixed until recently. Not blaming you, just to
> >> be clear.
> >
> > It was reported a long time ago.  By you and others.  But there wasn't
> > the means -- or rather the energy from my part -- to fix it.  I couldn't
> > just have truncated that information.  So I enhanced ElDoc instead with
> > the :echo option.
>
> Aha, so the :echo thingy made it possible. Gotcha.

Right, which is also the reason it makes even _less_ sense to bring Eglot
1.15 into Emacs 29 _without_ ElDoc (I hope that plan is now completely off
the table).

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 19/04/2023 00:15, João Távora wrote:
>> Aha, so the :echo thingy made it possible. Gotcha.
> Right, which is also the reason it makes even_less_  sense to bring Eglot
> 1.15 into Emacs 29_without_  ElDoc (I hope that plan is now completely off
> the table).

Eh, sure.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 00:20:21 +0300
> Cc: Eli Zaretskii <eliz@gnu.org>, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 19/04/2023 00:15, João Távora wrote:
> >> Aha, so the :echo thingy made it possible. Gotcha.
> > Right, which is also the reason it makes even_less_  sense to bring Eglot
> > 1.15 into Emacs 29_without_  ElDoc (I hope that plan is now completely off
> > the table).
> 
> Eh, sure.

It isn't my call in this case, but FWIW: I still have no idea why
wouldn't we want Eglot 1.14 or 1.15 to be in Emacs 29.1.  I didn't
hear any serious argument against doing that; every reason that was
raised was almost immediately explained away as not being a hard
limitation.

And mind you: Emacs 29.1 will not be released tomorrow or the day
after.  We still have at least several weeks till then, with at least
one more pretest.  So the decision whether to import a newer Eglot
into the release branch doesn't have to be today.  However, the
argument against updating Eglot on the release branch, such as they
were, are of some vaguely "fundamental" nature, so I'm not sure a few
more weeks of time will change the decision.  No one said something
like "if Emacs 29.1 were to be released in NN weeks or more, it would
be okay to update Eglot on the release branch."  But then I already
admitted to not understanding those reasons, so maybe I'm missing
something here.

So there you are.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 1:05 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > Date: Wed, 19 Apr 2023 00:20:21 +0300
> > Cc: Eli Zaretskii <eliz@gnu.org>, rpluim@gmail.com, philipk@posteo.net,
> >  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> > From: Dmitry Gutov <dmitry@gutov.dev>
> >
> > On 19/04/2023 00:15, João Távora wrote:
> > >> Aha, so the :echo thingy made it possible. Gotcha.
> > > Right, which is also the reason it makes even_less_  sense to bring Eglot
> > > 1.15 into Emacs 29_without_  ElDoc (I hope that plan is now completely off
> > > the table).
> >
> > Eh, sure.
>
> It isn't my call in this case, but FWIW: I still have no idea why
> wouldn't we want Eglot 1.14 or 1.15 to be in Emacs 29.1.  I didn't
> hear any serious argument against doing that; every reason that was
> raised was almost immediately explained away as not being a hard
> limitation.
>
> And mind you: Emacs 29.1 will not be released tomorrow or the day
> after.  We still have at least several weeks till then, with at least
> one more pretest.  So the decision whether to import a newer Eglot
> into the release branch doesn't have to be today.  However, the
> argument against updating Eglot on the release branch, such as they
> were, are of some vaguely "fundamental" nature, so I'm not sure a few
> more weeks of time will change the decision.  No one said something
> like "if Emacs 29.1 were to be released in NN weeks or more, it would
> be okay to update Eglot on the release branch."  But then I already
> admitted to not understanding those reasons, so maybe I'm missing
> something here.
>
> So there you are.

Eli, do you want Eglot 1.14 (or 1.15, or 1.16 or whatever version
is the latest Eglot release "several weeks" from now) to be in Emacs 29?

From your writings, I'm assuming you do.  Let's call that version Eglot
1.1x with x > 14.  If we did that, we would have two options:

1. Bundle Eglot 1.1x with Emacs 29 and all its up-to-date dependencies, so
that, right at the moment of the Emacs 29 release, Eglot would function
exactly the same on Emacs 28 + package-install.

2. Bundle a "Frankenglot" with Emacs 29 that has all the
lisp/progmodes/eglot.el code of the future Eglot version, advertises
itself as Eglot 1.1x, probably doesn't break, but does _not_ provide
the same experience as Emacs 28 + package-install

Both options are bad, IMO, but 2 is worse.

The first reason that both options are bad is that you're discarding
whatever value the pretest phase brings to the stability of Eglot's code.
Eglot, being a part of Emacs the Emacs code base, benefits from the same
testing all its code does.  You're discarding that value, and I think
it's bad, because the pretest is supposedly there for a good reason. A bug in
Eglot 1.1x will just escape us and there's no time to fix it.

The second reason only applies to option 2. It would completely confuse
users.  A user running Emacs 28 would see a much better 1.1x than
the 1.1x bundled in Emacs 29. Her configuration for Eglot 1.1x
could simply break in 1.1x. Eglot 1.1x was never designed to be
run in such a hampered environment.

They are "fundamental" reasons indeed.  Are they now less vague
and more concrete?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 14:04:10 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> Eli, do you want Eglot 1.14 (or 1.15, or 1.16 or whatever version
> is the latest Eglot release "several weeks" from now) to be in Emacs 29?

What I want in Emacs 29.1 (and any future release of Emacs) is the
latest version of Eglot that you, as Eglot developer, consider stable
enough to recommend to users of a stable Emacs.  Which version is that
is your decision.  But to make sense to me, the decision you make
should be consistent: if a version X.Y of Eglot is "stable enough" for
you to recommend that users of Emacs 29.n upgrade to it, then that
version X.Y can be part of Emacs 29.n.  (Of course, if you decided
that X.Y is stable enough _after_ Emacs 29.n was released, then X.Y
will be able to become part of Emacs only in Emacs 29.n+1 and later.)

I say above "to make sense to me", and I mean that, and only that.
That is, you _can_ decide that you don't agree with my definition of
consistency, and therefore Eglot X.Y will be recommended to users of
Emacs 29.n, but at the same time you don't want it to be in Emacs
29.n; such a decision will not "make sense to me", but we will still
act according to your decision in this matter.

I hope I clarified my position in this regard.

And before you draw too far-fetched conclusions from the above: I'm
saying this about Eglot, and only about Eglot.  Similar decisions
about other packages, and the conditions for including those other
packages in a stable Emacs versions, could very well be different.

> 1. Bundle Eglot 1.1x with Emacs 29 and all its up-to-date dependencies, so
> that, right at the moment of the Emacs 29 release, Eglot would function
> exactly the same on Emacs 28 + package-install.

That is probably not acceptable, although I'd need to know exactly
which versions of what other packages need to be imported into the
emacs-29 branch, to give a definitive answer.

> 2. Bundle a "Frankenglot" with Emacs 29 that has all the
> lisp/progmodes/eglot.el code of the future Eglot version, advertises
> itself as Eglot 1.1x, probably doesn't break, but does _not_ provide
> the same experience as Emacs 28 + package-install
> 
> Both options are bad, IMO, but 2 is worse.

I don't see why option 2 would be bad, let alone worse.  See below.

> The first reason that both options are bad is that you're discarding
> whatever value the pretest phase brings to the stability of Eglot's code.

Eglot itself isn't my problem.  My problem is the other packages that
upgrading Eglot, per option 1 above, will require to update: those
other packages usually affect much more than just Eglot, and therefore
bringing their potentially less stable code into emacs-29 might break
much more than just Eglot.

> Eglot, being a part of Emacs the Emacs code base, benefits from the same
> testing all its code does.  You're discarding that value, and I think
> it's bad, because the pretest is supposedly there for a good reason. A bug in
> Eglot 1.1x will just escape us and there's no time to fix it.

Please leave this consideration to me, it is exactly the kind of
judgment call I make every several days when someone asks whether a
particular change is OK for the release branch.  And in the case of
Eglot I already said that IMO we should include in Emacs 29.1 the
latest stable version of Eglot, thus my decision about that was
already made in favor of upgrading Eglot on emacs-29.  But I won't
insist if you are uncomfortable with that.

> The second reason only applies to option 2. It would completely confuse
> users.  A user running Emacs 28 would see a much better 1.1x than
> the 1.1x bundled in Emacs 29.

No, users of Emacs 28 who installed the latest stable Eglot from ELPA
should see almost the same version of Eglot as users of Emacs 29.1 if
that will come with the same stable Eglot.  Unless you intend to
somehow degenerate important parts of Eglot in what you call
"Frankenglot", whatever that means.

My interpretation of option 2 is that we get a newer Eglot (1.14 or
1.15, whichever you decide is stable enough) with various minor
fallbacks intended to work around the fact that dependency packages
are not necessarily at their versions for which Eglot 1.14/1.15 was
designed to work, if the versions of those dependencies in Emacs 29.1
are older.  That is, 1.14/1.15 without the mandatory requirement o
upgrade any other package already in Emacs.  If that is not what you
mean, then perhaps consider this as option 3, which is the one that
makes the most sense to me, and if possible and agreed upon, will be
accepted into emacs-29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 2:35 PM Eli Zaretskii <eliz@gnu.org> wrote:

> What I want in Emacs 29.1 (and any future release of Emacs) is the
> latest version of Eglot that you, as Eglot developer, consider stable
> enough to recommend to users of a stable Emacs.  Which version is that
> is your decision.  But to make sense to me, the decision you make
> should be consistent: if a version X.Y of Eglot is "stable enough" for
> you to recommend that users of Emacs 29.n upgrade to it, then that
> version X.Y can be part of Emacs 29.n.  (Of course, if you decided
> that X.Y is stable enough _after_ Emacs 29.n was released, then X.Y
> will be able to become part of Emacs only in Emacs 29.n+1 and later.)

OK. Then I will tell you that "stable enough", for, me means "as stable
as possible", because I like things to be as stable as possible.
And, to me (and I would think to most people), the most stable
version of a program is the one which has seen the most testing. So,
I see no reason to give Eglot a shorter pretest period than the
rest of Emacs gets.  Thus Eglot 1.12.29 it is.

> I hope I clarified my position in this regard.

OK, it seems you're giving the call to me.  Then I hope I've also
clarified my decision and the criteria I used in reaching it.

> > 1. Bundle Eglot 1.1x with Emacs 29 and all its up-to-date dependencies, so
> > that, right at the moment of the Emacs 29 release, Eglot would function
> > exactly the same on Emacs 28 + package-install.
>
> That is probably not acceptable, although I'd need to know exactly
> which versions of what other packages need to be imported into the
> emacs-29 branch, to give a definitive answer.

I could teach you how to figure that out (it's very simple), but
since I don't want this either, and you just gave the call to me,
there's probably no point.

> > 2. Bundle a "Frankenglot" with Emacs 29 that has all the
> > lisp/progmodes/eglot.el code of the future Eglot version, advertises
> > itself as Eglot 1.1x, probably doesn't break, but does _not_ provide
> > the same experience as Emacs 28 + package-install
> >
> > Both options are bad, IMO, but 2 is worse.
>
> I don't see why option 2 would be bad, let alone worse.  See below.
>
> > The first reason that both options are bad is that you're discarding
> > whatever value the pretest phase brings to the stability of Eglot's code.
>
> Eglot itself isn't my problem.  My problem is the other packages that
> upgrading Eglot, per option 1 above, will require to update: those
> other packages usually affect much more than just Eglot, and therefore
> bringing their potentially less stable code into emacs-29 might break
> much more than just Eglot.

Yes, as I said option 1 is bad.  Option 2 is worse, but 1 is pretty
bad too, partly for the reasons you state (yes you're totally right
"it's much more than Eglot").

> > Eglot, being a part of Emacs the Emacs code base, benefits from the same
> > testing all its code does.  You're discarding that value, and I think
> > it's bad, because the pretest is supposedly there for a good reason. A bug in
> > Eglot 1.1x will just escape us and there's no time to fix it.
>
> Please leave this consideration to me, it is exactly the kind of
> judgment call I make every several days when someone asks whether a
> particular change is OK for the release branch.  And in the case of
> Eglot I already said that IMO we should include in Emacs 29.1 the
> latest stable version of Eglot, thus my decision about that was
> already made in favor of upgrading Eglot on emacs-29.  But I won't
> insist if you are uncomfortable with that.

There is no one "stable".  You yourself are talking about stability
gradation.   Eglot 1.12.29 is "stablest".  Eglot 1.14 is "stable". Eglot
in master has recently gone two days with a semi-serious live bug (now
fixed) Let's call it "unstable" anyway.  Before I release 1.15, I will
try to make
sure it is as stable as 1.14 (or more).

By the way, the ElDoc change about the non-blinking is "safe for
the release branch" IMHO.  Just in case you didn't get my opinion
on that part of ElDoc of which I am the author and the maintainer
since 2018.

> > The second reason only applies to option 2. It would completely confuse
> > users.  A user running Emacs 28 would see a much better 1.1x than
> > the 1.1x bundled in Emacs 29.
>
> No, users of Emacs 28 who installed the latest stable Eglot from ELPA
> should see almost the same version of Eglot as users of Emacs 29.1 if
> that will come with the same stable Eglot.  Unless you intend to
> somehow degenerate important parts of Eglot in what you call
> "Frankenglot", whatever that means.

Frankenglot 1.1x is just Eglot without the resources that 1.1x would
have in Emacs 28.  Don't you understand?  The "good" Eglot experience
requires things that are not in lisp/progmodes/eglot.el.  It requires,
for example, :echo support in ElDoc's eldoc-documentation-functions
so that echo areas aren't flooded by some LSP servers.
eglot.el can function without it, but users would be surprised.  They would
ask: hey didn't you fix this echo area flooding, back in Eglot 1.14???
And I would have to explain: this is not really Eglot 1.1x, this is
Frankenglot 1.1x:  the content of the eglot.el file is the same but
it's missing a lot of stuff.  And then I wouldn't even know how to
teach them to upgrade.

In so many software packages I know outside of Emacs, bugs and
features are fixed and added just by bumping a dependency.  Surely
you must have seen this, too.

> My interpretation of option 2 is that we get a newer Eglot (1.14 or
> 1.15, whichever you decide is stable enough) with various minor
> fallbacks intended to work around the fact that dependency packages
> are not necessarily at their versions for which Eglot 1.14/1.15 was
> designed to work, if the versions of those dependencies in Emacs 29.1
> are older.

Why put ourselves (mostly myself) through these chores??  Just so that
two weeks later after whatever Emacs 29 is officially released a more
recent,  "stable"  version is already available? And pay the price of
discarding the value of the pretest period?  Just because Eglot is now
much harder to  upgrade?  Then just make it easy to upgrade: it's in your
control.

There there are 0 downsides, as far as I can tell.  As far as anyone
who has commented that patch can tell.   I've seen 0 contestation
of the patch I proposed before, which I've grown almost tired of
linking to.  That patch was engineered to answer precisely your
objections to previous patches.  It was engineered to make _you_
happy and make me happy, and make Eglot users happy.  And you're
ignoring it.

 That is, 1.14/1.15 without the mandatory requirement o
> upgrade any other package already in Emacs.  If that is not what you
> mean, then perhaps consider this as option 3

No, you're just describing option 2, as far as I can see.
But probably no "various minor fallbacks" would be needed, because
I am careful with the interfaces.


-- 
João Távora

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 19/04/2023 15:05, Eli Zaretskii wrote:
> It isn't my call in this case, but FWIW: I still have no idea why
> wouldn't we want Eglot 1.14 or 1.15 to be in Emacs 29.1.  I didn't
> hear any serious argument against doing that; every reason that was
> raised was almost immediately explained away as not being a hard
> limitation.

Okay, let me try to answer this: since the goal is (apparently) to have 
a stable version of Eglot in emacs-29, we don't know yet whether 1.14 or 
1.15 is "stable".

> And mind you: Emacs 29.1 will not be released tomorrow or the day
> after.  We still have at least several weeks till then, with at least
> one more pretest.  So the decision whether to import a newer Eglot
> into the release branch doesn't have to be today.  However, the
> argument against updating Eglot on the release branch, such as they
> were, are of some vaguely "fundamental" nature, so I'm not sure a few
> more weeks of time will change the decision.  No one said something
> like "if Emacs 29.1 were to be released in NN weeks or more, it would
> be okay to update Eglot on the release branch."  But then I already
> admitted to not understanding those reasons, so maybe I'm missing
> something here.

Let's imagine I was making this choice.

I would include (or propose for inclusion) Eglot 1.x.y in Emacs 29 only 
N weeks after it has been tagged on master and thus published to ELPA, 
on the condition that no major bugs have been discovered in the meantime 
that required major reworks (because any bugfix would reset the timer to 
L weeks where L < N, but a major change would reset it to N weeks 
again). That would be the general guideline. Add to that the 
maintainer's best judgment, who would be able to reduce or extend these 
periods on case by case basis as well, according to the changes that 
went into every release.

To answer the original question: N weeks still haven't passed (I guess) 
since 1.15 was tagged, so we don't quite know whether it's acceptable 
for emacs-29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 15:04:30 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Wed, Apr 19, 2023 at 2:35 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > My interpretation of option 2 is that we get a newer Eglot (1.14 or
> > 1.15, whichever you decide is stable enough) with various minor
> > fallbacks intended to work around the fact that dependency packages
> > are not necessarily at their versions for which Eglot 1.14/1.15 was
> > designed to work, if the versions of those dependencies in Emacs 29.1
> > are older.
> 
> Why put ourselves (mostly myself) through these chores??

Because I thought we agreed that requiring newer versions of other
packages where that could be avoided (note: "could be avoided", not
"is nice to have") is a Good Thing, and is well worth these chores.
But if you don't agree, fine; just one more disagreement between us.

> Just so that two weeks later after whatever Emacs 29 is officially
> released a more recent, "stable" version is already available?

No, just so users of Emacs 29 could have a better Eglot without any
complications.

But again, have it your way.  I said I won't argue with your decision
in this matter.  I just wanted you and everyone else to understand my
position on this.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 18:48:00 +0300
> Cc: joaotavora@gmail.com, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > And mind you: Emacs 29.1 will not be released tomorrow or the day
> > after.  We still have at least several weeks till then, with at least
> > one more pretest.  So the decision whether to import a newer Eglot
> > into the release branch doesn't have to be today.  However, the
> > argument against updating Eglot on the release branch, such as they
> > were, are of some vaguely "fundamental" nature, so I'm not sure a few
> > more weeks of time will change the decision.  No one said something
> > like "if Emacs 29.1 were to be released in NN weeks or more, it would
> > be okay to update Eglot on the release branch."  But then I already
> > admitted to not understanding those reasons, so maybe I'm missing
> > something here.
> 
> Let's imagine I was making this choice.
> 
> I would include (or propose for inclusion) Eglot 1.x.y in Emacs 29 only 
> N weeks after it has been tagged on master and thus published to ELPA, 
> on the condition that no major bugs have been discovered in the meantime 
> that required major reworks (because any bugfix would reset the timer to 
> L weeks where L < N, but a major change would reset it to N weeks 
> again). That would be the general guideline. Add to that the 
> maintainer's best judgment, who would be able to reduce or extend these 
> periods on case by case basis as well, according to the changes that 
> went into every release.
> 
> To answer the original question: N weeks still haven't passed (I guess) 
> since 1.15 was tagged, so we don't quite know whether it's acceptable 
> for emacs-29.

That is fine with me, assuming N has some reasonable value.  It means
I could ask you again before the next pretest, and then again before
RC, and perhaps you'd then agree to import a newer Eglot.

But note that this is not what João is saying.  He says 1.14 will not
be in Emacs 29.1, period.  No matter how long I will drag the pretest.
He certainly doesn't want to invest the effort of making Eglot 1.14
less dependent on latest changes in other packages, so as to make sure
we could drop Eglot 1.14 into Emacs 29 without risking any problems
elsewhere.  And that more or less seals the issue, effectively setting
your N to infinity.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 5:01 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 19 Apr 2023 15:04:30 +0100
> > Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net,
> >       62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> >
> > On Wed, Apr 19, 2023 at 2:35 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > My interpretation of option 2 is that we get a newer Eglot (1.14 or
> > > 1.15, whichever you decide is stable enough) with various minor
> > > fallbacks intended to work around the fact that dependency packages
> > > are not necessarily at their versions for which Eglot 1.14/1.15 was
> > > designed to work, if the versions of those dependencies in Emacs 29.1
> > > are older.
> >
> > Why put ourselves (mostly myself) through these chores??
>
> Because I thought we agreed that requiring newer versions of other
> packages where that could be avoided (note: "could be avoided", not
> "is nice to have") is a Good Thing, and is well worth these chores.
> But if you don't agree, fine; just one more disagreement between us.

I gave plenty of arguments (which you didn't contest) for why doing
this is a very bad thing (in my opinion of course).  If your utmost priority
is to not require newer versions of other packages, then 1.12.29 should be
just fine!  It will work without an internet connection.  And it will be much
more consistent and well tested than Frankenglot 1.1x, because it will
go through the pretest period.

> > Just so that two weeks later after whatever Emacs 29 is officially
> > released a more recent, "stable" version is already available?
>
> No, just so users of Emacs 29 could have a better Eglot without any
> complications.

It wouldn't be better.  It would be a different thing, a maintenance
nightmare for one.  What if the user then _did_ explicitly install of
those dependencies?  Installing a package and its dependencies isn't a
complication, it's how it has worked for years.  _You've_ decided it is
a complication (and made it more complicated) for Eglot Emacs 29. But
people have been doing it for years.


João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 5:10 PM Eli Zaretskii <eliz@gnu.org> wrote:

> But note that this is not what João is saying.  He says 1.14 will not
> be in Emacs 29.1, period.  No matter how long I will drag the pretest.
> He certainly doesn't want to invest the effort of making Eglot 1.14
> less dependent on latest changes in other packages, so as to make sure
> we could drop Eglot 1.14 into Emacs 29 without risking any problems
> elsewhere.  And that more or less seals the issue, effectively setting
> your N to infinity.

This is extremely odd for me.  Weren't you the one very, very sternly
asking for almost _no_ changes to go into Elisp code of the Emacs 29
now that it is in pretest.  And now you're liberally and casually
suggesting way-over-last minute changes and work to go into that same
version?

This just doesn't make any sense to me.  Can't you understand that other
maintainers also value stability for their packages?

It's certainly NOT about "not wanting to invest the effort".  That effort
would amount to forking Eglot in its feature set so that effort would be
a disservice to everybody.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 17:23:19 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Wed, Apr 19, 2023 at 5:10 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > But note that this is not what João is saying.  He says 1.14 will not
> > be in Emacs 29.1, period.  No matter how long I will drag the pretest.
> > He certainly doesn't want to invest the effort of making Eglot 1.14
> > less dependent on latest changes in other packages, so as to make sure
> > we could drop Eglot 1.14 into Emacs 29 without risking any problems
> > elsewhere.  And that more or less seals the issue, effectively setting
> > your N to infinity.
> 
> This is extremely odd for me.  Weren't you the one very, very sternly
> asking for almost _no_ changes to go into Elisp code of the Emacs 29
> now that it is in pretest.  And now you're liberally and casually
> suggesting way-over-last minute changes and work to go into that same
> version?

I'm talking only about Eglot, not in general.  I believe I've stressed
that point already.  So what I said or did or asked to do regarding
other packages or other places in Emacs is not necessarily relevant
for this discussion.  The decisions about these issues are always on a
case by case basis, so you cannot compare different cases and expect
them to yield the same decisions.

> This just doesn't make any sense to me.  Can't you understand that other
> maintainers also value stability for their packages?

Of course I can.  But once again: if Eglot 1.14 is not stable enough,
then why do we recommend users of Emacs 29 to update their bundled
Eglot to v1.14?  This is inconsistent: if 1.14 is not stable enough to
be in Emacs 29, we should only recommend it for users of Emacs 30.

I already explained this inconsistency more than once.  Why do you
keep bringing it up time and again?  You might disagree, but why do
you insist that I accept what I perceive as inconsistent logic?  Just
let it go and accept that we disagree about this.

> It's certainly NOT about "not wanting to invest the effort".  That effort
> would amount to forking Eglot in its feature set so that effort would be
> a disservice to everybody.

No, it doesn't require any forks.  It requires more cautious
introduction of new features into Eglot on master.  And yes, it's
extra effort.  But IMNSHO, users will benefit, so in my book it's
worth it.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 19/04/2023 19:10, Eli Zaretskii wrote:
>> To answer the original question: N weeks still haven't passed (I guess)
>> since 1.15 was tagged, so we don't quite know whether it's acceptable
>> for emacs-29.
> That is fine with me, assuming N has some reasonable value.  It means
> I could ask you again before the next pretest, and then again before
> RC, and perhaps you'd then agree to import a newer Eglot.

More or less.

> But note that this is not what João is saying.  He says 1.14 will not
> be in Emacs 29.1, period.  No matter how long I will drag the pretest.
> He certainly doesn't want to invest the effort of making Eglot 1.14
> less dependent on latest changes in other packages, so as to make sure
> we could drop Eglot 1.14 into Emacs 29 without risking any problems
> elsewhere.  And that more or less seals the issue, effectively setting
> your N to infinity.

I think you're simply talking past each other, and will essentially 
agree on that "N weeks" thing outside of this discussion.

But also note that you added a complication: avoid bumping the required 
dependencies. If the backport is not performed as-is and needs 
additional changes (with extra shims, stubbed new features, etc, instead 
of simply using new features from the latest eldoc.el), then you are not 
just asking whether the maintainer thinks the new version is good and 
stable, but also whether they are willing to expend extra effort 
altering it and maintaining diverging versions of code.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 5:50 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > This just doesn't make any sense to me.  Can't you understand that other
> > maintainers also value stability for their packages?
>
> Of course I can.  But once again: if Eglot 1.14 is not stable enough,

Stable enough for whom, or for what?  Stability is quantity in a spectrum.

I think Emacs releases should come with the most well tested code.  No
program is perfect.  Eglot 1.14 is stable, but it's not _as_ stable as Eglot
1.12 because the latter has seen more testing?

So different users and different use cases will accept different things.

> > It's certainly NOT about "not wanting to invest the effort".  That effort
> > would amount to forking Eglot in its feature set so that effort would be
> > a disservice to everybody.
>
> No, it doesn't require any forks.  It requires more cautious
> introduction of new features into Eglot on master.  And yes, it's
> extra effort.  But IMNSHO, users will benefit, so in my book it's
> worth it.

AFAICT you suggest a different eglot.el in Emacs 29 which has the
same features as eglot.el in Emacs master, but without needing
the dependencies that the master version can enjoy.  That's a
fork if I've ever seen one.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 20:23:05 +0300
> Cc: joaotavora@gmail.com, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > But note that this is not what João is saying.  He says 1.14 will not
> > be in Emacs 29.1, period.  No matter how long I will drag the pretest.
> > He certainly doesn't want to invest the effort of making Eglot 1.14
> > less dependent on latest changes in other packages, so as to make sure
> > we could drop Eglot 1.14 into Emacs 29 without risking any problems
> > elsewhere.  And that more or less seals the issue, effectively setting
> > your N to infinity.
> 
> I think you're simply talking past each other, and will essentially 
> agree on that "N weeks" thing outside of this discussion.

I wish.

> But also note that you added a complication: avoid bumping the required 
> dependencies. If the backport is not performed as-is and needs 
> additional changes (with extra shims, stubbed new features, etc, instead 
> of simply using new features from the latest eldoc.el), then you are not 
> just asking whether the maintainer thinks the new version is good and 
> stable, but also whether they are willing to expend extra effort 
> altering it and maintaining diverging versions of code.

Yes, noted.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 18:27:08 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Wed, Apr 19, 2023 at 5:50 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > This just doesn't make any sense to me.  Can't you understand that other
> > > maintainers also value stability for their packages?
> >
> > Of course I can.  But once again: if Eglot 1.14 is not stable enough,
> 
> Stable enough for whom, or for what?  Stability is quantity in a spectrum.

Stable for us.  We are not talking about absolutes here, we are
talking about relative stability.  I'm saying that if some package is
stable enough to be used with Emacs version X.Y, it is by definition
also stable enough to be included in Emacs version X.Y.  The relative
stability levels of these two cases must be the same, or else we are
inconsistent in our own judgment of stability.

> I think Emacs releases should come with the most well tested code.  No
> program is perfect.  Eglot 1.14 is stable, but it's not _as_ stable as Eglot
> 1.12 because the latter has seen more testing?

Then how come we tell users of this same Emacs 29 to update to Eglot
1.14 without too much thought?  And you even insist on making that
automatic when packages are updated at startup.  Won't that
destabilize their Emacs?

> > > It's certainly NOT about "not wanting to invest the effort".  That effort
> > > would amount to forking Eglot in its feature set so that effort would be
> > > a disservice to everybody.
> >
> > No, it doesn't require any forks.  It requires more cautious
> > introduction of new features into Eglot on master.  And yes, it's
> > extra effort.  But IMNSHO, users will benefit, so in my book it's
> > worth it.
> 
> AFAICT you suggest a different eglot.el in Emacs 29 which has the
> same features as eglot.el in Emacs master, but without needing
> the dependencies that the master version can enjoy.  That's a
> fork if I've ever seen one.

No, I suggest that you make changes on master so that these problems
are avoided in the first place.  Changes in a core package on the
Emacs master branch should be done while keeping in mind that this
same version of a package will be on ELPA and users of older Emacsen
will install that newer version.  So the newer version on master
should avoid making changes which would mandate newer versions of
other packages, by providing the necessary compatibility fallbacks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 6:59 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 19 Apr 2023 18:27:08 +0100
> > Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net,
> >       62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> >
> > On Wed, Apr 19, 2023 at 5:50 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > > This just doesn't make any sense to me.  Can't you understand that other
> > > > maintainers also value stability for their packages?
> > >
> > > Of course I can.  But once again: if Eglot 1.14 is not stable enough,
> >
> > Stable enough for whom, or for what?  Stability is quantity in a spectrum.
>
> Stable for us.  We are not talking about absolutes here, we are
> talking about relative stability.  I'm saying that if some package is
> stable enough to be used with Emacs version X.Y, it is by definition
> also stable enough to be included in Emacs version X.Y.  The relative
> stability levels of these two cases must be the same, or else we are
> inconsistent in our own judgment of stability.
>
> > I think Emacs releases should come with the most well tested code.  No
> > program is perfect.  Eglot 1.14 is stable, but it's not _as_ stable as Eglot
> > 1.12 because the latter has seen more testing?
>
> Then how come we tell users of this same Emacs 29 to update to Eglot
> 1.14 without too much thought?  And you even insist on making that
> automatic when packages are updated at startup.  Won't that
> destabilize their Emacs?

I don't dictate to users how much to think :-)  I just assume that
they've been thinking about it in Emacs 28, and they are happy
with this amount of thinking, else they would be complaining
about stability, and they aren't.  Maybe some users are holding
on to Emacs 26.3 + Eglot 1.0 for dear life, who knows? It's very
simple to me.

To summarize and to hopefully answer the question that you repeatedly
put before me, this is my recommendation to users:

 "Dear user,

  Emacs 29 will come with the most stable version I can offer, and I
  can tell you, prospective user, that it has been through as much
  of the Emacs 29 pretest period as possible. If you, prospective user,
  are interested in the features listed in etc/EGLOT-NEWS, you may try
  upgrading to the latest version (I hope it will be easy, good luck!)
  In the unlikely event that things go sour for your particular
  LSP server and use case, you have my deepest apologies, and you
  should revert back to the version bundled with Emacs 29.

  Thank you,
  Your maintainer"

The second part of this recommendation is what I've always recommended.
I've never had to phrase it like this, but this has always been my
recommendation.

> No, I suggest that you make changes on master so that these problems
> are avoided in the first place.  Changes in a core package on the
> Emacs master branch should be done while keeping in mind that this
> same version of a package will be on ELPA and users of older Emacsen
> will install that newer version.  So the newer version on master
> should avoid making changes which would mandate newer versions of
> other packages, by providing the necessary compatibility fallbacks.

So, if I want to do a feature on master that depends on some
new infrastructure on package X that appeared on master (say Xref's
upcoming support for "find any type of thing declaration/macro/etc"),
I have to do no less than duplicate that new infrastructure in Eglot.el
and do a runtime check.  That's... not good, to say the least.

But noted.  Now I (think I) understand what you suggest.

AFAICT basically suggesting all Package-Requires are eliminated,
to get rid of package.el dependency system.  You should argue for
that in emacs-devel explicitly so, I think, because that will
probably elicit some interesting reactions from :core package
developers. "Package-Requires" is used liberally now AFAIK

It's the most basic underpinning of Eglot's design philosophy,
if there such a thing.  That's why Eglot is called "minimal".  It's
not minimal in functionality!  It's minimal in size and complexity
because it reuses code and keeps itself simple.  And if I may be a bit less
humble about it, I've gotten a few compliments about it.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 19:27:11 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Wed, Apr 19, 2023 at 6:59 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > No, I suggest that you make changes on master so that these problems
> > are avoided in the first place.  Changes in a core package on the
> > Emacs master branch should be done while keeping in mind that this
> > same version of a package will be on ELPA and users of older Emacsen
> > will install that newer version.  So the newer version on master
> > should avoid making changes which would mandate newer versions of
> > other packages, by providing the necessary compatibility fallbacks.
> 
> So, if I want to do a feature on master that depends on some
> new infrastructure on package X that appeared on master (say Xref's
> upcoming support for "find any type of thing declaration/macro/etc"),
> I have to do no less than duplicate that new infrastructure in Eglot.el
> and do a runtime check.

Not necessarily duplicate.  That is only one alternative, and not the
best one, IMO.  Other alternatives could be:

  . decide that users who use this new version of Eglot without that
    new version of Xref will not have this new Eglot feature
  . provide a less functional and simpler replacement for that Xref
    feature
  . find a way of providing the new Eglot feature without relying on
    Xref, but via some alternative solution

Whether each one of the above could be relevant and reasonable depends
on what exactly are those Eglot and Xref features.

I'm sure you don't need the above, you know that better than I do.
It's stuff we do in Emacs every day, and core packages consider these
factors since we began having core packages.  The question is just how
far to go in that direction.  I'm trying to make a point that going as
far as is practical will allow us to bundle at least some packages in
newer versions, which in the end will benefit users.

> AFAICT basically suggesting all Package-Requires are eliminated,
> to get rid of package.el dependency system.

No!  Not "all dependencies eliminated", that's impractical.  I'm
talking only about dependencies on core packages, and I'm asking to
avoid mandatory updates to newer versions.  The dependencies should
still exist, but the minimum supported version of each dependency
should ideally barely change, or at least change as infrequently as
possible.

> You should argue for that in emacs-devel explicitly so

That's what I did.  I'm not to bl;ame that you keep responding here
when I asked a day ago to respond to the discussion I started on
emacs-devel.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 01:06:10 +0300
> Cc: Eli Zaretskii <eliz@gnu.org>, arne_bab@web.de, jporterbugs@gmail.com,
>  emacs-devel <emacs-devel@gnu.org>
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> OK then, I think have to re-evaluate my position on this. Previously, I 
> guess, I made some hasty conclusions from how the discussion went on 
> without refreshing the exact details about package.el and use-package 
> (the latter I never knew to begin with). Apologies.
> 
> Eli, let me know if we should take this back to the bug tracker instead.

I've moved this back to the bug-tracker.  Please post all further
replies about this particular issue, i.e. updating of built-in
packages with package.el, to this bug and not to emacs-devel.

> So I would suggest to focus on functions that don't work as intended. 
> Namely:
> 
> - Add a user option for the list of builtin packages which would be 
> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x 
> package-upgrade-all' (nee package-update-all). Maybe make it nil by 
> default, or maybe add 'eglot' to it. I don't have a strong opinion.
> 
> - Fix 'M-x package-upgrade' (nee package-update) to suggest Eglot as one 
> of the options and actually perform the upgrade. That shouldn't require 
> changes to 'package-install' because, as we already know, the user can 
> already install a newer version of Eglot using the 'list-packages' menu 
> (and picking the exact version manually). That execution path is going 
> to go through 'package-install' as well, so it must be suitable already.
> 
> - Revert 580d8278c5f48 because it creates odd semantics (upgrading 
> certain packages that are already installed but not others) and it 
> doesn't solve the issue with (use-package 'eglot :ensure t) anyway. We 
> could keep it, but seems like a half-measure that didn't make anyone 
> happy anyway. OTOH, it could minimize the rewrites of CI scripts.

I don't think we are ready to make any new decisions in this matter.
I think we don't even have a comprehensive and detailed picture of the
problems with updating/upgrading built-in packages in Emacs 29.
People are still discovering facts and subtleties of various
package.el commands and features, and are still arguing what exactly
happens in this or that scenario.

So before we discuss solutions, we need a full and detailed
description of the problems to solve.  If someone can do the footwork
of collecting this information and posting it here, please do, and
TIA.

I will say up front that, given what I already read here and in the
related thread on emacs-devel, there seem to be too many
inconsistencies and dark corners in this, in particular when built-in
packages are involved.  We will probably be unable to solve them in
time for Emacs 29.1, certainly not all of them.  So don't raise your
expectations too high.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 11:02 AM Eli Zaretskii <eliz@gnu.org> wrote:

> I will say up front that, given what I already read here and in the
> related thread on emacs-devel, there seem to be too many
> inconsistencies and dark corners in this, in particular when built-in
> packages are involved.  We will probably be unable to solve them in
> time for Emacs 29.1, certainly not all of them.  So don't raise your
> expectations too high.

Yes, I don't have these expectations.  Please in the meantime, allow
me add to eglot.el:

 (defun eglot-update ()  "Update Eglot regardless of package.el policy."
    (interactive)
    (unless package-archive-contents (package-refresh-contents))
    (package-install (cadr (assoc 'eglot package-archive-contents))

This will allow at least Eglot users to weather the storm until Emacs
30.  I know you forbade this in the past, but now that more "facts and
subtleties" have come to light, maybe you can reconsider?  I'll readily
deprecate this function when a solution has been found for the next
Emacs.  It doesn't really affect anyone but Eglot users and these
are, in your own admission, already negatively affected anyway, so
what is there to lose?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 11:31:52 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, jporterbugs@gmail.com, 62720@debbugs.gnu.org, 
> 	philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> 
> On Thu, Apr 20, 2023 at 11:02 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Please in the meantime, allow me add to eglot.el:
> 
>  (defun eglot-update ()  "Update Eglot regardless of package.el policy."
>     (interactive)
>     (unless package-archive-contents (package-refresh-contents))
>     (package-install (cadr (assoc 'eglot package-archive-contents))
> 
> This will allow at least Eglot users to weather the storm until Emacs
> 30.  I know you forbade this in the past, but now that more "facts and
> subtleties" have come to light, maybe you can reconsider?  I'll readily
> deprecate this function when a solution has been found for the next
> Emacs.  It doesn't really affect anyone but Eglot users and these
> are, in your own admission, already negatively affected anyway, so
> what is there to lose?

I didn't forbid it.  I said I'd prefer not to have it in Emacs, and I
still do.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 12:49 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Please in the meantime, allow me add to eglot.el:
> >
> >  (defun eglot-update ()  "Update Eglot regardless of package.el policy."
> >     (interactive)
> >     (unless package-archive-contents (package-refresh-contents))
> >     (package-install (cadr (assoc 'eglot package-archive-contents))
> >
> > This will allow at least Eglot users to weather the storm until Emacs
> > 30.  I know you forbade this in the past, but now that more "facts and
> > subtleties" have come to light, maybe you can reconsider?  I'll readily
> > deprecate this function when a solution has been found for the next
> > Emacs.  It doesn't really affect anyone but Eglot users and these
> > are, in your own admission, already negatively affected anyway, so
> > what is there to lose?
>
> I didn't forbid it.  I said I'd prefer not to have it in Emacs, and I
> still do.


OK.  Thanks for clarifying.  I would _also_ prefer not to.  And
if a good solution is found for Emacs 29, I will take it out or just
deprecate it.  But given this situation, I'm just going to
add it.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 12:53:35 +0100
> Cc: dmitry@gutov.dev, jporterbugs@gmail.com, 62720@debbugs.gnu.org, 
> 	philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> 
> On Thu, Apr 20, 2023 at 12:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I didn't forbid it.  I said I'd prefer not to have it in Emacs, and I
> > still do.
> 
> 
> OK.  Thanks for clarifying.  I would _also_ prefer not to.  And
> if a good solution is found for Emacs 29, I will take it out or just
> deprecate it.  But given this situation, I'm just going to
> add it.

Sigh...

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 13:02, Eli Zaretskii wrote:
>> Date: Thu, 20 Apr 2023 01:06:10 +0300
>> Cc: Eli Zaretskii <eliz@gnu.org>, arne_bab@web.de, jporterbugs@gmail.com,
>>   emacs-devel <emacs-devel@gnu.org>
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> OK then, I think have to re-evaluate my position on this. Previously, I
>> guess, I made some hasty conclusions from how the discussion went on
>> without refreshing the exact details about package.el and use-package
>> (the latter I never knew to begin with). Apologies.
>>
>> Eli, let me know if we should take this back to the bug tracker instead.
> 
> I've moved this back to the bug-tracker.  Please post all further
> replies about this particular issue, i.e. updating of built-in
> packages with package.el, to this bug and not to emacs-devel.

Noted.

>> So I would suggest to focus on functions that don't work as intended.
>> Namely:
>>
>> - Add a user option for the list of builtin packages which would be
>> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x
>> package-upgrade-all' (nee package-update-all). Maybe make it nil by
>> default, or maybe add 'eglot' to it. I don't have a strong opinion.
>>
>> - Fix 'M-x package-upgrade' (nee package-update) to suggest Eglot as one
>> of the options and actually perform the upgrade. That shouldn't require
>> changes to 'package-install' because, as we already know, the user can
>> already install a newer version of Eglot using the 'list-packages' menu
>> (and picking the exact version manually). That execution path is going
>> to go through 'package-install' as well, so it must be suitable already.
>>
>> - Revert 580d8278c5f48 because it creates odd semantics (upgrading
>> certain packages that are already installed but not others) and it
>> doesn't solve the issue with (use-package 'eglot :ensure t) anyway. We
>> could keep it, but seems like a half-measure that didn't make anyone
>> happy anyway. OTOH, it could minimize the rewrites of CI scripts.
> 
> I don't think we are ready to make any new decisions in this matter.
> I think we don't even have a comprehensive and detailed picture of the
> problems with updating/upgrading built-in packages in Emacs 29.
> People are still discovering facts and subtleties of various
> package.el commands and features, and are still arguing what exactly
> happens in this or that scenario.

True.

> So before we discuss solutions, we need a full and detailed
> description of the problems to solve.  If someone can do the footwork
> of collecting this information and posting it here, please do, and
> TIA.

I think I have made a fair attempt at this, though. Here's an update:

- package-upgrade (nee package-update) doesn't upgrade builtin packages 
that never been upgraded before. It's a bug. Hopefully not too hard to fix.

- package-menu-mark-upgrades and package-update-all don't upgrade them 
either. That's not necessarily a bug, but a problem nevertheless. A new 
user option could help.

- Fixing package-update should also obviate the need for eglot-update. 
Though perhaps the latter could still be useful as a single entry point 
to recommend to both users of Emacs 28 and 29+.

- The current fix (commit 580d8278c5f48) is not comprehensive WRT to 
Joao's scenario because use-package-ensure-elpa short-circuits when it 
find that the package is installed ('package-installed-p' returns t). So 
(use-package eglot :ensure t) does not upgrade Eglot even when 
package-install-upgrade-built-in is t.

- package-install-upgrade-built-in is not nuanced: if we suggest the 
users to set it to t, that can result in making _all_ builtin packages 
upgradable with 'package-install'. Whereas I think we originally only 
wanted that for Eglot and maybe for use-package. For this and other 
minor reasons I would suggest reverting 580d8278c5f48. But I suppose we 
could also try to make it more granular (e.g. turn the boolean option 
into a list). I'm not sure it's a good direction overall, however.

- According to Jim P., package pinning doesn't work for builtin packages 
either. Which could be a decent solution as well, e.g. putting something 
like this in the docs: (use-package eglot :ensure t :pin gnu) if the 
users want the Eglot version from ELPA -- and this form might even be 
compatible with Emacs 28. I'm not sure how difficult it would be to fix 
package pinning, however.

I haven't spent as much time on this bug as some others here, though, so 
corrections and additions are welcome.

> I will say up front that, given what I already read here and in the
> related thread on emacs-devel, there seem to be too many
> inconsistencies and dark corners in this, in particular when built-in
> packages are involved.  We will probably be unable to solve them in
> time for Emacs 29.1, certainly not all of them.  So don't raise your
> expectations too high.

The reasons for me to be hopeful are:

- The functions are propose to fix are "leaves": those that are supposed 
to be used interactively, without (many) known callers in-tree. E.g. 
fixing package-update-all shouldn't affect some other part of Emacs as 
much as changing package-install could.

- I imagine the diffs will be rather short. I haven't tried to take a 
stab at this yet, though, and if somebody wants to take the initiative, 
they're very welcome.

The reasons for me to be less hopeful:

- We seem to be unable to come to any agreement here. :-D

Joao's approval for the above list would go a long way.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 2:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> The reasons for me to be less hopeful:
>
> - We seem to be unable to come to any agreement here. :-D
>
> Joao's approval for the above list would go a long way.

First of all, thank you Dmitry for making such an extensive
and detailed summary of the simualtion.  It's really
impressive.

What you say about a fixed package-update obviating the need
for eglot-update is totally true.  In fact I proposed a fixed
package-update much much earlier in this thread.  If you
manage to do that, then I'm more than happy to remove
eglot-update.

What more approval do you want? :-) I mean, I've resigned
myself to a bad situation already (but if you can make
package-install DTRT, I'm of course even happier).  So
package-update or eglot-update -- anything that I can put
on the README .

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 2:56 PM João Távora <joaotavora@gmail.com> wrote:
>
> On Thu, Apr 20, 2023 at 2:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> > The reasons for me to be less hopeful:
> >
> > - We seem to be unable to come to any agreement here. :-D
> >
> > Joao's approval for the above list would go a long way.
>
> First of all, thank you Dmitry for making such an extensive
> and detailed summary of the simualtion.  It's really
> impressive.
>
> What you say about a fixed package-update obviating the need
> for eglot-update is totally true.  In fact I proposed a fixed
> package-update much much earlier in this thread.  If you
> manage to do that, then I'm more than happy to remove
> eglot-update.

Whoops, sorry, made a thinko.  package-update _doesn't_ really
obviate the need for eglot-update, because it won't be callable
in Emacs 26/27/28, and eglot-update will.

...unless package.el itself becomes a :core package and
self-updates through ELPA, which I think is being proposed
around here somewhere.  Could happen in master (and leave
Emacs's 29 version untouched).

So for the moment eglot-update is my best out.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 16:39:20 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > So before we discuss solutions, we need a full and detailed
> > description of the problems to solve.  If someone can do the footwork
> > of collecting this information and posting it here, please do, and
> > TIA.
> 
> I think I have made a fair attempt at this, though. Here's an update:

Thanks.

What I miss in this description is something a bit higher-level: the
list of all the ways/commands/methods people use to install and
upgrade the packages.  We must have this list before our eyes to make
sure we review all the behaviors related to these activities, if we
want them all to eventually behave consistently.

E.g., you don't mention any commands invoked from the package menu.

> - package-upgrade (nee package-update) doesn't upgrade builtin packages 
> that never been upgraded before. It's a bug. Hopefully not too hard to fix.

I'm okay with adding the same prefix argument to package-upgrade,
which would then allow upgrading a built-in package.  IOW, a change
similar to what we did in package-install -- provided that the change
is safe enough to go into Emacs 29.

But note that AFAIU there's a relatively easy workaround for this:
install the later version of the package manually, just once.

> - package-menu-mark-upgrades and package-update-all don't upgrade them 
> either. That's not necessarily a bug, but a problem nevertheless. A new 
> user option could help.

A very relevant question is: can this wait till Emacs 30?

> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to 
> Joao's scenario because use-package-ensure-elpa short-circuits when it 
> find that the package is installed ('package-installed-p' returns t). So 
> (use-package eglot :ensure t) does not upgrade Eglot even when 
> package-install-upgrade-built-in is t.

I don't think (use-package eglot :ensure t) should automatically
upgrade built-in packages.  We could make it do that if
package-install-upgrade-built-in is non-nil -- again, if such a change
could be safe enough.  If not, then the same workaround as for
package-upgrade would do here, I suppose?

> - package-install-upgrade-built-in is not nuanced: if we suggest the 
> users to set it to t, that can result in making _all_ builtin packages 
> upgradable with 'package-install'.

It should be set to t only by users who indeed want all built-in
packages to be updated.  That's why the default is nil.

> Whereas I think we originally only wanted that for Eglot and maybe
> for use-package.

"We" never did want that.  João did, for obvious reasons, but that was
never my intent.  The issue is indeed more general: what should
package-install and package.el in general do with built-in packages
for which a newer version is on ELPA?  We don't yet know the answer to
that, and no hope of obtaining one before Emacs 29.1 is released
(barring any calamities).

> For this and other minor reasons I would suggest reverting
> 580d8278c5f48.

Not going to happen, not unless someone comes up with a better
solution that is much better and still safe enough.  Personally, I
don't believe such a solution exists, since we don't really know the
answer to the above question.

> But I suppose we could also try to make it more granular (e.g. turn
> the boolean option into a list). I'm not sure it's a good direction
> overall, however.

I don't think we should go in this direction, precisely because we
don't know it's a good one.

> - According to Jim P., package pinning doesn't work for builtin packages 
> either. Which could be a decent solution as well, e.g. putting something 
> like this in the docs: (use-package eglot :ensure t :pin gnu) if the 
> users want the Eglot version from ELPA -- and this form might even be 
> compatible with Emacs 28. I'm not sure how difficult it would be to fix 
> package pinning, however.

As long as this is optional behavior (and AFAIU it is), I won't
object, but again, provided that the addition of this will not touch
any other code in unsafe ways.

> The reasons for me to be hopeful are:
> 
> - The functions are propose to fix are "leaves": those that are supposed 
> to be used interactively, without (many) known callers in-tree. E.g. 
> fixing package-update-all shouldn't affect some other part of Emacs as 
> much as changing package-install could.

Experience until now in this thread indicates otherwise: almost every
suggested change touched low-level code with many callers.  It isn't
an accident that arriving at what was finally installed took so many
iterations for such a simple change.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 16:56, João Távora wrote:
> First of all, thank you Dmitry for making such an extensive
> and detailed summary of the simualtion.  It's really
> impressive.
> 
> What you say about a fixed package-update obviating the need
> for eglot-update is totally true.  In fact I proposed a fixed
> package-update much much earlier in this thread.  If you
> manage to do that, then I'm more than happy to remove
> eglot-update.
> 
> What more approval do you want? 😄

An explicit approval always helps anyway. Thanks!

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 17:25, João Távora wrote:
> Whoops, sorry, made a thinko.  package-update_doesn't_  really
> obviate the need for eglot-update, because it won't be callable
> in Emacs 26/27/28, and eglot-update will.
> 
> ...unless package.el itself becomes a :core package and
> self-updates through ELPA, which I think is being proposed
> around here somewhere.  Could happen in master (and leave
> Emacs's 29 version untouched).
> 
> So for the moment eglot-update is my best out.

We could add it and then make a tentative plan (with a TODO comment) to 
remove it as soon as a version of Emacs with package-upgrade that 
supports upgrading builtins becomes the older supported version.

That would be in line with our general policy of developing :core 
packages anyway.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 3:31 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 20/04/2023 17:25, João Távora wrote:
> > Whoops, sorry, made a thinko.  package-update_doesn't_  really
> > obviate the need for eglot-update, because it won't be callable
> > in Emacs 26/27/28, and eglot-update will.
> >
> > ...unless package.el itself becomes a :core package and
> > self-updates through ELPA, which I think is being proposed
> > around here somewhere.  Could happen in master (and leave
> > Emacs's 29 version untouched).
> >
> > So for the moment eglot-update is my best out.
>
> We could add it and then make a tentative plan (with a TODO comment) to
> remove it as soon as a version of Emacs with package-upgrade that
> supports upgrading builtins becomes the older supported version.
>
> That would be in line with our general policy of developing :core
> packages anyway.

Yes, for sure, we'll deprecate it and make it re-route to
package-upgrade.  And what do you think of the idea of package.el
becoming :core itself.  It doesn't seem to have many dependencies.
If that were to happen and the fixed package.el you (or someone else)
is going to eventually propose to Emacs 29 was out in the open,
eglot-update wouldn't be needed.  And we would never have these
discussions under the shadow of the no-more-changes and
the pretest's pressure.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 15:25:00 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, jporterbugs@gmail.com, 62720@debbugs.gnu.org, 
> 	philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> 
> Whoops, sorry, made a thinko.  package-update _doesn't_ really
> obviate the need for eglot-update, because it won't be callable
> in Emacs 26/27/28, and eglot-update will.
> 
> ...unless package.el itself becomes a :core package and
> self-updates through ELPA, which I think is being proposed
> around here somewhere.  Could happen in master (and leave
> Emacs's 29 version untouched).
> 
> So for the moment eglot-update is my best out.

Of course, introducing and advertising eglot-update will mean users of
Emacs 29+ will need to use a different command to upgrade Eglot than
users of previous Emacs versions -- the same problem that you consider
so awful in the change we made in package-update, where these users
will now have to specify a prefix argument (and do it just once,
AFAIU).  That is, of course, a very consistent logic on your part.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

(I haven't had the time to follow the thread in detail, so I might have
missed some context)

João Távora <joaotavora@gmail.com> writes:

> On Thu, Apr 20, 2023 at 2:56 PM João Távora <joaotavora@gmail.com> wrote:
>>
>> On Thu, Apr 20, 2023 at 2:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>>
>> > The reasons for me to be less hopeful:
>> >
>> > - We seem to be unable to come to any agreement here. :-D
>> >
>> > Joao's approval for the above list would go a long way.
>>
>> First of all, thank you Dmitry for making such an extensive
>> and detailed summary of the simualtion.  It's really
>> impressive.
>>
>> What you say about a fixed package-update obviating the need
>> for eglot-update is totally true.  In fact I proposed a fixed
>> package-update much much earlier in this thread.  If you
>> manage to do that, then I'm more than happy to remove
>> eglot-update.
>
> Whoops, sorry, made a thinko.  package-update _doesn't_ really
> obviate the need for eglot-update, because it won't be callable
> in Emacs 26/27/28, and eglot-update will.
>
> ...unless package.el itself becomes a :core package and
> self-updates through ELPA, which I think is being proposed
> around here somewhere.  Could happen in master (and leave
> Emacs's 29 version untouched).

BTW, everything that this would require has been done so this is not a
far-off solution.

> So for the moment eglot-update is my best out.
>
> João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 3:49 PM Eli Zaretskii <eliz@gnu.org> wrote:

> Of course, introducing and advertising eglot-update will mean users of
> Emacs 29+ will need to use a different command to upgrade Eglot than
> users of previous Emacs versions -- the same problem that you consider
> so awful in the change we made in package-update, where these users
> will now have to specify a prefix argument (and do it just once,
> AFAIU).  That is, of course, a very consistent logic on your part.

Sarcasm?  It's my best out.  eglot-update will eventually be available
to users of previous Emacs versions.  And I can precisely control
its behaviour to "always update Eglot, no matter what" (which
package-install didn't do -- BTW you mean package-install I think).
And it will work non-interactively in a config. And the change you're
referring to has been proposed for reversion anyway, because problems.
I don't appreciate eglot-update especially -- I've said that.  But
it's the best out and won't harm anyone.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Thu, 20 Apr 2023 17:25:49 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> "We" never did want that.  João did, for obvious reasons, but that was
    Eli> never my intent.  The issue is indeed more general: what should
    Eli> package-install and package.el in general do with built-in packages
    Eli> for which a newer version is on ELPA?  We don't yet know the answer to
    Eli> that, and no hope of obtaining one before Emacs 29.1 is released
    Eli> (barring any calamities).

"We" may not know that, but by the principle of not giving me things *I*
didnʼt ask for, *I* donʼt want ':core' packages being automatically
upgraded, unless I either

1. explicitly ask for such an upgrade
2. have myself somehow installed a newer version already, in which case
  itʼs no longer a ':core' package

Iʼve not checked, but does whatʼs currently in emacs-29 not give us at
least [1]?

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Thu, 20 Apr 2023 17:25:49 +0300, Eli Zaretskii <eliz@gnu.org> said:
>
>     Eli> "We" never did want that.  João did, for obvious reasons, but that was
>     Eli> never my intent.  The issue is indeed more general: what should
>     Eli> package-install and package.el in general do with built-in packages
>     Eli> for which a newer version is on ELPA?  We don't yet know the answer to
>     Eli> that, and no hope of obtaining one before Emacs 29.1 is released
>     Eli> (barring any calamities).
>
> "We" may not know that, but by the principle of not giving me things *I*
> didnʼt ask for, *I* donʼt want ':core' packages being automatically
> upgraded, unless I either
>
> 1. explicitly ask for such an upgrade
> 2. have myself somehow installed a newer version already, in which case
>   itʼs no longer a ':core' package
>
> Iʼve not checked, but does whatʼs currently in emacs-29 not give us at
> least [1]?

Yes, this is currently provided by C-u M-x package-install.  2. should
(?) have always been the case.

> Robert

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 7:08 PM Robert Pluim <rpluim@gmail.com> wrote:

> "We" may not know that, but by the principle of not giving me things *I*
> didnʼt ask for, *I* donʼt want ':core' packages being automatically
> upgraded, unless I either
>
> 1. explicitly ask for such an upgrade
> 2. have myself somehow installed a newer version already, in which case
>   itʼs no longer a ':core' package
>
> Iʼve not checked, but does whatʼs currently in emacs-29 not give us at
> least [1]?

Not when package dependencies are involved.  If you weren't aware,
in Emacs 26 (including 29), if you explicitly ask to install package A
and it depends on :core package B, which you didn't ask to install,
package B gets upgraded.

In another data point, the last two patches I proposed to package-install
are similar and only extend the "upgrade-even-if-:core" behaviour
to two packages that weren't core are now :core.  Those two packages
are Eglot and Use-Package.  The rationale I used to develop these patches
was to protect users like you (inasmuch as Emacs 28 already protects
you) _and_ to protect Eglot users.  I am of course presuming that
you aren't/weren't also an Eglot user, otherwise you would have been
hit by these package upgrades related to dependencies in Emacs 28
and would have noticed them.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 17:40, João Távora wrote:
> Yes, for sure, we'll deprecate it and make it re-route to
> package-upgrade.  And what do you think of the idea of package.el
> becoming :core itself.  It doesn't seem to have many dependencies.
> If that were to happen and the fixed package.el you (or someone else)
> is going to eventually propose to Emacs 29 was out in the open,
> eglot-update wouldn't be needed.  And we would never have these
> discussions under the shadow of the no-more-changes and
> the pretest's pressure.

I don't know, someone should really test that idea, hard.

It could exhibit the same problem that some package upgrades do: the 
installed packages is not properly reloaded, and the new version is not 
quite usable until Emacs' restart (I recall you saw something like this 
with project.el).

For such a central piece of infrastructure this could be a bigger 
problem by itself, and could even be made worse by the fact that the 
code which installs the new version of package.el also belongs to it, so 
it'll be hotswapped (right?) sometime during its execution.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 17:25, Eli Zaretskii wrote:

> What I miss in this description is something a bit higher-level: the
> list of all the ways/commands/methods people use to install and
> upgrade the packages.  We must have this list before our eyes to make
> sure we review all the behaviors related to these activities, if we
> want them all to eventually behave consistently.

Why list the commands people use to install packages if we're talking 
about upgrading them? Hopefully we'll decide that 'package-install' 
won't upgrade packages because it hasn't done that in the past. Or if 
it's still going to do that because we don't want to revert commits, 
we'll just avoid touching that.

> E.g., you don't mention any commands invoked from the package menu.

The command which people use to upgrade packages from the package menu 
is called package-menu-mark-upgrades (bound to 'U'), mentioned here a 
few times. I also said that I hope it's *the* main way people use to 
upgrade packages.

The others would be package-update and package-update-all (can we please 
do the rename now? I know they've been in for a year, but just as this 
very discussion has shown, most people weren't even aware of their 
existence).

>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
> 
> I'm okay with adding the same prefix argument to package-upgrade,
> which would then allow upgrading a built-in package.  IOW, a change
> similar to what we did in package-install -- provided that the change
> is safe enough to go into Emacs 29.

If we agree it's a bug, why don't we just fix it? 'package-update' is an 
interactive function which itself it called from only one place: 
'package-update-all', and since the plan is to improve both, we can make 
sure they only do what we ask of them: package-update will upgrade 
builtins when invoked directly, and package-update-all will upgrade them 
only when the builtin has been upgraded before (making it not a builtin 
anymore), or a new user option is set.

> But note that AFAIU there's a relatively easy workaround for this:
> install the later version of the package manually, just once.

It's indeed a workaround that exists, and if we were satisfied with it, 
we would have closed this bug as "wontfix". But it's not a very friendly 
way to do that (the user must hunt for the appropriate version in the 
packages menu), and it's very poorly scriptable (you won't be able to 
provide an easy and readable snippet for the user to evaluate or put in 
their init script).

>> - package-menu-mark-upgrades and package-update-all don't upgrade them
>> either. That's not necessarily a bug, but a problem nevertheless. A new
>> user option could help.
> 
> A very relevant question is: can this wait till Emacs 30?

If you ask my opinion, then I already wrote it. The only thing that 
looks potentially complex is fixing the pinning of packages to archives. 
The rest should be fairly trivial.

Should we leave bugfixing and easy upgrading of builtin packages to 
Emacs 30? I suppose we could. Let me know if it's your foregone 
conclusion, then I'll just walk away now. This issue doesn't solve any 
problems for me personally, to be clear.

>> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
>> Joao's scenario because use-package-ensure-elpa short-circuits when it
>> find that the package is installed ('package-installed-p' returns t). So
>> (use-package eglot :ensure t) does not upgrade Eglot even when
>> package-install-upgrade-built-in is t.
> 
> I don't think (use-package eglot :ensure t) should automatically
> upgrade built-in packages.

I don't think it should, either.

But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.

> We could make it do that if
> package-install-upgrade-built-in is non-nil -- again, if such a change
> could be safe enough.  If not, then the same workaround as for
> package-upgrade would do here, I suppose?

What workaround would that be? use-package is not invoked interactively 
-- there is no prefix argument to pass.

>> - package-install-upgrade-built-in is not nuanced: if we suggest the
>> users to set it to t, that can result in making _all_ builtin packages
>> upgradable with 'package-install'.
> 
> It should be set to t only by users who indeed want all built-in
> packages to be updated.  That's why the default is nil.

Sounds unnecessarily dangerous.

>> Whereas I think we originally only wanted that for Eglot and maybe
>> for use-package.
> 
> "We" never did want that.  João did, for obvious reasons, but that was
> never my intent.  The issue is indeed more general: what should
> package-install and package.el in general do with built-in packages
> for which a newer version is on ELPA?

It could continue doing what it's done before: when a package is already 
installed, abort. For upgrading, we should recommend commands with 
"upgrade" in their names.

> We don't yet know the answer to
> that, and no hope of obtaining one before Emacs 29.1 is released
> (barring any calamities).

I'm not sure how we're going to get the answer to that question after 
29.1 is released, too. Similar to that thing with treesit modes and 
auto-mode-alist.

>> For this and other minor reasons I would suggest reverting
>> 580d8278c5f48.
> 
> Not going to happen, not unless someone comes up with a better
> solution that is much better and still safe enough.  Personally, I
> don't believe such a solution exists, since we don't really know the
> answer to the above question.

Could you specify which problem it's currently solving? Some particular 
scenario.

Then I could confidently propose an alternative (or not).

>> But I suppose we could also try to make it more granular (e.g. turn
>> the boolean option into a list). I'm not sure it's a good direction
>> overall, however.
> 
> I don't think we should go in this direction, precisely because we
> don't know it's a good one.

If we made it more granular, its use could become a reasonable option, 
then we'd be able to get some feedback from people who end up using it. 
I don't know what kind of people that will be, though.

>> - According to Jim P., package pinning doesn't work for builtin packages
>> either. Which could be a decent solution as well, e.g. putting something
>> like this in the docs: (use-package eglot :ensure t :pin gnu) if the
>> users want the Eglot version from ELPA -- and this form might even be
>> compatible with Emacs 28. I'm not sure how difficult it would be to fix
>> package pinning, however.
> 
> As long as this is optional behavior (and AFAIU it is), I won't
> object, but again, provided that the addition of this will not touch
> any other code in unsafe ways.

Yes, the use of pinning is and will remain optional. I'm not sure what 
code it will touch and how much, however. There is not a single function 
called "pin that" which needs to be fixed, the fix(es) will be somewhere 
on the common path(s) of package installation and upgrades, I think.

>> The reasons for me to be hopeful are:
>>
>> - The functions are propose to fix are "leaves": those that are supposed
>> to be used interactively, without (many) known callers in-tree. E.g.
>> fixing package-update-all shouldn't affect some other part of Emacs as
>> much as changing package-install could.
> 
> Experience until now in this thread indicates otherwise: almost every
> suggested change touched low-level code with many callers.  It isn't
> an accident that arriving at what was finally installed took so many
> iterations for such a simple change.

You might be correct at that, but note that I'm suggesting a slightly 
different set of goals than what had been discussed here before. So the 
changes will likewise need to be different.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 21 Apr 2023 03:50:52 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/04/2023 17:25, Eli Zaretskii wrote:
> 
> > What I miss in this description is something a bit higher-level: the
> > list of all the ways/commands/methods people use to install and
> > upgrade the packages.  We must have this list before our eyes to make
> > sure we review all the behaviors related to these activities, if we
> > want them all to eventually behave consistently.
> 
> Why list the commands people use to install packages if we're talking 
> about upgrading them?

I said "to install and upgrade", not just "install".

And I do think we should also consider commands that install packages
because they are part of this issue, and in particular, should be
consistent with the upgrade commands.  Also, package-install was the
original cause of this thread.

> Hopefully we'll decide that 'package-install' 
> won't upgrade packages because it hasn't done that in the past.

But it does upgrade non-built-in packages, doesn't it?  And at least
João (and I think others as well) expected it to upgrade Eglot even
though it is now built in.

> > E.g., you don't mention any commands invoked from the package menu.
> 
> The command which people use to upgrade packages from the package menu 
> is called package-menu-mark-upgrades (bound to 'U'), mentioned here a 
> few times. I also said that I hope it's *the* main way people use to 
> upgrade packages.
> 
> The others would be package-update and package-update-all (can we please 
> do the rename now? I know they've been in for a year, but just as this 
> very discussion has shown, most people weren't even aware of their 
> existence).

If the list you provided covers all of the relevant commands and
functions, fine.  Still, for completeness' sake, we should have the
higher-level description as well, as a framework against which to
examine the proposed solutions.  And I'm still not convinced that we
covered all the relevant package-menu commands.

> >> - package-upgrade (nee package-update) doesn't upgrade builtin packages
> >> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
> > 
> > I'm okay with adding the same prefix argument to package-upgrade,
> > which would then allow upgrading a built-in package.  IOW, a change
> > similar to what we did in package-install -- provided that the change
> > is safe enough to go into Emacs 29.
> 
> If we agree it's a bug, why don't we just fix it?

Precisely because, as with package-install, this is a bug for some and
a feature for others, depending on whether people do or don't want the
built-in packages to be upgraded by default.

> 'package-update' is an 
> interactive function which itself it called from only one place: 
> 'package-update-all', and since the plan is to improve both, we can make 
> sure they only do what we ask of them: package-update will upgrade 
> builtins when invoked directly, and package-update-all will upgrade them 
> only when the builtin has been upgraded before (making it not a builtin 
> anymore), or a new user option is set.

This is one possibility, and it might make sense to some users.  But I
don't think we can be sure it will make sense to an overwhelming
majority of the users.

> > But note that AFAIU there's a relatively easy workaround for this:
> > install the later version of the package manually, just once.
> 
> It's indeed a workaround that exists, and if we were satisfied with it, 
> we would have closed this bug as "wontfix". But it's not a very friendly 
> way to do that (the user must hunt for the appropriate version in the 
> packages menu), and it's very poorly scriptable (you won't be able to 
> provide an easy and readable snippet for the user to evaluate or put in 
> their init script).

For Emacs 29, we don't have the luxury of rejecting slightly awkward
workarounds.

> >> - package-menu-mark-upgrades and package-update-all don't upgrade them
> >> either. That's not necessarily a bug, but a problem nevertheless. A new
> >> user option could help.
> > 
> > A very relevant question is: can this wait till Emacs 30?
> 
> If you ask my opinion, then I already wrote it.

I'm asking everyone who reads this.

> Should we leave bugfixing and easy upgrading of builtin packages to 
> Emacs 30? I suppose we could. Let me know if it's your foregone 
> conclusion, then I'll just walk away now.

It isn't a forgone conclusion, because no one has yet shown the code
to fix the issues you mention.  If the code is safe enough and doesn't
risk breaking some reasonable workflows, it could be considered for
Emacs 29.

> >> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
> >> Joao's scenario because use-package-ensure-elpa short-circuits when it
> >> find that the package is installed ('package-installed-p' returns t). So
> >> (use-package eglot :ensure t) does not upgrade Eglot even when
> >> package-install-upgrade-built-in is t.
> > 
> > I don't think (use-package eglot :ensure t) should automatically
> > upgrade built-in packages.
> 
> I don't think it should, either.
> 
> But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.

No, it was supposed to allow the user to invoke package-install in a
way that upgrades built-in packages, something that doesn't happen by
default.  IOW, it was a partial solution to the original problem which
started this discussion, see

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5

> > We could make it do that if
> > package-install-upgrade-built-in is non-nil -- again, if such a change
> > could be safe enough.  If not, then the same workaround as for
> > package-upgrade would do here, I suppose?
> 
> What workaround would that be? use-package is not invoked interactively 
> -- there is no prefix argument to pass.

The workaround is to manually install the package from ELPA, once,
using "C-u M-x package-install RET".

> >> - package-install-upgrade-built-in is not nuanced: if we suggest the
> >> users to set it to t, that can result in making _all_ builtin packages
> >> upgradable with 'package-install'.
> > 
> > It should be set to t only by users who indeed want all built-in
> > packages to be updated.  That's why the default is nil.
> 
> Sounds unnecessarily dangerous.

Which is why it shouldn't be the default.

> >> Whereas I think we originally only wanted that for Eglot and maybe
> >> for use-package.
> > 
> > "We" never did want that.  João did, for obvious reasons, but that was
> > never my intent.  The issue is indeed more general: what should
> > package-install and package.el in general do with built-in packages
> > for which a newer version is on ELPA?
> 
> It could continue doing what it's done before: when a package is already 
> installed, abort. For upgrading, we should recommend commands with 
> "upgrade" in their names.

If people agree with that, I don't think I'll object.  But this is in
a sense a breaking change: package-install will only install, and
thereafter users will need to use package-upgrade.  Some might dislike
such behavior changes.  And we will need to make sure that all the
available methods of "installing" do not "upgrade", for consistency.

> > We don't yet know the answer to
> > that, and no hope of obtaining one before Emacs 29.1 is released
> > (barring any calamities).
> 
> I'm not sure how we're going to get the answer to that question after 
> 29.1 is released, too. Similar to that thing with treesit modes and 
> auto-mode-alist.

Time will tell.  If we have no significant new information and data
points by the time the release of Emacs 30 is on the table, we will
have to use our best judgment at that time.

> >> For this and other minor reasons I would suggest reverting
> >> 580d8278c5f48.
> > 
> > Not going to happen, not unless someone comes up with a better
> > solution that is much better and still safe enough.  Personally, I
> > don't believe such a solution exists, since we don't really know the
> > answer to the above question.
> 
> Could you specify which problem it's currently solving? Some particular 
> scenario.

The scenario which started this bug report, see the message whose URL
I mentioned above.  IOW, we now allow users to explicitly request that
package-install includes built-in packages in the list of candidates,
and will therefore allow to upgrade them.

> >> But I suppose we could also try to make it more granular (e.g. turn
> >> the boolean option into a list). I'm not sure it's a good direction
> >> overall, however.
> > 
> > I don't think we should go in this direction, precisely because we
> > don't know it's a good one.
> 
> If we made it more granular, its use could become a reasonable option, 
> then we'd be able to get some feedback from people who end up using it. 
> I don't know what kind of people that will be, though.

It could be a reasonable option, or it could be a nuisance because of
a gratuitous proliferation of too-granular options.  To make a good
decision, we need data and experience we don't have yet.  So let's not
rush into solutions about which we are uncertain.  We can always make
a boolean option have more values in the future, this kind of change
is usually easy and backward-compatible.

> >> - The functions are propose to fix are "leaves": those that are supposed
> >> to be used interactively, without (many) known callers in-tree. E.g.
> >> fixing package-update-all shouldn't affect some other part of Emacs as
> >> much as changing package-install could.
> > 
> > Experience until now in this thread indicates otherwise: almost every
> > suggested change touched low-level code with many callers.  It isn't
> > an accident that arriving at what was finally installed took so many
> > iterations for such a simple change.
> 
> You might be correct at that, but note that I'm suggesting a slightly 
> different set of goals than what had been discussed here before. So the 
> changes will likewise need to be different.

It's possible.  I said what I said not as an up-front decision, but as
a warning: we should not raise our expectations too high.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 21/04/2023 09:37, Eli Zaretskii wrote:

>> Why list the commands people use to install packages if we're talking
>> about upgrading them?
> 
> I said "to install and upgrade", not just "install".

I listed the upgrade commands.

> And I do think we should also consider commands that install packages
> because they are part of this issue, and in particular, should be
> consistent with the upgrade commands.  Also, package-install was the
> original cause of this thread.

Okay, we also have package-menu-mark-install and package-menu-execute.

>> Hopefully we'll decide that 'package-install'
>> won't upgrade packages because it hasn't done that in the past.
> 
> But it does upgrade non-built-in packages, doesn't it?

Apparently not. I didn't remember whether it does, and I deduced that it 
does just from reading this discussion previously, but it does not.

E.g. just now pressing 'U' after 'M-x list-packages' showed me that I 
have available upgrades for a lot of packages. But still if I evaluate

   (package-install 'sml-mode)

where sml-mode is one of said packages, I just get the message:

   ‘sml-mode’ is already installed

> And at least
> João (and I think others as well) expected it to upgrade Eglot even
> though it is now built in.

I think he wants that because this way (package-install 'eglot) and 
(use-package eglot :ensure t) could match the behavior of Emacs 28 with 
an empty init directory. Backward compatibility and all that.

But I think that's questionable, semantically. Given that Eglot is 
already "installed". Though, of course, one could argue that a bundled 
package is not exactly installed, but then we should change what 
'package-installed-p' does as well. And think hard before doing that.

>> The others would be package-update and package-update-all (can we please
>> do the rename now? I know they've been in for a year, but just as this
>> very discussion has shown, most people weren't even aware of their
>> existence).
> 
> If the list you provided covers all of the relevant commands and
> functions, fine.  Still, for completeness' sake, we should have the
> higher-level description as well, as a framework against which to
> examine the proposed solutions.  And I'm still not convinced that we
> covered all the relevant package-menu commands.

It might be easier if you just ask additional question during review.

>>>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
>>>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
>>>
>>> I'm okay with adding the same prefix argument to package-upgrade,
>>> which would then allow upgrading a built-in package.  IOW, a change
>>> similar to what we did in package-install -- provided that the change
>>> is safe enough to go into Emacs 29.
>>
>> If we agree it's a bug, why don't we just fix it?
> 
> Precisely because, as with package-install, this is a bug for some and
> a feature for others, depending on whether people do or don't want the
> built-in packages to be upgraded by default.

I'm having a hard time imagining someone evaluating (package-upgrade 
'eglot) without intention to upgrade it to the latest version. Or 
invoking it interactively with same argument, expecting a different result.

I don't think anyone will put it straight this way into their init 
script either.

>> 'package-update' is an
>> interactive function which itself it called from only one place:
>> 'package-update-all', and since the plan is to improve both, we can make
>> sure they only do what we ask of them: package-update will upgrade
>> builtins when invoked directly, and package-update-all will upgrade them
>> only when the builtin has been upgraded before (making it not a builtin
>> anymore), or a new user option is set.
> 
> This is one possibility, and it might make sense to some users.  But I
> don't think we can be sure it will make sense to an overwhelming
> majority of the users.

Hence the user option?

But okay, this particular addition, though trivial, we could probably 
postpone until Emacs 30, or even avoid adding at all. It is indeed not 
obvious that people will really need it, although

   (setq package-upgradable-builtins '(eglot use-package))
   (package-upgrade-all)
   ;; or M-x package-upgrade-all
   ;; or 'U' in the list-packages menu

seems like a plausible scenario for a certain kind of user. Because 
package-upgrade does not have a menu entry, or a button anywhere, 
whereas package-upgradable-builtins can be altered from the Customize UI.

>>>> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
>>>> Joao's scenario because use-package-ensure-elpa short-circuits when it
>>>> find that the package is installed ('package-installed-p' returns t). So
>>>> (use-package eglot :ensure t) does not upgrade Eglot even when
>>>> package-install-upgrade-built-in is t.
>>>
>>> I don't think (use-package eglot :ensure t) should automatically
>>> upgrade built-in packages.
>>
>> I don't think it should, either.
>>
>> But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.
> 
> No, it was supposed to allow the user to invoke package-install in a
> way that upgrades built-in packages, something that doesn't happen by
> default.  IOW, it was a partial solution to the original problem which
> started this discussion, see
> 
>    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5

Fair enough.

>>> We could make it do that if
>>> package-install-upgrade-built-in is non-nil -- again, if such a change
>>> could be safe enough.  If not, then the same workaround as for
>>> package-upgrade would do here, I suppose?
>>
>> What workaround would that be? use-package is not invoked interactively
>> -- there is no prefix argument to pass.
> 
> The workaround is to manually install the package from ELPA, once,
> using "C-u M-x package-install RET".

That's not the use-package workflow.

>>>> Whereas I think we originally only wanted that for Eglot and maybe
>>>> for use-package.
>>>
>>> "We" never did want that.  João did, for obvious reasons, but that was
>>> never my intent.  The issue is indeed more general: what should
>>> package-install and package.el in general do with built-in packages
>>> for which a newer version is on ELPA?
>>
>> It could continue doing what it's done before: when a package is already
>> installed, abort. For upgrading, we should recommend commands with
>> "upgrade" in their names.
> 
> If people agree with that, I don't think I'll object.  But this is in
> a sense a breaking change: package-install will only install, and
> thereafter users will need to use package-upgrade.  Some might dislike
> such behavior changes.  And we will need to make sure that all the
> available methods of "installing" do not "upgrade", for consistency.

Yeah, apparently it won't be a breaking change, or a change at all.

>>>> For this and other minor reasons I would suggest reverting
>>>> 580d8278c5f48.
>>>
>>> Not going to happen, not unless someone comes up with a better
>>> solution that is much better and still safe enough.  Personally, I
>>> don't believe such a solution exists, since we don't really know the
>>> answer to the above question.
>>
>> Could you specify which problem it's currently solving? Some particular
>> scenario.
> 
> The scenario which started this bug report, see the message whose URL
> I mentioned above.  IOW, we now allow users to explicitly request that
> package-install includes built-in packages in the list of candidates,
> and will therefore allow to upgrade them.

After we fix 'package-upgrade', users will be able to 'M-x 
package-upgrade RET eglot RET'.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 21 Apr 2023 13:19:39 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 21/04/2023 09:37, Eli Zaretskii wrote:
> 
> >> Why list the commands people use to install packages if we're talking
> >> about upgrading them?
> > 
> > I said "to install and upgrade", not just "install".
> 
> I listed the upgrade commands.

Yes, and I think we should consider both install and upgrade commands.

> >> Hopefully we'll decide that 'package-install'
> >> won't upgrade packages because it hasn't done that in the past.
> > 
> > But it does upgrade non-built-in packages, doesn't it?
> 
> Apparently not. I didn't remember whether it does, and I deduced that it 
> does just from reading this discussion previously, but it does not.
> 
> E.g. just now pressing 'U' after 'M-x list-packages' showed me that I 
> have available upgrades for a lot of packages. But still if I evaluate
> 
>    (package-install 'sml-mode)
> 
> where sml-mode is one of said packages, I just get the message:
> 
>    ‘sml-mode’ is already installed
> 
> > And at least
> > João (and I think others as well) expected it to upgrade Eglot even
> > though it is now built in.
> 
> I think he wants that because this way (package-install 'eglot) and 
> (use-package eglot :ensure t) could match the behavior of Emacs 28 with 
> an empty init directory. Backward compatibility and all that.

But if, with older Emacsen, package-install would refuse to update to
a newer version of Eglot if _some_ older version of Eglot is already
installed, then where's the problem with the default behavior of
package-install? it behaves exactly like in previous versions of
Emacs.  And why is this a problem for users of Eglot, if they couldn't
use package-install more than once for Eglot anyway?

Something is amiss here.

> But I think that's questionable, semantically. Given that Eglot is 
> already "installed". Though, of course, one could argue that a bundled 
> package is not exactly installed, but then we should change what 
> 'package-installed-p' does as well. And think hard before doing that.

I'd question why we have two commands instead of just one, but that's
probably water under the bridge at this point.

> >>>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
> >>>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
> >>>
> >>> I'm okay with adding the same prefix argument to package-upgrade,
> >>> which would then allow upgrading a built-in package.  IOW, a change
> >>> similar to what we did in package-install -- provided that the change
> >>> is safe enough to go into Emacs 29.
> >>
> >> If we agree it's a bug, why don't we just fix it?
> > 
> > Precisely because, as with package-install, this is a bug for some and
> > a feature for others, depending on whether people do or don't want the
> > built-in packages to be upgraded by default.
> 
> I'm having a hard time imagining someone evaluating (package-upgrade 
> 'eglot) without intention to upgrade it to the latest version. Or 
> invoking it interactively with same argument, expecting a different result.

In interactive invocation, package-upgrade calls completing-read with
its 4th argument non-nil, so you cannot select a package which is not
in the collection returned by package--updateable-packages.  What I
meant above is to allow that collection to include built-in packages
as optional behavior.  I just tried invoking package-update for ElDoc,
and I get "No match" after typing "eldoc" to its prompt, although
eldoc version 1.14.0 is in the list presented by list-packages as
"available".

> >> 'package-update' is an
> >> interactive function which itself it called from only one place:
> >> 'package-update-all', and since the plan is to improve both, we can make
> >> sure they only do what we ask of them: package-update will upgrade
> >> builtins when invoked directly, and package-update-all will upgrade them
> >> only when the builtin has been upgraded before (making it not a builtin
> >> anymore), or a new user option is set.
> > 
> > This is one possibility, and it might make sense to some users.  But I
> > don't think we can be sure it will make sense to an overwhelming
> > majority of the users.
> 
> Hence the user option?

Which one?  Are you suggesting to add a new one?  If so, why not use
the one we already added, package-install-upgrade-built-in?

> But okay, this particular addition, though trivial, we could probably 
> postpone until Emacs 30, or even avoid adding at all. It is indeed not 
> obvious that people will really need it, although

If by "this particular addition" you mean to allow package-update to
update built-in packages, then I thought adding that for consistency
with package-install was one of your main bothers?  Or what am I
missing?

>    (setq package-upgradable-builtins '(eglot use-package))
>    (package-upgrade-all)
>    ;; or M-x package-upgrade-all
>    ;; or 'U' in the list-packages menu
> 
> seems like a plausible scenario for a certain kind of user.

Why not treat the fact that some version was already installed from
ELPA as an indication that the user wants this?

> Because 
> package-upgrade does not have a menu entry, or a button anywhere, 
> whereas package-upgradable-builtins can be altered from the Customize UI.

Maybe marking a package in the list for update could be interpreted as
"upgrade that, no questions asked", and we will need no user options?

> >>> We could make it do that if
> >>> package-install-upgrade-built-in is non-nil -- again, if such a change
> >>> could be safe enough.  If not, then the same workaround as for
> >>> package-upgrade would do here, I suppose?
> >>
> >> What workaround would that be? use-package is not invoked interactively
> >> -- there is no prefix argument to pass.
> > 
> > The workaround is to manually install the package from ELPA, once,
> > using "C-u M-x package-install RET".
> 
> That's not the use-package workflow.

The use-package workflow should perhaps get a separate and different
solution.

> >>>> Whereas I think we originally only wanted that for Eglot and maybe
> >>>> for use-package.
> >>>
> >>> "We" never did want that.  João did, for obvious reasons, but that was
> >>> never my intent.  The issue is indeed more general: what should
> >>> package-install and package.el in general do with built-in packages
> >>> for which a newer version is on ELPA?
> >>
> >> It could continue doing what it's done before: when a package is already
> >> installed, abort. For upgrading, we should recommend commands with
> >> "upgrade" in their names.
> > 
> > If people agree with that, I don't think I'll object.  But this is in
> > a sense a breaking change: package-install will only install, and
> > thereafter users will need to use package-upgrade.  Some might dislike
> > such behavior changes.  And we will need to make sure that all the
> > available methods of "installing" do not "upgrade", for consistency.
> 
> Yeah, apparently it won't be a breaking change, or a change at all.

I'm not sure, see above.

Also, when you mark packages for update from the list presented by
list-packages, the menu entry says

  i    Mark for Install

and its help-echo says "Mark a package for installation and move to
the next line", so we already confuse "install" and "upgrade".

> >>>> For this and other minor reasons I would suggest reverting
> >>>> 580d8278c5f48.
> >>>
> >>> Not going to happen, not unless someone comes up with a better
> >>> solution that is much better and still safe enough.  Personally, I
> >>> don't believe such a solution exists, since we don't really know the
> >>> answer to the above question.
> >>
> >> Could you specify which problem it's currently solving? Some particular
> >> scenario.
> > 
> > The scenario which started this bug report, see the message whose URL
> > I mentioned above.  IOW, we now allow users to explicitly request that
> > package-install includes built-in packages in the list of candidates,
> > and will therefore allow to upgrade them.
> 
> After we fix 'package-upgrade', users will be able to 'M-x 
> package-upgrade RET eglot RET'.

This goes back to the issue of having two confusingly-similar but
different commands, as mentioned above.  I guess we should first make
up our minds what, if anything, we want to do about this.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 21/04/2023 14:05, Eli Zaretskii wrote:

>>> And at least
>>> João (and I think others as well) expected it to upgrade Eglot even
>>> though it is now built in.
>>
>> I think he wants that because this way (package-install 'eglot) and
>> (use-package eglot :ensure t) could match the behavior of Emacs 28 with
>> an empty init directory. Backward compatibility and all that.
> 
> But if, with older Emacsen, package-install would refuse to update to
> a newer version of Eglot if _some_ older version of Eglot is already
> installed, then where's the problem with the default behavior of
> package-install? it behaves exactly like in previous versions of
> Emacs.  And why is this a problem for users of Eglot, if they couldn't
> use package-install more than once for Eglot anyway?
> 
> Something is amiss here.

The above scenarios (package-install or (use-package eglot :ensure t)) 
when ~/.emacs.d/elpa is empty, result in the very latest Eglot being 
installed when using Emacs 28. And will result in something different 
with Emacs 29 (not the latest version). That's not nothing: the CI 
scripts will have to be updated, and the user instructions for reporting 
problems will have to be made more complicated as well. Some 
possibilities will simply be gone (the user won't be able to upgrade 
Eglot to the very latest by deleting it from ~/.emacs.d/elpa and calling 
package-install, for example). This *is* a non-backward compatible 
change from the perspective of Eglot's maintainer.

I, personally, don't really buy this kind of argument, but I figured you 
might. After all, it's rather in line with reasoning we've seen voiced 
around these parts many times ("X has worked this way for Y years, let's 
never change it from now on"). Classic https://xkcd.com/1172/.

>> But I think that's questionable, semantically. Given that Eglot is
>> already "installed". Though, of course, one could argue that a bundled
>> package is not exactly installed, but then we should change what
>> 'package-installed-p' does as well. And think hard before doing that.
> 
> I'd question why we have two commands instead of just one, but that's
> probably water under the bridge at this point.

Either way would be fine, IMO, as long as the behavior is logical and 
matches documentation. But having a separate command to upgrade now lets 
us fix it separately without worry of breaking something more globally.

>>>>>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
>>>>>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
>>>>>
>>>>> I'm okay with adding the same prefix argument to package-upgrade,
>>>>> which would then allow upgrading a built-in package.  IOW, a change
>>>>> similar to what we did in package-install -- provided that the change
>>>>> is safe enough to go into Emacs 29.
>>>>
>>>> If we agree it's a bug, why don't we just fix it?
>>>
>>> Precisely because, as with package-install, this is a bug for some and
>>> a feature for others, depending on whether people do or don't want the
>>> built-in packages to be upgraded by default.
>>
>> I'm having a hard time imagining someone evaluating (package-upgrade
>> 'eglot) without intention to upgrade it to the latest version. Or
>> invoking it interactively with same argument, expecting a different result.
> 
> In interactive invocation, package-upgrade calls completing-read with
> its 4th argument non-nil, so you cannot select a package which is not
> in the collection returned by package--updateable-packages.  What I
> meant above is to allow that collection to include built-in packages
> as optional behavior.  I just tried invoking package-update for ElDoc,
> and I get "No match" after typing "eldoc" to its prompt, although
> eldoc version 1.14.0 is in the list presented by list-packages as
> "available".

That's what I imagined: adding a new optional argument to 
package--updateable-packages which would include builtins in the result.

And only pass it when called from package-upgrade.

Hopefully that's the kind of optional that you meant.

>>>> 'package-update' is an
>>>> interactive function which itself it called from only one place:
>>>> 'package-update-all', and since the plan is to improve both, we can make
>>>> sure they only do what we ask of them: package-update will upgrade
>>>> builtins when invoked directly, and package-update-all will upgrade them
>>>> only when the builtin has been upgraded before (making it not a builtin
>>>> anymore), or a new user option is set.
>>>
>>> This is one possibility, and it might make sense to some users.  But I
>>> don't think we can be sure it will make sense to an overwhelming
>>> majority of the users.
>>
>> Hence the user option?
> 
> Which one?  Are you suggesting to add a new one?  If so, why not use
> the one we already added, package-install-upgrade-built-in?

The user option I was thinking of would probably be called a little 
shorter: package-upgrade-built-in. And it would only affect the upgrader 
commands.

I'm not sure tying that optional behavior to the new optional behavior 
of package-install is a good idea (one upgrades, another installs). But 
I'm also not sure keeping the latter around is a good idea, as you know.

>> But okay, this particular addition, though trivial, we could probably
>> postpone until Emacs 30, or even avoid adding at all. It is indeed not
>> obvious that people will really need it, although
> 
> If by "this particular addition" you mean to allow package-update to
> update built-in packages, then I thought adding that for consistency
> with package-install was one of your main bothers?  Or what am I
> missing?

That particular addition would be the option package-upgrade-built-in 
proposed above. One that would affect package-upgrade-all and 
package-menu-mark-upgrades only.

So the plan minimum is to fix package-upgrade.

>>     (setq package-upgradable-builtins '(eglot use-package))
>>     (package-upgrade-all)
>>     ;; or M-x package-upgrade-all
>>     ;; or 'U' in the list-packages menu
>>
>> seems like a plausible scenario for a certain kind of user.
> 
> Why not treat the fact that some version was already installed from
> ELPA as an indication that the user wants this?

That's how it works now, indeed. And we _could_ leave it at that.

>> Because
>> package-upgrade does not have a menu entry, or a button anywhere,
>> whereas package-upgradable-builtins can be altered from the Customize UI.
> 
> Maybe marking a package in the list for update could be interpreted as
> "upgrade that, no questions asked", and we will need no user options?

There is no handy "upgrade that" binding in the packages menu. The only 
command that's available there related to upgrading is 
package-menu-mark-upgrades, which does that to all packages (except 
builtins).

To manually execute an upgrade of one package, one needs to both mark 
the new version for installation (after first scrolling down the list to 
find it), and mark the current version for deletion. This is what 
currently an upgrade consists of. If they just do the first step, they 
end up with two installed versions, one of them "obsolete". package.el 
will hopefully handle this fine during activation after restart 
(ignoring the older one), but it's not a great configuration to leave 
the user in.

All this is to say, the first step (upgrading Eglot to the version from 
ELPA) will be less user-friendly compared to the other UIs we have. But 
it's probably manageable, especially if documented well.

>>>>> We could make it do that if
>>>>> package-install-upgrade-built-in is non-nil -- again, if such a change
>>>>> could be safe enough.  If not, then the same workaround as for
>>>>> package-upgrade would do here, I suppose?
>>>>
>>>> What workaround would that be? use-package is not invoked interactively
>>>> -- there is no prefix argument to pass.
>>>
>>> The workaround is to manually install the package from ELPA, once,
>>> using "C-u M-x package-install RET".
>>
>> That's not the use-package workflow.
> 
> The use-package workflow should perhaps get a separate and different
> solution.

Possibly (I suppose people are welcome to submit related proposals -- 
the only reasonable one I heard is related to pinning, see previous msgs).

My point here, however, is that commit 580d8278c5f48 improved the 
situation to a lesser degree that some people here might have expected.

>>>> It could continue doing what it's done before: when a package is already
>>>> installed, abort. For upgrading, we should recommend commands with
>>>> "upgrade" in their names.
>>>
>>> If people agree with that, I don't think I'll object.  But this is in
>>> a sense a breaking change: package-install will only install, and
>>> thereafter users will need to use package-upgrade.  Some might dislike
>>> such behavior changes.  And we will need to make sure that all the
>>> available methods of "installing" do not "upgrade", for consistency.
>>
>> Yeah, apparently it won't be a breaking change, or a change at all.
> 
> I'm not sure, see above.
> 
> Also, when you mark packages for update from the list presented by
> list-packages, the menu entry says
> 
>    i    Mark for Install
> 
> and its help-echo says "Mark a package for installation and move to
> the next line", so we already confuse "install" and "upgrade".

But that's the thing: we don't have a separate action in the packages 
list to "upgrade" a single package. The bindings that you see in the 
menu are "Mark for [i]nstall" and "Mark for [d]eletion". The first marks 
a specific (usually newer) version of a package for installation without 
deleting an existing one. The second marks a version for deletion. 
Neither means "upgrade".

The fact that having multiple versions installed will (probably) result 
in the newest one being used after a restart is great, of course, but 
that's still not a proper upgrade.

But we can conclude that we do have two different installation actions:

  - package-install which will install the latest version of a package, 
but only if it's not installed;
  - package-menu-mark-install, which will mark a specific version for 
installation, disregarding whether some earlier version is already 
installed; the previous version will remain installed still.

>>>>>> For this and other minor reasons I would suggest reverting
>>>>>> 580d8278c5f48.
>>>>>
>>>>> Not going to happen, not unless someone comes up with a better
>>>>> solution that is much better and still safe enough.  Personally, I
>>>>> don't believe such a solution exists, since we don't really know the
>>>>> answer to the above question.
>>>>
>>>> Could you specify which problem it's currently solving? Some particular
>>>> scenario.
>>>
>>> The scenario which started this bug report, see the message whose URL
>>> I mentioned above.  IOW, we now allow users to explicitly request that
>>> package-install includes built-in packages in the list of candidates,
>>> and will therefore allow to upgrade them.
>>
>> After we fix 'package-upgrade', users will be able to 'M-x
>> package-upgrade RET eglot RET'.
> 
> This goes back to the issue of having two confusingly-similar but
> different commands, as mentioned above.  I guess we should first make
> up our minds what, if anything, we want to do about this.

Sure.

But even if we decide that we want to eliminate that split, doing *that* 
would really be a breaking change. I don't have a reasonable plan to 
present for doing that in Emacs 29, so far.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 02:12, Dmitry Gutov wrote:
>> In interactive invocation, package-upgrade calls completing-read with
>> its 4th argument non-nil, so you cannot select a package which is not
>> in the collection returned by package--updateable-packages.  What I
>> meant above is to allow that collection to include built-in packages
>> as optional behavior.  I just tried invoking package-update for ElDoc,
>> and I get "No match" after typing "eldoc" to its prompt, although
>> eldoc version 1.14.0 is in the list presented by list-packages as
>> "available".
> 
> That's what I imagined: adding a new optional argument to 
> package--updateable-packages which would include builtins in the result.
> 
> And only pass it when called from package-upgrade.
> 
> Hopefully that's the kind of optional that you meant.

Here's a patch which does that. The diff could be reduced (the 
package-update part) by binding the new option 
(package-install-upgrade-built-in), but I figured it's better to avoid 
interdependency while we're still deciding what to keep.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ffa6272dd1f..1f0a47f6b6a 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2270,17 +2270,21 @@ package-update
    "Update package NAME if a newer version exists."
    (interactive
     (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: " (package--updateable-packages t) nil t)))
    (let* ((package (if (symbolp name)
                        name
                      (intern name)))
           (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
+    (if (and pkg-desc (package-vc-p pkg-desc))
          (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
-
-(defun package--updateable-packages ()
+      (when pkg-desc
+        (package-delete pkg-desc 'force))
+      (package-install-from-archive
+       (car (last (seq-sort-by #'package-desc-priority-version
+                               #'version-list-<
+                               (cdr (assq package 
package-archive-contents)))))))))
+
+(defun package--updateable-packages (&optional allow-builtins)
    ;; Initialize the package system to get the list of package
    ;; symbols for completion.
    (package--archives-initialize)
@@ -2291,11 +2295,21 @@ package--updateable-packages
        (or (let ((available
                   (assq (car elt) package-archive-contents)))
              (and available
-                 (version-list-<
-                  (package-desc-version (cadr elt))
-                  (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+                 (or (and
+                      allow-builtins
+                      (not (package-desc-version (cadr elt))))
+                     (version-list-<
+                      (package-desc-version (cadr elt))
+                      (package-desc-version (cadr available))))))
+          (package-vc-p (cadr elt))))
+    (if allow-builtins
+        (append package-alist
+                (mapcan
+                 (lambda (elt)
+                   (when (not (assq (car elt) package-alist))
+                     (list (list (car elt) (package--from-builtin elt)))))
+                 package--builtins))
+      package-alist))))

  ;;;###autoload
  (defun package-update-all (&optional query)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 22, 2023 at 12:12 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> I, personally, don't really buy this kind of argument, but I figured you
> might. After all, it's rather in line with reasoning we've seen voiced
> around these parts many times ("X has worked this way for Y years, let's
> never change it from now on"). Classic https://xkcd.com/1172/.

The CI + user instructions having to be updated is unfortunate, but I can
change those (and M-x eglot-update is a simple way to give predictable
consistent semantics for "bringing Eglot to latest").

Then there are the things I can't change, like users trying out
a new init.el file, a very common operation -- be it for bug
reproduction or just to try something out.  Also note that
package-delete + package-install interactively is a  pretty good way
to update packages in Emacs 28: no need to nuke the home directory.

In general, it's hard to predict the damage of hard-to-explain different
behaviour of these forms depending on whether you're on 28 or 29.
Or, the way this seems to be going, 30.

If 29 ships with this bug, it's going to linger for a good while,
and get worse as time passes.

If you "dont buy this", that's OK :-) I'm not trying to sell it to you
specifically.  I don't think we'll have torches and pitchforks either,
but we're not really talking about "spacebar overheating" here.

Note that I also don't really buy, personally, the "furtive update of
:core packages" argument either.  My reasoning is that that cat has
been out of the bag for a long long, just because of dependencies which
are absolutely liberally by package.el and the fact tha I've never seen
a bug reported about this.

I buy even less that people using M-x package-update in Emacs 29
don't want to update :core packages just because it doesn't do
that right now.  For me, it's obvious people weren't using to
update :core packages because that command lived in master
for the large part of its short life and in master :core packages
don't need any updating, by definition. But of course fact that I
don't buy it  shouldn't mean that it should be disregarded (and
noone is advocating for that).

The last two patches I provided aim to aid Eglot users (lightly,
as you've discovered, since the use-package use case is not well
covered in the first one, and there's still the error behaviour
to ponde).  But, more importantly they are designed so that no
cats that weren't previously out of the bag make it outside
the bag

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 02:12:23 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 21/04/2023 14:05, Eli Zaretskii wrote:
> 
> >>> And at least
> >>> João (and I think others as well) expected it to upgrade Eglot even
> >>> though it is now built in.
> >>
> >> I think he wants that because this way (package-install 'eglot) and
> >> (use-package eglot :ensure t) could match the behavior of Emacs 28 with
> >> an empty init directory. Backward compatibility and all that.
> > 
> > But if, with older Emacsen, package-install would refuse to update to
> > a newer version of Eglot if _some_ older version of Eglot is already
> > installed, then where's the problem with the default behavior of
> > package-install? it behaves exactly like in previous versions of
> > Emacs.  And why is this a problem for users of Eglot, if they couldn't
> > use package-install more than once for Eglot anyway?
> > 
> > Something is amiss here.
> 
> The above scenarios (package-install or (use-package eglot :ensure t)) 
> when ~/.emacs.d/elpa is empty, result in the very latest Eglot being 
> installed when using Emacs 28. And will result in something different 
> with Emacs 29 (not the latest version). That's not nothing: the CI 
> scripts will have to be updated, and the user instructions for reporting 
> problems will have to be made more complicated as well. Some 
> possibilities will simply be gone (the user won't be able to upgrade 
> Eglot to the very latest by deleting it from ~/.emacs.d/elpa and calling 
> package-install, for example). This *is* a non-backward compatible 
> change from the perspective of Eglot's maintainer.

We were talking about users installing and updating packages.  The CI
scenario doesn't belong here.  It is also much less important one
(test suites are always required to chase the changes in development).

Let's not complicate an already complicated set of issues by bringing
up unimportant secondary use cases.

> I, personally, don't really buy this kind of argument, but I figured you 
> might. After all, it's rather in line with reasoning we've seen voiced 
> around these parts many times ("X has worked this way for Y years, let's 
> never change it from now on"). Classic https://xkcd.com/1172/.

If you allude to my reasoning, then it is never that simplistic, and
always considers each case separately, not "by analogy".

> >> But I think that's questionable, semantically. Given that Eglot is
> >> already "installed". Though, of course, one could argue that a bundled
> >> package is not exactly installed, but then we should change what
> >> 'package-installed-p' does as well. And think hard before doing that.
> > 
> > I'd question why we have two commands instead of just one, but that's
> > probably water under the bridge at this point.
> 
> Either way would be fine, IMO, as long as the behavior is logical and 
> matches documentation. But having a separate command to upgrade now lets 
> us fix it separately without worry of breaking something more globally.

Except that, based on what we have (see below) ,we don't really have
an "upgrade" operation, we only have "install" and "delete"
(i.e. "uninstall").  So maybe we should preserve that, to minimize
problems and user surprise/confusion.

> > In interactive invocation, package-upgrade calls completing-read with
> > its 4th argument non-nil, so you cannot select a package which is not
> > in the collection returned by package--updateable-packages.  What I
> > meant above is to allow that collection to include built-in packages
> > as optional behavior.  I just tried invoking package-update for ElDoc,
> > and I get "No match" after typing "eldoc" to its prompt, although
> > eldoc version 1.14.0 is in the list presented by list-packages as
> > "available".
> 
> That's what I imagined: adding a new optional argument to 
> package--updateable-packages which would include builtins in the result.
> 
> And only pass it when called from package-upgrade.
> 
> Hopefully that's the kind of optional that you meant.

Yes, something like that.  Presumably activated by the same new option
introduced for package-installed.

> >>>> 'package-update' is an
> >>>> interactive function which itself it called from only one place:
> >>>> 'package-update-all', and since the plan is to improve both, we can make
> >>>> sure they only do what we ask of them: package-update will upgrade
> >>>> builtins when invoked directly, and package-update-all will upgrade them
> >>>> only when the builtin has been upgraded before (making it not a builtin
> >>>> anymore), or a new user option is set.
> >>>
> >>> This is one possibility, and it might make sense to some users.  But I
> >>> don't think we can be sure it will make sense to an overwhelming
> >>> majority of the users.
> >>
> >> Hence the user option?
> > 
> > Which one?  Are you suggesting to add a new one?  If so, why not use
> > the one we already added, package-install-upgrade-built-in?
> 
> The user option I was thinking of would probably be called a little 
> shorter: package-upgrade-built-in. And it would only affect the upgrader 
> commands.

We could rename the existing option, if the name is the problem.
Otherwise, I don't see why we would need two separate options: they do
the same job and have the same meaning from the user's POV.

> >> Because
> >> package-upgrade does not have a menu entry, or a button anywhere,
> >> whereas package-upgradable-builtins can be altered from the Customize UI.
> > 
> > Maybe marking a package in the list for update could be interpreted as
> > "upgrade that, no questions asked", and we will need no user options?
> 
> There is no handy "upgrade that" binding in the packages menu. The only 
> command that's available there related to upgrading is 
> package-menu-mark-upgrades, which does that to all packages (except 
> builtins).
> 
> To manually execute an upgrade of one package, one needs to both mark 
> the new version for installation (after first scrolling down the list to 
> find it), and mark the current version for deletion. This is what 
> currently an upgrade consists of.

Yes.  So users could manually mark an new version for installation,
and if needed mark the old version for deletion, thus indicating that
they want this upgrade.  No separate option to indicate the same is
needed.

> All this is to say, the first step (upgrading Eglot to the version from 
> ELPA) will be less user-friendly compared to the other UIs we have. But 
> it's probably manageable, especially if documented well.

I don't see why it would be less user-friendly.

> My point here, however, is that commit 580d8278c5f48 improved the 
> situation to a lesser degree that some people here might have expected.

One again: commit 580d8278c5f48 solved precisely the problem which
opened this bug report, nothing more, nothing less.  The rest are late
additions, which were not on the table when the original bug was
discussed.  So the above complaint is at least unfair.

> >>> If people agree with that, I don't think I'll object.  But this is in
> >>> a sense a breaking change: package-install will only install, and
> >>> thereafter users will need to use package-upgrade.  Some might dislike
> >>> such behavior changes.  And we will need to make sure that all the
> >>> available methods of "installing" do not "upgrade", for consistency.
> >>
> >> Yeah, apparently it won't be a breaking change, or a change at all.
> > 
> > I'm not sure, see above.
> > 
> > Also, when you mark packages for update from the list presented by
> > list-packages, the menu entry says
> > 
> >    i    Mark for Install
> > 
> > and its help-echo says "Mark a package for installation and move to
> > the next line", so we already confuse "install" and "upgrade".
> 
> But that's the thing: we don't have a separate action in the packages 
> list to "upgrade" a single package.

We agree.  I'm saying that your suggestion to make package-upgrade a
more important command is by itself a breaking change: there's no
"upgrade" operation, per se, in the current code.

> But we can conclude that we do have two different installation actions:
> 
>   - package-install which will install the latest version of a package, 
> but only if it's not installed;
>   - package-menu-mark-install, which will mark a specific version for 
> installation, disregarding whether some earlier version is already 
> installed; the previous version will remain installed still.

Which is again a breaking behavior change, AFAIU.  Is this a good idea
so late in the development of Emacs 29?

> >>>> Could you specify which problem it's currently solving? Some particular
> >>>> scenario.
> >>>
> >>> The scenario which started this bug report, see the message whose URL
> >>> I mentioned above.  IOW, we now allow users to explicitly request that
> >>> package-install includes built-in packages in the list of candidates,
> >>> and will therefore allow to upgrade them.
> >>
> >> After we fix 'package-upgrade', users will be able to 'M-x
> >> package-upgrade RET eglot RET'.
> > 
> > This goes back to the issue of having two confusingly-similar but
> > different commands, as mentioned above.  I guess we should first make
> > up our minds what, if anything, we want to do about this.
> 
> Sure.
> 
> But even if we decide that we want to eliminate that split, doing *that* 
> would really be a breaking change. I don't have a reasonable plan to 
> present for doing that in Emacs 29, so far.

There's no "split".  What I wanted to point out is that we don't seem
to have a clear vision of these two commands, since they are confusing
intertwined.  In fact, one could argue that package-upgrade in its
current form is simply a convenience shortcut for "delete old and
install new".

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 03:57:03 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> > That's what I imagined: adding a new optional argument to 
> > package--updateable-packages which would include builtins in the result.
> > 
> > And only pass it when called from package-upgrade.
> > 
> > Hopefully that's the kind of optional that you meant.
> 
> Here's a patch which does that. The diff could be reduced (the 
> package-update part) by binding the new option 
> (package-install-upgrade-built-in), but I figured it's better to avoid 
> interdependency while we're still deciding what to keep.

Thanks, but this is not what was being discussed, AFAIU.  What I said
I'd agree to is to have package-update accept a prefix argument and
heed package-install-upgrade-built-in (perhaps renamed), and only then
update built-in packages.

I also don't think I like the significant changes in package-update,
nor understand why they are needed.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 11:33, Eli Zaretskii wrote:
>> Date: Sat, 22 Apr 2023 03:57:03 +0300
>> From: Dmitry Gutov<dmitry@gutov.dev>
>> Cc:jporterbugs@gmail.com,philipk@posteo.net,62720@debbugs.gnu.org,
>>   joaotavora@gmail.com,larsi@gnus.org,monnier@iro.umontreal.ca
>>
>>> That's what I imagined: adding a new optional argument to
>>> package--updateable-packages which would include builtins in the result.
>>>
>>> And only pass it when called from package-upgrade.
>>>
>>> Hopefully that's the kind of optional that you meant.
>> Here's a patch which does that. The diff could be reduced (the
>> package-update part) by binding the new option
>> (package-install-upgrade-built-in), but I figured it's better to avoid
>> interdependency while we're still deciding what to keep.
> Thanks, but this is not what was being discussed, AFAIU.  What I said
> I'd agree to is to have package-update accept a prefix argument and
> heed package-install-upgrade-built-in (perhaps renamed),

I think I explained in the previous email why reusing 
package-install-upgrade-built-in doesn't seem like a good idea.

> and only then
> update built-in packages.

I asked what plausible scenario you think might be broken by having 
package-update upgrade builtin package by default.

Do you want to answer that?

> I also don't think I like the significant changes in package-update,
> nor understand why they are needed.

Like I said: the changes are to avoid relying on package-install being 
able to install a package that's already installed. Which currently 
works only for builtins and when only a user option is set. It's a mess.

And to "avoid interdependency".

Just to be clear, we are talking about the 4 lines at the end, right?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 11:26, Eli Zaretskii wrote:

>> The above scenarios (package-install or (use-package eglot :ensure t))
>> when ~/.emacs.d/elpa is empty, result in the very latest Eglot being
>> installed when using Emacs 28. And will result in something different
>> with Emacs 29 (not the latest version). That's not nothing: the CI
>> scripts will have to be updated, and the user instructions for reporting
>> problems will have to be made more complicated as well. Some
>> possibilities will simply be gone (the user won't be able to upgrade
>> Eglot to the very latest by deleting it from ~/.emacs.d/elpa and calling
>> package-install, for example). This *is* a non-backward compatible
>> change from the perspective of Eglot's maintainer.
> 
> We were talking about users installing and updating packages.  The CI
> scenario doesn't belong here.  It is also much less important one
> (test suites are always required to chase the changes in development).
> 
> Let's not complicate an already complicated set of issues by bringing
> up unimportant secondary use cases.

You asked. I relayed what's already been said on the matter by Joao.

>> I, personally, don't really buy this kind of argument, but I figured you
>> might. After all, it's rather in line with reasoning we've seen voiced
>> around these parts many times ("X has worked this way for Y years, let's
>> never change it from now on"). Classic https://xkcd.com/1172/.
> 
> If you allude to my reasoning, then it is never that simplistic, and
> always considers each case separately, not "by analogy".

Of course that's a simplification. Hence the "might" anyway.

>> Either way would be fine, IMO, as long as the behavior is logical and
>> matches documentation. But having a separate command to upgrade now lets
>> us fix it separately without worry of breaking something more globally.
> 
> Except that, based on what we have (see below) ,we don't really have
> an "upgrade" operation, we only have "install" and "delete"
> (i.e. "uninstall").  So maybe we should preserve that, to minimize
> problems and user surprise/confusion.

We do. We have commands for upgrading, both in "list-packages", and used 
interactively. Which do the thing of installing the new version and 
removing the old one.

Which is what upgrading means in various tools, e.g. 'apt'.

>>> In interactive invocation, package-upgrade calls completing-read with
>>> its 4th argument non-nil, so you cannot select a package which is not
>>> in the collection returned by package--updateable-packages.  What I
>>> meant above is to allow that collection to include built-in packages
>>> as optional behavior.  I just tried invoking package-update for ElDoc,
>>> and I get "No match" after typing "eldoc" to its prompt, although
>>> eldoc version 1.14.0 is in the list presented by list-packages as
>>> "available".
>>
>> That's what I imagined: adding a new optional argument to
>> package--updateable-packages which would include builtins in the result.
>>
>> And only pass it when called from package-upgrade.
>>
>> Hopefully that's the kind of optional that you meant.
> 
> Yes, something like that.  Presumably activated by the same new option
> introduced for package-installed.

Okay, then it's something different, and you didn't answer my question.

>>>>>> 'package-update' is an
>>>>>> interactive function which itself it called from only one place:
>>>>>> 'package-update-all', and since the plan is to improve both, we can make
>>>>>> sure they only do what we ask of them: package-update will upgrade
>>>>>> builtins when invoked directly, and package-update-all will upgrade them
>>>>>> only when the builtin has been upgraded before (making it not a builtin
>>>>>> anymore), or a new user option is set.
>>>>>
>>>>> This is one possibility, and it might make sense to some users.  But I
>>>>> don't think we can be sure it will make sense to an overwhelming
>>>>> majority of the users.
>>>>
>>>> Hence the user option?
>>>
>>> Which one?  Are you suggesting to add a new one?  If so, why not use
>>> the one we already added, package-install-upgrade-built-in?
>>
>> The user option I was thinking of would probably be called a little
>> shorter: package-upgrade-built-in. And it would only affect the upgrader
>> commands.
> 
> We could rename the existing option, if the name is the problem.
> Otherwise, I don't see why we would need two separate options: they do
> the same job and have the same meaning from the user's POV.

The name is a problem, yes. What could also be a problem is a user that 
customizes this option to have package-update update builtin packages (a 
reasonable behavior that should be on by default anyway), will also 
automatically have change the behavior of package-install to be more 
surprising (install an already installed package).

Further, if we have a user option affect package-update, we'll have to 
alter package-update-all and package-manu-mark-for-update in the same 
patch (otherwise we'll have more nonsense on our hands). Whereas the 
first version I sent is more minimal.

>> All this is to say, the first step (upgrading Eglot to the version from
>> ELPA) will be less user-friendly compared to the other UIs we have. But
>> it's probably manageable, especially if documented well.
> 
> I don't see why it would be less user-friendly.

The same reason we do have commands with "upgrade" in their names, 
rather than force everybody to use the "install" and "delete" ones.

>> My point here, however, is that commit 580d8278c5f48 improved the
>> situation to a lesser degree that some people here might have expected.
> 
> One again: commit 580d8278c5f48 solved precisely the problem which
> opened this bug report, nothing more, nothing less.

It doesn't seem like the originator of the report agrees with that.

>> But we can conclude that we do have two different installation actions:
>>
>>    - package-install which will install the latest version of a package,
>> but only if it's not installed;
>>    - package-menu-mark-install, which will mark a specific version for
>> installation, disregarding whether some earlier version is already
>> installed; the previous version will remain installed still.
> 
> Which is again a breaking behavior change, AFAIU.  Is this a good idea
> so late in the development of Emacs 29?

The above is not a breaking change, it's how things work already. And 
have been working for quite a while.

>> But even if we decide that we want to eliminate that split, doing *that*
>> would really be a breaking change. I don't have a reasonable plan to
>> present for doing that in Emacs 29, so far.
> 
> There's no "split".  What I wanted to point out is that we don't seem
> to have a clear vision of these two commands, since they are confusing
> intertwined.  In fact, one could argue that package-upgrade in its
> current form is simply a convenience shortcut for "delete old and
> install new".

What should an upgrade command do, in your opinion, if not "delete old 
and install new"?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 13:30:41 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > Thanks, but this is not what was being discussed, AFAIU.  What I said
> > I'd agree to is to have package-update accept a prefix argument and
> > heed package-install-upgrade-built-in (perhaps renamed),
> 
> I think I explained in the previous email why reusing 
> package-install-upgrade-built-in doesn't seem like a good idea.

And I thought I've explained why I didn't see a need for another
option.

> > and only then
> > update built-in packages.
> 
> I asked what plausible scenario you think might be broken by having 
> package-update upgrade builtin package by default.

That's obvious: this is how package-update behaved until now.

> > I also don't think I like the significant changes in package-update,
> > nor understand why they are needed.
> 
> Like I said: the changes are to avoid relying on package-install being 
> able to install a package that's already installed. Which currently 
> works only for builtins and when only a user option is set. It's a mess.
> 
> And to "avoid interdependency".

Why does this have to be in Emacs 29?  It's a cleanup, right?

> Just to be clear, we are talking about the 4 lines at the end, right?

Yes, and also the (somewhat mysterious) additions of tests for
pkg-desc.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 13:48:41 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> The user option I was thinking of would probably be called a little
> >> shorter: package-upgrade-built-in. And it would only affect the upgrader
> >> commands.
> > 
> > We could rename the existing option, if the name is the problem.
> > Otherwise, I don't see why we would need two separate options: they do
> > the same job and have the same meaning from the user's POV.
> 
> The name is a problem, yes. What could also be a problem is a user that 
> customizes this option to have package-update update builtin packages (a 
> reasonable behavior that should be on by default anyway), will also 
> automatically have change the behavior of package-install to be more 
> surprising (install an already installed package).

It's the same change in behavior, since for built-in packages
"install" and "upgrade" is the same.

> Further, if we have a user option affect package-update, we'll have to 
> alter package-update-all and package-manu-mark-for-update in the same 
> patch (otherwise we'll have more nonsense on our hands). Whereas the 
> first version I sent is more minimal.

How is this relevant to whether we need one or two separate user
options?

> >> All this is to say, the first step (upgrading Eglot to the version from
> >> ELPA) will be less user-friendly compared to the other UIs we have. But
> >> it's probably manageable, especially if documented well.
> > 
> > I don't see why it would be less user-friendly.
> 
> The same reason we do have commands with "upgrade" in their names, 
> rather than force everybody to use the "install" and "delete" ones.

I still don't think this is less user-friendly.

> > One again: commit 580d8278c5f48 solved precisely the problem which
> > opened this bug report, nothing more, nothing less.
> 
> It doesn't seem like the originator of the report agrees with that.

I'm aware of that.  But you are talking to me, not to him, and the
above is my opinion.  I also agree that the solution is not ideal,
just the best I could agree to.

> >>    - package-install which will install the latest version of a package,
> >> but only if it's not installed;
> >>    - package-menu-mark-install, which will mark a specific version for
> >> installation, disregarding whether some earlier version is already
> >> installed; the previous version will remain installed still.
> > 
> > Which is again a breaking behavior change, AFAIU.  Is this a good idea
> > so late in the development of Emacs 29?
> 
> The above is not a breaking change, it's how things work already. And 
> have been working for quite a while.

That's not what I understand.  E.g., package-install will install even
if the package is already installed.

But if this already works, then why are you bringing this up?

> >> But even if we decide that we want to eliminate that split, doing *that*
> >> would really be a breaking change. I don't have a reasonable plan to
> >> present for doing that in Emacs 29, so far.
> > 
> > There's no "split".  What I wanted to point out is that we don't seem
> > to have a clear vision of these two commands, since they are confusing
> > intertwined.  In fact, one could argue that package-upgrade in its
> > current form is simply a convenience shortcut for "delete old and
> > install new".
> 
> What should an upgrade command do, in your opinion, if not "delete old 
> and install new"?

Why is this question important, in the context of the current
discussion?  It's a tangent.  All I wanted to point out is that IMO
there's no "split" that we want to eliminate.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 14:11, Eli Zaretskii wrote:
>> Date: Sat, 22 Apr 2023 13:30:41 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> Thanks, but this is not what was being discussed, AFAIU.  What I said
>>> I'd agree to is to have package-update accept a prefix argument and
>>> heed package-install-upgrade-built-in (perhaps renamed),
>>
>> I think I explained in the previous email why reusing
>> package-install-upgrade-built-in doesn't seem like a good idea.
> 
> And I thought I've explained why I didn't see a need for another
> option.

I also don't see the point of using an option here.

Also think forward to Emacs 30: I think the most reasonable choice would 
be to have package-update upgrade builtins by default, whereas 
package-update-all and package-menu-mark-for-upgrades probably still 
need to be preffed off (not sure, but we won't be able to make the 
choice until later, I think).

But if to make package-update behave properly we need to flip the 
default of the said option, it will flip the behavior of 
package-update-all and package-menu-mark-for-upgrades as well.

>>> and only then
>>> update built-in packages.
>>
>> I asked what plausible scenario you think might be broken by having
>> package-update upgrade builtin package by default.
> 
> That's obvious: this is how package-update behaved until now.

That's not an answer to the question.

>>> I also don't think I like the significant changes in package-update,
>>> nor understand why they are needed.
>>
>> Like I said: the changes are to avoid relying on package-install being
>> able to install a package that's already installed. Which currently
>> works only for builtins and when only a user option is set. It's a mess.
>>
>> And to "avoid interdependency".
> 
> Why does this have to be in Emacs 29?  It's a cleanup, right?

Not a cleanup, no. If I just keep the previous version of the code, I 
get "package xxx is already installed". Because when upgrading a builtin 
package, the "current" version is not deleted.

So we need to compute the exact version to install (then package-install 
does not say "it's already installed" because the installed version is 
different). The use of package-install-from-archive might have been a 
mistake, though, (in case dependencies need to be updated too) I'm 
looking into that now.

Alternatively, we could add an optional argument to package-install 
which would mean "install the latest version anyway".

>> Just to be clear, we are talking about the 4 lines at the end, right?
> 
> Yes, and also the (somewhat mysterious) additions of tests for
> pkg-desc.

pkg-desc is nil for builtin packages in this case (they are not in 
package-alist, so (assq package package-alist) returns nil).

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

On 22/04/2023 14:24, Dmitry Gutov wrote:
>>>> I also don't think I like the significant changes in package-update,
>>>> nor understand why they are needed.
>>>
>>> Like I said: the changes are to avoid relying on package-install being
>>> able to install a package that's already installed. Which currently
>>> works only for builtins and when only a user option is set. It's a mess.
>>>
>>> And to "avoid interdependency".
>>
>> Why does this have to be in Emacs 29?  It's a cleanup, right?
> 
> Not a cleanup, no. If I just keep the previous version of the code, I 
> get "package xxx is already installed". Because when upgrading a builtin 
> package, the "current" version is not deleted.
> 
> So we need to compute the exact version to install (then package-install 
> does not say "it's already installed" because the installed version is 
> different). The use of package-install-from-archive might have been a 
> mistake, though, (in case dependencies need to be updated too) I'm 
> looking into that now.

Here's an updated patch that's a little closer to what's been there before.

> Alternatively, we could add an optional argument to package-install 
> which would mean "install the latest version anyway".

[-- Attachment #2: package-update-fix.diff --]
[-- Type: text/x-patch, Size: 2476 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ffa6272dd1f..f9ad83d7650 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2270,17 +2270,22 @@ package-update
   "Update package NAME if a newer version exists."
   (interactive
    (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: " (package--updateable-packages t) nil t)))
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
          (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
+    (if (and pkg-desc (package-vc-p pkg-desc))
         (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
-
-(defun package--updateable-packages ()
+      (when pkg-desc
+        (package-delete pkg-desc 'force))
+      (package-install
+       (car (last (seq-sort-by #'package-desc-priority-version
+                               #'version-list-<
+                               (cdr (assq package package-archive-contents)))))
+       'dont-select))))
+
+(defun package--updateable-packages (&optional allow-builtins)
   ;; Initialize the package system to get the list of package
   ;; symbols for completion.
   (package--archives-initialize)
@@ -2291,11 +2296,21 @@ package--updateable-packages
       (or (let ((available
                  (assq (car elt) package-archive-contents)))
             (and available
-                 (version-list-<
-                  (package-desc-version (cadr elt))
-                  (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+                 (or (and
+                      allow-builtins
+                      (not (package-desc-version (cadr elt))))
+                     (version-list-<
+                      (package-desc-version (cadr elt))
+                      (package-desc-version (cadr available))))))
+          (package-vc-p (cadr elt))))
+    (if allow-builtins
+        (append package-alist
+                (mapcan
+                 (lambda (elt)
+                   (when (not (assq (car elt) package-alist))
+                     (list (list (car elt) (package--from-builtin elt)))))
+                 package--builtins))
+      package-alist))))
 
 ;;;###autoload
 (defun package-update-all (&optional query)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 03:57, João Távora wrote:

> Then there are the things I can't change, like users trying out
> a new init.el file, a very common operation -- be it for bug
> reproduction or just to try something out.  Also note that
> package-delete + package-install interactively is a  pretty good way
> to update packages in Emacs 28: no need to nuke the home directory.

It's as good a workaround for ignorance as any, but if we think it's 
okay for a user to do that (a relatively large manual operation), 
perhaps we can guide them to do a 'package-install' from list-packages 
instead.

But the difference from Emacs 28 is indeed unfortunate, no argument there.

Or if the new option stays around, I guess you could be recommending 
they customize it first?

> If you "dont buy this", that's OK :-) I'm not trying to sell it to you
> specifically.  I don't think we'll have torches and pitchforks either,
> but we're not really talking about "spacebar overheating" here.
> 
> Note that I also don't really buy, personally, the "furtive update of
> :core packages" argument either.  My reasoning is that that cat has
> been out of the bag for a long long, just because of dependencies which
> are absolutely liberally by package.el and the fact tha I've never seen
> a bug reported about this.

I don't have a strong opinion one way or the other, but this part seems 
to be worth treating conservatively, I think. This close to release, etc.

We might as well switch to the "update all the cores!" model later, but 
hopefully by that time we have a good CI setup that tests the 
compatibility of all core packages with all Emacs versions they are 
supposed to support.

> I buy even less that people using M-x package-update in Emacs 29
> don't want to update :core packages just because it doesn't do
> that right now.  For me, it's obvious people weren't using to
> update :core packages because that command lived in master
> for the large part of its short life and in master :core packages
> don't need any updating, by definition. But of course fact that I
> don't buy it  shouldn't mean that it should be disregarded (and
> noone is advocating for that).

This one I'm trying to fix, currently.

> The last two patches I provided aim to aid Eglot users (lightly,
> as you've discovered, since the use-package use case is not well
> covered in the first one, and there's still the error behaviour
> to ponde).  But, more importantly they are designed so that no
> cats that weren't previously out of the bag make it outside
> the bag

Do you prefer that package-install-upgrade-built-in (together with the 
behavior of package-install it enables) stays around?

One possible alternative I mentioned, is package-install could grow 
another programmatic parameter which would mean "install the very latest 
version". The users will then need to evaluate (package-install 'eglot 
nil t) for that to happen. Also a breaking change in instructions, 
though, since in Emacs 28 this will error.

Another gotcha with the latter, is that if some version from ELPA is 
already installed, they will get two versions installed as a result.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 14:24:47 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 22/04/2023 14:11, Eli Zaretskii wrote:
> >> Date: Sat, 22 Apr 2023 13:30:41 +0300
> >> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
> >>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >>
> >>> Thanks, but this is not what was being discussed, AFAIU.  What I said
> >>> I'd agree to is to have package-update accept a prefix argument and
> >>> heed package-install-upgrade-built-in (perhaps renamed),
> >>
> >> I think I explained in the previous email why reusing
> >> package-install-upgrade-built-in doesn't seem like a good idea.
> > 
> > And I thought I've explained why I didn't see a need for another
> > option.
> 
> I also don't see the point of using an option here.

We must not change past behavior unconditionally and by default, not
this close to the release.

> Also think forward to Emacs 30: I think the most reasonable choice would 
> be to have package-update upgrade builtins by default, whereas 
> package-update-all and package-menu-mark-for-upgrades probably still 
> need to be preffed off (not sure, but we won't be able to make the 
> choice until later, I think).

I don't see why package-update and package-update-all should behave
differently wrt core packages.  If the user expresses his/her will to
update core packages, then package-update-all should do this for all
of them.

> But if to make package-update behave properly we need to flip the 
> default of the said option, it will flip the behavior of 
> package-update-all and package-menu-mark-for-upgrades as well.

Which is how it should be, IMO.

> >>> and only then
> >>> update built-in packages.
> >>
> >> I asked what plausible scenario you think might be broken by having
> >> package-update upgrade builtin package by default.
> > 
> > That's obvious: this is how package-update behaved until now.
> 
> That's not an answer to the question.

It is for me (and I'm quite surprised that it is not for you).

> >>> I also don't think I like the significant changes in package-update,
> >>> nor understand why they are needed.
> >>
> >> Like I said: the changes are to avoid relying on package-install being
> >> able to install a package that's already installed. Which currently
> >> works only for builtins and when only a user option is set. It's a mess.
> >>
> >> And to "avoid interdependency".
> > 
> > Why does this have to be in Emacs 29?  It's a cleanup, right?
> 
> Not a cleanup, no. If I just keep the previous version of the code, I 
> get "package xxx is already installed". Because when upgrading a builtin 
> package, the "current" version is not deleted.
> 
> So we need to compute the exact version to install (then package-install 
> does not say "it's already installed" because the installed version is 
> different). The use of package-install-from-archive might have been a 
> mistake, though, (in case dependencies need to be updated too) I'm 
> looking into that now.

With prefix argument, or when package-update-built-in is non-nil, the
behavior of package-install is different.  So just use this, instead
of changing the code of package-update.

> Alternatively, we could add an optional argument to package-install 
> which would mean "install the latest version anyway".

There is already such an option, added as part of fixing this bug.

> >> Just to be clear, we are talking about the 4 lines at the end, right?
> > 
> > Yes, and also the (somewhat mysterious) additions of tests for
> > pkg-desc.
> 
> pkg-desc is nil for builtin packages in this case (they are not in 
> package-alist, so (assq package package-alist) returns nil).

This should either be commented, or a variable with a telltale name
added to reflect this subtlety.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 14:29:46 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> Here's an updated patch that's a little closer to what's been there before.

I suggest not to rush with posting patches until we agree about what
the modified code should do.  I would like to minimize the waste of
your time.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 22, 2023 at 12:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> But the difference from Emacs 28 is indeed unfortunate, no argument there.
>
> Or if the new option stays around, I guess you could be recommending
> they customize it first?

No idea yet. I'm leaning eglot-update as it guaranteedly the most
consistent.  That or straight.el or elpaca.el.  Or no recommendation
whatsoever.

I just got my first bug report about a user where I can't tell
the version of Eglot being used but which is consistent with an
older version being used, perhaps unknowingly. This is, of course,
not unheard of,  so let's see.

> We might as well switch to the "update all the cores!" model later, but
> hopefully by that time we have a good CI setup that tests the
> compatibility of all core packages with all Emacs versions they are
> supposed to support.

Right, that makes sense.  If it helps, Eglot's CI has been doing that
for year now.  It tests Eglot -- and its up-to-date dependencies --
in Emacs 26, 27 and 28.  It's not very complex to set up.

> Do you prefer that package-install-upgrade-built-in (together with the
> behavior of package-install it enables) stays around?

No, my opinion is that I find this user unfriendly, and of course
even more so with the default value.  I've already answered this
early on in the thread.

Creating customization points as a "cure" for haphazard design is
not a good practice.  It's like there are two package managers
in Emacs. The package menu and the package-install.  The
"upgrade :core" cat has always been out of the bag in the
first one fully, and in the second almost fully.  And then
there's the third "package manager", i.e. use-package's specific
package-install behaviour.   And nwo there is package-update which
is almost a newborn, but is already inexplicably broken. Putting
more customization variables into the mix is just adding variations
to a very crowded field, more questions to ask, more
fear-uncertainty-doubt.

The "update all cores" is frankly the safest model IMO.  Of course,
I understand that others disagree -- and I even see their well.
So knowing emacs-devel, I don't have any illusions.

But I suspect they see this "update all cores" idea as some kind of
gargantuan jump from what is in Emacs 28, when in reality it's not
(again, because dependencies).  And I also tend to think they
disagree because they haven't been working with this and with users
for as long as I have (and you have), so from afar it seems dangerous.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 15:00, Eli Zaretskii wrote:
>> Also think forward to Emacs 30: I think the most reasonable choice would
>> be to have package-update upgrade builtins by default, whereas
>> package-update-all and package-menu-mark-for-upgrades probably still
>> need to be preffed off (not sure, but we won't be able to make the
>> choice until later, I think).
> I don't see why package-update and package-update-all should behave
> differently wrt core packages.  If the user expresses his/her will to
> update core packages, then package-update-all should do this for all
> of them.

The idea is that if the user invokes 'M-x package-update' interactively 
and inputs the name of the package, they express their intention to 
update said package this way. Whether it's core or not.

With package-update-all or package-menu-mark-for-update, the user does 
not have a chance to specify which of the core packages they want to 
upgrade, if any.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 15:14:01 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 22/04/2023 15:00, Eli Zaretskii wrote:
> >> Also think forward to Emacs 30: I think the most reasonable choice would
> >> be to have package-update upgrade builtins by default, whereas
> >> package-update-all and package-menu-mark-for-upgrades probably still
> >> need to be preffed off (not sure, but we won't be able to make the
> >> choice until later, I think).
> > I don't see why package-update and package-update-all should behave
> > differently wrt core packages.  If the user expresses his/her will to
> > update core packages, then package-update-all should do this for all
> > of them.
> 
> The idea is that if the user invokes 'M-x package-update' interactively 
> and inputs the name of the package, they express their intention to 
> update said package this way. Whether it's core or not.
> 
> With package-update-all or package-menu-mark-for-update, the user does 
> not have a chance to specify which of the core packages they want to 
> upgrade, if any.

That's why we should have both the prefix-arg and the user option: one
is for one-off update of a single package, the other for updating many
of them.  It's the same logic as for package-install in those cases.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

Philip,

On 13/04/2023 21:49, Philip Kaludercic wrote:
> +    (when (and (or current-prefix-arg package-install-upgrade-built-in)
> +               (package--active-built-in-p pkg))
> +      (setq pkg (cadr (assq name package-archive-contents))))

How sure are you that the first element in (cdr (assq name 
package-archive-contents)) will contain the latest version?

When reading the code I stumbled on these two items:

- Comment in package-compute-transaction:

                 ;; pkg-descs is sorted by priority, not version, so
                 ;; don't error just yet.

- package-menu--find-upgrades iterates through installed packages and 
checks for new versions with this condition:

              (version-list-< (package-desc-priority-version pkg-desc)
                              (package-desc-priority-version avail-pkg))

That's why I came with the more complex way to choose the appropriate 
package-desc in my latest attempt:

  (car
   (last (seq-sort-by #'package-desc-priority-version
                      #'version-list-<
                      (cdr (assq package package-archive-contents)))))

Would be happy to hear it's an overkill and we can use the more simple 
version.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2245 bytes --]

On 22/04/2023 15:00, Eli Zaretskii wrote:

>>>> I think I explained in the previous email why reusing
>>>> package-install-upgrade-built-in doesn't seem like a good idea.
>>>
>>> And I thought I've explained why I didn't see a need for another
>>> option.
>>
>> I also don't see the point of using an option here.
> 
> We must not change past behavior unconditionally and by default, not
> this close to the release.

We're still allowed bugfixing, I believe.

>>>>> and only then
>>>>> update built-in packages.
>>>>
>>>> I asked what plausible scenario you think might be broken by having
>>>> package-update upgrade builtin package by default.
>>>
>>> That's obvious: this is how package-update behaved until now.
>>
>> That's not an answer to the question.
> 
> It is for me (and I'm quite surprised that it is not for you).

Not sure why you're surprised, you know my approach toward backward 
compatibility. Never a fan of enshrining problems in amber.

But in this case it's also a function that's never been in a released 
Emacs, so the formal conditions are lacking as well.

And it's okay to use the time since the code was added as a rough proxy 
for stability, but when it's pretty clear (just from the comments in 
this very thread) that most people never noticed or forgot that it's 
there. So it's obviously not very well tested.

Just from reading it code and testing, I see another bug: it removes the 
updated package from package-selected-packages because it doesn't pass 
NOSAVE to package-delete. Meaning, 'M-x package-autoremove' at any time 
later will delete it.

>> Alternatively, we could add an optional argument to package-install
>> which would mean "install the latest version anyway".
> 
> There is already such an option, added as part of fixing this bug.

Ok, since you insist. See attached.

>>>> Just to be clear, we are talking about the 4 lines at the end, right?
>>>
>>> Yes, and also the (somewhat mysterious) additions of tests for
>>> pkg-desc.
>>
>> pkg-desc is nil for builtin packages in this case (they are not in
>> package-alist, so (assq package package-alist) returns nil).
> 
> This should either be commented, or a variable with a telltale name
> added to reflect this subtlety.

Not a problem.

[-- Attachment #2: package-update-fix.diff --]
[-- Type: text/x-patch, Size: 2661 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ffa6272dd1f..2c8eec998c1 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2270,17 +2270,23 @@ package-update
   "Update package NAME if a newer version exists."
   (interactive
    (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: " (package--updateable-packages t) nil t)))
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
+         (pkg-desc (cadr (assq package package-alist)))
+         (package-install-upgrade-built-in (not pkg-desc)))
+    ;; `pkg-desc' will be nil when the package is an "active built-in".
+    (if (and pkg-desc (package-vc-p pkg-desc))
         (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
-
-(defun package--updateable-packages ()
+      (when pkg-desc
+        (package-delete pkg-desc 'force 'nosave))
+      (package-install package
+                       ;; An active built-in has never been "selected"
+                       ;; before.  Mark it as installed explicitly.
+                       (and pkg-desc 'dont-select)))))
+
+(defun package--updateable-packages (&optional allow-builtins)
   ;; Initialize the package system to get the list of package
   ;; symbols for completion.
   (package--archives-initialize)
@@ -2291,11 +2297,21 @@ package--updateable-packages
       (or (let ((available
                  (assq (car elt) package-archive-contents)))
             (and available
-                 (version-list-<
-                  (package-desc-version (cadr elt))
-                  (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+                 (or (and
+                      allow-builtins
+                      (not (package-desc-version (cadr elt))))
+                     (version-list-<
+                      (package-desc-version (cadr elt))
+                      (package-desc-version (cadr available))))))
+          (package-vc-p (cadr elt))))
+    (if allow-builtins
+        (append package-alist
+                (mapcan
+                 (lambda (elt)
+                   (when (not (assq (car elt) package-alist))
+                     (list (list (car elt) (package--from-builtin elt)))))
+                 package--builtins))
+      package-alist))))
 
 ;;;###autoload
 (defun package-update-all (&optional query)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sun, 23 Apr 2023 02:46:05 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >>> And I thought I've explained why I didn't see a need for another
> >>> option.
> >>
> >> I also don't see the point of using an option here.
> > 
> > We must not change past behavior unconditionally and by default, not
> > this close to the release.
> 
> We're still allowed bugfixing, I believe.

Not when fixing it might introduce another bug or break someone's
workflow.

> >>>>> and only then
> >>>>> update built-in packages.
> >>>>
> >>>> I asked what plausible scenario you think might be broken by having
> >>>> package-update upgrade builtin package by default.
> >>>
> >>> That's obvious: this is how package-update behaved until now.
> >>
> >> That's not an answer to the question.
> > 
> > It is for me (and I'm quite surprised that it is not for you).
> 
> Not sure why you're surprised, you know my approach toward backward 
> compatibility. Never a fan of enshrining problems in amber.
> 
> But in this case it's also a function that's never been in a released 
> Emacs, so the formal conditions are lacking as well.
> 
> And it's okay to use the time since the code was added as a rough proxy 
> for stability, but when it's pretty clear (just from the comments in 
> this very thread) that most people never noticed or forgot that it's 
> there. So it's obviously not very well tested.

What is obvious to you is not obvious to me.

> Just from reading it code and testing, I see another bug: it removes the 
> updated package from package-selected-packages because it doesn't pass 
> NOSAVE to package-delete. Meaning, 'M-x package-autoremove' at any time 
> later will delete it.

If there are bugs, we need to fix them.

Philip, any comments regarding this particular issue with
package-autoremove after package-update?

> >> Alternatively, we could add an optional argument to package-install
> >> which would mean "install the latest version anyway".
> > 
> > There is already such an option, added as part of fixing this bug.
> 
> Ok, since you insist. See attached.

I don't understand how this patch is supposed to be any progress in
this discussion.  I see no prefix argument handling in package-update
and no change in behavior when package-install-upgrade-built-in is
non-nil.  So why did you think this brings the code closer to what I
suggested?

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 09:39, Eli Zaretskii wrote:
> I don't understand how this patch is supposed to be any progress in
> this discussion.  I see no prefix argument handling in package-update
> and no change in behavior when package-install-upgrade-built-in is
> non-nil.  So why did you think this brings the code closer to what I
> suggested?

It addressed two last points from your previous email, obviously.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Philip,
>
> On 13/04/2023 21:49, Philip Kaludercic wrote:
>> +    (when (and (or current-prefix-arg package-install-upgrade-built-in)
>> +               (package--active-built-in-p pkg))
>> +      (setq pkg (cadr (assq name package-archive-contents))))
>
> How sure are you that the first element in (cdr (assq name
> package-archive-contents)) will contain the latest version?

This is not certain, but the same approach is used in other places in
package.el, so I just stocked to it.

> When reading the code I stumbled on these two items:
>
> - Comment in package-compute-transaction:
>
>                 ;; pkg-descs is sorted by priority, not version, so
>                 ;; don't error just yet.
>
> - package-menu--find-upgrades iterates through installed packages and
>   checks for new versions with this condition:
>
>              (version-list-< (package-desc-priority-version pkg-desc)
>                              (package-desc-priority-version avail-pkg))
>
> That's why I came with the more complex way to choose the appropriate
> package-desc in my latest attempt:
>
>  (car
>   (last (seq-sort-by #'package-desc-priority-version
>                      #'version-list-<
>                      (cdr (assq package package-archive-contents)))))

If we insist on package.el installing the newest version, then this
would make sense.  AFAIK there is no guarantees on the order of packages
in `package-archive-contents'.  This shouldn't be an issue for Eglot,
but might be one for use-package.  I would actually pull it out into a
separate utility function that we would re-use in other places.

> Would be happy to hear it's an overkill and we can use the more simple
> version.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sun, 23 Apr 2023 14:58:33 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 23/04/2023 09:39, Eli Zaretskii wrote:
> > I don't understand how this patch is supposed to be any progress in
> > this discussion.  I see no prefix argument handling in package-update
> > and no change in behavior when package-install-upgrade-built-in is
> > non-nil.  So why did you think this brings the code closer to what I
> > suggested?
> 
> It addressed two last points from your previous email, obviously.

Which points are those?  Please help me identify them.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Just from reading it code and testing, I see another bug: it removes the 
>> updated package from package-selected-packages because it doesn't pass 
>> NOSAVE to package-delete. Meaning, 'M-x package-autoremove' at any time 
>> later will delete it.
>
> If there are bugs, we need to fix them.
>
> Philip, any comments regarding this particular issue with
> package-autoremove after package-update?

If this is an issue, I think it would be nice if it could be reported in
a separate issue to ease tracking, so that we can solve it in time for
Emacs 29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 16:05, Philip Kaludercic wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>>> Just from reading it code and testing, I see another bug: it removes the
>>> updated package from package-selected-packages because it doesn't pass
>>> NOSAVE to package-delete. Meaning, 'M-x package-autoremove' at any time
>>> later will delete it.
>> If there are bugs, we need to fix them.
>>
>> Philip, any comments regarding this particular issue with
>> package-autoremove after package-update?
> If this is an issue, I think it would be nice if it could be reported in
> a separate issue to ease tracking, so that we can solve it in time for
> Emacs 29.

I've included the fix in the patch.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 16:02, Eli Zaretskii wrote:
>> Date: Sun, 23 Apr 2023 14:58:33 +0300
>> Cc:jporterbugs@gmail.com,philipk@posteo.net,62720@debbugs.gnu.org,
>>   joaotavora@gmail.com,larsi@gnus.org,monnier@iro.umontreal.ca
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 23/04/2023 09:39, Eli Zaretskii wrote:
>>> I don't understand how this patch is supposed to be any progress in
>>> this discussion.  I see no prefix argument handling in package-update
>>> and no change in behavior when package-install-upgrade-built-in is
>>> non-nil.  So why did you think this brings the code closer to what I
>>> suggested?
>> It addressed two last points from your previous email, obviously.
> Which points are those?  Please help me identify them.

These two:

 > There is already such an option, added as part of fixing this bug.

The binding was added, so now we straight away delegate to package-install.

 > This should either be commented, or a variable with a telltale name
added to reflect this subtlety.

A comment was added as well.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sun, 23 Apr 2023 16:11:44 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> It addressed two last points from your previous email, obviously.
> > Which points are those?  Please help me identify them.
> 
> These two:
> 
>  > There is already such an option, added as part of fixing this bug.
> 
> The binding was added, so now we straight away delegate to package-install.

I meant to give the user the control on whether package-update will
update built-in packages, like we did with package-install: either via
prefix argument or by customizing the user option.

>  > This should either be commented, or a variable with a telltale name
> added to reflect this subtlety.
> 
> A comment was added as well.

This one I did see, thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 16:02, Philip Kaludercic wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> Philip,
>>
>> On 13/04/2023 21:49, Philip Kaludercic wrote:
>>> +    (when (and (or current-prefix-arg package-install-upgrade-built-in)
>>> +               (package--active-built-in-p pkg))
>>> +      (setq pkg (cadr (assq name package-archive-contents))))
>>
>> How sure are you that the first element in (cdr (assq name
>> package-archive-contents)) will contain the latest version?
> 
> This is not certain, but the same approach is used in other places in
> package.el, so I just stocked to it.

Perhaps they conceal latent bugs as well. I don't know the codebase too 
well, but the places I did examine used the comparison together with the 
priority.

>> That's why I came with the more complex way to choose the appropriate
>> package-desc in my latest attempt:
>>
>>   (car
>>    (last (seq-sort-by #'package-desc-priority-version
>>                       #'version-list-<
>>                       (cdr (assq package package-archive-contents)))))
> 
> If we insist on package.el installing the newest version, then this
> would make sense.

What's the alternative? Upgrading to "some version"?

> AFAIK there is no guarantees on the order of packages
> in `package-archive-contents'.  This shouldn't be an issue for Eglot,
> but might be one for use-package.

Ah, it seems they removed it from Melpa, that shouldn't surprise me.

Eglot is in the GNU-devel archive as well, though. So there is some 
potential for conflict.

 > I would actually pull it out into a
 > separate utility function that we would re-use in other places.

Since I had to drop the respective code from the latest version of the 
patch, I guess you could put it in package-install instead.

A reusable helper is fine, of course, but what are the other places we 
would use it at?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 17:24, Eli Zaretskii wrote:
>> Date: Sun, 23 Apr 2023 16:11:44 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> It addressed two last points from your previous email, obviously.
>>> Which points are those?  Please help me identify them.
>>
>> These two:
>>
>>   > There is already such an option, added as part of fixing this bug.
>>
>> The binding was added, so now we straight away delegate to package-install.
> 
> I meant to give the user the control on whether package-update will
> update built-in packages, like we did with package-install: either via
> prefix argument or by customizing the user option.

That would be a different change, though.

So what are we guarding against here? That the user will choose a 
built-in package to upgrade by accident?

And we'll show her an error, saying "use the prefix argument"?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Thu, 20 Apr 2023 19:53:16 +0100, João Távora <joaotavora@gmail.com> said:

    João> On Thu, Apr 20, 2023 at 7:08 PM Robert Pluim <rpluim@gmail.com> wrote:
    >> "We" may not know that, but by the principle of not giving me things *I*
    >> didnʼt ask for, *I* donʼt want ':core' packages being automatically
    >> upgraded, unless I either
    >> 
    >> 1. explicitly ask for such an upgrade
    >> 2. have myself somehow installed a newer version already, in which case
    >> itʼs no longer a ':core' package
    >> 
    >> Iʼve not checked, but does whatʼs currently in emacs-29 not give us at
    >> least [1]?

    João> Not when package dependencies are involved.  If you weren't aware,
    João> in Emacs 26 (including 29), if you explicitly ask to install package A
    João> and it depends on :core package B, which you didn't ask to install,
    João> package B gets upgraded.

I think thatʼs OK as a behaviour

    João> In another data point, the last two patches I proposed to package-install
    João> are similar and only extend the "upgrade-even-if-:core" behaviour
    João> to two packages that weren't core are now :core.  Those two packages
    João> are Eglot and Use-Package.  The rationale I used to develop these patches
    João> was to protect users like you (inasmuch as Emacs 28 already protects
    João> you) _and_ to protect Eglot users.  I am of course presuming that
    João> you aren't/weren't also an Eglot user, otherwise you would have been
    João> hit by these package upgrades related to dependencies in Emacs 28
    João> and would have noticed them.

I use eglot on emacs-29, but have never installed it (and I never run
emacs-28 any more :-))

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

On Mon, Apr 24, 2023, 08:48 Robert Pluim <rpluim@gmail.com> wrote:

>
>
>     João> Not when package dependencies are involved.  If you weren't
> aware,
>     João> in Emacs 26 (including 29), if you explicitly ask to install
> package A
>     João> and it depends on :core package B, which you didn't ask to
> install,
>     João> package B gets upgraded.
>
> I think thatʼs OK as a behaviour


Noted :)

  .  I am of course presuming that
>     João> you aren't/weren't also an Eglot user, otherwise you would have
> been
>     João> hit by these package upgrades related to dependencies in Emacs 28
>     João> and would have noticed them.
>
> I use eglot on emacs-29, but have never installed it (and I never run
> emacs-28 any more :-))
>

Also noted. Eglot on Emacs 29 is generally the most stable version of it
you can get. However is has less features. Expect the gap to grow with
time. Some fringe LSP servers are also better supported on later versions.

If you never use package-install on Eglot, then there's even less reason to
worry about my patch, which only affects Eglot.

(Meaning, FTR, that we're still missing an account from at least one user
who would be perturbed by my patch.)

Robert, do you use package-install at all? On what packages?

João

[-- Attachment #2: Type: text/html, Size: 2332 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Mon, 24 Apr 2023 09:57:43 +0100, João Távora <joaotavora@gmail.com> said:
    >> I use eglot on emacs-29, but have never installed it (and I never run
    >> emacs-28 any more :-))
    >> 

    João> Also noted. Eglot on Emacs 29 is generally the most stable version of it
    João> you can get. However is has less features. Expect the gap to grow with
    João> time. Some fringe LSP servers are also better supported on later versions.

There are too many things going on to attempt to keep up to date with
them all :-)

    João> If you never use package-install on Eglot, then there's even less reason to
    João> worry about my patch, which only affects Eglot.

    João> (Meaning, FTR, that we're still missing an account from at least one user
    João> who would be perturbed by my patch.)

    João> Robert, do you use package-install at all? On what packages?

Explicitly in my init file, no. The only package installation and
upgrading I do is via `list-packages', for the usual suspects like
magit and helm. And even that I do reluctantly, since something always
seems to break.

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Mon, Apr 24, 2023 at 10:38 AM Robert Pluim <rpluim@gmail.com> wrote:
>
> >>>>> On Mon, 24 Apr 2023 09:57:43 +0100, João Távora <joaotavora@gmail.com> said:
>     >> I use eglot on emacs-29, but have never installed it (and I never run
>     >> emacs-28 any more :-))
>     >>
>
>     João> Also noted. Eglot on Emacs 29 is generally the most stable version of it
>     João> you can get. However is has less features. Expect the gap to grow with
>     João> time. Some fringe LSP servers are also better supported on later versions.
>
> There are too many things going on to attempt to keep up to date with
> them all :-)

Perfectly reasonable.  It would be rather unwise to force a newer
version of Eglot on you.  Do let me know if you find any bugs in
the Emacs 29 eglot.el, which is version 1.12.29. The pretest is here
for that.

>     João> If you never use package-install on Eglot, then there's even less reason to
>     João> worry about my patch, which only affects Eglot.
>
>     João> (Meaning, FTR, that we're still missing an account from at least one user
>     João> who would be perturbed by my patch.)
>
>     João> Robert, do you use package-install at all? On what packages?
>
> Explicitly in my init file, no. The only package installation and
> upgrading I do is via `list-packages', for the usual suspects like
> magit and helm. And even that I do reluctantly, since something always
> seems to break.

That's likely something to bring up with the maintainers of
said packages (which aren't GNU ELPA btw).  I don't even use
them (I use stock VC and stock M-x fido-vertical-mode).

But do you have any suspicion if that is related to upgrade of
dependencies, or -- more to the point of this bug report --
to upgrade of :core dependencies?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Mon, 24 Apr 2023 00:53:30 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 23/04/2023 17:24, Eli Zaretskii wrote:
> >> Date: Sun, 23 Apr 2023 16:11:44 +0300
> >> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
> >>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >>
> >>>> It addressed two last points from your previous email, obviously.
> >>> Which points are those?  Please help me identify them.
> >>
> >> These two:
> >>
> >>   > There is already such an option, added as part of fixing this bug.
> >>
> >> The binding was added, so now we straight away delegate to package-install.
> > 
> > I meant to give the user the control on whether package-update will
> > update built-in packages, like we did with package-install: either via
> > prefix argument or by customizing the user option.
> 
> That would be a different change, though.
> 
> So what are we guarding against here? That the user will choose a 
> built-in package to upgrade by accident?

Yes.  Also, against invocations of this command from other commands
and from the menu.

> And we'll show her an error, saying "use the prefix argument"?

No, I think we'll do whatever the code does today when passed a
built-in package.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Mon, 24 Apr 2023 12:43:21 +0100, João Távora <joaotavora@gmail.com> said:

    João> On Mon, Apr 24, 2023 at 10:38 AM Robert Pluim <rpluim@gmail.com> wrote:
    >> 
    >> >>>>> On Mon, 24 Apr 2023 09:57:43 +0100, João Távora <joaotavora@gmail.com> said:
    >> >> I use eglot on emacs-29, but have never installed it (and I never run
    >> >> emacs-28 any more :-))
    >> >>
    >> 
    João> Also noted. Eglot on Emacs 29 is generally the most stable version of it
    João> you can get. However is has less features. Expect the gap to grow with
    João> time. Some fringe LSP servers are also better supported on later versions.
    >> 
    >> There are too many things going on to attempt to keep up to date with
    >> them all :-)

    João> Perfectly reasonable.  It would be rather unwise to force a newer
    João> version of Eglot on you.  Do let me know if you find any bugs in
    João> the Emacs 29 eglot.el, which is version 1.12.29. The pretest is here
    João> for that.

I will, although I use clangd which I suspect is pretty heavily tested
anyway.

    >> Explicitly in my init file, no. The only package installation and
    >> upgrading I do is via `list-packages', for the usual suspects like
    >> magit and helm. And even that I do reluctantly, since something always
    >> seems to break.

    João> That's likely something to bring up with the maintainers of
    João> said packages (which aren't GNU ELPA btw).  I don't even use
    João> them (I use stock VC and stock M-x fido-vertical-mode).

I donʼt think magit or helm are to blame here.

    João> But do you have any suspicion if that is related to upgrade of
    João> dependencies, or -- more to the point of this bug report --
    João> to upgrade of :core dependencies?

`list-packages' seems to get confused about which dependent packages
need to be uninstalled, and sometimes uninstalls too many (or maybe it
fails to upgrade them, Iʼm not sure). I then get errors about 'package
bar requires package foo version 1.12, only found 1.11' or similar.

Every time it happens I manually install the packages that should not
have been uninstalled, and get on with other things, since I donʼt
normally have time to spend on creating a minimal test case.

Now you understand why I only upgrade my packages every couple of
months.

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Jim Porter <jporterbugs@gmail.com>,  62720@debbugs.gnu.org,  "Philip K."
>  <philipk@posteo.net>,  Dmitry Gutov <dmitry@gutov.dev>,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  Eli Zaretskii <eliz@gnu.org>,  Lars
>  Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 24 Apr 2023 15:01:59 +0200
> 
>     João> But do you have any suspicion if that is related to upgrade of
>     João> dependencies, or -- more to the point of this bug report --
>     João> to upgrade of :core dependencies?
> 
> `list-packages' seems to get confused about which dependent packages
> need to be uninstalled, and sometimes uninstalls too many (or maybe it
> fails to upgrade them, Iʼm not sure). I then get errors about 'package
> bar requires package foo version 1.12, only found 1.11' or similar.

This could be related to the fact that I see here some packages as
"incompat".  Eglot, Flymake, Project, and others are in that category.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Mon, 24 Apr 2023 16:08:57 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Jim Porter <jporterbugs@gmail.com>,  62720@debbugs.gnu.org,  "Philip K."
    >> <philipk@posteo.net>,  Dmitry Gutov <dmitry@gutov.dev>,  Stefan Monnier
    >> <monnier@iro.umontreal.ca>,  Eli Zaretskii <eliz@gnu.org>,  Lars
    >> Ingebrigtsen <larsi@gnus.org>
    >> Date: Mon, 24 Apr 2023 15:01:59 +0200
    >> 
    João> But do you have any suspicion if that is related to upgrade of
    João> dependencies, or -- more to the point of this bug report --
    João> to upgrade of :core dependencies?
    >> 
    >> `list-packages' seems to get confused about which dependent packages
    >> need to be uninstalled, and sometimes uninstalls too many (or maybe it
    >> fails to upgrade them, Iʼm not sure). I then get errors about 'package
    >> bar requires package foo version 1.12, only found 1.11' or similar.

    Eli> This could be related to the fact that I see here some packages as
    Eli> "incompat".  Eglot, Flymake, Project, and others are in that category.

I donʼt see that, but I do have eg:

  helm-core                      20230317.1729  available    melpa    Development files for Helm
  helm-core                      20220425.1625  dependency            Development files for Helm
  helm-core                      20220824.1925  obsolete              Development files for Helm

Maybe I should trim my `package-archives'

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 24/04/2023 16:01, Robert Pluim wrote:
> Every time it happens I manually install the packages that should not
> have been uninstalled, and get on with other things, since I donʼt
> normally have time to spend on creating a minimal test case.

We might have some more bugs with package being removed from 
package-selected-packages during installation/upgrade. I just found one 
in 'M-x package-update' (discussed within the context of the patch for it).

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Mon, Apr 24, 2023 at 2:02 PM Robert Pluim <rpluim@gmail.com> wrote:

> Now you understand why I only upgrade my packages every couple of
> months.

It doesn't seem like you have any problems with the packages
themselves but rather with a significantly buggy idiosyncratic
package management system(s).

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2076 bytes --]

On 24/04/2023 14:58, Eli Zaretskii wrote:
>> Date: Mon, 24 Apr 2023 00:53:30 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 23/04/2023 17:24, Eli Zaretskii wrote:
>>>> Date: Sun, 23 Apr 2023 16:11:44 +0300
>>>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>>>    joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>>>> From: Dmitry Gutov <dmitry@gutov.dev>
>>>>
>>>>>> It addressed two last points from your previous email, obviously.
>>>>> Which points are those?  Please help me identify them.
>>>>
>>>> These two:
>>>>
>>>>    > There is already such an option, added as part of fixing this bug.
>>>>
>>>> The binding was added, so now we straight away delegate to package-install.
>>>
>>> I meant to give the user the control on whether package-update will
>>> update built-in packages, like we did with package-install: either via
>>> prefix argument or by customizing the user option.
>>
>> That would be a different change, though.
>>
>> So what are we guarding against here? That the user will choose a
>> built-in package to upgrade by accident?
> 
> Yes.  Also, against invocations of this command from other commands
> and from the menu.

It's not in the menu. There are also no known callers aside from 
package-update-all.

>> And we'll show her an error, saying "use the prefix argument"?
> 
> No, I think we'll do whatever the code does today when passed a
> built-in package.

Very well, here's the next version. It adds a new optional argument to 
the function (so that people can evaluate e.g. (package-update 'eglot 
t)). When called interactively, it is determined by current-prefix-argument.

Also please review the docstring change.

Regarding obeying package-install-upgrade-built-in, I think it would 
need to be renamed, and both package-update-all and 
package-menu-mark-upgrades would need to be made obey it too. All that 
could be done in a subsequent change.

[-- Attachment #2: package-update-fix.diff --]
[-- Type: text/x-patch, Size: 3312 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ffa6272dd1f..b72bf86c773 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2266,21 +2266,39 @@ package-install
 (declare-function package-vc-update "package-vc" (pkg))
 
 ;;;###autoload
-(defun package-update (name)
-  "Update package NAME if a newer version exists."
+(defun package-update (name &optional update-built-ins)
+  "Update package NAME if a newer version exists.
+
+Only packages installed from ELPA are allowed to be updated this
+way.
+
+But if UPDATE-BUILT-INS is non-nil, or if the command is invoked
+interactively with a prefix argument, it will allow upgrading of
+active built-in packages."
   (interactive
    (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: " (package--updateable-packages
+                              current-prefix-arg)
+          nil t)
+         current-prefix-arg))
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
+         (pkg-desc (cadr (assq package package-alist)))
+         (package-install-upgrade-built-in (and
+                                            update-built-ins
+                                            (not pkg-desc))))
+    ;; `pkg-desc' will be nil when the package is an "active built-in".
+    (if (and pkg-desc (package-vc-p pkg-desc))
         (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
-
-(defun package--updateable-packages ()
+      (when pkg-desc
+        (package-delete pkg-desc 'force 'nosave))
+      (package-install package
+                       ;; An active built-in has never been "selected"
+                       ;; before.  Mark it as installed explicitly.
+                       (and pkg-desc 'dont-select)))))
+
+(defun package--updateable-packages (&optional allow-builtins)
   ;; Initialize the package system to get the list of package
   ;; symbols for completion.
   (package--archives-initialize)
@@ -2291,11 +2309,21 @@ package--updateable-packages
       (or (let ((available
                  (assq (car elt) package-archive-contents)))
             (and available
-                 (version-list-<
-                  (package-desc-version (cadr elt))
-                  (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+                 (or (and
+                      allow-builtins
+                      (not (package-desc-version (cadr elt))))
+                     (version-list-<
+                      (package-desc-version (cadr elt))
+                      (package-desc-version (cadr available))))))
+          (package-vc-p (cadr elt))))
+    (if allow-builtins
+        (append package-alist
+                (mapcan
+                 (lambda (elt)
+                   (when (not (assq (car elt) package-alist))
+                     (list (list (car elt) (package--from-builtin elt)))))
+                 package--builtins))
+      package-alist))))
 
 ;;;###autoload
 (defun package-update-all (&optional query)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Tue, 25 Apr 2023 02:45:46 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> So what are we guarding against here? That the user will choose a
> >> built-in package to upgrade by accident?
> > 
> > Yes.  Also, against invocations of this command from other commands
> > and from the menu.
> 
> It's not in the menu.

There's this:

  <menu-bar> <package> <Mark All Available Upgrades> runs the command
  package-menu-mark-upgrades (found in package-menu-mode-map), which is
  an interactive byte-compiled Lisp function in ‘package.el’.

  It is bound to U.

  (package-menu-mark-upgrades)

  Mark all upgradable packages in the Package Menu.
  For each installed package with a newer version available, place
  an (I)nstall flag on the available version and a (D)elete flag on
  the installed version.  A subsequent x
  call will upgrade the package.

I also envision that we will at some point have an "upgrade" menu
item, because it make little sense to have this command, but not to be
able to invoke it from the menu.

> There are also no known callers aside from package-update-all.

One caller is enough, IMO.

> Very well, here's the next version. It adds a new optional argument to 
> the function (so that people can evaluate e.g. (package-update 'eglot 
> t)). When called interactively, it is determined by current-prefix-argument.

Thanks, this is very close to what I had in mind.  The only thing that
is missing is the support for user option, which should then avoid the
need to invoke the command with a prefix argument.

> Also please review the docstring change.

It looks OK to me.

> Regarding obeying package-install-upgrade-built-in, I think it would 
> need to be renamed, and both package-update-all and 
> package-menu-mark-upgrades would need to be made obey it too. All that 
> could be done in a subsequent change.

If the option will affect more than just package-install, it should
indeed be renamed.

> -(defun package-update (name)
> -  "Update package NAME if a newer version exists."
> +(defun package-update (name &optional update-built-ins)
> +  "Update package NAME if a newer version exists.
> +
> +Only packages installed from ELPA are allowed to be updated this
> +way.

I'm not sure I understand where this restriction comes from.  Did the
original code enforce it?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 25/04/2023 10:47, Eli Zaretskii wrote:
>> Date: Tue, 25 Apr 2023 02:45:46 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> So what are we guarding against here? That the user will choose a
>>>> built-in package to upgrade by accident?
>>>
>>> Yes.  Also, against invocations of this command from other commands
>>> and from the menu.
>>
>> It's not in the menu.
> 
> There's this:
> 
>    <menu-bar> <package> <Mark All Available Upgrades> runs the command
>    package-menu-mark-upgrades (found in package-menu-mode-map), which is
>    an interactive byte-compiled Lisp function in ‘package.el’.
> 
>    It is bound to U.
> 
>    (package-menu-mark-upgrades)
> 
>    Mark all upgradable packages in the Package Menu.
>    For each installed package with a newer version available, place
>    an (I)nstall flag on the available version and a (D)elete flag on
>    the installed version.  A subsequent x
>    call will upgrade the package.

package-menu-mark-upgrades does not delegate to package-update, for 
better or worse.

As discussed previously, if we're going to change its behavior (of 
package-menu-mark-upgrades), we *will* hide that behind the pref.

> I also envision that we will at some point have an "upgrade" menu
> item, because it make little sense to have this command, but not to be
> able to invoke it from the menu.

We might. But since that will only happen in the future, by that time 
there won't be any users who are accustomed to using that menu item and 
thus possibly inconvenienced by the change in behavior.

>> Very well, here's the next version. It adds a new optional argument to
>> the function (so that people can evaluate e.g. (package-update 'eglot
>> t)). When called interactively, it is determined by current-prefix-argument.
> 
> Thanks, this is very close to what I had in mind.  The only thing that
> is missing is the support for user option, which should then avoid the
> need to invoke the command with a prefix argument.
> 
>> Also please review the docstring change.
> 
> It looks OK to me.
> 
>> -(defun package-update (name)
>> -  "Update package NAME if a newer version exists."
>> +(defun package-update (name &optional update-built-ins)
>> +  "Update package NAME if a newer version exists.
>> +
>> +Only packages installed from ELPA are allowed to be updated this
>> +way.
> 
> I'm not sure I understand where this restriction comes from.  Did the
> original code enforce it?

I'm not sure what you mean about "enforce it". That's the essence of the 
bug here: this function's inability to upgrade built-in packages 
(packages installed not from ELPA). Since you are asking to keep that 
behavior by default, it now needs to be documented.

 >> Regarding obeying package-install-upgrade-built-in, I think it would
 >> need to be renamed, and both package-update-all and
 >> package-menu-mark-upgrades would need to be made obey it too. All that
 >> could be done in a subsequent change.
 >
 > If the option will affect more than just package-install, it should
 > indeed be renamed.

That will require some more work. On package-menu-mark-upgrades in 
particular.

TBH, I'm getting more doubts about this change now.

What will we do in Emacs 30? If we add the new argument, it will be hard 
to back out of it, to default to the proper behavior.

Perhaps we should just wait and then fix it on master properly. 
Workarounds exist, after all.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

> Perhaps we should just wait and then fix it on master properly.
> Workarounds exist, after all.

+1

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 23/04/2023 16:02, Philip Kaludercic wrote:
>> Dmitry Gutov <dmitry@gutov.dev> writes:
>> 
>>> Philip,
>>>
>>> On 13/04/2023 21:49, Philip Kaludercic wrote:
>>>> +    (when (and (or current-prefix-arg package-install-upgrade-built-in)
>>>> +               (package--active-built-in-p pkg))
>>>> +      (setq pkg (cadr (assq name package-archive-contents))))
>>>
>>> How sure are you that the first element in (cdr (assq name
>>> package-archive-contents)) will contain the latest version?
>> This is not certain, but the same approach is used in other places
>> in
>> package.el, so I just stocked to it.
>
> Perhaps they conceal latent bugs as well. I don't know the codebase
> too well, but the places I did examine used the comparison together
> with the priority.

I will look into this.

>>> That's why I came with the more complex way to choose the appropriate
>>> package-desc in my latest attempt:
>>>
>>>   (car
>>>    (last (seq-sort-by #'package-desc-priority-version
>>>                       #'version-list-<
>>>                       (cdr (assq package package-archive-contents)))))
>> If we insist on package.el installing the newest version, then this
>> would make sense.
>
> What's the alternative? Upgrading to "some version"?

Respect `package-archive-priorities'.

>> AFAIK there is no guarantees on the order of packages
>> in `package-archive-contents'.  This shouldn't be an issue for Eglot,
>> but might be one for use-package.
>
> Ah, it seems they removed it from Melpa, that shouldn't surprise me.

Oh, I did not know it was on MELPA in the first place.  I haven't been
using the archive for a while.

> Eglot is in the GNU-devel archive as well, though. So there is some
> potential for conflict.

Hmm, I guess that is true, but I have never been a fan of advertising
the -devel archives for day-to-day usage.

>> I would actually pull it out into a
>> separate utility function that we would re-use in other places.
>
> Since I had to drop the respective code from the latest version of the
> patch, I guess you could put it in package-install instead.
>
> A reusable helper is fine, of course, but what are the other places we
> would use it at?

At any place where (cadr ...) is used, I would assume.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Tue, 25 Apr 2023 15:08:15 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> -(defun package-update (name)
> >> -  "Update package NAME if a newer version exists."
> >> +(defun package-update (name &optional update-built-ins)
> >> +  "Update package NAME if a newer version exists.
> >> +
> >> +Only packages installed from ELPA are allowed to be updated this
> >> +way.
> > 
> > I'm not sure I understand where this restriction comes from.  Did the
> > original code enforce it?
> 
> I'm not sure what you mean about "enforce it". That's the essence of the 
> bug here: this function's inability to upgrade built-in packages 
> (packages installed not from ELPA). Since you are asking to keep that 
> behavior by default, it now needs to be documented.

Oh, then I misunderstood what that says.  I thought is says ELPA as
opposed to, say, MELPA.

So I think we need to rephrase that.  Something like

  Packages which are part of the Emacs distribution cannot be updated
  that way.

>  >> Regarding obeying package-install-upgrade-built-in, I think it would
>  >> need to be renamed, and both package-update-all and
>  >> package-menu-mark-upgrades would need to be made obey it too. All that
>  >> could be done in a subsequent change.
>  >
>  > If the option will affect more than just package-install, it should
>  > indeed be renamed.
> 
> That will require some more work. On package-menu-mark-upgrades in 
> particular.
> 
> TBH, I'm getting more doubts about this change now.
> 
> What will we do in Emacs 30? If we add the new argument, it will be hard 
> to back out of it, to default to the proper behavior.

I thought that in Emacs 30 we could make the user option be non-nil by
default, assuming we will decide not to treat built-in packages
specially in this regard.  Then the additional argument will become
much less important, perhaps for some rare situations or something.

> Perhaps we should just wait and then fix it on master properly. 
> Workarounds exist, after all.

I won't object, but I thought you and others wanted to have
package-install and package-update to behave consistently in this
respect.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 25/04/2023 15:43, Eli Zaretskii wrote:
>> Date: Tue, 25 Apr 2023 15:08:15 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> -(defun package-update (name)
>>>> -  "Update package NAME if a newer version exists."
>>>> +(defun package-update (name &optional update-built-ins)
>>>> +  "Update package NAME if a newer version exists.
>>>> +
>>>> +Only packages installed from ELPA are allowed to be updated this
>>>> +way.
>>>
>>> I'm not sure I understand where this restriction comes from.  Did the
>>> original code enforce it?
>>
>> I'm not sure what you mean about "enforce it". That's the essence of the
>> bug here: this function's inability to upgrade built-in packages
>> (packages installed not from ELPA). Since you are asking to keep that
>> behavior by default, it now needs to be documented.
> 
> Oh, then I misunderstood what that says.  I thought is says ELPA as
> opposed to, say, MELPA.

No, just ELPA in general.

> So I think we need to rephrase that.  Something like
> 
>    Packages which are part of the Emacs distribution cannot be updated
>    that way.

This is probably better. As long as we understand it to read "packages 
which are part ... and were never upgraded to a version from ELPA".

>>   >> Regarding obeying package-install-upgrade-built-in, I think it would
>>   >> need to be renamed, and both package-update-all and
>>   >> package-menu-mark-upgrades would need to be made obey it too. All that
>>   >> could be done in a subsequent change.
>>   >
>>   > If the option will affect more than just package-install, it should
>>   > indeed be renamed.
>>
>> That will require some more work. On package-menu-mark-upgrades in
>> particular.
>>
>> TBH, I'm getting more doubts about this change now.
>>
>> What will we do in Emacs 30? If we add the new argument, it will be hard
>> to back out of it, to default to the proper behavior.
> 
> I thought that in Emacs 30 we could make the user option be non-nil by
> default, assuming we will decide not to treat built-in packages
> specially in this regard.  Then the additional argument will become
> much less important, perhaps for some rare situations or something.

It would remain an odd vestige, and it might be difficult to repurpose 
for something more useful (e.g. being able to pick a specific version to 
upgrade/downgrade to?)

>> Perhaps we should just wait and then fix it on master properly.
>> Workarounds exist, after all.
> 
> I won't object, but I thought you and others wanted to have
> package-install and package-update to behave consistently in this
> respect.

Having package-install and package-update behave the same was never the 
goal, at least not for me.

Quite the opposite: package-install doesn't install anything when the 
package is already installed (and the argument is a package name symbol, 
which is the case for interactive invocations).

package-update/upgrade, OTOH, is supposed to upgrade already installed 
packages. Which I'm assuming is the category we are going to assign the 
built-ins to, after all.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 25/04/2023 21:35, Dmitry Gutov wrote:
>> So I think we need to rephrase that.  Something like
>>
>>    Packages which are part of the Emacs distribution cannot be updated
>>    that way.
> 
> This is probably better. As long as we understand it to read "packages 
> which are part ... and were never upgraded to a version from ELPA".

Anyway, if we agree to keep this unfixed, we should probably add this 
text somewhere: either into the docstring anyway, or to etc/PROBLEMS.

Not sure how to avoid the impression that this is the intended behavior 
and not something we're looking to change in the future, though.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Thu, 27 Apr 2023 02:05:21 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
> 
> On 25/04/2023 21:35, Dmitry Gutov wrote:
> >> So I think we need to rephrase that.  Something like
> >>
> >>    Packages which are part of the Emacs distribution cannot be updated
> >>    that way.
> > 
> > This is probably better. As long as we understand it to read "packages 
> > which are part ... and were never upgraded to a version from ELPA".
> 
> Anyway, if we agree to keep this unfixed, we should probably add this 
> text somewhere: either into the docstring anyway, or to etc/PROBLEMS.

Do we agree to leave package-update unchanged?

And what about the renaming you suggested in another bug report?

I'd like to make a new pretest soon, with all these issues finalized
as best as we can.

As for PROBLEMS, I'm not sure.  We usually put there only something
that people actually complain about, not proactively.  How about
mentioning this in the doc string instead?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 27/04/2023 08:41, Eli Zaretskii wrote:
>> Date: Thu, 27 Apr 2023 02:05:21 +0300
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
>>
>> On 25/04/2023 21:35, Dmitry Gutov wrote:
>>>> So I think we need to rephrase that.  Something like
>>>>
>>>>     Packages which are part of the Emacs distribution cannot be updated
>>>>     that way.
>>>
>>> This is probably better. As long as we understand it to read "packages
>>> which are part ... and were never upgraded to a version from ELPA".
>>
>> Anyway, if we agree to keep this unfixed, we should probably add this
>> text somewhere: either into the docstring anyway, or to etc/PROBLEMS.
> 
> Do we agree to leave package-update unchanged?

Apparently so.

> And what about the renaming you suggested in another bug report?

Still waiting for your go-ahead. Although it sounds like I have it.

> I'd like to make a new pretest soon, with all these issues finalized
> as best as we can.

Very good.

> As for PROBLEMS, I'm not sure.  We usually put there only something
> that people actually complain about, not proactively.  How about
> mentioning this in the doc string instead?

Sure. Hopefully it won't create an impression that the described 
behavior is _intended_.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Thu, 27 Apr 2023 12:00:51 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > And what about the renaming you suggested in another bug report?
> 
> Still waiting for your go-ahead. Although it sounds like I have it.

No one objected, AFAIU.  So I think you can install that.

> > As for PROBLEMS, I'm not sure.  We usually put there only something
> > that people actually complain about, not proactively.  How about
> > mentioning this in the doc string instead?
> 
> Sure. Hopefully it won't create an impression that the described 
> behavior is _intended_.

We can say something like "Currently, ..." to make that clear.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 27/04/2023 13:44, Eli Zaretskii wrote:
>> Date: Thu, 27 Apr 2023 12:00:51 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> And what about the renaming you suggested in another bug report?
>>
>> Still waiting for your go-ahead. Although it sounds like I have it.
> 
> No one objected, AFAIU.  So I think you can install that.

Thank you, done.

>>> As for PROBLEMS, I'm not sure.  We usually put there only something
>>> that people actually complain about, not proactively.  How about
>>> mentioning this in the doc string instead?
>>
>> Sure. Hopefully it won't create an impression that the described
>> behavior is _intended_.
> 
> We can say something like "Currently, ..." to make that clear.

Very good, I've used that and an additional clarification for a 
recommended alternative. Please see how you like it, or whether it needs 
improving.

Before the next pretest is cut, I'd also like to bring up the question 
again of backporting the patch for bug#62816 (with Joao's support: 
previously mentioned in 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#517, for example).

It improves the part of Eldoc that is not used by default 
(eldoc-documentation-strategy has a different value) but which is used 
by Eglot since it changes that variable in its managed buffers.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 28 Apr 2023 02:51:19 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > We can say something like "Currently, ..." to make that clear.
> 
> Very good, I've used that and an additional clarification for a 
> recommended alternative. Please see how you like it, or whether it needs 
> improving.

I've made a few minor improvements, thanks.

> Before the next pretest is cut, I'd also like to bring up the question 
> again of backporting the patch for bug#62816 (with Joao's support: 
> previously mentioned in 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#517, for example).
> 
> It improves the part of Eldoc that is not used by default 
> (eldoc-documentation-strategy has a different value) but which is used 
> by Eglot since it changes that variable in its managed buffers.

That patch didn't yet accumulate enough time for me to consider it
safe for the release branch.  Depending on when Emacs 29.1 is released
and whether we hear some downsides of the change, it could need to
wait for Emacs 29.2 or for Emacs 30.1.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> If we insist on package.el installing the newest version, then this
> would make sense.

I don't see any reason we should ignore priorities here, so we shouldn't
blindly aim for the "newest version".


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 28/04/2023 08:19, Eli Zaretskii wrote:
>> Date: Fri, 28 Apr 2023 02:51:19 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> We can say something like "Currently, ..." to make that clear.
>>
>> Very good, I've used that and an additional clarification for a
>> recommended alternative. Please see how you like it, or whether it needs
>> improving.
> 
> I've made a few minor improvements, thanks.

Thank you.

>> Before the next pretest is cut, I'd also like to bring up the question
>> again of backporting the patch for bug#62816 (with Joao's support:
>> previously mentioned in
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#517, for example).
>>
>> It improves the part of Eldoc that is not used by default
>> (eldoc-documentation-strategy has a different value) but which is used
>> by Eglot since it changes that variable in its managed buffers.
> 
> That patch didn't yet accumulate enough time for me to consider it
> safe for the release branch.  Depending on when Emacs 29.1 is released
> and whether we hear some downsides of the change, it could need to
> wait for Emacs 29.2 or for Emacs 30.1.

Okay.

Let's get back to the previous topic. What about the previous fix for 
package-upgrade that I posted, one that makes it unconditionally upgrade 
built-in packages when invoked?

The one attached here: https://debbugs.gnu.org/62720#718

Can we put it on master now, or do we have to wait for some feedback 
from Emacs 29 first?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 5 May 2023 02:58:25 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> Let's get back to the previous topic. What about the previous fix for 
> package-upgrade that I posted, one that makes it unconditionally upgrade 
> built-in packages when invoked?
> 
> The one attached here: https://debbugs.gnu.org/62720#718
> 
> Can we put it on master now, or do we have to wait for some feedback 
> from Emacs 29 first?

I'd prefer the latter.  I'd prefer even more to have same behavior in
Emacs 20 and Emacs 30, which could be possible if we decide to make
this change in Emacs 29.2, based on feedback.  Because is it really a
good idea to have the master and the release branch behave so
differently in this regard?  People who use both branches, or switch
from one to the other, will become confused.

Philip, Stefan: WDYT about this?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 5 May 2023 02:58:25 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> Let's get back to the previous topic. What about the previous fix for 
>> package-upgrade that I posted, one that makes it unconditionally upgrade 
>> built-in packages when invoked?
>> 
>> The one attached here: https://debbugs.gnu.org/62720#718
>> 
>> Can we put it on master now, or do we have to wait for some feedback 
>> from Emacs 29 first?
>
> I'd prefer the latter.  I'd prefer even more to have same behavior in
> Emacs 20 and Emacs 30, which could be possible if we decide to make
> this change in Emacs 29.2, based on feedback.  Because is it really a
> good idea to have the master and the release branch behave so
> differently in this regard?  People who use both branches, or switch
> from one to the other, will become confused.
>
> Philip, Stefan: WDYT about this?

I am fine with any change, as long as package-upgrade-all does not
automatically switch from built-in packages to a different version from ELPA.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> People who use both branches, or switch from one to the other, will
> become confused.

I'm not terribly worried about that.

> Philip, Stefan: WDYT about this?

I'll second Philip's reply here.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dmitry@gutov.dev>,  jporterbugs@gmail.com,
>   philipk@posteo.net,  62720@debbugs.gnu.org,  larsi@gnus.org,
>   joaotavora@gmail.com
> Date: Fri, 05 May 2023 09:58:44 -0400
> 
> > People who use both branches, or switch from one to the other, will
> > become confused.
> 
> I'm not terribly worried about that.
> 
> > Philip, Stefan: WDYT about this?
> 
> I'll second Philip's reply here.

So I guess the consensus here is that this change is fine for master,
so let's install it there.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 05/05/2023 17:05, Eli Zaretskii wrote:
>> From: Stefan Monnier<monnier@iro.umontreal.ca>
>> Cc: Dmitry Gutov<dmitry@gutov.dev>,jporterbugs@gmail.com,
>>    philipk@posteo.net,62720@debbugs.gnu.org,larsi@gnus.org,
>>    joaotavora@gmail.com
>> Date: Fri, 05 May 2023 09:58:44 -0400
>>
>>> People who use both branches, or switch from one to the other, will
>>> become confused.
>> I'm not terribly worried about that.
>>
>>> Philip, Stefan: WDYT about this?
>> I'll second Philip's reply here.
> So I guess the consensus here is that this change is fine for master,
> so let's install it there.

Thanks all, done.

I've also installed the minor fix discussed previously on emacs-29.