bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Hello,

I've been meaning to report this for some time.  I really think this
should be fixed in Emacs 29 before it ships.

src/emacs -Q # Emacs 29

M-x package-install RET eglot RET

Echoes "No match", even though eglot 1.14 is available from GNU ELPA.
That's because Eglot is a :core ELPA package and it's already in Emacs
29, so package.el thinks there is nothing to install.

Evaluating (package-install 'eglot) also doesn't work.  Errors with
"'eglot' is already installed"

In Emacs 28, both these alternatives work as expected (because Eglot
isn't a :core package there).

It's not impossible to upgrade in Emacs 29, of course.  The only way I
know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
it with 'i' and confirm installation with 'x'.  But it is very awkward.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> It's not impossible to upgrade in Emacs 29, of course.  The only way I
> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
> it with 'i' and confirm installation with 'x'.  But it is very awkward.

Maybe we should add a `package-upgrade` command?


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> It's not impossible to upgrade in Emacs 29, of course.  The only way I
>> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
>> it with 'i' and confirm installation with 'x'.  But it is very awkward.
>
> Maybe we should add a `package-upgrade` command?

Just for :core packages or how would this differ from package-update?

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 8, 2023 at 11:43 AM Philip Kaludercic <philipk@posteo.net> wrote:
>
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
> >> It's not impossible to upgrade in Emacs 29, of course.  The only way I
> >> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
> >> it with 'i' and confirm installation with 'x'.  But it is very awkward.
> >
> > Maybe we should add a `package-upgrade` command?
>
> Just for :core packages or how would this differ from package-update?

Just a note that I forgot M-x package-update existed.  I was
hoping it wouldn't suffer the same bug as package-install, but
alas it does.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Sat, Apr 8, 2023 at 11:43 AM Philip Kaludercic <philipk@posteo.net> wrote:
>>
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>>
>> >> It's not impossible to upgrade in Emacs 29, of course.  The only way I
>> >> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
>> >> it with 'i' and confirm installation with 'x'.  But it is very awkward.
>> >
>> > Maybe we should add a `package-upgrade` command?
>>
>> Just for :core packages or how would this differ from package-update?
>
> Just a note that I forgot M-x package-update existed.  I was
> hoping it wouldn't suffer the same bug as package-install, but
> alas it does.

As package-update just defers to package-install, I don't find this
surprising.  What you want is that package.el always treats a built-in
package as upgradable to a package from ELPA.  It seems like this should
be possible by adjusting the interactive spec of package-install and
modifying package-compute-transaction or even package-built-in-p.

> João

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 8, 2023 at 3:42 PM Philip Kaludercic <philipk@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > On Sat, Apr 8, 2023 at 11:43 AM Philip Kaludercic <philipk@posteo.net> wrote:
> >>
> >> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> >>
> >> >> It's not impossible to upgrade in Emacs 29, of course.  The only way I
> >> >> know is to M-x package-list-packages, find Eglot 1.14 in the list, mark
> >> >> it with 'i' and confirm installation with 'x'.  But it is very awkward.
> >> >
> >> > Maybe we should add a `package-upgrade` command?
> >>
> >> Just for :core packages or how would this differ from package-update?
> >
> > Just a note that I forgot M-x package-update existed.  I was
> > hoping it wouldn't suffer the same bug as package-install, but
> > alas it does.
>
> As package-update just defers to package-install, I don't find this
> surprising.  What you want is that package.el always treats a built-in
> package as upgradable to a package from ELPA.  It seems like this should
> be possible by adjusting the interactive spec of package-install and
> modifying package-compute-transaction or even package-built-in-p.

Sounds good.  Can you propose a patch so we have something to look at
and test?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> As package-update just defers to package-install, I don't find this
> surprising.  What you want is that package.el always treats a built-in
> package as upgradable to a package from ELPA.  It seems like this should
> be possible by adjusting the interactive spec of package-install and
> modifying package-compute-transaction or even package-built-in-p.

I agree, tho I don't understand why you say "built-in" above.  Isn't the
problem also present for already-installed-but-out-of-date ELPA packages?


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 8, 2023 at 4:45 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > As package-update just defers to package-install, I don't find this
> > surprising.  What you want is that package.el always treats a built-in
> > package as upgradable to a package from ELPA.  It seems like this should
> > be possible by adjusting the interactive spec of package-install and
> > modifying package-compute-transaction or even package-built-in-p.
>
> I agree, tho I don't understand why you say "built-in" above.  Isn't the
> problem also present for already-installed-but-out-of-date ELPA packages?

I bit the bullet and went after this in package.el.  The answer is no,
it's not present for those packages.

But Philip's imagined fix seems to be more complicated than it needs
to be. Here's a simple patch that works well in my tests.  It makes
M-x package-update also update built-in-packages.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..f54b6f39e40 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2243,11 +2243,16 @@ package-update
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
-        (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
+         (nonbuiltin (assq package package-alist)))
+    (cond (nonbuiltin
+           (let ((desc (cadr nonbuiltin)))
+             (if (package-vc-p desc)
+                 (package-vc-update desc)
+               (package-delete desc 'force)
+               (package-install package 'dont-select))))
+          (t
+           (package-install
+            (cadr (assq package package-archive-contents)))))))

 (defun package--updateable-packages ()
   ;; Initialize the package system to get the list of package
@@ -2261,10 +2266,14 @@ package--updateable-packages
                  (assq (car elt) package-archive-contents)))
             (and available
                  (version-list-<
-                  (package-desc-version (cadr elt))
+                  (if (vectorp (cdr elt)) (aref (cdr elt) 0)
+                    (package-desc-version (cadr elt)))
                   (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+          (and (consp (cdr elt))
+               (package-desc-p (cadr elt))
+               (package-vc-p (cadr elt)))))
+    (seq-union package-alist package--builtins
+               (lambda (a b) (eq (car a) (car b)))))))

 ;;;###autoload
 (defun package-update-all (&optional query)

The only thing that's slightly inelegant/hard-to-follow is that
the format of package-alist is different than package--builtins.
Lots of consp, cdr-taking, vectorp and so on.  Besides that, it's
a question of taking the union of the two sets and operating on that.

This also relies on seq-union's undocumented behavior of keeping
the first of any duplicates.  Feel free to rewrite.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Sat, Apr 8, 2023 at 4:45 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>
>> > As package-update just defers to package-install, I don't find this
>> > surprising.  What you want is that package.el always treats a built-in
>> > package as upgradable to a package from ELPA.  It seems like this should
>> > be possible by adjusting the interactive spec of package-install and
>> > modifying package-compute-transaction or even package-built-in-p.
>>
>> I agree, tho I don't understand why you say "built-in" above.  Isn't the
>> problem also present for already-installed-but-out-of-date ELPA packages?
>
> I bit the bullet and went after this in package.el.  The answer is no,
> it's not present for those packages.
>
> But Philip's imagined fix seems to be more complicated than it needs
> to be. Here's a simple patch that works well in my tests.  It makes
> M-x package-update also update built-in-packages.
>
> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index f92afe56b76..f54b6f39e40 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -2243,11 +2243,16 @@ package-update
>    (let* ((package (if (symbolp name)
>                        name
>                      (intern name)))
> -         (pkg-desc (cadr (assq package package-alist))))
> -    (if (package-vc-p pkg-desc)
> -        (package-vc-update pkg-desc)
> -      (package-delete pkg-desc 'force)
> -      (package-install package 'dont-select))))
> +         (nonbuiltin (assq package package-alist)))
> +    (cond (nonbuiltin
> +           (let ((desc (cadr nonbuiltin)))
> +             (if (package-vc-p desc)
> +                 (package-vc-update desc)
> +               (package-delete desc 'force)
> +               (package-install package 'dont-select))))
> +          (t
> +           (package-install
> +            (cadr (assq package package-archive-contents)))))))
>
>  (defun package--updateable-packages ()
>    ;; Initialize the package system to get the list of package
> @@ -2261,10 +2266,14 @@ package--updateable-packages
>                   (assq (car elt) package-archive-contents)))
>              (and available
>                   (version-list-<
> -                  (package-desc-version (cadr elt))
> +                  (if (vectorp (cdr elt)) (aref (cdr elt) 0)
> +                    (package-desc-version (cadr elt)))
>                    (package-desc-version (cadr available)))))
> -          (package-vc-p (cadr (assq (car elt) package-alist)))))
> -    package-alist)))
> +          (and (consp (cdr elt))
> +               (package-desc-p (cadr elt))
> +               (package-vc-p (cadr elt)))))
> +    (seq-union package-alist package--builtins
> +               (lambda (a b) (eq (car a) (car b)))))))

Will this not affect `package-update-all'?  I don't if we want that the
command installs all packages from ELPA that it can find.

>  ;;;###autoload
>  (defun package-update-all (&optional query)
>
> The only thing that's slightly inelegant/hard-to-follow is that
> the format of package-alist is different than package--builtins.
> Lots of consp, cdr-taking, vectorp and so on.  Besides that, it's
> a question of taking the union of the two sets and operating on that.

That is necessary complexity, so I don't think there is any way around it.

> This also relies on seq-union's undocumented behavior of keeping
> the first of any duplicates.  Feel free to rewrite.
>
> João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2275 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Will this not affect `package-update-all'?  I don't if we want that the
> command installs all packages from ELPA that it can find.

Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
updates the built-in and the manually installed packages that can be
updated.  It _doesn't_ install any packages that weren't installed yet,
of course.

In this case, in a bare emacs-29/src/emacs -Q, it updates the Eglot
package, the Eldoc package and few others.  I think this is exactly what
M-x package-update-all is supposed to do (but see notes at the end of
this email).

So I think my change to the existing package-update and
package--updateable-packages fixes the bug cleanly.

>> Lots of consp, cdr-taking, vectorp and so on.  Besides that, it's
>> a question of taking the union of the two sets and operating on that.
> That is necessary complexity, so I don't think there is any way around it.

There is, but that's just an improvement, in this case to the
type-starved 'package--bi-desc' structure and the 'package--builtins'
built by finder.el.  A common structure format should be used.  Or,
better yet, CLOS.

Anyway, with the version of the patch I posted earlier, there are the 6
(six) packages updated currently from a "bare" emacs 29.

    (csharp-mode eglot eldoc jsonrpc transient verilog-mode)

Of these, csharp-mode and transient are mistakes.  But that's not my
patch's fault :-) 

- csharp-mode.el is merely missing version information, so package.el
  thinks that the ELPA version supersedes it (when in fact it doesn't:
  it's older). The patch adds version information to csharp-mode.el to
  fix that.

- transient.el has version information but in a header that finder.el
  doesn't recognize.  The patch has a minimal fix for that, too.

So the final patch that I'm proposing for emacs 29 is attached.  M-x
package-update-all fixes those cases and correctly finds and updates 4
packages to their newest released versions, exactly as it should.

   (eglot eldoc jsonrpc verilog-mode)

Eli, what do you think?  Who is package.el's main maintainer?  Everyone?
Lars added M-x package-update (for Emacs 29) so I'm pinging him as well.

João


[-- Attachment #2: 0001-Add-ability-to-update-built-in-packages-bug-62720.patch --]
[-- Type: text/x-patch, Size: 4009 bytes --]

From 65e811a0fcf9ffd1f12b8b2a2d9d8a0474543b36 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20T=C3=A1vora?= <joaotavora@gmail.com>
Date: Tue, 11 Apr 2023 11:52:46 +0100
Subject: [PATCH] Add ability to update built-in packages (bug#62720)

Previously package.el's M-x package-update command completely ignored
built-in packages.  With this patch in place, it updates them along
with any other manually installed non-built-in packages, as long as
the version available from ELPA is newer.

To prevent misupdates of the 'transient.el' and 'csharp-mode.el'
packages, which are built into emacs-29, version information is now
correctly collected from these two.

* lisp/finder.el (finder-compile-keywords): Be aware of
"Package-Version" header.

* lisp/emacs-lisp/package.el (package-update): Rework.
(package--updateable-packages): Rework.

* lisp/progmodes/csharp-mode.el: Add version information.
---
 lisp/emacs-lisp/package.el    | 26 ++++++++++++++++++--------
 lisp/finder.el                |  2 +-
 lisp/progmodes/csharp-mode.el |  1 +
 3 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..286583100c3 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2243,11 +2243,16 @@ package-update
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
-        (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
+         (nonbuiltin (assq package package-alist)))
+    (cond (nonbuiltin
+           (let ((desc (cadr nonbuiltin)))
+             (if (package-vc-p desc)
+                 (package-vc-update desc)
+               (package-delete desc 'force)
+               (package-install package 'dont-select))))
+          (t
+           (package-install
+            (cadr (assq package package-archive-contents)))))))
 
 (defun package--updateable-packages ()
   ;; Initialize the package system to get the list of package
@@ -2261,10 +2266,15 @@ package--updateable-packages
                  (assq (car elt) package-archive-contents)))
             (and available
                  (version-list-<
-                  (package-desc-version (cadr elt))
+                  (if (vectorp (cdr elt))
+                      (aref (cdr elt) 0)
+                    (package-desc-version (cadr elt)))
                   (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+          (and (consp (cdr elt))
+               (package-desc-p (cadr elt))
+               (package-vc-p (cadr elt)))))
+    (seq-union package-alist package--builtins
+               (lambda (a b) (eq (car a) (car b)))))))
 
 ;;;###autoload
 (defun package-update-all (&optional query)
diff --git a/lisp/finder.el b/lisp/finder.el
index 5aec0149b89..ddc6d6f03da 100644
--- a/lisp/finder.el
+++ b/lisp/finder.el
@@ -231,7 +231,7 @@ finder-compile-keywords
 		  summary  (or (cdr
 			        (assq package finder--builtins-descriptions))
 			       (lm-synopsis))
-		  version  (lm-header "version")))
+		  version  (or (lm-header "package-version") (lm-header "version"))))
 	  (when summary
 	    (setq version (or (ignore-errors (version-to-list version))
                               (alist-get package package--builtin-versions)))
diff --git a/lisp/progmodes/csharp-mode.el b/lisp/progmodes/csharp-mode.el
index 47cd13e7fdb..cd045cd14d1 100644
--- a/lisp/progmodes/csharp-mode.el
+++ b/lisp/progmodes/csharp-mode.el
@@ -8,6 +8,7 @@
 ;;              Jostein Kjønigsen <jostein@kjonigsen.net>
 ;; Created    : September 2022
 ;; Keywords   : c# languages oop
+;; Version    : 3.0.0
 
 ;; This file is part of GNU Emacs.
 
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: eliz@gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>,
>   62720@debbugs.gnu.org, Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 11 Apr 2023 12:02:48 +0100
> 
> So the final patch that I'm proposing for emacs 29 is attached.  M-x
> package-update-all fixes those cases and correctly finds and updates 4
> packages to their newest released versions, exactly as it should.
> 
>    (eglot eldoc jsonrpc verilog-mode)
> 
> Eli, what do you think?

I'd prefer it to go to master, not to emacs-29.  The problem is not
grave enough and OTOH the workaround is simple enough.  So changing
package.el in such non-trivial ways is not something I'd like to risk
now.

> Who is package.el's main maintainer?  Everyone?
> Lars added M-x package-update (for Emacs 29) so I'm pinging him as well.

I think it's mostly Philip and Stefan.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: eliz@gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>,
>>   62720@debbugs.gnu.org, Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 11 Apr 2023 12:02:48 +0100
>> 
>> So the final patch that I'm proposing for emacs 29 is attached.  M-x
>> package-update-all fixes those cases and correctly finds and updates 4
>> packages to their newest released versions, exactly as it should.
>> 
>>    (eglot eldoc jsonrpc verilog-mode)
>> 
>> Eli, what do you think?
>
> I'd prefer it to go to master, not to emacs-29.  The problem is not
> grave enough and OTOH the workaround is simple enough.  So changing
> package.el in such non-trivial ways is not something I'd like to risk
> now.

Please reconsider.  If we do this, than Emacs 29 users will be almost
locked out of upgrading Eglot and a lot of other built-in packages.
I'll have to teach people that workaround in the manual, where such
workarounds don't really belong.

Note that Eglot moved from ELPA to core, but it had (and has) many users
on Emacs 26, 27 and 28.  Eglot is getting regular new features in
master, the bundled Emacs 29 version is now already pretty "old".  When
migrating to Emacs 29, these users will expect to keep being able to
update to the latest version, and will likely be baffled that it doesn't
work as smoothly as it used to.

M-x package-update and M-x package-update-all are new in Emacs 29.
They're buggy, so why ship them buggy?  The change I'm proposing it not
really "non-trivial".  I can walk you or anybody through the code, or
write tests if that would improve the outlook.

>> Who is package.el's main maintainer?  Everyone?
>> Lars added M-x package-update (for Emacs 29) so I'm pinging him as well.
>
> I think it's mostly Philip and Stefan.

Let's hear from them, to see if there's some kind of subtlety I might
have missed.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Tue, 11 Apr 2023 13:52:36 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Eli, what do you think?
> >
> > I'd prefer it to go to master, not to emacs-29.  The problem is not
> > grave enough and OTOH the workaround is simple enough.  So changing
> > package.el in such non-trivial ways is not something I'd like to risk
> > now.
> 
> Please reconsider.  If we do this, than Emacs 29 users will be almost
> locked out of upgrading Eglot and a lot of other built-in packages.
> I'll have to teach people that workaround in the manual, where such
> workarounds don't really belong.

OK, I looked closer at the patch and the code involved in this, and
also re-read this discussion.  I cannot agree with installing your
patch, as submitted, on the emacs-29 branch, sorry.  It modifies code
that affects "normal" invocations of package-update, and also numerous
other functions in package.el (via the change in
package--updateable-packages), in ways that are very hard for me to
audit.  It is hard to audit because there are parts of it that read
like some kind of "black magic":

> +         (nonbuiltin (assq package package-alist)))

Why is the return value of assq the sign that the package is
"nonbuiltin"?

> +    (cond (nonbuiltin
> +           (let ((desc (cadr nonbuiltin)))
> +             (if (package-vc-p desc)
> +                 (package-vc-update desc)
> +               (package-delete desc 'force)
> +               (package-install package 'dont-select))))
> +          (t
> +           (package-install
> +            (cadr (assq package package-archive-contents)))))))

Why the different way of calling package-install for "built-in"
packages?

> -                  (package-desc-version (cadr elt))
> +                  (if (vectorp (cdr elt))
> +                      (aref (cdr elt) 0)
> +                    (package-desc-version (cadr elt)))

What is the significance of the (vectorp (cdr elt)) test?

> -          (package-vc-p (cadr (assq (car elt) package-alist)))))
> -    package-alist)))
> +          (and (consp (cdr elt))
> +               (package-desc-p (cadr elt))
> +               (package-vc-p (cadr elt)))))
> +    (seq-union package-alist package--builtins
> +               (lambda (a b) (eq (car a) (car b)))))))

What is the significance of the (consp (cdr elt)) test?  And why do we
need to add package--builtins to the list?

How am I supposed to assess the safety of this patch, given all this
semi-obfuscated code, and given that I'm not the every-day maintainer
of package.el and am not familiar with all the quirks of its code?
(It is quite possible that this obfuscated nature of the code is not
your fault, but is caused by how package.el is implemented.  In which
case I hope that we could clean up the code of package.el on master to
allow updating :core packages more seamlessly and with simpler code.)

OTOH, the workaround you described in

   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5

doesn't sound too awful to me, given that this problem exists for a
while and is not specific to Eglot.

However, if we still want to have a better solution that will be safe
enough to be installed on emacs-29, I can suggest two alternatives:

 . add a prefix argument to package-update, which would mean to update
   a package unconditionally, even if it is a built-in or of an older
   version, and make this special case be handled by code that is
   completely independent of the code we have in package-update now,
   so that the "normal" case is unaffected; or
 . add a new command, say, package-update-core-package, which will
   then be used only for :core packages

OK?

> M-x package-update and M-x package-update-all are new in Emacs 29.

They might be new, but package-update was virtually unmodified since a
year ago, during which time it was used by many people, and so its
current code can be considered to be well tested.  Your modifications
are by contrast completely new.

> The change I'm proposing it not really "non-trivial".  I can walk
> you or anybody through the code, or write tests if that would
> improve the outlook.

See above.  Given the problems I mentioned, I'm allowed to doubt that
you yourself understand the changes well enough to vouch for them.
And even if you did vouch, my gray hair won't believe you.  So I
prefer to go for much safer, if slightly less clean, changes.  I hope
one of the two alternatives I suggested will be acceptable.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Tue, Apr 11, 2023 at 6:55 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Please reconsider.  If we do this, than Emacs 29 users will be almost
> > locked out of upgrading Eglot and a lot of other built-in packages.
> > I'll have to teach people that workaround in the manual, where such
> > workarounds don't really belong.
>
> OK, I looked closer at the patch and the code involved in this, and
> also re-read this discussion.  I cannot agree with installing your
> patch, as submitted, on the emacs-29 branch, sorry.  It modifies code
> that affects "normal" invocations of package-update, and also numerous
> other functions in package.el (via the change in
> package--updateable-packages),

I don't understand.  package--updateable-packages is an internal helper
that only has two users, both of which I tested.  That's not "numerous".

> in ways that are very hard for me to
> audit.  It is hard to audit because there are parts of it that read
> like some kind of "black magic":
>
> > +         (nonbuiltin (assq package package-alist)))
>
> Why is the return value of assq the sign that the package is
> "nonbuiltin"?

Because package-alist only contains packages that were installed
by the user explicitly.

>
> > +    (cond (nonbuiltin
> > +           (let ((desc (cadr nonbuiltin)))
> > +             (if (package-vc-p desc)
> > +                 (package-vc-update desc)
> > +               (package-delete desc 'force)
> > +               (package-install package 'dont-select))))
> > +          (t
> > +           (package-install
> > +            (cadr (assq package package-archive-contents)))))))
>
> Why the different way of calling package-install for "built-in"
> packages?

1. Because built-in packages cannot be deleted.  2. Because built-in
packages aren't described the same way that explicitly installed packages.
The description of a built-in package is much poorer in information.
To make package-install work with a built-in package, you have to give it
the richer description of the package that you want to install, fresh
from package-archive-contents.

> > -                  (package-desc-version (cadr elt))
> > +                  (if (vectorp (cdr elt))
> > +                      (aref (cdr elt) 0)
> > +                    (package-desc-version (cadr elt)))
>
> What is the significance of the (vectorp (cdr elt)) test?

It tells if the current element being iterated has, in its cdr
an object of type package--bi-desc.  That struct, is implemented
via a vector, and so, unfortunately has no recognizer predicate.

> > -          (package-vc-p (cadr (assq (car elt) package-alist)))))
> > -    package-alist)))
> > +          (and (consp (cdr elt))
> > +               (package-desc-p (cadr elt))
> > +               (package-vc-p (cadr elt)))))
> > +    (seq-union package-alist package--builtins
> > +               (lambda (a b) (eq (car a) (car b)))))))
>
> What is the significance of the (consp (cdr elt)) test?  And why do we
> need to add package--builtins to the list?

package-alist's form is

   ((SYM PACKAGE-DESC)...)

while package--builtins is

   ((SYM . PACKAGE--BI-DESC) ...)

> How am I supposed to assess the safety of this patch, given all this
> semi-obfuscated code, and given that I'm not the every-day maintainer
> of package.el and am not familiar with all the quirks of its code?
> (It is quite possible that this obfuscated nature of the code is not
> your fault, but is caused by how package.el is implemented.  In which
> case I hope that we could clean up the code of package.el on master to
> allow updating :core packages more seamlessly and with simpler code.)

Yes, quite so.  That was my point to Philip earlier: this code is awful
to read, but when you read it, you'll notice that it's not really
rocket science going on there.  That's why I think this is simple enough
patch to go for emacs 29.

I do hope Stefan and Philip can chime in.

Do note that if this change goes to master and not to emacs-29, people
will only be effectively testing the new functionality when the emacs-30
branch is cut.

> OTOH, the workaround you described in
>
>    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5
>
> doesn't sound too awful to me, given that this problem exists for a
> while and is not specific to Eglot.

As I explained, I don't think there were ever :core packages
as popular as Eglot.  There is also the fact that many people are
using non-package.el package managers, which is a maintenance burden
for me.  I always recommend package.el, the official package manager,
since I don't have the resources to learn about those other package
managers (some of which have brought problems in the past).

That workaround is awful to use, BTW.  It's quite slow,
(M-x package-list-packages takes ages, like almost a minute here).
Then you have to C-s and find a million false positives eglot-something
packages and then you have to know the `i` and `x` shortcuts, which
aren't really something Emacs newcomers know about.

On the other hand, M-x package-update gives you a completion
list of the packages you have already.

> See above.  Given the problems I mentioned, I'm allowed to doubt that
> you yourself understand the changes well enough to vouch for them.
> And even if you did vouch, my gray hair won't believe you.  So I
> prefer to go for much safer, if slightly less clean, changes.  I hope
> one of the two alternatives I suggested will be acceptable.

If this change can't go into emacs-29, I think it's better to add
an M-x eglot-update to eglot.el.  That's discoverable, easy to remember
and the absolute safest, as package.el is absolutely unchanged.

(defun eglot-update () "Update Eglot to latest version."
  (interactive)
  (unless package-archive-contents (package-refresh-contents))
  (package-install (assq 'eglot package-archive-contents)))

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 11 Apr 2023 19:31:09 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> > See above.  Given the problems I mentioned, I'm allowed to doubt that
> > you yourself understand the changes well enough to vouch for them.
> > And even if you did vouch, my gray hair won't believe you.  So I
> > prefer to go for much safer, if slightly less clean, changes.  I hope
> > one of the two alternatives I suggested will be acceptable.
> 
> If this change can't go into emacs-29, I think it's better to add
> an M-x eglot-update to eglot.el.

That's the worst of all worlds.

What is the problem with the two possible solutions I suggested?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Tue, Apr 11, 2023, 19:51 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Tue, 11 Apr 2023 19:31:09 +0100
> > Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org,
>
> >       larsi@gnus.org
> >
> > > See above.  Given the problems I mentioned, I'm allowed to doubt that
> > > you yourself understand the changes well enough to vouch for them.
> > > And even if you did vouch, my gray hair won't believe you.  So I
> > > prefer to go for much safer, if slightly less clean, changes.  I hope
> > > one of the two alternatives I suggested will be acceptable.
> >
> > If this change can't go into emacs-29, I think it's better to add
> > an M-x eglot-update to eglot.el.
>
> That's the worst of all worlds.
>

Why? It's the safest option. Absolutely no package.el regression possible,
and doesn't solve a problem where you don't think I've exists.

>
> What is the problem with the two possible solutions I suggested?
>

They are incongruent and not very user friendly IMO.

João

>

[-- Attachment #2: Type: text/html, Size: 2306 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

LOn Tue, Apr 11, 2023, 21:08 João Távora <joaotavora@gmail.com> wrote:

> .
>> >
>> > If this change can't go into emacs-29, I think it's better to add
>> > an M-x eglot-update to eglot.el.
>>
>> That's the worst of all worlds.
>>
>
> Why? It's the safest option. Absolutely no package.el regression possible,
> and doesn't solve a problem where you don't think I've exists.
>

Sorry, I meant to write "where you don't think one exists".

>
>> What is the problem with the two possible solutions I suggested?
>>
>
> They are incongruent and not very user friendly IMO.
>

Admittedly, eglot-update is also incongruent, of course, but at least that
problem is confined to a single package.

João

>

[-- Attachment #2: Type: text/html, Size: 2130 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 11 Apr 2023 21:08:32 +0100
> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
>  > If this change can't go into emacs-29, I think it's better to add
>  > an M-x eglot-update to eglot.el.
> 
>  That's the worst of all worlds.
> 
> Why? It's the safest option. Absolutely no package.el regression possible, and doesn't solve a problem
> where you don't think I've exists.

From your POV of the Eglot maintainer, it might make sense.  But from
my POV, it doesn't: the problem here is general, and a solution is at
hand that will give you what you want and also support all the other
core packages.

>  What is the problem with the two possible solutions I suggested?
> 
> They are incongruent and not very user friendly IMO.

The one which proposed a new command is a natural generalization of
your eglot-update proposal, so I think it is as user-friendly as it
gets.  As for congruency, Stefan just expressed his intuition about
that, and I tend to agree with him: upgrades of core packages should
indeed be handled by separate command(s) with somewhat different rules
and perhaps also a different UI.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Tue, 11 Apr 2023 21:08:32 +0100
>> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org, 
>> 	Lars Ingebrigtsen <larsi@gnus.org>
>> 
>>  > If this change can't go into emacs-29, I think it's better to add
>>  > an M-x eglot-update to eglot.el.
>> 
>>  That's the worst of all worlds.
>> 
>> Why? It's the safest option. Absolutely no package.el regression possible, and doesn't solve a problem
>> where you don't think I've exists.
>
> From your POV of the Eglot maintainer, it might make sense.  But from
> my POV, it doesn't: the problem here is general, and a solution is at
> hand that will give you what you want and also support all the other
> core packages.

Your solution doesn't "give me what I want".  If I add 'eglot-update' it
will work as a single command on every Emacs version from Emacs 26.3
onwards.  This new command you're proposing is for Emacs 29 only (and
will presumably be deprecated soon).

So "what I want(ed)" is for `M-x package-install RET eglot` to keep
working smoothly as it did up to here and forever onwards.  That's want
many package managers do (the "install" verb upgrades, or offers to
upgrade).  That not being possible, I had settled for a decent working
`M-x package-update RET eglot` instead.

So this is "what I want": smooth user experience with no new commands or
at least simple ones that don't require understanding emacs dev
concepts.

>>  What is the problem with the two possible solutions I suggested?
>> 
>> They are incongruent and not very user friendly IMO.
>
> The one which proposed a new command is a natural generalization of
> your eglot-update proposal, so I think it is as user-friendly as it
> gets.  As for congruency, Stefan just expressed his intuition about
> that, and I tend to agree with him: upgrades of core packages should
> indeed be handled by separate command(s) with somewhat different rules
> and perhaps also a different UI.

I think 'package-update-core-package' is just unfortunate, because the
regular user doesn't care what the heck is core, and can you blame her?

I can't stop you from adding it, of course.  Have you thought how it
should behave when the package is no longer a core package, i.e. has
already updated?  I.e. should 'package-update-core-package' update a
non-core package in certain situations?  If so, then we're inches away
from 'M-x package-update-any-package', which is just a fixed `M-x
package-update` as I proposed.  If not, then it's really quite strange
to have to run one command for one update and another command for the
next one.

I have to ask (though I can guess the answer): may I add 'eglot-update'
anyway to emacs-29 as a no-brainer shortcut in the meantime?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Wed, 12 Apr 2023 08:58:23 +0100
> 
> > From your POV of the Eglot maintainer, it might make sense.  But from
> > my POV, it doesn't: the problem here is general, and a solution is at
> > hand that will give you what you want and also support all the other
> > core packages.
> 
> Your solution doesn't "give me what I want".  If I add 'eglot-update' it
> will work as a single command on every Emacs version from Emacs 26.3
> onwards.  This new command you're proposing is for Emacs 29 only (and
> will presumably be deprecated soon).

Since you started by proposing a patch to package.el, that solution
had the same issues with older Emacsen.  IOW, that is a separate
problem.  In particular, Eglot is not bundled in those older Emacsen,
so package.el should support updating Eglot just fine for those older
versions.

> So this is "what I want": smooth user experience with no new commands or
> at least simple ones that don't require understanding emacs dev
> concepts.

If we want the solution be general, rather than having a separate
update command for each core package, then there's a limit to what we
can do to give this smooth user experience to users of older Emacsen.
However, I don't think in this case there's a problem, see above.

> I think 'package-update-core-package' is just unfortunate, because the
> regular user doesn't care what the heck is core, and can you blame her?

I disagree with your assessment.  Moreover, I think such a command is
needed anyway, for reasons other than the ones which prompted this bug
report.

> I can't stop you from adding it, of course.  Have you thought how it
> should behave when the package is no longer a core package, i.e. has
> already updated?

Yes, see my other messages.

> I have to ask (though I can guess the answer): may I add 'eglot-update'
> anyway to emacs-29 as a no-brainer shortcut in the meantime?

I'd prefer not to have package-specific upgrade commands.  I hope we
will soon add to package.el on emacs-29 a new command that will allow
users to update core packages, including Eglot, and that will solve
the problem for users of Emacs 29 and later, where Eglot is bundled.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 9:19 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Your solution doesn't "give me what I want".  If I add 'eglot-update' it
> > will work as a single command on every Emacs version from Emacs 26.3
> > onwards.  This new command you're proposing is for Emacs 29 only (and
> > will presumably be deprecated soon).
>
> Since you started by proposing a patch to package.el, that solution
> had the same issues with older Emacsen.

In onler Emacsen, M-x package-install works.  I was going to also propose
that M-x package-install on Emacs 29 offers to upgrade.  But if even a
"fixed" (IMO, of course) M-x package-update isn't accepted, then I won't
even bother proposing fixing M-x package-install on Emacs 29.

> IOW, that is a separate
> problem.  In particular, Eglot is not bundled in those older Emacsen,
> so package.el should support updating Eglot just fine for those older
> versions.

Yes, but the command that two users using, say, Emacs 28 and Emacs 29
will be different.  And happen to think that's unfortunate.

> > So this is "what I want": smooth user experience with no new commands or
> > at least simple ones that don't require understanding emacs dev
> > concepts.
>
> If we want the solution be general, rather than having a separate
> update command for each core package, then there's a limit to what we
> can do to give this smooth user experience to users of older Emacsen.
> However, I don't think in this case there's a problem, see above.

I don't think this "limit" really exists.  As far as I understand, if
I had detected this problem earlier or your perception of the stability
of my patch for emacs-29 were different, then this problem would be well
on its way to non-existence.

> > I think 'package-update-core-package' is just unfortunate, because the
> > regular user doesn't care what the heck is core, and can you blame her?
>
> I disagree with your assessment.  Moreover, I think such a command is
> needed anyway, for reasons other than the ones which prompted this bug
> report.
>
> > I can't stop you from adding it, of course.  Have you thought how it
> > should behave when the package is no longer a core package, i.e. has
> > already updated?
>
> Yes, see my other messages.

I can't well understand what the behaviour will be, I'll wait to see the
final version.

> > I have to ask (though I can guess the answer): may I add 'eglot-update'
> > anyway to emacs-29 as a no-brainer shortcut in the meantime?
>
> I'd prefer not to have package-specific upgrade commands.  I hope we
> will soon add to package.el on emacs-29 a new command that will allow
> users to update core packages, including Eglot, and that will solve
> the problem for users of Emacs 29 and later, where Eglot is bundled.

FTR I would also 100% also "prefer not to have package-specific upgrade
commands" :-) That would be my preference too. But again, I really have
to ask :-), may I add it or would you revert the commit immediately?
Again, I'm just looking to have a congruent, simple way to tell users
how to ensure latest Eglot version without needing to know if they have
Emacs 29, 27 or 47.  This command, while also not being my first "preference",
fulfills that requirement.  Can we have it?  It's only 4 lines of code.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 09:51:08 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> > > I have to ask (though I can guess the answer): may I add 'eglot-update'
> > > anyway to emacs-29 as a no-brainer shortcut in the meantime?
> >
> > I'd prefer not to have package-specific upgrade commands.  I hope we
> > will soon add to package.el on emacs-29 a new command that will allow
> > users to update core packages, including Eglot, and that will solve
> > the problem for users of Emacs 29 and later, where Eglot is bundled.
> 
> FTR I would also 100% also "prefer not to have package-specific upgrade
> commands" :-) That would be my preference too. But again, I really have
> to ask :-), may I add it or would you revert the commit immediately?

Why do you insist on threats of reverting being the only efficient
means of communications about these matters?  This is supposed to be a
collaborative effort, not a confrontation.

If we are going to have a special command for upgrading core packages
(and it currently looks like we are), then having an additional
Eglot-specific command for a similar purpose would be against our
common goal of having a UX that changes as little as possible, given
the restrictions.  What else can I say to express clearly that I would
like such a command to not be installed?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 11:23 AM Eli Zaretskii <eliz@gnu.org> wrote:

> Why do you insist on threats of reverting being the only efficient
> means of communications about these matters?  This is supposed to be a
> collaborative effort, not a confrontation.

Of course.  But we don't agree Eli, and that's a normal part of
working with someone.  No need to use the words "threads" or
"confrontation", or suggest this nasty ill will on my end. No one
is threatening anybody. I was just asking nicely if I can add the
command: you're the head maintainer, you have to make a decision at
some point.

> If we are going to have a special command for upgrading core packages
> (and it currently looks like we are), then having an additional
> Eglot-specific command for a similar purpose would be against our
> common goal of having a UX that changes as little as possible, given
> the restrictions.  What else can I say to express clearly that I would
> like such a command to not be installed?

I wouldn't want it either, but it seems the least bad of all solutions
here.  Package.el is in my honest opinion very inconsistent already.
M-x package-isntall does one thing, package-update does another, then
there's the "package upgrades" in the menu which use yet different
logic.  I am beginning to understand why people are flocking away to
straight.el and elpaca.el.  Given this poor situation, I think that
adding yet _more_ complexity to package.el UX is worse.

But I don't want to stop you from doing that if you really think it's
the correct way.  No confrontation here, Eli.  And in fact, if I'm wrong
and people love `package-update-core-package` and find it consistent,
then `eglot-update` will just fall into oblivion and be forgotten,
and we can deprecate it just like we did `eglot-manual`.  So please,
work with me and give me the benefit of the doubt that this 4-line
helper command in Eglot won't hurt anybody and might actually
help somebody.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 11:38:35 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> On Wed, Apr 12, 2023 at 11:23 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Why do you insist on threats of reverting being the only efficient
> > means of communications about these matters?  This is supposed to be a
> > collaborative effort, not a confrontation.
> 
> Of course.  But we don't agree Eli, and that's a normal part of
> working with someone.

We are talking here about relative merits of two mostly equivalent
commands, one of which is more general than the other and will allow
users to perform what you want, among other things.  The disagreement
is about how serious is the problem of having the UI change for Emacs
users from Emacs 29 onwards.  (Btw, having a special Eglot command
also constitutes a change in the UI.)  When this kind of matters are
in disagreement (as opposed to design and implementation of package
which you develop and maintain), I expect my opinion to have a
slightly larger weight.

> But I don't want to stop you from doing that if you really think it's
> the correct way.  No confrontation here, Eli.  And in fact, if I'm wrong
> and people love `package-update-core-package` and find it consistent,
> then `eglot-update` will just fall into oblivion and be forgotten,
> and we can deprecate it just like we did `eglot-manual`.  So please,
> work with me and give me the benefit of the doubt that this 4-line
> helper command in Eglot won't hurt anybody and might actually
> help somebody.

I think adding such a command would be a mistake, so I'm asking you
not to add such a command to Emacs.  Please!

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> the restrictions.  What else can I say to express clearly that I would
> like such a command to not be installed?

Had another idea: what about this very tiny patch, then?  It makes `M-x
package-install` work for installing a :core package.  This also rhymes
exactly with Stefan's intution/feeling that :core packages need to be
"installed" to promote them to installable.  The current M-x
package-install recommendation could remain flawlessly and then you can
do whatever you think is best for M-x package-update & friends.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..134505a96af 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2164,8 +2164,10 @@ package-installed-p
        (and pkg-descs
             (version-list-<= min-version
                              (package-desc-version (car pkg-descs)))))
-     ;; Also check built-in packages.
-     (package-built-in-p package min-version)))))
+     ;; Also check built-in packages.  But if min-version is nil, just
+     ;; act as if a package-desc had been passed (bug#62720)
+     (and min-version (package-built-in-p package min-version))))))
 
 (defun package-download-transaction (packages)
   "Download and install all the packages in PACKAGES.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Wed, 12 Apr 2023 12:00:04 +0100
> 
> Had another idea: what about this very tiny patch, then?  It makes `M-x
> package-install` work for installing a :core package.  This also rhymes
> exactly with Stefan's intution/feeling that :core packages need to be
> "installed" to promote them to installable.  The current M-x
> package-install recommendation could remain flawlessly and then you can
> do whatever you think is best for M-x package-update & friends.

This has the same problem: it modifies a function that is called in
too many places.  package-installed-p has half a dozen callers in
package.el alone.  The change is tiny, but what about its
implications on every use case where it is involved?

So I still think a special command for updating a core package will be
a better, safer solution, and the reasons for that are more than just
this one, as mentioned up-thread.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 12:07 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
> >   larsi@gnus.org
> > Date: Wed, 12 Apr 2023 12:00:04 +0100
> >
> > Had another idea: what about this very tiny patch, then?  It makes `M-x
> > package-install` work for installing a :core package.  This also rhymes
> > exactly with Stefan's intution/feeling that :core packages need to be
> > "installed" to promote them to installable.  The current M-x
> > package-install recommendation could remain flawlessly and then you can
> > do whatever you think is best for M-x package-update & friends.
>
> This has the same problem: it modifies a function that is called in
> too many places.  package-installed-p has half a dozen callers in
> package.el alone.  The change is tiny, but what about its
> implications on every use case where it is involved?

What about them?  I volunteer to help test whichever cases you (or
anyone else) think may be problematic.

Can we put this tiny patch on master then?  Maybe there's half a
chance for a backport until the Emacs 29.1 release, or a quarter
of a chance that some future Emacs 29.2 will also get it.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 12:15:11 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> > This has the same problem: it modifies a function that is called in
> > too many places.  package-installed-p has half a dozen callers in
> > package.el alone.  The change is tiny, but what about its
> > implications on every use case where it is involved?
> 
> What about them?  I volunteer to help test whichever cases you (or
> anyone else) think may be problematic.

We will never be able to reveal them all, let alone test them, before
Emacs 29.1 is released.

> Can we put this tiny patch on master then?

Yes, if Philip and Stefan don't object.  But, since there will be a
command for updating core packages, doesn't this go against your
desire not to change the UX?

> Maybe there's half a chance for a backport until the Emacs 29.1
> release, or a quarter of a chance that some future Emacs 29.2 will
> also get it.

I don't think there is such a chance, and I'm not sure we even want
that, given that we are leaning towards a new special command.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 12:21 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 12 Apr 2023 12:15:11 +0100
> > Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org,
> >       larsi@gnus.org
> >
> > > This has the same problem: it modifies a function that is called in
> > > too many places.  package-installed-p has half a dozen callers in
> > > package.el alone.  The change is tiny, but what about its
> > > implications on every use case where it is involved?
> >
> > What about them?  I volunteer to help test whichever cases you (or
> > anyone else) think may be problematic.
>
> We will never be able to reveal them all, let alone test them, before
> Emacs 29.1 is released.

Were there even :core packages back in 2015 when this package-install
safeguard was added in package-installed-p?  I haven't checked, but
I think not. I think this is just a plain oversight in an irregularly
maintained library, if I've ever seen one.  Why not just fix it?
The other use cases of package-installed-p you mention pass a
package-desc structure, and there this safeguard doesn't
even exist (see the docstring).

> > Can we put this tiny patch on master then?
>
> Yes, if Philip and Stefan don't object.  But, since there will be a
> command for updating core packages, doesn't this go against your
> desire not to change the UX?

IMO The user experience is bad/broken.  So at least I want to change it,
if minimally.  That's what any bug report is about, to change UX for the
better.  Here, to let people not to have to go through odd hoops big or
small to install packages or know if something is built-in/core or not.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 12:35:23 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> > > Can we put this tiny patch on master then?
> >
> > Yes, if Philip and Stefan don't object.  But, since there will be a
> > command for updating core packages, doesn't this go against your
> > desire not to change the UX?
> 
> IMO The user experience is bad/broken.

We don't yet have the other command, let alone any user feedback.  So
I don't think you can know this is bad/broken yet.  You may think so,
but I happen to think differently, so let's not behave as if this is a
fact already.

> So at least I want to change it, if minimally.  That's what any bug
> report is about, to change UX for the better.  Here, to let people
> not to have to go through odd hoops big or small to install packages
> or know if something is built-in/core or not.

Using a specialized command for a special job is nowhere near jumping
through hoops.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 12:46 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 12 Apr 2023 12:35:23 +0100
> > Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org,
> >       larsi@gnus.org
> >
> > > > Can we put this tiny patch on master then?
> > >
> > > Yes, if Philip and Stefan don't object.  But, since there will be a
> > > command for updating core packages, doesn't this go against your
> > > desire not to change the UX?
> >
> > IMO The user experience is bad/broken.
>
> We don't yet have the other command, let alone any user feedback.  So
> I don't think you can know this is bad/broken yet.  You may think so,
> but I happen to think differently, so let's not behave as if this is a
> fact already.

I was talking about the current state, obviously.  It's bad/broken,
IMO, as I described the original bug report, and noone really contested
that I think.  As to the new command you're proposing I think we'll both
agree that it's adding _new_ UX to the Emacs 29 pretest, something
which is at least infrequent.

> > So at least I want to change it, if minimally.  That's what any bug
> > report is about, to change UX for the better.  Here, to let people
> > not to have to go through odd hoops big or small to install packages
> > or know if something is built-in/core or not.
>
> Using a specialized command for a special job is nowhere near jumping
> through hoops.

Right.  But we fundamentally disagree that this is a "special job".
There's 0 reason why it should be.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Wed, 12 Apr 2023 12:15:11 +0100
>> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
>> 	larsi@gnus.org
>> 
>> > This has the same problem: it modifies a function that is called in
>> > too many places.  package-installed-p has half a dozen callers in
>> > package.el alone.  The change is tiny, but what about its
>> > implications on every use case where it is involved?
>> 
>> What about them?  I volunteer to help test whichever cases you (or
>> anyone else) think may be problematic.
>
> We will never be able to reveal them all, let alone test them, before
> Emacs 29.1 is released.
>
>> Can we put this tiny patch on master then?
>
> Yes, if Philip and Stefan don't object.  But, since there will be a
> command for updating core packages, doesn't this go against your
> desire not to change the UX?

After thinking about this for a bit, I think that the right approach is
to use package-install instead of writing a separate command.  After
all, this will make the behaviour of package-install consistent with
that of the package menu.

It might work but it should be tested somewhat thoroughly before the
patch is applied.  In the meantime, I just finished a similar approach
that does not modify `package-installed-p', but just adds another
utility function:


[-- Attachment #2: Type: text/plain, Size: 1807 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..5a79c277f0d 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,14 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package-core-p (package)
+  "Return non-nil the built-in version of PACKAGE is loaded."
+  (let ((package (if (package-desc-p package)
+                     (package-desc-name package)
+                   package)))
+    (and (assq package (package--alist))
+         (package-built-in-p package))))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -1908,7 +1916,7 @@ package-compute-transaction
                  (package-version-join (package-desc-version already)))))
       (cond
        (already nil)
-       ((package-installed-p next-pkg next-version) nil)
+       ((package-core-p (car next-pkg))) ;done
 
        (t
         ;; A package is required, but not installed.  It might also be
@@ -2205,11 +2213,9 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (seq-filter
+                     (lambda (elt) (package-core-p (car elt)))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)

[-- Attachment #3: Type: text/plain, Size: 291 bytes --]


>> Maybe there's half a chance for a backport until the Emacs 29.1
>> release, or a quarter of a chance that some future Emacs 29.2 will
>> also get it.
>
> I don't think there is such a chance, and I'm not sure we even want
> that, given that we are leaning towards a new special command.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Philip Kaludercic <philipk@posteo.net> writes:

>>> Can we put this tiny patch on master then?
>>
>> Yes, if Philip and Stefan don't object.  But, since there will be a
>> command for updating core packages, doesn't this go against your
>> desire not to change the UX?
>
> After thinking about this for a bit, I think that the right approach is
> to use package-install instead of writing a separate command.  After
> all, this will make the behaviour of package-install consistent with
> that of the package menu.

+1000

> It might work but it should be tested somewhat thoroughly before the
> patch is applied.  In the meantime, I just finished a similar approach
> that does not modify `package-installed-p', but just adds another
> utility function:

I've just applied it, re-made emacs, tested with Emacs -Q and now M-x
package-install doesn't offer _any_ completions to install, core or
non-core.  Is this what you intended?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1158 bytes --]

João Távora <joaotavora@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>>> Can we put this tiny patch on master then?
>>>
>>> Yes, if Philip and Stefan don't object.  But, since there will be a
>>> command for updating core packages, doesn't this go against your
>>> desire not to change the UX?
>>
>> After thinking about this for a bit, I think that the right approach is
>> to use package-install instead of writing a separate command.  After
>> all, this will make the behaviour of package-install consistent with
>> that of the package menu.
>
> +1000
>
>> It might work but it should be tested somewhat thoroughly before the
>> patch is applied.  In the meantime, I just finished a similar approach
>> that does not modify `package-installed-p', but just adds another
>> utility function:
>
> I've just applied it, re-made emacs, tested with Emacs -Q and now M-x
> package-install doesn't offer _any_ completions to install, core or
> non-core.  Is this what you intended?

No, that was my mistake (I changed something last minute before applying
the patch).  This patch should behave properly:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages.patch --]
[-- Type: text/x-diff, Size: 3497 bytes --]

From 8c742056f2669a21de16488652e114cba8c8147a Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 12 Apr 2023 14:26:39 +0200
Subject: [PATCH] Allow upgrading built-in packages

* lisp/emacs-lisp/package.el (package-core-p): Add new utility
function.
(package-compute-transaction): Check if an installed package is
built-in while resolving dependencies and allow it to be installed.
(package-install): Suggest upgrading built-in packages in the
interactive specification.
---
 lisp/emacs-lisp/package.el | 36 ++++++++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 6 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..91fbbb11f68 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,20 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package-core-p (package)
+  "Return non-nil the built-in version of PACKAGE is loaded.
+This command differs from `package-built-in-p' in that it only
+returns a non-nil value if the user has not installed a more a
+more recent version of the package from a package archive."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -1908,7 +1922,16 @@ package-compute-transaction
                  (package-version-join (package-desc-version already)))))
       (cond
        (already nil)
-       ((package-installed-p next-pkg next-version) nil)
+       ;; If a package is installed, we don't need to continue.
+       ;; Built-in packages constitute an exception, because we want
+       ;; to allow the user to "upgrade" from a built-in version to a
+       ;; potentially newer version available on ELPA (bug#62720).
+       ((and (package-installed-p next-pkg next-version)
+             (not (package-core-p next-pkg))))
+       ;; The pseudo-package Emacs is always installed and built-in.
+       ;; It cannot be upgraded, so we make sure not to proceed beyond
+       ;; this point when resolving dependencies.
+       ((eq next-pkg 'emacs))
 
        (t
         ;; A package is required, but not installed.  It might also be
@@ -2205,11 +2228,12 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (not (package-installed-p (car elt)))
+                                (package-core-p (car elt)))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 1:28 PM Philip Kaludercic <philipk@posteo.net> wrote:

> > I've just applied it, re-made emacs, tested with Emacs -Q and now M-x
> > package-install doesn't offer _any_ completions to install, core or
> > non-core.  Is this what you intended?
>
> No, that was my mistake (I changed something last minute before applying
> the patch).  This patch should behave properly:

Can confirm it does for M-x package-install.

I hope this patch goes far, but maybe rename package-core-p to
package--core-p and there's a duplicate "a more" in its docstring.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>   monnier@iro.umontreal.ca,
>   62720@debbugs.gnu.org,  larsi@gnus.org
> Date: Wed, 12 Apr 2023 12:00:09 +0000
> 
> > Yes, if Philip and Stefan don't object.  But, since there will be a
> > command for updating core packages, doesn't this go against your
> > desire not to change the UX?
> 
> After thinking about this for a bit, I think that the right approach is
> to use package-install instead of writing a separate command.  After
> all, this will make the behaviour of package-install consistent with
> that of the package menu.

Is this for master or for the release branch?

And I thought we all agreed built-in packages need special treatment
anyway, didn't we?  Then why having a separate command is not a
natural next step?

> It might work but it should be tested somewhat thoroughly before the
> patch is applied.  In the meantime, I just finished a similar approach
> that does not modify `package-installed-p', but just adds another
> utility function:

A new utility function is fine by me, even if this is e branch.  But I
don't quite understand how this is supposed to work in package-install
to allow updating built-in packages, and do that in a way that will
not touch the existing code for non-built-in packages in significant
ways (assuming you propose this from the release branch).  Can you
elaborate on that?

> +(defun package-core-p (package)
> +  "Return non-nil the built-in version of PACKAGE is loaded."

Didn't you say the "core" terminology was confusing people?

> +  (let ((package (if (package-desc-p package)
> +                     (package-desc-name package)
> +                   package)))
> +    (and (assq package (package--alist))
> +         (package-built-in-p package))))

It sounds like this doesn't check whether a package is "core", it
checks whether it's built-in and can be updated?  So maybe the name
should be changed to reflect that?  And the doc string as well (what
it means by "is loaded")?

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 3486 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: João Távora <joaotavora@gmail.com>,
>>   monnier@iro.umontreal.ca,
>>   62720@debbugs.gnu.org,  larsi@gnus.org
>> Date: Wed, 12 Apr 2023 12:00:09 +0000
>> 
>> > Yes, if Philip and Stefan don't object.  But, since there will be a
>> > command for updating core packages, doesn't this go against your
>> > desire not to change the UX?
>> 
>> After thinking about this for a bit, I think that the right approach is
>> to use package-install instead of writing a separate command.  After
>> all, this will make the behaviour of package-install consistent with
>> that of the package menu.
>
> Is this for master or for the release branch?

Personally I am indifferent, it should be compatible with both

> And I thought we all agreed built-in packages need special treatment
> anyway, didn't we?  Then why having a separate command is not a
> natural next step?

I don't necessarily agree that "special treatment" requires a separate
command.  I think it is wrong the assume that an built-in package should
automatically be updated to a ELPA package whenever possible.

It might be that I misunderstood something about your long-term plan
where package-install wouldn't suffice.  I'll go re-read the thread to
check.

>> It might work but it should be tested somewhat thoroughly before the
>> patch is applied.  In the meantime, I just finished a similar approach
>> that does not modify `package-installed-p', but just adds another
>> utility function:
>
> A new utility function is fine by me, even if this is e branch.  But I
> don't quite understand how this is supposed to work in package-install
> to allow updating built-in packages, and do that in a way that will
> not touch the existing code for non-built-in packages in significant
> ways (assuming you propose this from the release branch).  Can you
> elaborate on that?

The only reason we couldn't install built-in packages is that when
planning to install packages `package-compute-transaction' believes that
if a built-in package is provided, then everything is fine and we don't
need to proceed with installing any packages.  All I propose is to lift
this assumption, then this works fine.

One point that might be deliberated is that this means all built-in
dependencies are also installed, even if these are not strictly
necessary.  It shouldn't matter that much, since most users would
upgrade them eventually, but worth mentioning I guess? 

>> +(defun package-core-p (package)
>> +  "Return non-nil the built-in version of PACKAGE is loaded."
>
> Didn't you say the "core" terminology was confusing people?

TBH I am not really satisfied with the name (so any other suggestion is
just as fine for me), and as Joao said it would be better to make the
predicate as internal so that users are not expected to deal with it.

>> +  (let ((package (if (package-desc-p package)
>> +                     (package-desc-name package)
>> +                   package)))
>> +    (and (assq package (package--alist))
>> +         (package-built-in-p package))))
>
> It sounds like this doesn't check whether a package is "core", it
> checks whether it's built-in and can be updated?  So maybe the name
> should be changed to reflect that?  And the doc string as well (what
> it means by "is loaded")?

Right the "loaded" doesn't make sense.  How about this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages.patch --]
[-- Type: text/x-diff, Size: 3577 bytes --]

From 12e0b209992675a042112e790571d427a003c30d Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 12 Apr 2023 14:26:39 +0200
Subject: [PATCH] Allow upgrading built-in packages

* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
utility predicate.
(package-compute-transaction): Check if an installed package is
built-in while resolving dependencies and allow it to be installed.
(package-install): Suggest upgrading built-in packages in the
interactive specification.  Allow upgrading built-in packages
---
 lisp/emacs-lisp/package.el | 36 ++++++++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 6 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..2cf98290bba 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,20 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Check if a built-in PACKAGE can be upgraded.
+This command differs from `package-built-in-p' in that it only
+returns a non-nil value if the user has not installed a more
+recent version of the package from a package archive."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -1908,7 +1922,16 @@ package-compute-transaction
                  (package-version-join (package-desc-version already)))))
       (cond
        (already nil)
-       ((package-installed-p next-pkg next-version) nil)
+       ;; If a package is installed, we don't need to continue.
+       ;; Built-in packages constitute an exception, because we want
+       ;; to allow the user to "upgrade" from a built-in version to a
+       ;; potentially newer version available on ELPA (bug#62720).
+       ((and (not (package--upgradable-built-in-p next-pkg))
+             (package-installed-p next-pkg next-version)))
+       ;; The pseudo-package Emacs is always installed and built-in.
+       ;; It cannot be upgraded, so we make sure not to proceed beyond
+       ;; this point when resolving dependencies.
+       ((eq next-pkg 'emacs))
 
        (t
         ;; A package is required, but not installed.  It might also be
@@ -2205,11 +2228,12 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (package--upgradable-built-in-p (car elt))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> +(defun package-core-p (package)
>>> +  "Return non-nil the built-in version of PACKAGE is loaded."
>>
>> Didn't you say the "core" terminology was confusing people?
>
> TBH I am not really satisfied with the name (so any other suggestion is
> just as fine for me), and as Joao said it would be better to make the
> predicate as internal so that users are not expected to deal with it.

Beside "core" and "built-in" (with or without dash), another name that's
been used over the years is "bundled".  I don't have a strong preference
between those.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 3:11 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> >>> +(defun package-core-p (package)
> >>> +  "Return non-nil the built-in version of PACKAGE is loaded."
> >>
> >> Didn't you say the "core" terminology was confusing people?
> >
> > TBH I am not really satisfied with the name (so any other suggestion is
> > just as fine for me), and as Joao said it would be better to make the
> > predicate as internal so that users are not expected to deal with it.
>
> Beside "core" and "built-in" (with or without dash), another name that's
> been used over the years is "bundled".  I don't have a strong preference
> between those.

Isn't "bundled" better left for when non-core, non-builtin ELPA packages
are somehow thrown together with the Emacs release tarball or the
platform-specific
builds we make of them?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Isn't "bundled" better left for when non-core, non-builtin ELPA
> packages are somehow thrown together with the Emacs release tarball or
> the platform-specific builds we make of them?

I can't see any reason why that would make a difference here.

AFAICT this only matters in terms of which Git repository/branch holds
the source code and how it gets included into the Emacs distribution,
but after that it's an "internal detail".


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 3:17 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > Isn't "bundled" better left for when non-core, non-builtin ELPA
> > packages are somehow thrown together with the Emacs release tarball or
> > the platform-specific builds we make of them?
>
> I can't see any reason why that would make a difference here.
>
> AFAICT this only matters in terms of which Git repository/branch holds
> the source code and how it gets included into the Emacs distribution,
> but after that it's an "internal detail".

Fair enough, and lets hope updating them will be easy.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Wed, 12 Apr 2023 13:42:56 +0000
> 
> >> After thinking about this for a bit, I think that the right approach is
> >> to use package-install instead of writing a separate command.  After
> >> all, this will make the behaviour of package-install consistent with
> >> that of the package menu.
> >
> > Is this for master or for the release branch?
> 
> Personally I am indifferent, it should be compatible with both

If we want to install this on the release branch, the changes must be
very safe, which in this case basically means "do not touch any code
used when updating non-core packages".

If the patch you presented is all there is to it, then I'm afraid it
doesn't satisfy the above condition, because it does affect the use
case of updating an ELPA package that is not in core.  So I cannot
agree to installing this on emacs-29 in the form you posted, sorry.

> > And I thought we all agreed built-in packages need special treatment
> > anyway, didn't we?  Then why having a separate command is not a
> > natural next step?
> 
> I don't necessarily agree that "special treatment" requires a separate
> command.

Even if you don't agree with that in general, having a separate
command would allow us to install that command on emacs-29 without any
fears.  So that alone is a significant advantage, even if the rest are
not yet agreed upon.

> I think it is wrong the assume that an built-in package should
> automatically be updated to a ELPA package whenever possible.

This seems to be an argument in favor of a separate command?  Or what
did you mean by that, and how is it related to the issue at hand?

> >> It might work but it should be tested somewhat thoroughly before the
> >> patch is applied.  In the meantime, I just finished a similar approach
> >> that does not modify `package-installed-p', but just adds another
> >> utility function:
> >
> > A new utility function is fine by me, even if this is e branch.  But I
> > don't quite understand how this is supposed to work in package-install
> > to allow updating built-in packages, and do that in a way that will
> > not touch the existing code for non-built-in packages in significant
> > ways (assuming you propose this from the release branch).  Can you
> > elaborate on that?
> 
> The only reason we couldn't install built-in packages is that when
> planning to install packages `package-compute-transaction' believes that
> if a built-in package is provided, then everything is fine and we don't
> need to proceed with installing any packages.  All I propose is to lift
> this assumption, then this works fine.

My problem is _how_ to lift this assumption.  The way you propose
doing that affects updating non-core packages in ways that we will not
have enough time to test well enough, not compared to the year that
these commands exist in package.el and were used by many people in
many cases.  So we have the following alternatives for the way
forward:

 . install your changes on master only, and leave the problem of
   updating a core package unsolved in Emacs 29 (with the workaround
   mentioned in the beginning of this bug's discussion available to
   alleviate the problem to some extent)
 . come up with safer changes for package-install that could be
   installed on emacs-29
 . add a new command for updating a core package, which can then be
   safely installed on emacs-29

The last 2 alternatives can be for emacs-29 only, whereas on master we
install your proposed change (or something else).

For the 2nd alternative above to be acceptable, the added/modified
code must be completely separate from the code we have now, so that
any possibility of its destabilizing the current code could be
eliminated.  It could be a separate code, triggered by the prefix
argument, for example.

> One point that might be deliberated is that this means all built-in
> dependencies are also installed, even if these are not strictly
> necessary.  It shouldn't matter that much, since most users would
> upgrade them eventually, but worth mentioning I guess? 

That just confirms my fears that we are opening a Pandora's box.  We
have no idea what this will do, and no time to test the results.
Unintended consequences are abundant.  We must draw any such
consequences to the absolute minimum, at least the way the commands
work by default.  Even if the result is less than elegant.

> >> +(defun package-core-p (package)
> >> +  "Return non-nil the built-in version of PACKAGE is loaded."
> >
> > Didn't you say the "core" terminology was confusing people?
> 
> TBH I am not really satisfied with the name (so any other suggestion is
> just as fine for me), and as Joao said it would be better to make the
> predicate as internal so that users are not expected to deal with it.

The name should still be self-explanatory enough, because we the Emacs
maintainers will read this code and should be able to understand what
the function does without reading is source every time.

> 
> >> +  (let ((package (if (package-desc-p package)
> >> +                     (package-desc-name package)
> >> +                   package)))
> >> +    (and (assq package (package--alist))
> >> +         (package-built-in-p package))))
> >
> > It sounds like this doesn't check whether a package is "core", it
> > checks whether it's built-in and can be updated?  So maybe the name
> > should be changed to reflect that?  And the doc string as well (what
> > it means by "is loaded")?
> 
> Right the "loaded" doesn't make sense.  How about this:
> 
> +(defun package--upgradable-built-in-p (package)
> +  "Check if a built-in PACKAGE can be upgraded.
> +This command differs from `package-built-in-p' in that it only
> +returns a non-nil value if the user has not installed a more
> +recent version of the package from a package archive."

Note that what the doc string says is not what the name tells us.
"Upgradeable" and "user has not installed a more recent version" are
not the same.  What the doc string says calls for a name like
package--built-in-and-up-to-date or something.

> +  (and (not (assq (cond
> +                   ((package-desc-p package)
> +                    (package-desc-name package))
> +                   ((stringp package) (intern package))
> +                   ((symbolp package) package)
> +                   ((error "Unknown package format: %S" package)))
> +                  (package--alist)))
> +       (package-built-in-p package)))

Why do we need all these conditions, where we didn't need them in the
current code?

Also, package-alist is documented as "alist of all packages available
for activation", so it is not clear how the fact that assq returns nil
is evidence that "the user has not installed a more recent version".
Looking at what package--alist and package-load-all-descriptors do, it
looks like they just collect packages that were downloaded into the
relevant directories?  Is that enough to consider any package not in
the list to be "not installed"?  And what about the "more recent
version" condition -- this doesn't seem to be tested anywhere in
package--alist?

The above questions and undocumented subtleties is what scares me in
installing such changes at this late stage.  I'm not sure everyone
involved, yourself included, have a clear understanding of what the
modified code will do in each possible use case.  That is why I very
much prefer separate code, which will then free us from the need of
considering all these subtleties, as the last year of user's
experience with this code can vouch that it does its job correctly, by
and large.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 4:17 PM Eli Zaretskii <eliz@gnu.org> wrote:

> The above questions and undocumented subtleties is what scares me in
> installing such changes at this late stage.  I'm not sure everyone
> involved, yourself included, have a clear understanding of what the
> modified code will do in each possible use case.  That is why I very
> much prefer separate code, which will then free us from the need of
> considering all these subtleties, as the last year of user's
> experience with this code can vouch that it does its job correctly, by
> and large.

Alright, I've tried my hand at making this clean separation, so that
no logic of transaction or existing predicates is touched.  I tried to
make it as intelligible as possible perhaps overdoing the commentary
and the naming, but we can always trim it.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 685f983e285..51d633131d9 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2178,6 +2178,24 @@ package--archives-initialize
   (unless package-archive-contents
     (package-refresh-contents)))

+(defun package--vestal-builtins-available-for-installation ()
+  "Return built-in packages that can but have never been updated.
+The return value is a subset of `package-archive-contents', i.e.
+a list ((SYM PACKAGE-DESC) ...) where SYM names such a built-in
+package.  For bug#62720.
+
+Because we reject packages that are in `package--alist', this set
+is guaranteed to _not_ intersect with the subset of
+`package-archive-contents', which verifies `package-installed-p'.
+which, for historical reasons (and in suspicious conditions) says
+that built-in packages as installed."
+  (let (res)
+    (dolist (bi package--builtins)
+      (when-let ((probe (and (not (assq (car bi) (package--alist)))
+                             (assq (car bi) package-archive-contents))))
+        (push probe res)))
+    res))
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
@@ -2196,19 +2214,27 @@ package-install
 to install it but still mark it as selected."
   (interactive
    (progn
-     ;; Initialize the package system to get the list of package
-     ;; symbols for completion.
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (append
+                     (delq nil
+                           (mapcar (lambda (elt)
+                                     (unless (package-installed-p (car elt))
+                                       (symbol-name (car elt))))
+                                   package-archive-contents))
+                     (mapcar #'car
+
(package--vestal-builtins-available-for-installation)))
                     nil t))
            nil)))
   (package--archives-initialize)
+  ;; See docstring and code of
+  ;; `package--vestal-builtins-available-for-installation' to
+  ;; understand how this code can only kick in if indeed the user
+  ;; selected a such a package from the completion list. bug#62720.
+  (when-let ((name-and-desc
+              (assq pkg
(package--vestal-builtins-available-for-installation))))
+    (setq pkg (cadr name-and-desc)))
   (add-hook 'post-command-hook #'package-menu--post-refresh)
   (let ((name (if (package-desc-p pkg)
                   (package-desc-name pkg)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 5:13 PM João Távora <joaotavora@gmail.com> wrote:
> > modified code will do in each possible use case.  That is why I very
> > much prefer separate code, which will then free us from the need of
> Alright, I've tried my hand at making this clean separation, so that
> no logic of transaction or existing predicates is touched.  I tried to

>  (defun package-install (pkg &optional dont-select)
>    "Install the package PKG.
> @@ -2196,19 +2214,27 @@ package-install
>  to install it but still mark it as selected."
>    (interactive
>     (progn
> -     ;; Initialize the package system to get the list of package
> -     ;; symbols for completion.

Didn't mean to take out this comment, sorry, was an oversight.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 17:13:27 +0100
> Cc: Philip Kaludercic <philipk@posteo.net>, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> On Wed, Apr 12, 2023 at 4:17 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > The above questions and undocumented subtleties is what scares me in
> > installing such changes at this late stage.  I'm not sure everyone
> > involved, yourself included, have a clear understanding of what the
> > modified code will do in each possible use case.  That is why I very
> > much prefer separate code, which will then free us from the need of
> > considering all these subtleties, as the last year of user's
> > experience with this code can vouch that it does its job correctly, by
> > and large.
> 
> Alright, I've tried my hand at making this clean separation, so that
> no logic of transaction or existing predicates is touched.  I tried to
> make it as intelligible as possible perhaps overdoing the commentary
> and the naming, but we can always trim it.

Thanks, but this is not separate code.  It adds 21 built-in packages
to the list of completion candidates that the user can choose in
package-install, and who knows what kind of confusion this could cause
when users see packages like Xref and Package and Tramp and seq and
Python and Org in that list, and what accidents that could cause if
users select one of those by mistake, because they never saw them
there before?

PLEASE show me a completely separate code or a separate command, then
I will agree to this last-minute addition.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2286 bytes --]

On Wed, Apr 12, 2023, 17:52 Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 12 Apr 2023 17:13:27 +0100
> > Cc: Philip Kaludercic <philipk@posteo.net>, monnier@iro.umontreal.ca,
> 62720@debbugs.gnu.org,
> >       larsi@gnus.org
> >
> > On Wed, Apr 12, 2023 at 4:17 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > The above questions and undocumented subtleties is what scares me in
> > > installing such changes at this late stage.  I'm not sure everyone
> > > involved, yourself included, have a clear understanding of what the
> > > modified code will do in each possible use case.  That is why I very
> > > much prefer separate code, which will then free us from the need of
> > > considering all these subtleties, as the last year of user's
> > > experience with this code can vouch that it does its job correctly, by
> > > and large.
> >
> > Alright, I've tried my hand at making this clean separation, so that
> > no logic of transaction or existing predicates is touched.  I tried to
> > make it as intelligible as possible perhaps overdoing the commentary
> > and the naming, but we can always trim it.
>
> Thanks, but this is not separate code.  It adds 21 built-in packages
> to the list of completion candidates that the user can choose in
> package-install, and who knows what kind of confusion this could cause
> when users see packages like Xref and Package and Tramp and seq and
> Python and Org in that list, and what accidents that could cause if
> users select one of those by mistake, because they never saw them
> there before?
>

The intended behavior, of course. To install a newer version of the
package, of course. This is also what Philip's patch did, only with
different code.

PLEASE show me a completely separate code or a separate command, then
> I will agree to this last-minute addition.
>

The point is to change the broken behavior of the existing command,
package-install. Anything else is a waste of time, and noone except you has
demonstrated support for this separate command idea. Please, in normal
non-shouting case, explain to me how you think that the behavior of an
existing command can be changed with "completely separate code".

João

>

[-- Attachment #2: Type: text/html, Size: 3663 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 18:14:28 +0100
> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org, 
> 	Lars Ingebrigtsen <larsi@gnus.org>
> 
> Please, in normal non-shouting case, explain to me how you think that the behavior of an existing
> command can be changed with "completely separate code". 

I already did: either (1) add a prefix argument to an existing
command, which will then trigger the new behavior, or (2) add a
separate command.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 6:22 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 12 Apr 2023 18:14:28 +0100
> > Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org,
> >       Lars Ingebrigtsen <larsi@gnus.org>
> >
> > Please, in normal non-shouting case, explain to me how you think that the behavior of an existing
> > command can be changed with "completely separate code".
>
> I already did: either (1) add a prefix argument to an existing
> command, which will then trigger the new behavior, or (2) add a
> separate command.

We're just going in circles.  Those ideas don't solve this bug,
as hinted by the fact that noone is coming up with the patches
you want.
I guess people with `(package-install 'eglot)` in their configs
will just have to cope with whatever comes of this. As for me,
I've put too way too much time and effort into this already, so
good luck to all.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 12 Apr 2023 18:43:35 +0100
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org
> 
> As for me, I've put too way too much time and effort into this
> already, so good luck to all.

Thank you for your efforts.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Wed, 12 Apr 2023 18:14:28 +0100
>> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier
>> <monnier@iro.umontreal.ca>, 62720@debbugs.gnu.org,
>> 	Lars Ingebrigtsen <larsi@gnus.org>
>> 
>> Please, in normal non-shouting case, explain to me how you think
>> that the behavior of an existing
>> command can be changed with "completely separate code". 
>
> I already did: either (1) add a prefix argument to an existing
> command, which will then trigger the new behavior, or (2) add a
> separate command.

Here you have (1):


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages.patch --]
[-- Type: text/x-diff, Size: 4179 bytes --]

From 09a9e769ed0c7024c33bc135c57894daff8764af Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 12 Apr 2023 14:26:39 +0200
Subject: [PATCH] Allow upgrading built-in packages

* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
utility predicate.
(package-compute-transaction): Check if an installed package is
built-in while resolving dependencies and allow it to be installed.
(package-install): Suggest upgrading built-in packages in the
interactive specification, when invoked with a prefix argument.  Allow
upgrading built-in packages
---
 lisp/emacs-lisp/package.el | 41 +++++++++++++++++++++++++++++++-------
 1 file changed, 34 insertions(+), 7 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..03e63ee7227 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,20 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Check if a built-in PACKAGE can be upgraded.
+This command differs from `package-built-in-p' in that it only
+returns a non-nil value if the user has not installed a more
+recent version of the package from a package archive."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -1908,7 +1922,16 @@ package-compute-transaction
                  (package-version-join (package-desc-version already)))))
       (cond
        (already nil)
-       ((package-installed-p next-pkg next-version) nil)
+       ;; If a package is installed, we don't need to continue.
+       ;; Built-in packages constitute an exception, because we want
+       ;; to allow the user to "upgrade" from a built-in version to a
+       ;; potentially newer version available on ELPA (bug#62720).
+       ((and (not (package--upgradable-built-in-p next-pkg))
+             (package-installed-p next-pkg next-version)))
+       ;; The pseudo-package Emacs is always installed and built-in.
+       ;; It cannot be upgraded, so we make sure not to proceed beyond
+       ;; this point when resolving dependencies.
+       ((eq next-pkg 'emacs))
 
        (t
         ;; A package is required, but not installed.  It might also be
@@ -2187,7 +2210,9 @@ package-install
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
 available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+called interactively, prompt for the package name.  When invoked
+with a prefix argument, the prompt will include built-in packages
+that can be upgraded via an archive.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2205,11 +2230,13 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and current-prefix-arg
+                                     (package--upgradable-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>   monnier@iro.umontreal.ca,
>   62720@debbugs.gnu.org,  larsi@gnus.org
> Date: Wed, 12 Apr 2023 19:39:20 +0000
> 
> >> Please, in normal non-shouting case, explain to me how you think
> >> that the behavior of an existing
> >> command can be changed with "completely separate code". 
> >
> > I already did: either (1) add a prefix argument to an existing
> > command, which will then trigger the new behavior, or (2) add a
> > separate command.
> 
> Here you have (1):

Thanks.  This is almost on-target, but it modifies
package-compute-transaction.  Is that necessary?

> +(defun package--upgradable-built-in-p (package)
> +  "Check if a built-in PACKAGE can be upgraded.
> +This command differs from `package-built-in-p' in that it only
   ^^^^^^^^^^^^
This is not a command, this is a function.

Also, the name has a problem I pointed out earlier in this discussion:
"upgradeable" does not tell well enough what the function tests.

> @@ -2187,7 +2210,9 @@ package-install
>    "Install the package PKG.
>  PKG can be a `package-desc' or a symbol naming one of the
>  available packages in an archive in `package-archives'.  When
> -called interactively, prompt for the package name.
> +called interactively, prompt for the package name.  When invoked
> +with a prefix argument, the prompt will include built-in packages
> +that can be upgraded via an archive.

I wonder whether an invocation with the prefix argument should include
_only_ built-in packages in the prompt?  This could be a useful
feature regardless, and so would allow us to keep this option for
future uses.

Finally, there's still discussion going on whether built-in packages
should be handled only by package-update, not by package-install,
since built-in packages are always "installed".  WDYT?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: João Távora <joaotavora@gmail.com>,
>>   monnier@iro.umontreal.ca,
>>   62720@debbugs.gnu.org,  larsi@gnus.org
>> Date: Wed, 12 Apr 2023 19:39:20 +0000
>> 
>> >> Please, in normal non-shouting case, explain to me how you think
>> >> that the behavior of an existing
>> >> command can be changed with "completely separate code". 
>> >
>> > I already did: either (1) add a prefix argument to an existing
>> > command, which will then trigger the new behavior, or (2) add a
>> > separate command.
>> 
>> Here you have (1):
>
> Thanks.  This is almost on-target, but it modifies
> package-compute-transaction.  Is that necessary?

I have found an alternative that doesn't change the way
`package-compute-transaction' works, but requires a small change in
`package-install':


[-- Attachment #2: Type: text/plain, Size: 3483 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..461e92f27d8 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2187,7 +2198,9 @@ package-install
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
 available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+called interactively, prompt for the package name.  When invoked
+with a prefix argument, the prompt will include built-in packages
+that can be upgraded via an archive.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2205,11 +2218,13 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and current-prefix-arg
+                                     (package--upgradable-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2221,11 +2236,16 @@ package-install
       (package--save-selected-packages
        (cons name package-selected-packages)))
     (if-let* ((transaction
-               (if (package-desc-p pkg)
-                   (unless (package-installed-p pkg)
-                     (package-compute-transaction (list pkg)
-                                                  (package-desc-reqs pkg)))
-                 (package-compute-transaction () (list (list pkg))))))
+               (cond
+                ((package--upgradable-built-in-p pkg)
+                 (let ((desc (cadr (assq name package-archive-contents))))
+                   (package-compute-transaction
+                    (list desc) (package-desc-reqs desc))))
+                ((package-desc-p pkg)
+                 (and (not (package-installed-p pkg))
+                      (package-compute-transaction
+                       (list pkg) (package-desc-reqs pkg))))
+                ((package-compute-transaction () (list (list pkg)))))))
         (progn
           (package-download-transaction transaction)
           (package--quickstart-maybe-refresh)

[-- Attachment #3: Type: text/plain, Size: 1893 bytes --]


The idea here is that if we detect that a package is built-in, we
"pre-compute" part of the transaction and resolve the rest.  This does
not install any unnecessary dependencies.

When `package-install' is invoked interactively without a prefix
argument, the "new" code cannot run, since none of the completion
candidates satisfy `package--upgradable-built-in-p' (or whatever we end
up calling the predicate).  Note that (package-install 'eglot) does
download code, but I believe that this is the correct approach and would
align with what João wanted.

>> +(defun package--upgradable-built-in-p (package)
>> +  "Check if a built-in PACKAGE can be upgraded.
>> +This command differs from `package-built-in-p' in that it only
>    ^^^^^^^^^^^^
> This is not a command, this is a function.

I will address these issues as soon as we have working code.

> Also, the name has a problem I pointed out earlier in this discussion:
> "upgradeable" does not tell well enough what the function tests.
>
>> @@ -2187,7 +2210,9 @@ package-install
>>    "Install the package PKG.
>>  PKG can be a `package-desc' or a symbol naming one of the
>>  available packages in an archive in `package-archives'.  When
>> -called interactively, prompt for the package name.
>> +called interactively, prompt for the package name.  When invoked
>> +with a prefix argument, the prompt will include built-in packages
>> +that can be upgraded via an archive.
>
> I wonder whether an invocation with the prefix argument should include
> _only_ built-in packages in the prompt?  This could be a useful
> feature regardless, and so would allow us to keep this option for
> future uses.
>
> Finally, there's still discussion going on whether built-in packages
> should be handled only by package-update, not by package-install,
> since built-in packages are always "installed".  WDYT?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 07:38:28 +0000
> 
> > Thanks.  This is almost on-target, but it modifies
> > package-compute-transaction.  Is that necessary?
> 
> I have found an alternative that doesn't change the way
> `package-compute-transaction' works, but requires a small change in
> `package-install':

Thanks.

> @@ -2205,11 +2218,13 @@ package-install
>       (package--archives-initialize)
>       (list (intern (completing-read
>                      "Install package: "
> -                    (delq nil
> -                          (mapcar (lambda (elt)
> -                                    (unless (package-installed-p (car elt))
> -                                      (symbol-name (car elt))))
> -                                  package-archive-contents))
> +                    (mapcan
> +                     (lambda (elt)
> +                       (and (or (and current-prefix-arg
> +                                     (package--upgradable-built-in-p (car elt)))
> +                                (not (package-installed-p (car elt))))
> +                            (list (car elt))))
> +                     package-archive-contents)

Why did the original code use symbol-name, but the new one doesn't?

> @@ -2221,11 +2236,16 @@ package-install
>        (package--save-selected-packages
>         (cons name package-selected-packages)))
>      (if-let* ((transaction
> -               (if (package-desc-p pkg)
> -                   (unless (package-installed-p pkg)
> -                     (package-compute-transaction (list pkg)
> -                                                  (package-desc-reqs pkg)))
> -                 (package-compute-transaction () (list (list pkg))))))
> +               (cond
> +                ((package--upgradable-built-in-p pkg)
> +                 (let ((desc (cadr (assq name package-archive-contents))))
> +                   (package-compute-transaction
> +                    (list desc) (package-desc-reqs desc))))
> +                ((package-desc-p pkg)
> +                 (and (not (package-installed-p pkg))
> +                      (package-compute-transaction
> +                       (list pkg) (package-desc-reqs pkg))))
> +                ((package-compute-transaction () (list (list pkg)))))))

I think the first condition of 'cond' should be

    ((and current-prefix-arg (package--upgradable-built-in-p pkg))

to make sure we don't affect the non-prefix-arg invocations in any
way.

> Note that (package-install 'eglot) does download code, but I believe
> that this is the correct approach and would align with what João
> wanted.

I'm not sure I follow: which code does the above download?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 07:38:28 +0000
>> 
>> > Thanks.  This is almost on-target, but it modifies
>> > package-compute-transaction.  Is that necessary?
>> 
>> I have found an alternative that doesn't change the way
>> `package-compute-transaction' works, but requires a small change in
>> `package-install':
>
> Thanks.
>
>> @@ -2205,11 +2218,13 @@ package-install
>>       (package--archives-initialize)
>>       (list (intern (completing-read
>>                      "Install package: "
>> -                    (delq nil
>> -                          (mapcar (lambda (elt)
>> -                                    (unless (package-installed-p (car elt))
>> -                                      (symbol-name (car elt))))
>> -                                  package-archive-contents))
>> +                    (mapcan
>> +                     (lambda (elt)
>> +                       (and (or (and current-prefix-arg
>> +                                     (package--upgradable-built-in-p (car elt)))
>> +                                (not (package-installed-p (car elt))))
>> +                            (list (car elt))))
>> +                     package-archive-contents)
>
> Why did the original code use symbol-name, but the new one doesn't?

To my knowledge, completing-read given a collection of symbols will use
the symbol names as candidates, or is this more complicated?

>> @@ -2221,11 +2236,16 @@ package-install
>>        (package--save-selected-packages
>>         (cons name package-selected-packages)))
>>      (if-let* ((transaction
>> -               (if (package-desc-p pkg)
>> -                   (unless (package-installed-p pkg)
>> -                     (package-compute-transaction (list pkg)
>> -                                                  (package-desc-reqs pkg)))
>> -                 (package-compute-transaction () (list (list pkg))))))
>> +               (cond
>> +                ((package--upgradable-built-in-p pkg)
>> +                 (let ((desc (cadr (assq name package-archive-contents))))
>> +                   (package-compute-transaction
>> +                    (list desc) (package-desc-reqs desc))))
>> +                ((package-desc-p pkg)
>> +                 (and (not (package-installed-p pkg))
>> +                      (package-compute-transaction
>> +                       (list pkg) (package-desc-reqs pkg))))
>> +                ((package-compute-transaction () (list (list pkg)))))))
>
> I think the first condition of 'cond' should be
>
>     ((and current-prefix-arg (package--upgradable-built-in-p pkg))
>
> to make sure we don't affect the non-prefix-arg invocations in any
> way.

The issue here is that this breaks the non-interactive invocations like
(package-install 'eglot), unless they invoke the function while binding
`current-prefix-arg', which I don't think is a common practice.

>> Note that (package-install 'eglot) does download code, but I believe
>> that this is the correct approach and would align with what João
>> wanted.
>
> I'm not sure I follow: which code does the above download?

I did not change any of the code that downloads anything, all this does
is prompt the user for built-in packages when invoked interactively with
a prefix argument and if package-install is invoked with a built-in
package, then it will switch to the ELPA version.  This will not happen
in interactive usage, since `completing-read' is called with
REQUIRE-MATCH.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 11:23:12 +0000
> 
> > Why did the original code use symbol-name, but the new one doesn't?
> 
> To my knowledge, completing-read given a collection of symbols will use
> the symbol names as candidates, or is this more complicated?

So you are saying that the symbol-name call in the original code was
simply redundant?

> >> @@ -2221,11 +2236,16 @@ package-install
> >>        (package--save-selected-packages
> >>         (cons name package-selected-packages)))
> >>      (if-let* ((transaction
> >> -               (if (package-desc-p pkg)
> >> -                   (unless (package-installed-p pkg)
> >> -                     (package-compute-transaction (list pkg)
> >> -                                                  (package-desc-reqs pkg)))
> >> -                 (package-compute-transaction () (list (list pkg))))))
> >> +               (cond
> >> +                ((package--upgradable-built-in-p pkg)
> >> +                 (let ((desc (cadr (assq name package-archive-contents))))
> >> +                   (package-compute-transaction
> >> +                    (list desc) (package-desc-reqs desc))))
> >> +                ((package-desc-p pkg)
> >> +                 (and (not (package-installed-p pkg))
> >> +                      (package-compute-transaction
> >> +                       (list pkg) (package-desc-reqs pkg))))
> >> +                ((package-compute-transaction () (list (list pkg)))))))
> >
> > I think the first condition of 'cond' should be
> >
> >     ((and current-prefix-arg (package--upgradable-built-in-p pkg))
> >
> > to make sure we don't affect the non-prefix-arg invocations in any
> > way.
> 
> The issue here is that this breaks the non-interactive invocations like
> (package-install 'eglot), unless they invoke the function while binding
> `current-prefix-arg', which I don't think is a common practice.

Then let's add another optional argument, and let prefix arg set it.
Would that resolve this issue?

> >> Note that (package-install 'eglot) does download code, but I believe
> >> that this is the correct approach and would align with what João
> >> wanted.
> >
> > I'm not sure I follow: which code does the above download?
> 
> I did not change any of the code that downloads anything, all this does
> is prompt the user for built-in packages when invoked interactively with
> a prefix argument and if package-install is invoked with a built-in
> package, then it will switch to the ELPA version.  This will not happen
> in interactive usage, since `completing-read' is called with
> REQUIRE-MATCH.

So you are saying that non-interactive calls to package-install could
install Eglot from ELPA even without the changes, is that right?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 11:23:12 +0000
>> 
>> > Why did the original code use symbol-name, but the new one doesn't?
>> 
>> To my knowledge, completing-read given a collection of symbols will use
>> the symbol names as candidates, or is this more complicated?
>
> So you are saying that the symbol-name call in the original code was
> simply redundant?

Yes, it appears so.

>> >> @@ -2221,11 +2236,16 @@ package-install
>> >>        (package--save-selected-packages
>> >>         (cons name package-selected-packages)))
>> >>      (if-let* ((transaction
>> >> -               (if (package-desc-p pkg)
>> >> -                   (unless (package-installed-p pkg)
>> >> -                     (package-compute-transaction (list pkg)
>> >> -                                                  (package-desc-reqs pkg)))
>> >> -                 (package-compute-transaction () (list (list pkg))))))
>> >> +               (cond
>> >> +                ((package--upgradable-built-in-p pkg)
>> >> +                 (let ((desc (cadr (assq name package-archive-contents))))
>> >> +                   (package-compute-transaction
>> >> +                    (list desc) (package-desc-reqs desc))))
>> >> +                ((package-desc-p pkg)
>> >> +                 (and (not (package-installed-p pkg))
>> >> +                      (package-compute-transaction
>> >> +                       (list pkg) (package-desc-reqs pkg))))
>> >> +                ((package-compute-transaction () (list (list pkg)))))))
>> >
>> > I think the first condition of 'cond' should be
>> >
>> >     ((and current-prefix-arg (package--upgradable-built-in-p pkg))
>> >
>> > to make sure we don't affect the non-prefix-arg invocations in any
>> > way.
>> 
>> The issue here is that this breaks the non-interactive invocations like
>> (package-install 'eglot), unless they invoke the function while binding
>> `current-prefix-arg', which I don't think is a common practice.
>
> Then let's add another optional argument, and let prefix arg set it.
> Would that resolve this issue?

That could solve it, but a user option might be more elegant.  We could
set it to nil for now, and change it to non-nil for the next release.

>> >> Note that (package-install 'eglot) does download code, but I believe
>> >> that this is the correct approach and would align with what João
>> >> wanted.
>> >
>> > I'm not sure I follow: which code does the above download?
>> 
>> I did not change any of the code that downloads anything, all this does
>> is prompt the user for built-in packages when invoked interactively with
>> a prefix argument and if package-install is invoked with a built-in
>> package, then it will switch to the ELPA version.  This will not happen
>> in interactive usage, since `completing-read' is called with
>> REQUIRE-MATCH.
>
> So you are saying that non-interactive calls to package-install could
> install Eglot from ELPA even without the changes, is that right?

No, my proposed diff changes what package decides to download (the
planning phase), but doesn't touch anything after that.  The current
state is that (package-install 'eglot) just prints

  ‘eglot’ is already installed

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 15:10:01 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The issue here is that this breaks the non-interactive invocations like
> >> (package-install 'eglot), unless they invoke the function while binding
> >> `current-prefix-arg', which I don't think is a common practice.
> >
> > Then let's add another optional argument, and let prefix arg set it.
> > Would that resolve this issue?
> 
> That could solve it, but a user option might be more elegant.  We could
> set it to nil for now, and change it to non-nil for the next release.

Adding an option is fine by me, as long as its default preserves
previous behavior.

Just to be sure we are on the same page: you suggest _both_ prefix
argument and user option, where user option could be used to avoid the
need for prefix argument?

> >> >> Note that (package-install 'eglot) does download code, but I believe
> >> >> that this is the correct approach and would align with what João
> >> >> wanted.
> >> >
> >> > I'm not sure I follow: which code does the above download?
> >> 
> >> I did not change any of the code that downloads anything, all this does
> >> is prompt the user for built-in packages when invoked interactively with
> >> a prefix argument and if package-install is invoked with a built-in
> >> package, then it will switch to the ELPA version.  This will not happen
> >> in interactive usage, since `completing-read' is called with
> >> REQUIRE-MATCH.
> >
> > So you are saying that non-interactive calls to package-install could
> > install Eglot from ELPA even without the changes, is that right?
> 
> No, my proposed diff changes what package decides to download (the
> planning phase), but doesn't touch anything after that.  The current
> state is that (package-install 'eglot) just prints
> 
>   ‘eglot’ is already installed

And does nothing else?  You seem to be saying it still downloads
something, but what is that?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 15:10:01 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> The issue here is that this breaks the non-interactive invocations like
>> >> (package-install 'eglot), unless they invoke the function while binding
>> >> `current-prefix-arg', which I don't think is a common practice.
>> >
>> > Then let's add another optional argument, and let prefix arg set it.
>> > Would that resolve this issue?
>> 
>> That could solve it, but a user option might be more elegant.  We could
>> set it to nil for now, and change it to non-nil for the next release.
>
> Adding an option is fine by me, as long as its default preserves
> previous behavior.
>
> Just to be sure we are on the same page: you suggest _both_ prefix
> argument and user option, where user option could be used to avoid the
> need for prefix argument?

I was thinking about both, but I supposed that a user option would be
enough.

>> >> >> Note that (package-install 'eglot) does download code, but I believe
>> >> >> that this is the correct approach and would align with what João
>> >> >> wanted.
>> >> >
>> >> > I'm not sure I follow: which code does the above download?
>> >> 
>> >> I did not change any of the code that downloads anything, all this does
>> >> is prompt the user for built-in packages when invoked interactively with
>> >> a prefix argument and if package-install is invoked with a built-in
>> >> package, then it will switch to the ELPA version.  This will not happen
>> >> in interactive usage, since `completing-read' is called with
>> >> REQUIRE-MATCH.
>> >
>> > So you are saying that non-interactive calls to package-install could
>> > install Eglot from ELPA even without the changes, is that right?
>> 
>> No, my proposed diff changes what package decides to download (the
>> planning phase), but doesn't touch anything after that.  The current
>> state is that (package-install 'eglot) just prints
>> 
>>   ‘eglot’ is already installed
>
> And does nothing else?  You seem to be saying it still downloads
> something, but what is that?

No, it just prints that message but doesn't download anything.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 17:49:00 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Adding an option is fine by me, as long as its default preserves
> > previous behavior.
> >
> > Just to be sure we are on the same page: you suggest _both_ prefix
> > argument and user option, where user option could be used to avoid the
> > need for prefix argument?
> 
> I was thinking about both, but I supposed that a user option would be
> enough.

No, I think having both is better.  It is easier to say "C-u" than to
change the value of an option, so for one-off update of a single
package, the prefix argument is more convenient.

> >> No, my proposed diff changes what package decides to download (the
> >> planning phase), but doesn't touch anything after that.  The current
> >> state is that (package-install 'eglot) just prints
> >> 
> >>   ‘eglot’ is already installed
> >
> > And does nothing else?  You seem to be saying it still downloads
> > something, but what is that?
> 
> No, it just prints that message but doesn't download anything.

OK, then allowing to install such packages under the proposed changes
will improve that case as well.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 17:49:00 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Adding an option is fine by me, as long as its default preserves
>> > previous behavior.
>> >
>> > Just to be sure we are on the same page: you suggest _both_ prefix
>> > argument and user option, where user option could be used to avoid the
>> > need for prefix argument?
>> 
>> I was thinking about both, but I supposed that a user option would be
>> enough.
>
> No, I think having both is better.  It is easier to say "C-u" than to
> change the value of an option, so for one-off update of a single
> package, the prefix argument is more convenient.
>
>> >> No, my proposed diff changes what package decides to download (the
>> >> planning phase), but doesn't touch anything after that.  The current
>> >> state is that (package-install 'eglot) just prints
>> >> 
>> >>   ‘eglot’ is already installed
>> >
>> > And does nothing else?  You seem to be saying it still downloads
>> > something, but what is that?
>> 
>> No, it just prints that message but doesn't download anything.
>
> OK, then allowing to install such packages under the proposed changes
> will improve that case as well.

After having added the user option, I am not sure about the prefix
argument.  I see this as a temporary fix due to the time constraints of
releasing Emacs 29.  It is disabled for now, but can be enabled on
master to see if there are any problems.

But for now, this patch supports both the user option and the prefix
argument.  I am still not satisfied with the documentation, but cannot
come up with a better phrase than

  installing potentially newer versions of built-in packages from
  package archives

for explaining the issue without getting too technical.  Do you have any
ideas?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages-with-package-insta.patch --]
[-- Type: text/x-diff, Size: 4855 bytes --]

From 99e92d78560ef1aec7217d90ed3a8108bdf2924c Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 13 Apr 2023 20:13:59 +0200
Subject: [PATCH] Allow upgrading built-in packages with 'package-install'

* etc/NEWS: Mention the change
* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
predicate.
(package-install-upgrade-built-in): Add new user option to enable
feature.
(package-install): Respect new user option.
---
 etc/NEWS                   |  5 +++++
 lisp/emacs-lisp/package.el | 44 +++++++++++++++++++++++++++++++-------
 2 files changed, 41 insertions(+), 8 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 84dbb94a71a..a7834cd0d2b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1876,6 +1876,11 @@ package maintainers.
 By customizing this user option you can specify specific packages to
 install.
 
+---
+*** New user option 'package-install-upgrade-built-in'.
+When enabled, 'package-install' can be used to install potentially
+newer versions of built-in packages.
+
 ** Emacs Sessions (Desktop)
 
 +++
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..882db8db719 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--active-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2182,12 +2193,19 @@ package--archives-initialize
   (unless package-archive-contents
     (package-refresh-contents)))
 
+(defcustom package-install-upgrade-built-in nil
+  "Non-nil means that built-in packages can be upgraded via a package archive.
+If disabled, then `package-install' will not allow installing
+potentially newer versions of built-in packages from package
+archives."
+  :type 'boolean
+  :version "29.1")
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
-available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+available packages in an archive in `package-archives'.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2197,7 +2215,11 @@ package-install
 `package-selected-packages'.
 
 If PKG is a `package-desc' and it is already installed, don't try
-to install it but still mark it as selected."
+to install it but still mark it as selected.
+
+If the command is invoked with a prefix argument, the upgrading
+of built-in packages will be possible, as if
+`package-install-upgrade-built-in' had been enabled."
   (interactive
    (progn
      ;; Initialize the package system to get the list of package
@@ -2205,11 +2227,14 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and (or current-prefix-arg
+                                         package-install-upgrade-built-in)
+                                     (package--active-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (symbol-name (car elt)))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2245,9 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (and (or current-prefix-arg package-install-upgrade-built-in)
+               (package--active-built-in-p pkg))
+      (setq pkg (cadr (assq name package-archive-contents))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 18:49:05 +0000
> 
> After having added the user option, I am not sure about the prefix
> argument.  I see this as a temporary fix due to the time constraints of
> releasing Emacs 29.  It is disabled for now, but can be enabled on
> master to see if there are any problems.

Yes, we can change the default to t on master.  But that would also
require to adjust a few doc strings, see below.

> But for now, this patch supports both the user option and the prefix
> argument.  I am still not satisfied with the documentation, but cannot
> come up with a better phrase than
> 
>   installing potentially newer versions of built-in packages from
>   package archives
> 
> for explaining the issue without getting too technical.  Do you have any
> ideas?

See below.

> +*** New user option 'package-install-upgrade-built-in'.
> +When enabled, 'package-install' can be used to install potentially
> +newer versions of built-in packages.

I suggest

  When enabled, 'package-install' will include in the list of
  upgradeable packages those built-in packages (like Eglot, for
  example) for which a newer version is available on GNU ELPA.  By
  default, this is disabled; however, if 'package-install' is invoked
  with prefix argument, it will also include built-in packages in the
  list of packages which could be upgraded.

> +(defcustom package-install-upgrade-built-in nil
> +  "Non-nil means that built-in packages can be upgraded via a package archive.
> +If disabled, then `package-install' will not allow installing
> +potentially newer versions of built-in packages from package
> +archives."

The last sentence should be qualified by "...unless invoked with a
prefix argument".

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Fri, 14 Apr 2023 13:54:11 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Philip Kaludercic <philipk@posteo.net>
    >> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
    >> larsi@gnus.org
    >> Date: Thu, 13 Apr 2023 18:49:05 +0000
    >> 
    >> After having added the user option, I am not sure about the prefix
    >> argument.  I see this as a temporary fix due to the time constraints of
    >> releasing Emacs 29.  It is disabled for now, but can be enabled on
    >> master to see if there are any problems.

    Eli> Yes, we can change the default to t on master.  But that would also
    Eli> require to adjust a few doc strings, see below.

So on master if I upgrade all packages, ':core' packages would be
automatically upgraded as well? I strongly object to that as a
default; just because thereʼs a newer version on elpa of a :core
package doesnʼt mean emacs should upgrade to it unless *explicitly*
told to do so.

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 1:34 PM Robert Pluim <rpluim@gmail.com> wrote:

> So on master if I upgrade all packages, ':core' packages would be
> automatically upgraded as well?

By definition, all :core packages in master are already at their
newest version.

> I strongly object to that as a
> default; just because thereʼs a newer version on elpa of a :core

I really planned to sit this one out, but I'd like to make
sure people understand the implications of what they're asking for.

On Emacs 26, 27, 28 if the user has

  (package-install 'some-package-now-in-core)

in her configuration, it gets upgraded to the most recent version
there is.  In subsequent forms, the config can start doing stuff with
the variables and definitions in 'some-package-now-in-core', etc.
And the user can enjoy the newest features and bugfixes.

On Emacs 29 and later, the very same config will do nothing
and even probably/possibly break with an error.

Furthermore, the subtle problem will grow more serious and
bizarre as time goes on and "some-package-now-in-core" evolves.
It might not break for users who upgrade to 29 next month
and break for users who upgrade to 29 in 6 months' time, because
"some-package-now-in-core" will have evolved significantly.

> package doesnʼt mean emacs should upgrade to it unless *explicitly*
> told to do so.

I really don't understand why M-x package-install RET
<types-name-of-package> RET isn't explicit enough.  But I guess a
a confirmation prompt could be logical.  I haven't followed
all mails, maybe someone has proposed that?

As for non-interactive package-install, I guess that finding an
explicit `package-install` somewhere in the configuration is reason
enough to assume that the user meant for it to have the meaning
and effect it has always had before she upgraded to a version
where the same package happens to be in :core, and that meaning is
"upgrade to the newest".

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Fri, 14 Apr 2023 13:56:05 +0100, João Távora <joaotavora@gmail.com> said:

    João> On Fri, Apr 14, 2023 at 1:34 PM Robert Pluim <rpluim@gmail.com> wrote:
    >> So on master if I upgrade all packages, ':core' packages would be
    >> automatically upgraded as well?

    João> By definition, all :core packages in master are already at their
    João> newest version.

    >> I strongly object to that as a
    >> default; just because thereʼs a newer version on elpa of a :core

    João> I really planned to sit this one out, but I'd like to make
    João> sure people understand the implications of what they're asking for.

    João> On Emacs 26, 27, 28 if the user has

    João>   (package-install 'some-package-now-in-core)

    João> in her configuration, it gets upgraded to the most recent version
    João> there is.  In subsequent forms, the config can start doing stuff with
    João> the variables and definitions in 'some-package-now-in-core', etc.
    João> And the user can enjoy the newest features and bugfixes.

    João> On Emacs 29 and later, the very same config will do nothing
    João> and even probably/possibly break with an error.

    João> Furthermore, the subtle problem will grow more serious and
    João> bizarre as time goes on and "some-package-now-in-core" evolves.
    João> It might not break for users who upgrade to 29 next month
    João> and break for users who upgrade to 29 in 6 months' time, because
    João> "some-package-now-in-core" will have evolved significantly.

    >> package doesnʼt mean emacs should upgrade to it unless *explicitly*
    >> told to do so.

I have no objection to catering to people who have already asked for
the installation of a package that is now :core. But one that wasnʼt
installed explicitly (ie itʼs only there because Emacs now ships it)
shouldnʼt be upgraded.

    João> I really don't understand why M-x package-install RET
    João> <types-name-of-package> RET isn't explicit enough.  But I guess a
    João> a confirmation prompt could be logical.  I haven't followed
    João> all mails, maybe someone has proposed that?

I donʼt know. Iʼd be fine with that.

    João> As for non-interactive package-install, I guess that finding an
    João> explicit `package-install` somewhere in the configuration is reason
    João> enough to assume that the user meant for it to have the meaning
    João> and effect it has always had before she upgraded to a version
    João> where the same package happens to be in :core, and that meaning is
    João> "upgrade to the newest".

I think weʼre pretty much in agreement :-)

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 2:52 PM Robert Pluim <rpluim@gmail.com> wrote:

> I have no objection to catering to people who have already asked for
> the installation of a package that is now :core. But one that wasnʼt
> installed explicitly (ie itʼs only there because Emacs now ships it)
> shouldnʼt be upgraded.

I think I don't fully understand.

Maybe you can answer this: if a user is setting up Emacs 28 in company
laptop machines regularly and does M-x package-install RET eglot RET there,
or has a script with (package-install 'eglot), should or shouldn't this
user, in your opinion, be allowed to do exactly the same, with the same
predictable results (in this case getting the latest Eglot), when she starts
doing the same in Emacs 29, which now contains Eglot as a built-in?

What about other builtin packages that, contrary to Eglot, already existed
in Emacs 28?  If your answer is different now, maybe we should record
that difference somewhere, so we know what packages came from ELPA into
:core and when, and can decide accordingly.  Who knows maybe just
whitelisting Eglot and any other packages that weren't built-in in 28
and are now built-in Emacs 29 is the answer to all this.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Maybe you can answer this: if a user is setting up Emacs 28 in company
> laptop machines regularly and does M-x package-install RET eglot RET there,
> or has a script with (package-install 'eglot), should or shouldn't this
> user, in your opinion, be allowed to do exactly the same, with the same
> predictable results (in this case getting the latest Eglot), when she starts
> doing the same in Emacs 29, which now contains Eglot as a built-in?

Good point.  I think the answer is that we should distinguish "install"
from "upgrade" (and offer a way to do both, of course).

And I think we do want to break backward compatibility here (arguably we
even can't not break compatibility), because the Emacs<29 semantics of
`package-install` is "broken", since it does "install&upgrade" for
non-builtin packages but not for builtin packages: either we keep that
semantics and compatibility is broken when packages move to/from
builtin, or we change that semantics and compatibility is broken by the
change in semantics :-)


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 4:52 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > Maybe you can answer this: if a user is setting up Emacs 28 in company
> > laptop machines regularly and does M-x package-install RET eglot RET there,
> > or has a script with (package-install 'eglot), should or shouldn't this
> > user, in your opinion, be allowed to do exactly the same, with the same
> > predictable results (in this case getting the latest Eglot), when she starts
> > doing the same in Emacs 29, which now contains Eglot as a built-in?
>
> Good point.  I think the answer is that we should distinguish "install"
> from "upgrade" (and offer a way to do both, of course).
>
> And I think we do want to break backward compatibility here (arguably we
> even can't not break compatibility), because the Emacs<29 semantics of
> `package-install` is "broken", since it does "install&upgrade" for
> non-builtin packages but not for builtin packages: either we keep that
> semantics and compatibility is broken when packages move to/from
> builtin, or we change that semantics and compatibility is broken by the
> change in semantics :-)

I would think it's too late in the game to break compatibility.
Naming aside package-install has certain behaviour that for a certain
set of inputs used to produce predictable things.
Now, for the same inputs it does nothing on Emacs 29.

I think it should do the same thing, not only because it's
nicer for the unsuspecting user, but also because trying to
protect this user from "unintentional" upgrade of certain "unstable"
packages, as it seems to be the idea here, is a losing game
anyway, just because dependencies.  Presumably, the packaging
system where it takes less than a day to fix release a bug fix and
deliver this to the archives, should be the protection.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I would think it's too late in the game to break compatibility.
> Naming aside package-install has certain behaviour that for a certain
> set of inputs used to produce predictable things.
> Now, for the same inputs it does nothing on Emacs 29.

The only way I can think of to preserve compatibility is to change the
behavior so it doesn't pay attention to "is builtin or not" but to "used
to be builtin before Emacs-29".  This would make a bad semantics even
worse, so I'd rather we fix the semantics to something clean.

> I think it should do the same thing, not only because it's
> nicer for the unsuspecting user, but also because trying to
> protect this user from "unintentional" upgrade of certain "unstable"
> packages, as it seems to be the idea here, is a losing game
> anyway, just because dependencies.

You may be right: maybe the distinction between "install only" and
"install&upgrade" isn't worth the trouble.

I think to get closer to a useful "install only" behavior we'd want that
command to prompt the user before upgrading dependencies (tho probably
only for those in `package-selected-packages`).

BTW, for me the reluctance to upgrade when asked to install isn't due to
the risk that the package is "unstable".  I'm not completely sure what
is the reason, admittedly, but it's closer to viewing it as a silent
"change of distribution", or maybe it's because I like to know when
behavior may change and merely installing a package shouldn't change
Emacs's behavior.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 19:28, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
>> I would think it's too late in the game to break compatibility.
>> Naming aside package-install has certain behaviour that for a certain
>> set of inputs used to produce predictable things.
>> Now, for the same inputs it does nothing on Emacs 29.
> 
> The only way I can think of to preserve compatibility is to change the
> behavior so it doesn't pay attention to "is builtin or not" but to "used
> to be builtin before Emacs-29".  This would make a bad semantics even
> worse, so I'd rather we fix the semantics to something clean.

We could add a new attribute/package header which would mean "eagerly 
upgradable [even when] built-in".

>> I think it should do the same thing, not only because it's
>> nicer for the unsuspecting user, but also because trying to
>> protect this user from "unintentional" upgrade of certain "unstable"
>> packages, as it seems to be the idea here, is a losing game
>> anyway, just because dependencies.
> 
> You may be right: maybe the distinction between "install only" and
> "install&upgrade" isn't worth the trouble.
> 
> I think to get closer to a useful "install only" behavior we'd want that
> command to prompt the user before upgrading dependencies (tho probably
> only for those in `package-selected-packages`).
> 
> BTW, for me the reluctance to upgrade when asked to install isn't due to
> the risk that the package is "unstable".  I'm not completely sure what
> is the reason, admittedly, but it's closer to viewing it as a silent
> "change of distribution",

That's my feeling too.

 > or maybe it's because I like to know when
behavior may change and merely installing a package shouldn't change
Emacs's behavior.

This is probably a different question: whether 'M-x package-install' 
should do upgrades at all, whether the package is built-in or not.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 5:28 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> The only way I can think of to preserve compatibility is to change the
> behavior so it doesn't pay attention to "is builtin or not" but to "used
> to be builtin before Emacs-29".

Exactly, I suggested this via whiteslisting.  This would be blacklisting
and would be slightly cleaner.

> This would make a bad semantics even worse, so I'd rather we fix
> the semantics to something clean.

Bad semantics or not, at least it wouldn't break stuff, and that
blacklist would stay put.  Installing from it would require papal
benediction and that's it :-)

> > I think it should do the same thing, not only because it's
> > nicer for the unsuspecting user, but also because trying to
> > protect this user from "unintentional" upgrade of certain "unstable"
> > packages, as it seems to be the idea here, is a losing game
> > anyway, just because dependencies.
>
> You may be right: maybe the distinction between "install only" and
> "install&upgrade" isn't worth the trouble.
>
> I think to get closer to a useful "install only" behavior we'd want that
> command to prompt the user before upgrading dependencies (tho probably
> only for those in `package-selected-packages`).
>
> BTW, for me the reluctance to upgrade when asked to install isn't due to
> the risk that the package is "unstable".  I'm not completely sure what
> is the reason, admittedly, but it's closer to viewing it as a silent
> "change of distribution", or maybe it's because I like to know when
> behavior may change and merely installing a package shouldn't change
> Emacs's behavior.

I don't follow, they don't. Eexcept for major modes, that is, those autoload
changes into auto-mode-alist.  As for adding new commands, and
functionality to your existing modes, well you _did_ ask for the
package to be upgraded (yes I know the name is "install", but upgrading
is what it has always done).

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> BTW, for me the reluctance to upgrade when asked to install isn't due to
>> the risk that the package is "unstable".  I'm not completely sure what
>> is the reason, admittedly, but it's closer to viewing it as a silent
>> "change of distribution", or maybe it's because I like to know when
>> behavior may change and merely installing a package shouldn't change
>> Emacs's behavior.
[...]
> functionality to your existing modes, well you _did_ ask for the
> package to be upgraded

No, as the first line above shows, I'm talking about "asked to install",
not "asked to upgrade".


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 17:05:30 +0100
> Cc: Robert Pluim <rpluim@gmail.com>, 62720@debbugs.gnu.org, larsi@gnus.org, 
> 	Eli Zaretskii <eliz@gnu.org>, Philip Kaludercic <philipk@posteo.net>
> 
> On Fri, Apr 14, 2023 at 4:52 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > And I think we do want to break backward compatibility here (arguably we
> > even can't not break compatibility), because the Emacs<29 semantics of
> > `package-install` is "broken", since it does "install&upgrade" for
> > non-builtin packages but not for builtin packages: either we keep that
> > semantics and compatibility is broken when packages move to/from
> > builtin, or we change that semantics and compatibility is broken by the
> > change in semantics :-)
> 
> I would think it's too late in the game to break compatibility.
> Naming aside package-install has certain behaviour that for a certain
> set of inputs used to produce predictable things.
> Now, for the same inputs it does nothing on Emacs 29.

When you say "compatibility", you seem to have only one its aspect in
mind: that of Eglot.  But that is not the only aspect of the previous
behavior, and I, at least, must consider those other aspects as well.

That package-install doesn't upgrade core packages was how it behaved
in past versions.  In Emacs 29.1 I hope we will allow at least
overriding that by user-level means, so we will be closer to your
(Eglot-centric) ideal, without also breaking the other aspects of
previous behavior, since there are other core packages on ELPA besides
Eglot, and some of them were in that state before Emacs 29.

And that is all we can reasonably do at this time, guiven how close we
are to the release.

> I think it should do the same thing, not only because it's
> nicer for the unsuspecting user, but also because trying to
> protect this user from "unintentional" upgrade of certain "unstable"
> packages, as it seems to be the idea here, is a losing game
> anyway, just because dependencies.

"The same thing" for Eglot means "not the same thing" for other core
packages.  So you are in effect calling for breaking everyone else to
cater only to Eglot.  That is not going to happen, for more than one
reason.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 6:49 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > I think it should do the same thing, not only because it's
> > nicer for the unsuspecting user, but also because trying to
> > protect this user from "unintentional" upgrade of certain "unstable"
> > packages, as it seems to be the idea here, is a losing game
> > anyway, just because dependencies.
>
> "The same thing" for Eglot means "not the same thing" for other core
> packages.  So you are in effect calling for breaking everyone else to
> cater only to Eglot.  That is not going to happen, for more than one
> reason.

> when you say "compatibility", you seem to have only one its aspect in
> mind: that of Eglot.  But that is not the only aspect of the previous
> behavior, and I, at least, must consider those other aspects as well.

You seem to be worried about "everyone else" typing, say, M-x package-install
RET seq RET and getting an updated 'seq' as a result.  OK, but who does
this and why, in your opinion?  And who has `(package-install seq)` in
their config and why?  And won't they get the same updated 'seq'
"accidentally" if they install anything else that depends on newer seq,
which is likely a lot of non-core packages and likely to grow?  I just
can't see these "other aspects", i.e. who or whose use cases you wish to
protect with multiple user knobs to get a razor thin level of protection.

But even if these people and use cases did exist, you're still
plainly misrepresenting my position by writing that I'm "calling
for breaking" them.  I even proposed making a simple whitelist
of packages that have migrated from outside core to core.  And
I've proposed confirmation prompts for the interactive calls.
And others have proposed blacklists.  These things are trivial
to implement in Elisp.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 19:32:40 +0100
> Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org, philipk@posteo.net
> 
> On Fri, Apr 14, 2023 at 6:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > when you say "compatibility", you seem to have only one its aspect in
> > mind: that of Eglot.  But that is not the only aspect of the previous
> > behavior, and I, at least, must consider those other aspects as well.
> 
> You seem to be worried about "everyone else" typing, say, M-x package-install
> RET seq RET and getting an updated 'seq' as a result.

That, too.  But also everything else.  Any incompatible change of
behavior can potentially break someone's workflow.

> OK, but who does this and why, in your opinion?  And who has
> `(package-install seq)` in their config and why?  And won't they get
> the same updated 'seq' "accidentally" if they install anything else
> that depends on newer seq, which is likely a lot of non-core
> packages and likely to grow?

I don't know.  But we did have core packages that were also on ELPA
before Emacs 29, and people did get along with that.  So much so that
I don't recall any complaints about this, definitely not complaints
that claimed package.el is as badly broken as you seem to represent.

> But even if these people and use cases did exist, you're still
> plainly misrepresenting my position by writing that I'm "calling
> for breaking" them.

I said "in effect calling for breaking them".  You need to realize
this might be the outcome of what you are requesting, even if your
intentions are benign.

> I even proposed making a simple whitelist of packages that have
> migrated from outside core to core.  And I've proposed confirmation
> prompts for the interactive calls.  And others have proposed
> blacklists.  These things are trivial to implement in Elisp.

They are not trivial enough to be considered for emacs-29.  On master,
sure, feel free to install such changes, if the others agree.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 7:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 14 Apr 2023 19:32:40 +0100
> > Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, 62720@debbugs.gnu.org,
> >       larsi@gnus.org, philipk@posteo.net
> >
> > On Fri, Apr 14, 2023 at 6:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > when you say "compatibility", you seem to have only one its aspect in
> > > mind: that of Eglot.  But that is not the only aspect of the previous
> > > behavior, and I, at least, must consider those other aspects as well.
> >
> > You seem to be worried about "everyone else" typing, say, M-x package-install
> > RET seq RET and getting an updated 'seq' as a result.
>
> That, too.  But also everything else.  Any incompatible change of
> behavior can potentially break someone's workflow.

Of course.  But what is this "everything else" pandora's box that
the solution I'm proposing unleashes?  I just don't see the
connection.

> > OK, but who does this and why, in your opinion?  And who has
> > `(package-install seq)` in their config and why?  And won't they get
> > the same updated 'seq' "accidentally" if they install anything else
> > that depends on newer seq, which is likely a lot of non-core
> > packages and likely to grow?
>
> I don't know.  But we did have core packages that were also on ELPA
> before Emacs 29, and people did get along with that.  So much so that
> I don't recall any complaints about this, definitely not complaints
> that claimed package.el is as badly broken as you seem to represent.

I think it's simply and clearly because there weren't any non-core
packages that migrated to core before.  Were there?  Certainly
none with as fast a release cycle like Eglot's.

> > But even if these people and use cases did exist, you're still
> > plainly misrepresenting my position by writing that I'm "calling
> > for breaking" them.
>
> I said "in effect calling for breaking them".  You need to realize
> this might be the outcome of what you are requesting, even if your
> intentions are benign.

I _am_ realizing that, and even if I think the risks are minuscule,
I'm providing solutions that mitigate them.  And -- responsibly, I
think -- pointing to the fact that the risk surface is much larger
than you may think it is.

> > I even proposed making a simple whitelist of packages that have
> > migrated from outside core to core.  And I've proposed confirmation
> > prompts for the interactive calls.  And others have proposed
> > blacklists.  These things are trivial to implement in Elisp.
>
> They are not trivial enough to be considered for emacs-29.  On master,
> sure, feel free to install such changes, if the others agree.

Are you saying it's OK to propose this blacklist/whitelist idea?
I will work and prepare that patch if you promise to at least
look at the code, follow the logic with me and consider the
patch for backporting if it's as safe as relying on a simple
`(if (memq package package-safely-upgradeable-builtins) ...)`

If you don't want to do that, I'm not going to propose it, because
that in effect means I'm adding code that will only have any effect
in the Emacs 30 release, and the Emacs 29 will be a very odd
release in this aspect for several years to come.  And this is an even
worse outcome.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 20:03:31 +0100
> Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, 62720@debbugs.gnu.org, 
> 	larsi@gnus.org, philipk@posteo.net
> 
> > > OK, but who does this and why, in your opinion?  And who has
> > > `(package-install seq)` in their config and why?  And won't they get
> > > the same updated 'seq' "accidentally" if they install anything else
> > > that depends on newer seq, which is likely a lot of non-core
> > > packages and likely to grow?
> >
> > I don't know.  But we did have core packages that were also on ELPA
> > before Emacs 29, and people did get along with that.  So much so that
> > I don't recall any complaints about this, definitely not complaints
> > that claimed package.el is as badly broken as you seem to represent.
> 
> I think it's simply and clearly because there weren't any non-core
> packages that migrated to core before.  Were there?  Certainly
> none with as fast a release cycle like Eglot's.

You are missing the point.  My point is that users of all those other
core packages are used to the current situation, where package-install
doesn't update the built-ins.  They have workflows based on that.
Those are the workflows that I don't want to risk breaking.

Once the discussion with Philip runs to its completion, we _will_ have
in Emacs 29.1 a way of upgrading built-ins, only not by default, but
by an explicit user request.  Maybe this is not ideal, but we don't
have time to find out what is ideal for all those involved, and this
partial solution allows to get the job done without risking to break
anyone's workflow.  So this is a good compromise, given the time and
the constraints, at least from my POV.

> > I said "in effect calling for breaking them".  You need to realize
> > this might be the outcome of what you are requesting, even if your
> > intentions are benign.
> 
> I _am_ realizing that, and even if I think the risks are minuscule,
> I'm providing solutions that mitigate them.  And -- responsibly, I
> think -- pointing to the fact that the risk surface is much larger
> than you may think it is.

I respectfully disagree with your assessment of the risks.  And
eventually, it is my responsibility to do this job, so my assessment
of that is what counts more.  I've heard all of your arguments (and
also those of others), and considered all of them, and I still don't
think the risk is high enough to justify the potentially-breaking
changes that you propose, not on the release branch, not this late
into the release cycle.

> > > I even proposed making a simple whitelist of packages that have
> > > migrated from outside core to core.  And I've proposed confirmation
> > > prompts for the interactive calls.  And others have proposed
> > > blacklists.  These things are trivial to implement in Elisp.
> >
> > They are not trivial enough to be considered for emacs-29.  On master,
> > sure, feel free to install such changes, if the others agree.
> 
> Are you saying it's OK to propose this blacklist/whitelist idea?

For the release branch?  No.  For master?  Yes.

> I will work and prepare that patch if you promise to at least
> look at the code, follow the logic with me and consider the
> patch for backporting if it's as safe as relying on a simple
> `(if (memq package package-safely-upgradeable-builtins) ...)`

Backporting can be considered for Emacs 29.2, not before that.
Hopefully, by that time we will also have more data points regarding
the various uses of this, in particular with Eglot, so that our
decisions will be more informed.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 8:18 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 14 Apr 2023 20:03:31 +0100
> > Cc: monnier@iro.umontreal.ca, rpluim@gmail.com, 62720@debbugs.gnu.org,
> >       larsi@gnus.org, philipk@posteo.net
> >
> > > > OK, but who does this and why, in your opinion?  And who has
> > > > `(package-install seq)` in their config and why?  And won't they get
> > > > the same updated 'seq' "accidentally" if they install anything else
> > > > that depends on newer seq, which is likely a lot of non-core
> > > > packages and likely to grow?
> > >
> > > I don't know.  But we did have core packages that were also on ELPA
> > > before Emacs 29, and people did get along with that.  So much so that
> > > I don't recall any complaints about this, definitely not complaints
> > > that claimed package.el is as badly broken as you seem to represent.
> >
> > I think it's simply and clearly because there weren't any non-core
> > packages that migrated to core before.  Were there?  Certainly
> > none with as fast a release cycle like Eglot's.
>
> You are missing the point.  My point is that users of all those other
> core packages are used to the current situation, where package-install
> doesn't update the built-ins.  They have workflows based on that.
> Those are the workflows that I don't want to risk breaking.

The only workflows that traverse this use case I can think of:

1. they routinely invoke a useless M-x package-install RET seq RET
, it prints "already installed" and they breathe a daily sigh of relief.

2. they have `(package-install 'seq)` somewhere in their init file
which runs daily and does nothing.

> Once the discussion with Philip runs to its completion, we _will_ have
> in Emacs 29.1 a way of upgrading built-ins, only not by default, but
> by an explicit user request.  Maybe this is not ideal, but we don't
> have time to find out what is ideal for all those involved, and this
> partial solution allows to get the job done without risking to break
> anyone's workflow.  So this is a good compromise, given the time and
> the constraints, at least from my POV.

There is already this way, that was never in question.  Package menu,
M-: or anything else, it's not _that_ hard.  But it's a shame that
the canonical use-package recipe for Eglot (or for use-package itself)
it will break.

Looking at the patches, it seems, from my POV, that using a prefix
argument and an extra customization option is much, much more
complicated than a simple whitelist or blacklist that the user
needn't ever see.

> > I _am_ realizing that, and even if I think the risks are minuscule,
> > I'm providing solutions that mitigate them.  And -- responsibly, I
> > think -- pointing to the fact that the risk surface is much larger
> > than you may think it is.
> I respectfully disagree with your assessment of the risks.  And
> eventually, it is my responsibility to do this job, so my assessment
> of that is what counts more.  I've heard all of your arguments (and
> also those of others), and considered all of them, and I still don't
> think the risk is high enough to justify the potentially-breaking
> changes that you propose, not on the release branch, not this late
> into the release cycle.

And even though I also disagree, equally respectfully, with your
assessment of the risks, and still I'm providing a solution that
eliminates them totally.

> > I will work and prepare that patch if you promise to at least
> > look at the code, follow the logic with me and consider the
> > patch for backporting if it's as safe as relying on a simple
> > `(if (memq package package-safely-upgradeable-builtins) ...)`
>
> Backporting can be considered for Emacs 29.2, not before that.
> Hopefully, by that time we will also have more data points regarding
> the various uses of this, in particular with Eglot, so that our
> decisions will be more informed.

OK.  Fair enough. If there's half a chance for 29.2 I guess I can work on
it.  Not today though.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 18:34, João Távora wrote:
> On Fri, Apr 14, 2023 at 2:52 PM Robert Pluim<rpluim@gmail.com>  wrote:
> 
>> I have no objection to catering to people who have already asked for
>> the installation of a package that is now :core. But one that wasnʼt
>> installed explicitly (ie itʼs only there because Emacs now ships it)
>> shouldnʼt be upgraded.
> I think I don't fully understand.
> 
> Maybe you can answer this: if a user is setting up Emacs 28 in company
> laptop machines regularly and does M-x package-install RET eglot RET there,
> or has a script with (package-install 'eglot), should or shouldn't this
> user, in your opinion, be allowed to do exactly the same, with the same
> predictable results (in this case getting the latest Eglot), when she starts
> doing the same in Emacs 29, which now contains Eglot as a built-in?

I think Robert asked whether 'M-x package-menu-mark-upgrades' would mark 
Eglot for upgrade even if the user never marked it for upgrade 
explicitly before (e.g. with 'M-x package-install').

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 5:12 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 14/04/2023 18:34, João Távora wrote:
> > On Fri, Apr 14, 2023 at 2:52 PM Robert Pluim<rpluim@gmail.com>  wrote:
> >
> >> I have no objection to catering to people who have already asked for
> >> the installation of a package that is now :core. But one that wasnʼt
> >> installed explicitly (ie itʼs only there because Emacs now ships it)
> >> shouldnʼt be upgraded.
> > I think I don't fully understand.
> >
> > Maybe you can answer this: if a user is setting up Emacs 28 in company
> > laptop machines regularly and does M-x package-install RET eglot RET there,
> > or has a script with (package-install 'eglot), should or shouldn't this
> > user, in your opinion, be allowed to do exactly the same, with the same
> > predictable results (in this case getting the latest Eglot), when she starts
> > doing the same in Emacs 29, which now contains Eglot as a built-in?
>
> I think Robert asked whether 'M-x package-menu-mark-upgrades' would mark
> Eglot for upgrade even if the user never marked it for upgrade
> explicitly before (e.g. with 'M-x package-install').

Sorry, I missed that.  This package-menu interface really seems like
its own thing, a separate package manager with its own nomenclature
and rules.

BTW use-package also calls package-install, and
(use-package eglot :ensure t) is the number one installation form
for Eglot says google, chatGPT and 4 out of 5 bugs I get.
Unless special steps are taken there, it will likely be broken,
too.  That form will no longer give users the latest Eglot starting
Emacs 29.  Fortunately, though the use-package manual "expects that
most users will find [package.el] capable enough", it also
explains how to switch away from package.el into a third party
package-manager. Given the state of this bug and for their sake,
I hope they do.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

[...]

> Fortunately, though the use-package manual "expects that most users
> will find [package.el] capable enough", it also explains how to switch
> away from package.el into a third party package-manager. Given the
> state of this bug and for their sake, I hope they do.

As this issue only affects Eglot (a package that doesn't require that
much customisation to be used), I don't think the situation is as
drastic as you are portraying it to be.

If we all try to have a cooperative, non-alarmist attitude towards
solving the issue at hand, I am sure a suitable solution will be found.

Also keep in mind that I have proposed multiple patches that take
difference approaches.  Perhaps it would be of use to recapitulate them,
and you can explain which would be satisfying and which you think
wouldn't be as helpful:

- Use `package-install' to switch from a built-in package to the version
  from ELPA

  - Alternatively it has to be confirmed using a prefix argument
  - Alternatively it has to be enabled using a user option

- Use `package-update' (not `package-update-all') to switch from a
  built-in package to the version from ELPA

  (Same alternatives as above)

- Have both `package-update' and `package-update-all' switch potentially
  all packages from built-in versions to ELPA versions

  (Same alternatives as above)

- Provide a separate command to switch from a built-in version to a
  version from ELPA

That should be it?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 5:53 PM Philip Kaludercic <philipk@posteo.net> wrote:

> As this issue only affects Eglot

[Didn't other packages also make it into :core?]

> (a package that doesn't require that
> much customisation to be used), I don't think the situation is as
> drastic as you are portraying it to be.

> If we all try to have a cooperative, non-alarmist attitude towards
> solving the issue at hand, I am sure a suitable solution will be found.

It's true it doesn't have many customization now, so downright
erroring is probably rare.  But it's been advancing fast and
that may not always be the case. As I said Eglot 1.12 in Emacs
29 has lots of missing functionality and even bugs that are too
risky to fix in that version.

So I'm not being alarmist, I think.  With 5 years maintainership
of this package and well over 1000 issues, I think I make a fair
assessment of user's expectations.  As to being cooperative, I've
proposed 3 different patches already, answered every question about
them and proposed n other solutions.

But I like your optimism nonetheless :-)

> Also keep in mind that I have proposed multiple patches that take
> difference approaches.  Perhaps it would be of use to recapitulate them,
> and you can explain which would be satisfying and which you think
> wouldn't be as helpful:
>
> - Use `package-install' to switch from a built-in package to the version
>   from ELPA
>
>   - Alternatively it has to be confirmed using a prefix argument
>   - Alternatively it has to be enabled using a user option
>
> - Use `package-update' (not `package-update-all') to switch from a
>   built-in package to the version from ELPA
>
>   (Same alternatives as above)
>
> - Have both `package-update' and `package-update-all' switch potentially
>   all packages from built-in versions to ELPA versions
>
>   (Same alternatives as above)
>
> - Provide a separate command to switch from a built-in version to a
>   version from ELPA
>
> That should be it?

If it's not clear yet, I want(ed) something that users can use
_regardless_ of the version of Emacs to bring Eglot to the
latest version.  I want users to be able to do this easily
for obvious reasons.  The prime candidate for that "something"
is M-x package-install both in its command and non-interactive
form.  But if that isn't possible, the next best thing is
a 4 line eglot-update command in eglot.el which would eventually
boil down to the  same (because Emacs 26/27/28 users would eventually
also get it).  That idea was explicitly rejected, but I hope it
illustrates what I would prefer.

A command and function with enduring semantics, basically, the
kind you expect from a system like Emacs.

Any one of your or other's solutions that provide such a
command/function are most welcome.  Any other that doesn',
I'm indifferent to it.  Which doesn't mean "against" or "hostile".

And of course I thank you very much for your efforts in searching
for a solution.

Here's another solution you may want to consider.  package-install
in Emacs 29 updates built-ins non-interactively always.  Interactively
asks the user with a confirmation prompt.  This solution would
also work.

Here's yet another one. package-install like the above but, as has
been proposed, carefully vets the builtins that are subject to
from a controlled whitelist, which would include Eglot.  Stefan, Dmitry
and myself have proposed some variation of this.

Anyway, if my preference doesn't materialize, I'm going to recommend
in the manual:

M-: (package-install (alist-get 'eglot package-archive-contents) RET

which works interactively and non-interactively in every version
of Emacs.  It's not pretty but it's the best I have.  It's better
than to write "if you have emacs 29, do this, else do that".

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Fri, Apr 14, 2023 at 5:53 PM Philip Kaludercic <philipk@posteo.net> wrote:
>
>> As this issue only affects Eglot
>
> [Didn't other packages also make it into :core?]

I made a mistake in assuming that the rest are non-user facing
libraries, but there also is use-package.  But since that is also a
slower package, I don't think this is an emergency.

>> (a package that doesn't require that
>> much customisation to be used), I don't think the situation is as
>> drastic as you are portraying it to be.
>
>> If we all try to have a cooperative, non-alarmist attitude towards
>> solving the issue at hand, I am sure a suitable solution will be found.
>
> It's true it doesn't have many customization now, so downright
> erroring is probably rare.  But it's been advancing fast and
> that may not always be the case. As I said Eglot 1.12 in Emacs
> 29 has lots of missing functionality and even bugs that are too
> risky to fix in that version.
>
> So I'm not being alarmist, I think.  With 5 years maintainership
> of this package and well over 1000 issues, I think I make a fair
> assessment of user's expectations.  As to being cooperative, I've
> proposed 3 different patches already, answered every question about
> them and proposed n other solutions.
>
> But I like your optimism nonetheless :-)

It follows from my experience.

>> Also keep in mind that I have proposed multiple patches that take
>> difference approaches.  Perhaps it would be of use to recapitulate them,
>> and you can explain which would be satisfying and which you think
>> wouldn't be as helpful:
>>
>> - Use `package-install' to switch from a built-in package to the version
>>   from ELPA
>>
>>   - Alternatively it has to be confirmed using a prefix argument
>>   - Alternatively it has to be enabled using a user option
>>
>> - Use `package-update' (not `package-update-all') to switch from a
>>   built-in package to the version from ELPA
>>
>>   (Same alternatives as above)
>>
>> - Have both `package-update' and `package-update-all' switch potentially
>>   all packages from built-in versions to ELPA versions
>>
>>   (Same alternatives as above)
>>
>> - Provide a separate command to switch from a built-in version to a
>>   version from ELPA
>>
>> That should be it?
>
> If it's not clear yet, I want(ed) something that users can use
> _regardless_ of the version of Emacs to bring Eglot to the
> latest version.  I want users to be able to do this easily
> for obvious reasons.  The prime candidate for that "something"
> is M-x package-install both in its command and non-interactive
> form.  But if that isn't possible, the next best thing is
> a 4 line eglot-update command in eglot.el which would eventually
> boil down to the  same (because Emacs 26/27/28 users would eventually
> also get it).  That idea was explicitly rejected, but I hope it
> illustrates what I would prefer.
>
> A command and function with enduring semantics, basically, the
> kind you expect from a system like Emacs.
>
> Any one of your or other's solutions that provide such a
> command/function are most welcome.  Any other that doesn',
> I'm indifferent to it.  Which doesn't mean "against" or "hostile".
>
> And of course I thank you very much for your efforts in searching
> for a solution.
>
> Here's another solution you may want to consider.  package-install
> in Emacs 29 updates built-ins non-interactively always.  Interactively
> asks the user with a confirmation prompt.  This solution would
> also work.

What would the confirmation prompt ask the user?

> Here's yet another one. package-install like the above but, as has
> been proposed, carefully vets the builtins that are subject to
> from a controlled whitelist, which would include Eglot.  Stefan, Dmitry
> and myself have proposed some variation of this.
>
> Anyway, if my preference doesn't materialize, I'm going to recommend
> in the manual:
>
> M-: (package-install (alist-get 'eglot package-archive-contents)) RET

Are you sure?  The `alist-get' call returns a list, since
`package-archive-contents' is a map of package names to a list of
package descriptors.

> which works interactively and non-interactively in every version
> of Emacs.  It's not pretty but it's the best I have.  It's better
> than to write "if you have emacs 29, do this, else do that".
>
> João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 7:26 PM Philip Kaludercic <philipk@posteo.net> wrote:

> > Here's another solution you may want to consider.  package-install
> > in Emacs 29 updates built-ins non-interactively always.  Interactively
> > asks the user with a confirmation prompt.  This solution would
> > also work.
>
> What would the confirmation prompt ask the user?

"Are you sure you want to install Foo which is [also|now] a built-in
package in Emacs 29?"

> Are you sure?  The `alist-get' call returns a list, since
> `package-archive-contents' is a map of package names to a list of
> package descriptors.

Yeah, probably needs a cdr or car there for good measure.  Great.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> On Fri, Apr 14, 2023 at 7:26 PM Philip Kaludercic <philipk@posteo.net> wrote:
>
>> > Here's another solution you may want to consider.  package-install
>> > in Emacs 29 updates built-ins non-interactively always.  Interactively
>> > asks the user with a confirmation prompt.  This solution would
>> > also work.
>>
>> What would the confirmation prompt ask the user?
>
> "Are you sure you want to install Foo which is [also|now] a built-in
> package in Emacs 29?"

I feel this is confusing, if a user is not familiar with the way
built-in packages can be upgraded and are maintained.  Perhaps instead
of a simple yes-or-no-p, we could use read-multiple-choice to add some
more background?

>> Are you sure?  The `alist-get' call returns a list, since
>> `package-archive-contents' is a map of package names to a list of
>> package descriptors.
>
> Yeah, probably needs a cdr or car there for good measure.  Great.

Specifically (cadr (assq 'eglot package-archive-contents))

> João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 8:33 PM Philip Kaludercic <philipk@posteo.net> wrote:

> >> > Here's another solution you may want to consider.  package-install
> >> > in Emacs 29 updates built-ins non-interactively always.  Interactively
> >> > asks the user with a confirmation prompt.  This solution would
> >> > also work.
> >>
> >> What would the confirmation prompt ask the user?
> >
> > "Are you sure you want to install Foo which is [also|now] a built-in
> > package in Emacs 29?"
>
> I feel this is confusing, if a user is not familiar with the way
> built-in packages can be upgraded and are maintained.  Perhaps instead
> of a simple yes-or-no-p, we could use read-multiple-choice to add some
> more background?

Sure, I actually don't know what that does, but whatever you think
is best, I hope the point has gotten across.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 17:31:24 +0100
> Cc: Robert Pluim <rpluim@gmail.com>, Philip Kaludercic <philipk@posteo.net>, 62720@debbugs.gnu.org, 
> 	Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> BTW use-package also calls package-install, and
> (use-package eglot :ensure t) is the number one installation form
> for Eglot says google, chatGPT and 4 out of 5 bugs I get.
> Unless special steps are taken there, it will likely be broken,
> too.  That form will no longer give users the latest Eglot starting
> Emacs 29.

Is there any reason Eglot that will come with Emacs 29.1 will not be
the latest Eglot?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 6:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 14 Apr 2023 17:31:24 +0100
> > Cc: Robert Pluim <rpluim@gmail.com>, Philip Kaludercic <philipk@posteo.net>, 62720@debbugs.gnu.org,
> >       Eli Zaretskii <eliz@gnu.org>, larsi@gnus.org, monnier@iro.umontreal.ca
> >
> > BTW use-package also calls package-install, and
> > (use-package eglot :ensure t) is the number one installation form
> > for Eglot says google, chatGPT and 4 out of 5 bugs I get.
> > Unless special steps are taken there, it will likely be broken,
> > too.  That form will no longer give users the latest Eglot starting
> > Emacs 29.
>
> Is there any reason Eglot that will come with Emacs 29.1 will not be
> the latest Eglot?

Yes, it's not the latest Eglot already.  Eglot has been getting features
complex features and bugfixes in master that I haven't backported.
Eglot with Emacs 29.1 will be 1.12+simple bugfixes.

So if you're offline or don't like ELPA at all or are on sshing to a
machine, you can still have a decent LSP experience (well provided
you install a LSP server).


João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 19:44:35 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Fri, Apr 14, 2023 at 6:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Is there any reason Eglot that will come with Emacs 29.1 will not be
> > the latest Eglot?
> 
> Yes, it's not the latest Eglot already.  Eglot has been getting features
> complex features and bugfixes in master that I haven't backported.

Why not?

More generally, why does it make sense not to have in Emacs the latest
stable version of every core package at the time of the release?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 21:51, Eli Zaretskii wrote:
> More generally, why does it make sense not to have in Emacs the latest
> stable version of every core package at the time of the release?

Same reason we're now arguing about adding a simple change to package.el 
(which is also a core package)? Stability?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 14 Apr 2023 22:14:44 +0300
> Cc: rpluim@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/04/2023 21:51, Eli Zaretskii wrote:
> > More generally, why does it make sense not to have in Emacs the latest
> > stable version of every core package at the time of the release?
> 
> Same reason we're now arguing about adding a simple change to package.el 
> (which is also a core package)? Stability?

So Eglot 1.14 is not stable?  Does Eglot has a stable branch on ELPA?
I meant to include the latest stable release, not an unstable one.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 22:19, Eli Zaretskii wrote:
> So Eglot 1.14 is not stable?  Does Eglot has a stable branch on ELPA?

All Eglot development is led on master.

> I meant to include the latest stable release, not an unstable one.

We consider tagged releases (ones that bump the version header) as 
"stable" ones, and all intermediate commits as "unstable". Those are not 
the same kind of "stable" as Emacs releases, obviously.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 14 Apr 2023 22:21:41 +0300
> Cc: joaotavora@gmail.com, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/04/2023 22:19, Eli Zaretskii wrote:
> > So Eglot 1.14 is not stable?  Does Eglot has a stable branch on ELPA?
> 
> All Eglot development is led on master.
> 
> > I meant to include the latest stable release, not an unstable one.
> 
> We consider tagged releases (ones that bump the version header) as 
> "stable" ones, and all intermediate commits as "unstable". Those are not 
> the same kind of "stable" as Emacs releases, obviously.

So which version of Eglot is sufficiently "stable" to go with the
stable version of Emacs?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 8:21 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 14/04/2023 22:19, Eli Zaretskii wrote:
> > So Eglot 1.14 is not stable?  Does Eglot has a stable branch on ELPA?
>
> All Eglot development is led on master.
>
> > I meant to include the latest stable release, not an unstable one.
>
> We consider tagged releases (ones that bump the version header) as
> "stable" ones, and all intermediate commits as "unstable". Those are not
> the same kind of "stable" as Emacs releases, obviously.

Exactly this.  When I tag an Eglot release, I check that tests run.  I
run some manual tests and I come to a conclusion as to its stability.
But there is no pretest or "release candidate" or anything like that.
It's not feasible, I don't even know how that would work.
So I make the release, watch for feedback closely the next days or
weeks and fix bugs for the next release.  In the interim I give people
workarounds.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 7:51 PM Eli Zaretskii <eliz@gnu.org> wrote:

> Why not?
>
> More generally, why does it make sense not to have in Emacs the latest
> stable version of every core package at the time of the release?

Well, my answer to that would be: for the same reason it makes sense
for like new features not to go into Emacs release branches after
they have been cut.

The Eglot release 1.14 is not as stable as 1.12.29. It has new features.
Also a recent bugfix in commit a74403adda0 is quite complex and
I'm cautiously waiting for feedback.  Other simple bugfixes have
been backported.

So I'm not much different than most maintainers in that aspect, I suppose.
I also think stock Emacs releases should be very stable, despite
what you may think from my stance on this particular matter :-)

But if someone types M-x install-something they should get what
they ask for.  If they want to be 100% safe, they just shouldn't
invoke commands that download, compile and evaluate code.

Notwithstanding this personal opinion, I underline again that
it _is_ possible to craft a simple, emacs-29-safe modification,
to the package-install function that is even more cautious to
download certain types of things.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 20:20:20 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Fri, Apr 14, 2023 at 7:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> The Eglot release 1.14 is not as stable as 1.12.29. It has new features.
> Also a recent bugfix in commit a74403adda0 is quite complex and
> I'm cautiously waiting for feedback.  Other simple bugfixes have
> been backported.

So you don't recommend that users who want a stable Eglot upgrade to
1.14?  If so, why is it a problem that package-install by default
doesn't update built-in packages?  Users who want the cutting edge of
Eglot, and don't mind some instability, can always switch to the
master branch of Emacs, where we are free to change package-install to
upgrade core packages by default.

> But if someone types M-x install-something they should get what
> they ask for.  If they want to be 100% safe, they just shouldn't
> invoke commands that download, compile and evaluate code.

The logic should be consistent.  Emacs 29 is the stable branch of
Emacs, so it should come with the latest stable Eglot.  If that is
Eglot 1.12.29, then the fact that package-install won't upgrade it to
1.14 is consistent with the stability of Eglot's versions.  If, OTOH,
you think that it's imperative to allow _all_ users of Eglot with
Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
become available), then we should release Emacs 29 with 1.14.

> Notwithstanding this personal opinion, I underline again that
> it _is_ possible to craft a simple, emacs-29-safe modification,
> to the package-install function that is even more cautious to
> download certain types of things.

Philip presented such a safe modification, and we are in the final
stages of discussing its details, before it will be installed.  So
yes, it is possible.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Fri, Apr 14, 2023 at 8:28 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 14 Apr 2023 20:20:20 +0100
> > Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net,
> >       62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> >
> > On Fri, Apr 14, 2023 at 7:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > The Eglot release 1.14 is not as stable as 1.12.29. It has new features.
> > Also a recent bugfix in commit a74403adda0 is quite complex and
> > I'm cautiously waiting for feedback.  Other simple bugfixes have
> > been backported.
>
> So you don't recommend that users who want a stable Eglot upgrade to
> 1.14?

Depends on how "stable" they want it and how badly they want new
features.

> If so, why is it a problem that package-install by default
> doesn't update built-in packages?  Users who want the cutting edge of
> Eglot, and don't mind some instability, can always switch to the
> master branch of Emacs, where we are free to change package-install to
> upgrade core packages by default.

Users don't just switch to the master branch of Emacs.  Many just
can't because of enterprise complications, or just difficulty of compilation.
I've worked in companies using Emacs (some of them exclusively!!) for 20
years.  Most users are two, sometimes, three Emacs releases behind.
They are not remotely interested in updating.  "IT doesn't like it".
"It's not the official".  "This one is just fine".

But if a colleague goes to their workstation and shows them
M-x package-install RET very-fancy-nice-thing or sends them a
super-fancy init.el they will take it no problem, and buy you
coffee and rave about it. I can't be the only one who has
experienced this :-)

> > But if someone types M-x install-something they should get what
> > they ask for.  If they want to be 100% safe, they just shouldn't
> > invoke commands that download, compile and evaluate code.
>
> The logic should be consistent.  Emacs 29 is the stable branch of
> Emacs, so it should come with the latest stable Eglot.  If that is
> Eglot 1.12.29, then the fact that package-install won't upgrade it to
> 1.14 is consistent with the stability of Eglot's versions.  If, OTOH,
> you think that it's imperative to allow _all_ users of Eglot with
> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> become available), then we should release Emacs 29 with 1.14.

I think it's imperative to _allow_ -- as you say -- and also and
to _make easy_.  More importantly, and to the point, to _make it
as easy as it was in Emacs 26, 27 and 28_.  But I'd prefer not to force
Eglot 1.14 (or likely 1.15/1.16 by the time i expect the final RCs
to be around).

> > Notwithstanding this personal opinion, I underline again that
> > it _is_ possible to craft a simple, emacs-29-safe modification,
> > to the package-install function that is even more cautious to
> > download certain types of things.
>
> Philip presented such a safe modification, and we are in the final
> stages of discussing its details, before it will be installed.  So
> yes, it is possible.

As I've explained to Philip, the big drawback of that -- undoubtedly
safe -- modification is that it is not compatible to user's
configurations that have a (use-package 'eglot) or a
(package-install 'eglot) in them.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

>> > Notwithstanding this personal opinion, I underline again that
>> > it _is_ possible to craft a simple, emacs-29-safe modification,
>> > to the package-install function that is even more cautious to
>> > download certain types of things.
>>
>> Philip presented such a safe modification, and we are in the final
>> stages of discussing its details, before it will be installed.  So
>> yes, it is possible.

I might have missed a message, what was the last state here?

> As I've explained to Philip, the big drawback of that -- undoubtedly
> safe -- modification is that it is not compatible to user's
> configurations that have a (use-package 'eglot) or a
> (package-install 'eglot) in them.

Again: Are we sure about this?  After all, the package is installed
(which I think is the main thing), it just might not have the most
recent version.  Calling this "not compatible" seems excessive.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Philip Kaludercic <philipk@posteo.net> writes:

> João Távora <joaotavora@gmail.com> writes:
>>> Philip presented such a safe modification, and we are in the final
>>> stages of discussing its details, before it will be installed.  So
>>> yes, it is possible.
>
> I might have missed a message, what was the last state here?

Eli is talking about an interaction between you two, so you should be
able to figure out.

>> As I've explained to Philip, the big drawback of that -- undoubtedly
>> safe -- modification is that it is not compatible to user's
>> configurations that have a (use-package 'eglot) or a
>> (package-install 'eglot) in them.
>
> Again: Are we sure about this?  After all, the package is installed
> (which I think is the main thing), it just might not have the most
> recent version.  Calling this "not compatible" seems excessive.

I don't think we should look at this in legalese and disguise the fact
that the same form in Emacs 26, 27 and 28 has very different results.
In those versions (use-package 'eglot :ensure t) brings you always the
latest and greatest.  In Emacs 29, supposedly the more advanced version,
it will bring you a different, older, less powerful, and one that might
even be incompatible with the remainder of your configuration if you
aren't aware of this obscure quirk.

Furthermore the problem continues to worsen with time as packages
evolve.

This is incompatibility if there ever was such a thing.  So the "package
'eglot' is already installed" shown to the user, while technically
correct, is completely misleading.

The patch you're preparing, last I looked at it, doesn't solve this.  It
does makes Eglot slightly easier to upgrade, at least if the user is
aware of this quirky situation.  But it still doesn't fix the
use-package case, for example or the non-interactive config case.

Eli said something like this patch could be acceptable for master and
maybe 29.2.  

----------8<----------8<----------8<----------8<----------8<----------8<
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -652,6 +652,9 @@ package--builtins
 name (a symbol) and DESC is a `package--bi-desc' structure.")
 (put 'package--builtins 'risky-local-variable t)
 
+(defvar package--safely-upgradeable-builtins '(eglot use-package)
+  "See bug#62720 for longest docstring ever.")
+
 (defvar package-alist nil
   "Alist of all packages available for activation.
 Each element has the form (PKG . DESCS), where PKG is a package
@@ -2201,14 +2204,19 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
+                    (append
                      (delq nil
                            (mapcar (lambda (elt)
                                      (unless (package-installed-p (car elt))
                                        (symbol-name (car elt))))
                                    package-archive-contents))
+                     package--safely-upgradeable-builtins)
                     nil t))
            nil)))
   (package--archives-initialize)
+  (when-let ((desc (and (memq pkg package--safely-upgradeable-builtins)
+                        (cadr (assoc pkg package-archive-contents)))))
+    (setq pkg desc))
   (add-hook 'post-command-hook #'package-menu--post-refresh)
   (let ((name (if (package-desc-p pkg)
                   (package-desc-name pkg)
---------->8---------->8---------->8---------->8---------->8---------->8

1. Node code complexity.  It's 6 lines of trivial code.
2. Absolutely no risk of "silent installation of software that wasn't ever
   installed before"
3. Fixes all the aforementioned issues

Doesn't fix package-update, but that's minor in comparison, I didn't
want to add bloat this enormous patch with 2 more lines.

It is based on a hardcoded list.  Dmitry has suggested other ways to
whitelist.  There are infinite ways, of course.  Other have suggested
blacklists.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> João Távora <joaotavora@gmail.com> writes:
>>>> Philip presented such a safe modification, and we are in the final
>>>> stages of discussing its details, before it will be installed.  So
>>>> yes, it is possible.
>>
>> I might have missed a message, what was the last state here?
>
> Eli is talking about an interaction between you two, so you should be
> able to figure out.

I know, I have just lost track.


[...]

> Eli said something like this patch could be acceptable for master and
> maybe 29.2.  
>
> ----------8<----------8<----------8<----------8<----------8<----------8<
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -652,6 +652,9 @@ package--builtins
>  name (a symbol) and DESC is a `package--bi-desc' structure.")
>  (put 'package--builtins 'risky-local-variable t)
>  
> +(defvar package--safely-upgradeable-builtins '(eglot use-package)
> +  "See bug#62720 for longest docstring ever.")
> +
>  (defvar package-alist nil
>    "Alist of all packages available for activation.
>  Each element has the form (PKG . DESCS), where PKG is a package
> @@ -2201,14 +2204,19 @@ package-install
>       (package--archives-initialize)
>       (list (intern (completing-read
>                      "Install package: "
> +                    (append
>                       (delq nil
>                             (mapcar (lambda (elt)
>                                       (unless (package-installed-p (car elt))
>                                         (symbol-name (car elt))))
>                                     package-archive-contents))
> +                     package--safely-upgradeable-builtins)
>                      nil t))
>             nil)))
>    (package--archives-initialize)
> +  (when-let ((desc (and (memq pkg package--safely-upgradeable-builtins)
> +                        (cadr (assoc pkg package-archive-contents)))))
> +    (setq pkg desc))
>    (add-hook 'post-command-hook #'package-menu--post-refresh)
>    (let ((name (if (package-desc-p pkg)
>                    (package-desc-name pkg)
> ---------->8---------->8---------->8---------->8---------->8---------->8
>
> 1. Node code complexity.  It's 6 lines of trivial code.
> 2. Absolutely no risk of "silent installation of software that wasn't ever
>    installed before"
> 3. Fixes all the aforementioned issues

Personally I think this would be a fine solution considering the
constraints.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 15, 2023 at 11:40 AM Philip Kaludercic <philipk@posteo.net> wrote:

> > Eli said something like this patch could be acceptable for master and
> > maybe 29.2.
> >
> > ----------8<----------8<----------8<----------8<----------8<----------8<
> > --- a/lisp/emacs-lisp/package.el
> > +++ b/lisp/emacs-lisp/package.el
> > @@ -652,6 +652,9 @@ package--builtins
> >  name (a symbol) and DESC is a `package--bi-desc' structure.")
> >  (put 'package--builtins 'risky-local-variable t)
> >
> > +(defvar package--safely-upgradeable-builtins '(eglot use-package)
> > +  "See bug#62720 for longest docstring ever.")
> > +
> >  (defvar package-alist nil
> >    "Alist of all packages available for activation.
> >  Each element has the form (PKG . DESCS), where PKG is a package
> > @@ -2201,14 +2204,19 @@ package-install
> >       (package--archives-initialize)
> >       (list (intern (completing-read
> >                      "Install package: "
> > +                    (append
> >                       (delq nil
> >                             (mapcar (lambda (elt)
> >                                       (unless (package-installed-p (car elt))
> >                                         (symbol-name (car elt))))
> >                                     package-archive-contents))
> > +                     package--safely-upgradeable-builtins)
> >                      nil t))
> >             nil)))
> >    (package--archives-initialize)
> > +  (when-let ((desc (and (memq pkg package--safely-upgradeable-builtins)
> > +                        (cadr (assoc pkg package-archive-contents)))))
> > +    (setq pkg desc))
> >    (add-hook 'post-command-hook #'package-menu--post-refresh)
> >    (let ((name (if (package-desc-p pkg)
> >                    (package-desc-name pkg)
> > ---------->8---------->8---------->8---------->8---------->8---------->8
> >
> > 1. Node code complexity.  It's 6 lines of trivial code.
> > 2. Absolutely no risk of "silent installation of software that wasn't ever
> >    installed before"
> > 3. Fixes all the aforementioned issues
>
> Personally I think this would be a fine solution considering the
> constraints.

Me too, of course.  Good luck :-)

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 15/04/2023 13:40, Philip Kaludercic wrote:
>> Eli said something like this patch could be acceptable for master and
>> maybe 29.2.
>>
>> ----------8<----------8<----------8<----------8<----------8<----------8<
>> --- a/lisp/emacs-lisp/package.el
>> +++ b/lisp/emacs-lisp/package.el
>> @@ -652,6 +652,9 @@ package--builtins
>>   name (a symbol) and DESC is a `package--bi-desc' structure.")
>>   (put 'package--builtins 'risky-local-variable t)
>>   
>> +(defvar package--safely-upgradeable-builtins '(eglot use-package)
>> +  "See bug#62720 for longest docstring ever.")
>> +
>>   (defvar package-alist nil
>>     "Alist of all packages available for activation.
>>   Each element has the form (PKG . DESCS), where PKG is a package
>> @@ -2201,14 +2204,19 @@ package-install
>>        (package--archives-initialize)
>>        (list (intern (completing-read
>>                       "Install package: "
>> +                    (append
>>                        (delq nil
>>                              (mapcar (lambda (elt)
>>                                        (unless (package-installed-p (car elt))
>>                                          (symbol-name (car elt))))
>>                                      package-archive-contents))
>> +                     package--safely-upgradeable-builtins)
>>                       nil t))
>>              nil)))
>>     (package--archives-initialize)
>> +  (when-let ((desc (and (memq pkg package--safely-upgradeable-builtins)
>> +                        (cadr (assoc pkg package-archive-contents)))))
>> +    (setq pkg desc))
>>     (add-hook 'post-command-hook #'package-menu--post-refresh)
>>     (let ((name (if (package-desc-p pkg)
>>                     (package-desc-name pkg)
>> ---------->8---------->8---------->8---------->8---------->8---------->8
>>
>> 1. Node code complexity.  It's 6 lines of trivial code.
>> 2. Absolutely no risk of "silent installation of software that wasn't ever
>>     installed before"
>> 3. Fixes all the aforementioned issues
> Personally I think this would be a fine solution considering the
> constraints.

+1

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 14 Apr 2023 20:46:02 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Fri, Apr 14, 2023 at 8:28 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Date: Fri, 14 Apr 2023 20:20:20 +0100
> > > Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net,
> > >       62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> > >
> > > On Fri, Apr 14, 2023 at 7:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > The Eglot release 1.14 is not as stable as 1.12.29. It has new features.
> > > Also a recent bugfix in commit a74403adda0 is quite complex and
> > > I'm cautiously waiting for feedback.  Other simple bugfixes have
> > > been backported.
> >
> > So you don't recommend that users who want a stable Eglot upgrade to
> > 1.14?
> 
> Depends on how "stable" they want it and how badly they want new
> features.

You are dodging the question, which is unfortunate.  It is IMO an
important question, much more general and important than this
tempest-in-a-teapot issue we are discussing here.  We will need to
revisit this question seriously if we ever succeed in removing 'core'
packages from emacs.git and bundling them with release tarballs.  At
that time, we will need clear-cut criteria for how to select the
version of a package that is the most suitable one to go into the
tarball.  And that in turn will require the developers of each such
package to seriously address the questions I've asked: what are the
criteria for considering a package version "stable", and how is that
communicated via version numbers and/or repository branches.  For
example, Org has that encoded in its branches, so that the latest
changes from its stable branch are routinely merged into the stable
branch of Emacs, and the latest code from that branch will be in the
Emacs release tarball.

Based on your answer, it sounds like Eglot users are on their own in
this aspect: they have no real guidance from you which version is
currently considered "stable".

> > If so, why is it a problem that package-install by default
> > doesn't update built-in packages?  Users who want the cutting edge of
> > Eglot, and don't mind some instability, can always switch to the
> > master branch of Emacs, where we are free to change package-install to
> > upgrade core packages by default.
> 
> Users don't just switch to the master branch of Emacs.  Many just
> can't because of enterprise complications, or just difficulty of compilation.
> I've worked in companies using Emacs (some of them exclusively!!) for 20
> years.  Most users are two, sometimes, three Emacs releases behind.
> They are not remotely interested in updating.  "IT doesn't like it".
> "It's not the official".  "This one is just fine".
> 
> But if a colleague goes to their workstation and shows them
> M-x package-install RET very-fancy-nice-thing or sends them a
> super-fancy init.el they will take it no problem, and buy you
> coffee and rave about it. I can't be the only one who has
> experienced this :-)

I have no doubt there are such cases.  But I very much doubt they are
the majority.  Whatever we decide, it is clear that some use cases
will want a different behavior.  The issue discussed here is what
should be the default behavior, and whether the default behavior of
Emacs 29.1 would be wrong for most users of Eglot and of other core
packages.

To reiterate: I think each release of Emacs should ship with the
latest stable version of each core package.  If this is the case, the
need to upgrade core packages via package.el is not very important for
the majority of our users who prefer to use a stable Emacs.  Thus, the
arguments you present emphasize the needs of the minority, and
therefore I don't consider them strong enough to invalidate the
compromise solution to which we are converging.

> > > But if someone types M-x install-something they should get what
> > > they ask for.  If they want to be 100% safe, they just shouldn't
> > > invoke commands that download, compile and evaluate code.
> >
> > The logic should be consistent.  Emacs 29 is the stable branch of
> > Emacs, so it should come with the latest stable Eglot.  If that is
> > Eglot 1.12.29, then the fact that package-install won't upgrade it to
> > 1.14 is consistent with the stability of Eglot's versions.  If, OTOH,
> > you think that it's imperative to allow _all_ users of Eglot with
> > Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> > become available), then we should release Emacs 29 with 1.14.
> 
> I think it's imperative to _allow_ -- as you say -- and also and
> to _make easy_.

"C-u M-x package-install" is easy enough.

> More importantly, and to the point, to _make it as easy as it was in
> Emacs 26, 27 and 28_.

That's an impractical request, one that most probably prefers Eglot
users (and only part of them at that) to those of other core packages.
We must think about all the users, not just users of Eglot.  The price
of adaptation to the fact that Eglot is now a core package, while it
is not nil, is not high.  So once again, this solution is IMO a
reasonable compromise, given the constraints.

> > Philip presented such a safe modification, and we are in the final
> > stages of discussing its details, before it will be installed.  So
> > yes, it is possible.
> 
> As I've explained to Philip, the big drawback of that -- undoubtedly
> safe -- modification is that it is not compatible to user's
> configurations that have a (use-package 'eglot) or a
> (package-install 'eglot) in them.

That is not a problem serious enough to invalidate the solution.  We
can, for example, mention the change in NEWS, to alleviate some of the
costs.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> > So you don't recommend that users who want a stable Eglot upgrade to
>> > 1.14?
>> Depends on how "stable" they want it and how badly they want new
>> features.
> You are dodging the question, which is unfortunate.  

I'm not.  I don't have an answer.  I can't have one.  It depends on the
user's wishes.

Case 1: You use Emacs in a debian server VM you're setting up, perhaps
in a remote location, perhaps shared with other users.  Just 'sudo apt
install emacs clangd'.  No .emacs file at all and you can go edit little
C programs conveniently and quickly.  Then I recommend Eglot 1.12.29
bundled with emacs 29.

Case 2: For your "daily driver" home system, I recommend `M-x
package-install RET eglot' (at least I used to, that is ) since that
will bring you the latest nice features tested to a good degree, but not
as well tested as 1.12.29.

> important question, much more general and important than this
> tempest-in-a-teapot issue we are discussing here.  

If I were to use your rhetoric, I would say you are "shifting" the
question.  I've never had real problems about this.  But if you want to
"revisit this question seriously", I'll be there, of course.

> We will need to revisit this question seriously if we ever succeed in
> removing 'core'
...
> Based on your answer, it sounds like Eglot users are on their own in
> this aspect: they have no real guidance from you which version is
> currently considered "stable".

They have the NEWS file, the bug tracker database, and my best efforts.
I told you how I make releases.  I want them to be stable, but there is
no official pretest.  "Guidance"?  Are the two cases above I gave
guidance?  Just present some actual case and I'll give you my
recommendation.  Else it's like coming to a doctor and asking about
taking a drug: the doctor probably going to ask you questions.

>> But if a colleague goes to their workstation and shows them
>> M-x package-install RET very-fancy-nice-thing or sends them a
>> super-fancy init.el they will take it no problem, and buy you
>> coffee and rave about it. I can't be the only one who has
>> experienced this :-)
>
> I have no doubt there are such cases.  But I very much doubt they are
> the majority.  

I don't, whole '.emacs' get shared like crazy.  That's the number one
method of "getting a configuration". People see other's Emacs in
screenshots or over the shoulder and ask to see their init file.  Not
just Emacs of course, any util.  Then they copy over bits they think are
cool/useful.  What you think they won't copy "use-package" forms??

People don't read NEWS and they don't read the manual.  Most Emacs users
I knew don't even know NEWS exists, what with that strange all caps
name.  They probably think it's some major mode for journalists.

> To reiterate: I think each release of Emacs should ship with the
> latest stable version of each core package.  If this is the case, the
> need to upgrade core packages via package.el is not very important for
> the majority of our users who prefer to use a stable Emacs.  Thus, the
> arguments you present emphasize the needs of the minority, and
> therefore I don't consider them strong enough to invalidate the
> compromise solution to which we are converging.

Sorry, but IMO you come up with very complicated logic and premises
about stability just to arrive at where you want to arrive.

A simple, readily verifiable fact remains.  M-x package-install RET
eglot in Emacs 29 will bring you a older version than in Emacs 28.  If
not today, tomorrow, eventually it will.  And that's just bad in my
opinion.  And it will become worse.

If, in your opinion, this is somehow a good or indifferent thing, we can
stop the whole discussion right here.

Again: if you think it's a _good thing_ that, in Emacs 29

  (use-package eglot :ensure t) 

or

  (package-install 'eglot)

or 

  M-x package-install RET eglot

produces an older version of Eglot than the very same form in Emacs 26,
27 and 28, please do say so ASAP.

I was under the impression that you didn't prefer that, but I'm not sure
anymore after reading your complex last paragraph.

>> I think it's imperative to _allow_ -- as you say -- and also and
>> to _make easy_.
>
> "C-u M-x package-install" is easy enough.
>
>> More importantly, and to the point, to _make it as easy as it was in
>> Emacs 26, 27 and 28_.
>
> That's an impractical request, one that most probably prefers Eglot
> users (and only part of them at that) to those of other core packages.
> We must think about all the users, not just users of Eglot.  The price
> of adaptation to the fact that Eglot is now a core package, while it
> is not nil, is not high.  So once again, this solution is IMO a
> reasonable compromise, given the constraints.

It's _not_ impractical.  There doesn't have to be a "compromise
solution" at all.  There _can_ be a win-win solution (presuming I even
understand what counts as a win to you in this issue).

I just gave Philip a 7 line patch, my fourth in this discussion, that
does exactly I think answers all of your objections regarding
risk/stability.  Much simpler than any patch so far by _any_ measure.
Lines of code, cyclomatic complexity, number of user options, NEWS
changes.  Will you look at it?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: dmitry@gutov.dev,  rpluim@gmail.com,  philipk@posteo.net,
>   62720@debbugs.gnu.org,  larsi@gnus.org,  monnier@iro.umontreal.ca
> Date: Sat, 15 Apr 2023 11:24:41 +0100
> 
> A simple, readily verifiable fact remains.  M-x package-install RET
> eglot in Emacs 29 will bring you a older version than in Emacs 28.  If
> not today, tomorrow, eventually it will.  And that's just bad in my
> opinion.  And it will become worse.

It is not worse, it is better: they get Eglot integrated into Emacs.

> If, in your opinion, this is somehow a good or indifferent thing, we can
> stop the whole discussion right here.

Yes, let's stop.  There's no reason to keep it going.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Kévin Le Gouguec]

João Távora <joaotavora@gmail.com> writes:

> A simple, readily verifiable fact remains.  M-x package-install RET
> eglot in Emacs 29 will bring you a older version than in Emacs 28.  If
> not today, tomorrow, eventually it will.  And that's just bad in my
> opinion.  And it will become worse.
>
> If, in your opinion, this is somehow a good or indifferent thing, we can
> stop the whole discussion right here.
>
> Again: if you think it's a _good thing_ that, in Emacs 29
>
>   (use-package eglot :ensure t) 
>
> or
>
>   (package-install 'eglot)
>
> or 
>
>   M-x package-install RET eglot
>
> produces an older version of Eglot than the very same form in Emacs 26,
> 27 and 28, please do say so ASAP.
>
> I was under the impression that you didn't prefer that, but I'm not sure
> anymore after reading your complex last paragraph.

If I may throw in my ¢2: in Emacs ≤28, users never had a choice between
a "installing the newest Eglot from GNU ELPA" and "requiring that Eglot
be available, possibly not the latest & greatest".  They could only
request the former.

It's anyone's guess which of those two things users who cargo-cult those
configuration lines would prefer, now that the question is up in the air
for Emacs 29.  FWIW, I'd lean toward the latter: IMHO, package-install,
resp. (use-package … :ensure t), merely suggest ensuring availability,
not proactively ugprading to the latest (unlike say "package-update").

So I wouldn't be shocked for package-install to be a no-op for :core
packages, the semantics being "make sure the package is present,
favoring any built-in version which presumably underwent lots of
validation & stability fixes on the release branch".

With that perspective, I don't think the change in behaviour users will
observe in Emacs 29 re. which version of Eglot they get with `M-x
package-install eglot' is necessarily "worse": it depends on how much
one values "a release branch's worth of stability fixes" vs "a
development branch's worth of new features".

(
  Although I can understand that, in the _specific_ context of an LSP
  client that is in active development & "competing" with a MELPA-only
  alternative, it is a bit of a bummer that M-x package-install 𝒫 will
  yield something that users might consider "inferior" feature-wise when
  𝒫=eglot.  A bit of a bummer, but not a deal-breaker IMO; as long as
  "M-x package-list U x" brings the latest & greatest, I still think
  package-install's behaviour change re. eglot in Emacs 29 is
  defensible.
)

(
  Sorry for butting in and adding more words to this lengthy discussion;
  just thought that hearing the perspective from one random user might
  help.  I must also confess that I might not have read the whole thing
  as attentively as I perhaps should have; the parallel subthread
  between Eli & Philip re. changing package-install or package-update
  makes me unsure what "U x" will actually do with eglot in Emacs 29, so
  my previous parenthesized digression might be moot.
)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 15/04/2023 14:19, Kévin Le Gouguec wrote:

> If I may throw in my ¢2: in Emacs ≤28, users never had a choice between
> a "installing the newest Eglot from GNU ELPA" and "requiring that Eglot
> be available, possibly not the latest & greatest".  They could only
> request the former.
> 
> It's anyone's guess which of those two things users who cargo-cult those
> configuration lines would prefer, now that the question is up in the air
> for Emacs 29.  FWIW, I'd lean toward the latter: IMHO, package-install,
> resp. (use-package … :ensure t), merely suggest ensuring availability,
> not proactively ugprading to the latest (unlike say "package-update").
> 
> So I wouldn't be shocked for package-install to be a no-op for :core
> packages, the semantics being "make sure the package is present,
> favoring any built-in version which presumably underwent lots of
> validation & stability fixes on the release branch".

I can easily buy this argument (and made it myself in a different form, 
I think), but the problem is, there is no easily discoverable way for 
the user to force the upgrade. They'll need to hunt the docs for how to 
do that.

> (
>    Although I can understand that, in the _specific_ context of an LSP
>    client that is in active development & "competing" with a MELPA-only
>    alternative, it is a bit of a bummer that M-x package-install 𝒫 will
>    yield something that users might consider "inferior" feature-wise when
>    𝒫=eglot.  A bit of a bummer, but not a deal-breaker IMO; as long as
>    "M-x package-list U x" brings the latest & greatest, I still think
>    package-install's behaviour change re. eglot in Emacs 29 is
>    defensible.
> )

'M-x package-list U x' does not upgrade built-in Eglot either, AFAIU.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

>   yield something that users might consider "inferior" feature-wise when
>   𝒫=eglot.  
> A bit of a bummer, but not a deal-breaker IMO; 

It's not the end of the world.  But certainly not good, especially if
those users have just upgraded to Emacs 29 on some new machine and
ported over their Emacs 28 config.  It could even break.  In fact
eventually it will simply break, if Eglot 1.16 adds, say, some
eglot-fancy-eldoc-function and the config has

  (package-install 'eglot)
  (add-hook 'eldoc-documentation-functions 'eglot-fancy-eldoc-function)

This will break sooner or later, and bizzarely so, IMO.

Just as a data point, I got a lot of confused users just because of
bug#62576 and the missing "project-name" function.  And, mind, you
there, the simplest of workarounds, restarting Emacs fixes the issue.
But it still confused a load of users so eventually I tooks steps to
avoid it.

Just see
https://github.com/joaotavora/eglot/search?q=project-name&type=discussions

>   Sorry for butting in and adding more words to this lengthy discussion;
>   just thought that hearing the perspective from one random user might
>   help.  

No problem.  I always like reading your feedback :-)

Let's see where the tip of this discussion is heading.  Philip proposes:

> > - User option to enable upgrading built-in packages
> > - Prefix argument to enable upgrading built-in packages
> > - Always upgrade built-in packages

And Eli replies:

> The first two on emacs-29, the last one on master 

Now, this means that things like (use-package eglot :ensure t) in Emacs
26, 27 and 28 will rev up Eglot to the very latest, Emacs 29 will stay at
1.12.29 and Emacs 30, 31, 32 again to the very latest.

This is almost the mathematical definition of instability.  And the
amplitude of the version cliff will just keep growing.  I just don't
think it's a good thing, but if Eli and you think it is, that's
perfectly legitimate.

The key thing here is that there was a package hitherto with given
 update semantics.  That package wasn't in core and now is.  It's a clear
clash of update semantics, and noone (including me) was able to foresee
it.

I do appreciate all the other arguments for stability.  Because of that
I proposed a very simple patch that addresses everyone's concern but
unfortunately I don't think is going anywhere -- who knows for what
reason.  Even though Philip, Dmitry and I himself think it is the
cleanest solution for emacs 29.  See the patch here: it's 6-7 lines of
code.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#467

>   between Eli & Philip re. changing package-install or package-update
>   makes me unsure what "U x" will actually do with eglot in Emacs 29, so
>   my previous parenthesized digression might be moot.

Alas U x in the package menu _also_ doesn't upgrade Eglot.  And neither
does M-x package-update-all.  I don't see any plans for doing so.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

João Távora <joaotavora@gmail.com> writes:

>>   between Eli & Philip re. changing package-install or package-update
>>   makes me unsure what "U x" will actually do with eglot in Emacs 29, so
>>   my previous parenthesized digression might be moot.
>
> Alas U x in the package menu _also_ doesn't upgrade Eglot.  And neither
> does M-x package-update-all.  I don't see any plans for doing so.

But selecting the package with I and then installing it will "update"
it -- which is a good argument in favour of using `package-install'.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Kévin Le Gouguec]

João Távora <joaotavora@gmail.com> writes:

>>   between Eli & Philip re. changing package-install or package-update
>>   makes me unsure what "U x" will actually do with eglot in Emacs 29, so
>>   my previous parenthesized digression might be moot.
>
> Alas U x in the package menu _also_ doesn't upgrade Eglot.  And neither
> does M-x package-update-all.  I don't see any plans for doing so.

Interesting; thank you and Dmitry for confirming my impression.

This was perhaps the most surprising aspect of the equation IMO,
although now it does not sound so bad since, IIUC from reading
package.el, _once_ users manage to install eglot from ELPA (using
Philip's pending user option), _then_ either method should let them
fetch future upgrades transparently.

(
  Took me a couple of minutes to reach that conclusion; found it mildly
  confusing that package-update-all and package-menu-mark-upgrades each
  have their own heuristics for enumerating candidates.  IIUC the former
  iterates through package-alist; the latter looks at all *Packages*
  lines with status ∈ '("installed" "dependency" "unsigned" "external").

  So once eglot becomes "installed" under package-user-dir, both methods
  should allow users to stay on top of new versions… ?

  Hope that's not my optimism skewing my reading of the code.
)

(
  Neither here nor there, but this whole discussion reminds me of
  bug#59005, where another :core package (transient) gets silently
  updated from ELPA if it's a dependency of a package that the user asks
  to install (magit).  Assuming (a) I correctly understand what that
  other bug report is about (b) I correctly understand what Philip's
  patch does, ISTM that it will cause a behaviour change in that
  situation too?

  Not wholly sure about (a) nor (b) though, and even then, not wholly
  sure that the change would be for the worse, especially since there
  would now be a user option to make the desired behaviour explicit.
)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>>>   between Eli & Philip re. changing package-install or package-update
>>>   makes me unsure what "U x" will actually do with eglot in Emacs 29, so
>>>   my previous parenthesized digression might be moot.
>>
>> Alas U x in the package menu _also_ doesn't upgrade Eglot.  And neither
>> does M-x package-update-all.  I don't see any plans for doing so.
>
> Interesting; thank you and Dmitry for confirming my impression.
>
> This was perhaps the most surprising aspect of the equation IMO,
> although now it does not sound so bad since, IIUC from reading
> package.el, _once_ users manage to install eglot from ELPA (using
> Philip's pending user option), _then_ either method should let them
> fetch future upgrades transparently.

Yes, there are two sides to this question.

- User friendliness to interactive commands.  I think it's not very user
  friendly to do one behaviour in Emacs 28 and another in 29 and again
  another in 30.  But I've abandoned this battle.  When doing things
  interactively, users are more likely to ask themselves why and search
  for answers.  So the solution is not that bad.

- Non-interactive installations, such as the ones you get when trying
  someone's init files, setting up new systems after upgrade, CI system
  scripts etc.  Those will be broken, silently or violently, and the
  root cause will be much harder to find.  

  For example, a user enjoying, say, the bug fix to
  https://github.com/joaotavora/eglot/discussions/1206 in Emacs 28 with
  a simple (use-package eglot :ensure t) in her config, will suddenly
  see the bug pop up again in Emacs 29.  The same with a user enjoying
  the no-flooding-echo-area bugfix feature, which depended on changes to
  both Eglot and Eldoc.  The same for a user enjoying the new
  eglot-prefer-plaintext feature for bug#61373.

It's the second problem which is more serious IMO, and it's inevitable
it will keep geting worse.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 22:28, Eli Zaretskii wrote:
> If, OTOH,
> you think that it's imperative to allow_all_  users of Eglot with
> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> become available), then we should release Emacs 29 with 1.14.

Was this question about stability only?

Because since we've decided in favor of stability of package.el, and 
against eglot's easy upgradability, I would suggest to backport Eglot 
1.14 to emacs-29. Together with the fix for bug#62816. The improvement 
in eldoc's behavior is pretty stark.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sun, Apr 16, 2023 at 9:46 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 14/04/2023 22:28, Eli Zaretskii wrote:
> > If, OTOH,
> > you think that it's imperative to allow_all_  users of Eglot with
> > Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> > become available), then we should release Emacs 29 with 1.14.
> Was this question about stability only?
>
> Because since we've decided in favor of stability of package.el, and
> against eglot's easy upgradability,

Needlessly, I might add, since the simplest patch that did both things
was completely ignored (except for you and Philip).

> I would suggest to backport Eglot
> 1.14 to emacs-29. Together with the fix for bug#62816. The improvement
> in eldoc's behavior is pretty stark.

I'm against that (though I could be convinced).  Note that there are many
things to backport, such as non-bugfix improvements in Eglot's dependencies,
which would _also_ have to be backported.  In this case ElDoc 1.14.  And
if you want the full "stark improvement" package for the echo-area flooding
business, Flymake would likely also need to be backported.

Besides, by the time Emacs 29 is released.  Eglot will probably be at
1.15 or even more with the "breadcrumb headline" feature request of
bug#58431, a class hierarchy browser and a few others.

We could cherry-pick the backports you think are most valuable though,
like bug#62816 (ask to reopen there and convince Eli that the fix
is sound) and maybe a few more.  I'd support that.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sun, 16 Apr 2023 22:54:05 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> Besides, by the time Emacs 29 is released.  Eglot will probably be at
> 1.15 or even more with the "breadcrumb headline" feature request of
> bug#58431, a class hierarchy browser and a few others.

What would prevent us from upgrading Eglot on the release branch to
1.15, when that becomes available?  The Org folks already merge every
change on their stable branch to the Emacs release branch, right up to
the release date.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sun, 16 Apr 2023 23:46:37 +0300
> Cc: rpluim@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/04/2023 22:28, Eli Zaretskii wrote:
> > If, OTOH,
> > you think that it's imperative to allow_all_  users of Eglot with
> > Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> > become available), then we should release Emacs 29 with 1.14.
> 
> Was this question about stability only?

It was about the criteria for which versions of core packages to ship
with a release.

> Because since we've decided in favor of stability of package.el, and 
> against eglot's easy upgradability, I would suggest to backport Eglot 
> 1.14 to emacs-29.

I won't object.  In fact, I asked up front why not.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 17/04/2023 05:24, Eli Zaretskii wrote:
>> Date: Sun, 16 Apr 2023 23:46:37 +0300
>> Cc: rpluim@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 14/04/2023 22:28, Eli Zaretskii wrote:
>>> If, OTOH,
>>> you think that it's imperative to allow_all_  users of Eglot with
>>> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
>>> become available), then we should release Emacs 29 with 1.14.
>>
>> Was this question about stability only?
> 
> It was about the criteria for which versions of core packages to ship
> with a release.

I don't think we can get a single set of criteria across core packages.

E.g. Org is developed externally, has its own community of significant 
size, and does split off release branches (with additional testing, I',m 
guessing).

Eglot, OTOH, is developed only here, with no additional release workflow 
other than what MELPA/GNU ELPA historically provided: collect up some 
features/fixes, bump the Version header, and push a new release out to 
the users. The lack of extended testing period is made up for with the 
capability to push out a new fixed version overnight. That's why the 
difficulty in upgrading to the latest version (for Emacs 29 users) is 
going to hurt.

BTW, if you recall the threads before Eglot was added, I was against 
that, and one of the things I cited is an LSP client has inherently high 
development velocity. Maybe the LSP community will settle/mature/stop 
adding features one day, but it's not there yet.

>> Because since we've decided in favor of stability of package.el, and
>> against eglot's easy upgradability, I would suggest to backport Eglot
>> 1.14 to emacs-29.
> 
> I won't object.  In fact, I asked up front why not.

Note that that suggestion comes with a fix to eldoc which you so far 
have rejected for emacs-29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Tue, Apr 18, 2023 at 2:25 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > It was about the criteria for which versions of core packages to ship
> > with a release.
>
> I don't think we can get a single set of criteria across core packages.

Agree.  We should have categories and understand what is surprising
or workflow-breaking and what is not.

> BTW, if you recall the threads before Eglot was added, I was against
> that, and one of the things I cited is an LSP client has inherently high
> development velocity. Maybe the LSP community will settle/mature/stop
> adding features one day, but it's not there yet.

Very true, but the conclusion is only half-true.

It didn't have to be like this: Eglot _can_ grow rapidly in master
and and have its periodic stable releases.  And in the major Emacs
versions released to the public could have an even stabler release
(because it went through more testing).  This is just like any other
:core package until now.

The solution picked for this issue is bad in that it breaks some Eglot
users workflows and expectations when using very common configuration
recipes.  We should revert it and pick a fix that relies on recognizing
that there are different "sets of criteria" as you propose.  One
such fix remains uncriticized and unchallenged in this thread.

But if that doesn't happen, we shouldn't make a bad situation worse,
by backporting 100's of lines of code of Eglot and friends into Emacs 29.
That's the polar opposite in the pursuit of stability.  Hand-picked
bugfixes for problems manifesting themselves in Emacs 29, sure! But
wholesale changes are just asking for trouble and destroying the
value of the pretest and RC periods.  As bug#62907 shows, there are
certain edge-case bugs due to refactorings in upcoming Eglot 1.15
that are not in 1.12.29 bundled with Emacs 29.  Good! That's the way
it should be. Let's not ruin that.

> >> Because since we've decided in favor of stability of package.el, and
> >> against eglot's easy upgradability, I would suggest to backport Eglot
> >> 1.14 to emacs-29.
> >
> > I won't object.  In fact, I asked up front why not.
>
> Note that that suggestion comes with a fix to eldoc which you so far
> have rejected for emacs-29.

...to name but one change to the non-Eglot, already-there-in-Emacs-28,
libraries Eglot depends on (or will depend on).

I do think _that_ ElDoc fix should be just backported.  It's not
a complicated fix by any measure, it's easy to test, and it indeed
has value and safety.  Together with your similar fix for Company,
it'll make Emacs 29 users happier.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 18/04/2023 14:44, João Távora wrote:

>> BTW, if you recall the threads before Eglot was added, I was against
>> that, and one of the things I cited is an LSP client has inherently high
>> development velocity. Maybe the LSP community will settle/mature/stop
>> adding features one day, but it's not there yet.
> 
> Very true, but the conclusion is only half-true.
> 
> It didn't have to be like this: Eglot _can_ grow rapidly in master
> and and have its periodic stable releases.  And in the major Emacs
> versions released to the public could have an even stabler release
> (because it went through more testing).  This is just like any other
> :core package until now.

This particular one didn't have to, but it's a problem very 
characteristic of joining a strongly centralized project with ultimately 
one person having the last word in all major decisions. And it's not 
like Eli is being unreasonable: we do need a stability cutoff, and we're 
really long past it. These one-more-change kind of arguments repeat year 
over year, with reasonable, well-intentioned people on both sides.

I don't see any better solutions than better modularity (which delegates 
responsibility by its nature) and/or more frequent releases somehow.

> The solution picked for this issue is bad in that it breaks some Eglot
> users workflows and expectations when using very common configuration
> recipes.  We should revert it and pick a fix that relies on recognizing
> that there are different "sets of criteria" as you propose.  One
> such fix remains uncriticized and unchallenged in this thread.

Sure, and I agree, but I don't really see how to present that in terms 
Eli would feel suitable to accept. One "trick" that worked in the past 
was to somehow enumerate all potential execution flows (functions 
involved, etc) that would be affected by the change.

> But if that doesn't happen, we shouldn't make a bad situation worse,
> by backporting 100's of lines of code of Eglot and friends into Emacs 29.
> That's the polar opposite in the pursuit of stability.  Hand-picked
> bugfixes for problems manifesting themselves in Emacs 29, sure! But
> wholesale changes are just asking for trouble and destroying the
> value of the pretest and RC periods.  As bug#62907 shows, there are
> certain edge-case bugs due to refactorings in upcoming Eglot 1.15
> that are not in 1.12.29 bundled with Emacs 29.  Good! That's the way
> it should be. Let's not ruin that.

I don't insist, not at all. It was just my own impression of what would 
constitute a reasonable Eglot release that we could be satisfied with 
having a large number of people use without upgrading, for years. Issues 
like blinking eldoc messages, or eldoc messages that can take up half 
the height of the window seem like things that we wouldn't want in it.

Perhaps the second issue affects only a minority of servers, and I'm 
wrong to be worried. Because otherwise, I really don't understand why it 
hasn't been reported and fixed until recently. Not blaming you, just to 
be clear.

>>>> Because since we've decided in favor of stability of package.el, and
>>>> against eglot's easy upgradability, I would suggest to backport Eglot
>>>> 1.14 to emacs-29.
>>>
>>> I won't object.  In fact, I asked up front why not.
>>
>> Note that that suggestion comes with a fix to eldoc which you so far
>> have rejected for emacs-29.
> 
> ...to name but one change to the non-Eglot, already-there-in-Emacs-28,
> libraries Eglot depends on (or will depend on).
> 
> I do think _that_ ElDoc fix should be just backported.  It's not
> a complicated fix by any measure, it's easy to test, and it indeed
> has value and safety.  Together with your similar fix for Company,
> it'll make Emacs 29 users happier.

That's what I'm thinking, too.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Tue, Apr 18, 2023 at 9:38 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> This particular one didn't have to, but it's a problem very
> characteristic of joining a strongly centralized project with ultimately
> one person having the last word in all major decisions. And it's not
> like Eli is being unreasonable: we do need a stability cutoff, and we're
> really long past it. These one-more-change kind of arguments repeat year
> over year, with reasonable, well-intentioned people on both sides.

Yes. And here both people sides "one more change".  The change that _did_
make it in is more aggressive and more unstable than the one that didn't.

And I repeat it didn't (and doesn't) have to be like this for Eglot and
Emacs 29.  It could have been a minor decision, not a "major" one.

> Sure, and I agree, but I don't really see how to present that in terms
> Eli would feel suitable to accept. One "trick" that worked in the past
> was to somehow enumerate all potential execution flows (functions
> involved, etc) that would be affected by the change.

Right.  And IMO it's not a "trick", it's how it should be.  It's hard to prove a
negative, but at least it should be attempted.  Well, the patch I presented
(the one you +1'ed) makes it so that package-install keeps exactly its previous
behaviour unless its argument is one of (eglot use-package), which are arguments
that could not have ever been passed to that function as :core packages
in Emacs 28. M-x package-install RET seq or (package-install 'xref) keep EXACTLY
the same behaviour.  It's very clear to see from the minimal patch.

> I don't insist, not at all. It was just my own impression of what would
> constitute a reasonable Eglot release that we could be satisfied with
> having a large number of people use without upgrading, for years. Issues
> like blinking eldoc messages, or eldoc messages that can take up half
> the height of the window seem like things that we wouldn't want in it.

The issue has existed and has been worked around successfully for a long
period of time.  It's not actually a problem, it's a consequence of the
default values for eldoc-echo-area-use-multiline-p and max-mini-window-height,
both of which predate Eglot.

Of course I think the current behaviour is better.  But it's also different so
I don't think we should backport that particular one.  Even if so far the
only feedback we have had has been positive, it could well be that some
people _liked_ the half-the-window-height thing (after all their customization
options reflected that wish, even if by default).

And by the way, not really half-the-window.  I used this for a long time
without being much bothered by it.

> Perhaps the second issue affects only a minority of servers, and I'm
> wrong to be worried. Because otherwise, I really don't understand why it
> hasn't been reported and fixed until recently. Not blaming you, just to
> be clear.

It was reported a long time ago.  By you and others.  But there wasn't
the means -- or rather the energy from my part -- to fix it.  I couldn't
just have truncated that information.  So I enhanced ElDoc instead with
the :echo option.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 18/04/2023 23:56, João Távora wrote:
> On Tue, Apr 18, 2023 at 9:38 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> This particular one didn't have to, but it's a problem very
>> characteristic of joining a strongly centralized project with ultimately
>> one person having the last word in all major decisions. And it's not
>> like Eli is being unreasonable: we do need a stability cutoff, and we're
>> really long past it. These one-more-change kind of arguments repeat year
>> over year, with reasonable, well-intentioned people on both sides.
> 
> Yes. And here both people sides "one more change".  The change that _did_
> make it in is more aggressive and more unstable than the one that didn't.

Well, not really. It's by definition more conservative one. Problem is, 
it violates a practice established by third-party community outside of 
Emacs.

>> Sure, and I agree, but I don't really see how to present that in terms
>> Eli would feel suitable to accept. One "trick" that worked in the past
>> was to somehow enumerate all potential execution flows (functions
>> involved, etc) that would be affected by the change.
> 
> Right.  And IMO it's not a "trick", it's how it should be.  It's hard to prove a
> negative, but at least it should be attempted.  Well, the patch I presented
> (the one you +1'ed) makes it so that package-install keeps exactly its previous
> behaviour unless its argument is one of (eglot use-package), which are arguments
> that could not have ever been passed to that function as :core packages
> in Emacs 28. M-x package-install RET seq or (package-install 'xref) keep EXACTLY
> the same behaviour.  It's very clear to see from the minimal patch.

Perhaps a more structured/verbose outline of the same would help.

Although apparently the example I was thinking of 
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=37774#479 and the 
surrounding thread) occurred when the corresponding pretest hadn't 
started yet.

>> I don't insist, not at all. It was just my own impression of what would
>> constitute a reasonable Eglot release that we could be satisfied with
>> having a large number of people use without upgrading, for years. Issues
>> like blinking eldoc messages, or eldoc messages that can take up half
>> the height of the window seem like things that we wouldn't want in it.
> 
> The issue has existed and has been worked around successfully for a long
> period of time.  It's not actually a problem, it's a consequence of the
> default values for eldoc-echo-area-use-multiline-p and max-mini-window-height,
> both of which predate Eglot.

So there is a workaround for it anyway, thanks for the reminder.

> Of course I think the current behaviour is better.  But it's also different so
> I don't think we should backport that particular one.  Even if so far the
> only feedback we have had has been positive, it could well be that some
> people _liked_ the half-the-window-height thing (after all their customization
> options reflected that wish, even if by default).
> 
> And by the way, not really half-the-window.  I used this for a long time
> without being much bothered by it.

I guess it depends on the number of overloads for the function around point.

>> Perhaps the second issue affects only a minority of servers, and I'm
>> wrong to be worried. Because otherwise, I really don't understand why it
>> hasn't been reported and fixed until recently. Not blaming you, just to
>> be clear.
> 
> It was reported a long time ago.  By you and others.  But there wasn't
> the means -- or rather the energy from my part -- to fix it.  I couldn't
> just have truncated that information.  So I enhanced ElDoc instead with
> the :echo option.

Aha, so the :echo thingy made it possible. Gotcha.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Tue, Apr 18, 2023 at 10:06 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 18/04/2023 23:56, João Távora wrote:
> > On Tue, Apr 18, 2023 at 9:38 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> >
> >> This particular one didn't have to, but it's a problem very
> >> characteristic of joining a strongly centralized project with ultimately
> >> one person having the last word in all major decisions. And it's not
> >> like Eli is being unreasonable: we do need a stability cutoff, and we're
> >> really long past it. These one-more-change kind of arguments repeat year
> >> over year, with reasonable, well-intentioned people on both sides.
> >
> > Yes. And here both people sides "one more change".  The change that _did_
> > make it in is more aggressive and more unstable than the one that didn't.
>
> Well, not really. It's by definition more conservative one. Problem is,
> it violates a practice established by third-party community outside of
> Emacs.

Hence, more unstable.  There aren't watertight borders here.  These users
in the "third-party" community are using nothing but Emacs and GNU ELPA
Eglot, those are the users that we're hurting while to protect the
non Eglot users.  But why on earth can't we protect the non-Eglot users and
not screw the Eglot users.  We can, with a simpler change.

> >> Sure, and I agree, but I don't really see how to present that in terms
> >> Eli would feel suitable to accept. One "trick" that worked in the past
> >> was to somehow enumerate all potential execution flows (functions
> >> involved, etc) that would be affected by the change.
> >
> > Right.  And IMO it's not a "trick", it's how it should be.  It's hard to prove a
> > negative, but at least it should be attempted.  Well, the patch I presented
> > (the one you +1'ed) makes it so that package-install keeps exactly its previous
> > behaviour unless its argument is one of (eglot use-package), which are arguments
> > that could not have ever been passed to that function as :core packages
> > in Emacs 28. M-x package-install RET seq or (package-install 'xref) keep EXACTLY
> > the same behaviour.  It's very clear to see from the minimal patch.
>
> Perhaps a more structured/verbose outline of the same would help.

It's 7 lines of code.  Elisp should be trivial to read.  Eli read
much more complicated code in this thread.

> > Of course I think the current behaviour is better.  But it's also different so
> > I don't think we should backport that particular one.  Even if so far the
> > only feedback we have had has been positive, it could well be that some
> > people _liked_ the half-the-window-height thing (after all their customization
> > options reflected that wish, even if by default).
> >
> > And by the way, not really half-the-window.  I used this for a long time
> > without being much bothered by it.
>
> I guess it depends on the number of overloads for the function around point.

I'm using C++ ;-)

> >> Perhaps the second issue affects only a minority of servers, and I'm
> >> wrong to be worried. Because otherwise, I really don't understand why it
> >> hasn't been reported and fixed until recently. Not blaming you, just to
> >> be clear.
> >
> > It was reported a long time ago.  By you and others.  But there wasn't
> > the means -- or rather the energy from my part -- to fix it.  I couldn't
> > just have truncated that information.  So I enhanced ElDoc instead with
> > the :echo option.
>
> Aha, so the :echo thingy made it possible. Gotcha.

Right, which is also the reason it makes even _less_ sense to bring Eglot
1.15 into Emacs 29 _without_ ElDoc (I hope that plan is now completely off
the table).

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 19/04/2023 00:15, João Távora wrote:
>> Aha, so the :echo thingy made it possible. Gotcha.
> Right, which is also the reason it makes even_less_  sense to bring Eglot
> 1.15 into Emacs 29_without_  ElDoc (I hope that plan is now completely off
> the table).

Eh, sure.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 00:20:21 +0300
> Cc: Eli Zaretskii <eliz@gnu.org>, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 19/04/2023 00:15, João Távora wrote:
> >> Aha, so the :echo thingy made it possible. Gotcha.
> > Right, which is also the reason it makes even_less_  sense to bring Eglot
> > 1.15 into Emacs 29_without_  ElDoc (I hope that plan is now completely off
> > the table).
> 
> Eh, sure.

It isn't my call in this case, but FWIW: I still have no idea why
wouldn't we want Eglot 1.14 or 1.15 to be in Emacs 29.1.  I didn't
hear any serious argument against doing that; every reason that was
raised was almost immediately explained away as not being a hard
limitation.

And mind you: Emacs 29.1 will not be released tomorrow or the day
after.  We still have at least several weeks till then, with at least
one more pretest.  So the decision whether to import a newer Eglot
into the release branch doesn't have to be today.  However, the
argument against updating Eglot on the release branch, such as they
were, are of some vaguely "fundamental" nature, so I'm not sure a few
more weeks of time will change the decision.  No one said something
like "if Emacs 29.1 were to be released in NN weeks or more, it would
be okay to update Eglot on the release branch."  But then I already
admitted to not understanding those reasons, so maybe I'm missing
something here.

So there you are.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 1:05 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > Date: Wed, 19 Apr 2023 00:20:21 +0300
> > Cc: Eli Zaretskii <eliz@gnu.org>, rpluim@gmail.com, philipk@posteo.net,
> >  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> > From: Dmitry Gutov <dmitry@gutov.dev>
> >
> > On 19/04/2023 00:15, João Távora wrote:
> > >> Aha, so the :echo thingy made it possible. Gotcha.
> > > Right, which is also the reason it makes even_less_  sense to bring Eglot
> > > 1.15 into Emacs 29_without_  ElDoc (I hope that plan is now completely off
> > > the table).
> >
> > Eh, sure.
>
> It isn't my call in this case, but FWIW: I still have no idea why
> wouldn't we want Eglot 1.14 or 1.15 to be in Emacs 29.1.  I didn't
> hear any serious argument against doing that; every reason that was
> raised was almost immediately explained away as not being a hard
> limitation.
>
> And mind you: Emacs 29.1 will not be released tomorrow or the day
> after.  We still have at least several weeks till then, with at least
> one more pretest.  So the decision whether to import a newer Eglot
> into the release branch doesn't have to be today.  However, the
> argument against updating Eglot on the release branch, such as they
> were, are of some vaguely "fundamental" nature, so I'm not sure a few
> more weeks of time will change the decision.  No one said something
> like "if Emacs 29.1 were to be released in NN weeks or more, it would
> be okay to update Eglot on the release branch."  But then I already
> admitted to not understanding those reasons, so maybe I'm missing
> something here.
>
> So there you are.

Eli, do you want Eglot 1.14 (or 1.15, or 1.16 or whatever version
is the latest Eglot release "several weeks" from now) to be in Emacs 29?

From your writings, I'm assuming you do.  Let's call that version Eglot
1.1x with x > 14.  If we did that, we would have two options:

1. Bundle Eglot 1.1x with Emacs 29 and all its up-to-date dependencies, so
that, right at the moment of the Emacs 29 release, Eglot would function
exactly the same on Emacs 28 + package-install.

2. Bundle a "Frankenglot" with Emacs 29 that has all the
lisp/progmodes/eglot.el code of the future Eglot version, advertises
itself as Eglot 1.1x, probably doesn't break, but does _not_ provide
the same experience as Emacs 28 + package-install

Both options are bad, IMO, but 2 is worse.

The first reason that both options are bad is that you're discarding
whatever value the pretest phase brings to the stability of Eglot's code.
Eglot, being a part of Emacs the Emacs code base, benefits from the same
testing all its code does.  You're discarding that value, and I think
it's bad, because the pretest is supposedly there for a good reason. A bug in
Eglot 1.1x will just escape us and there's no time to fix it.

The second reason only applies to option 2. It would completely confuse
users.  A user running Emacs 28 would see a much better 1.1x than
the 1.1x bundled in Emacs 29. Her configuration for Eglot 1.1x
could simply break in 1.1x. Eglot 1.1x was never designed to be
run in such a hampered environment.

They are "fundamental" reasons indeed.  Are they now less vague
and more concrete?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 14:04:10 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> Eli, do you want Eglot 1.14 (or 1.15, or 1.16 or whatever version
> is the latest Eglot release "several weeks" from now) to be in Emacs 29?

What I want in Emacs 29.1 (and any future release of Emacs) is the
latest version of Eglot that you, as Eglot developer, consider stable
enough to recommend to users of a stable Emacs.  Which version is that
is your decision.  But to make sense to me, the decision you make
should be consistent: if a version X.Y of Eglot is "stable enough" for
you to recommend that users of Emacs 29.n upgrade to it, then that
version X.Y can be part of Emacs 29.n.  (Of course, if you decided
that X.Y is stable enough _after_ Emacs 29.n was released, then X.Y
will be able to become part of Emacs only in Emacs 29.n+1 and later.)

I say above "to make sense to me", and I mean that, and only that.
That is, you _can_ decide that you don't agree with my definition of
consistency, and therefore Eglot X.Y will be recommended to users of
Emacs 29.n, but at the same time you don't want it to be in Emacs
29.n; such a decision will not "make sense to me", but we will still
act according to your decision in this matter.

I hope I clarified my position in this regard.

And before you draw too far-fetched conclusions from the above: I'm
saying this about Eglot, and only about Eglot.  Similar decisions
about other packages, and the conditions for including those other
packages in a stable Emacs versions, could very well be different.

> 1. Bundle Eglot 1.1x with Emacs 29 and all its up-to-date dependencies, so
> that, right at the moment of the Emacs 29 release, Eglot would function
> exactly the same on Emacs 28 + package-install.

That is probably not acceptable, although I'd need to know exactly
which versions of what other packages need to be imported into the
emacs-29 branch, to give a definitive answer.

> 2. Bundle a "Frankenglot" with Emacs 29 that has all the
> lisp/progmodes/eglot.el code of the future Eglot version, advertises
> itself as Eglot 1.1x, probably doesn't break, but does _not_ provide
> the same experience as Emacs 28 + package-install
> 
> Both options are bad, IMO, but 2 is worse.

I don't see why option 2 would be bad, let alone worse.  See below.

> The first reason that both options are bad is that you're discarding
> whatever value the pretest phase brings to the stability of Eglot's code.

Eglot itself isn't my problem.  My problem is the other packages that
upgrading Eglot, per option 1 above, will require to update: those
other packages usually affect much more than just Eglot, and therefore
bringing their potentially less stable code into emacs-29 might break
much more than just Eglot.

> Eglot, being a part of Emacs the Emacs code base, benefits from the same
> testing all its code does.  You're discarding that value, and I think
> it's bad, because the pretest is supposedly there for a good reason. A bug in
> Eglot 1.1x will just escape us and there's no time to fix it.

Please leave this consideration to me, it is exactly the kind of
judgment call I make every several days when someone asks whether a
particular change is OK for the release branch.  And in the case of
Eglot I already said that IMO we should include in Emacs 29.1 the
latest stable version of Eglot, thus my decision about that was
already made in favor of upgrading Eglot on emacs-29.  But I won't
insist if you are uncomfortable with that.

> The second reason only applies to option 2. It would completely confuse
> users.  A user running Emacs 28 would see a much better 1.1x than
> the 1.1x bundled in Emacs 29.

No, users of Emacs 28 who installed the latest stable Eglot from ELPA
should see almost the same version of Eglot as users of Emacs 29.1 if
that will come with the same stable Eglot.  Unless you intend to
somehow degenerate important parts of Eglot in what you call
"Frankenglot", whatever that means.

My interpretation of option 2 is that we get a newer Eglot (1.14 or
1.15, whichever you decide is stable enough) with various minor
fallbacks intended to work around the fact that dependency packages
are not necessarily at their versions for which Eglot 1.14/1.15 was
designed to work, if the versions of those dependencies in Emacs 29.1
are older.  That is, 1.14/1.15 without the mandatory requirement o
upgrade any other package already in Emacs.  If that is not what you
mean, then perhaps consider this as option 3, which is the one that
makes the most sense to me, and if possible and agreed upon, will be
accepted into emacs-29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 2:35 PM Eli Zaretskii <eliz@gnu.org> wrote:

> What I want in Emacs 29.1 (and any future release of Emacs) is the
> latest version of Eglot that you, as Eglot developer, consider stable
> enough to recommend to users of a stable Emacs.  Which version is that
> is your decision.  But to make sense to me, the decision you make
> should be consistent: if a version X.Y of Eglot is "stable enough" for
> you to recommend that users of Emacs 29.n upgrade to it, then that
> version X.Y can be part of Emacs 29.n.  (Of course, if you decided
> that X.Y is stable enough _after_ Emacs 29.n was released, then X.Y
> will be able to become part of Emacs only in Emacs 29.n+1 and later.)

OK. Then I will tell you that "stable enough", for, me means "as stable
as possible", because I like things to be as stable as possible.
And, to me (and I would think to most people), the most stable
version of a program is the one which has seen the most testing. So,
I see no reason to give Eglot a shorter pretest period than the
rest of Emacs gets.  Thus Eglot 1.12.29 it is.

> I hope I clarified my position in this regard.

OK, it seems you're giving the call to me.  Then I hope I've also
clarified my decision and the criteria I used in reaching it.

> > 1. Bundle Eglot 1.1x with Emacs 29 and all its up-to-date dependencies, so
> > that, right at the moment of the Emacs 29 release, Eglot would function
> > exactly the same on Emacs 28 + package-install.
>
> That is probably not acceptable, although I'd need to know exactly
> which versions of what other packages need to be imported into the
> emacs-29 branch, to give a definitive answer.

I could teach you how to figure that out (it's very simple), but
since I don't want this either, and you just gave the call to me,
there's probably no point.

> > 2. Bundle a "Frankenglot" with Emacs 29 that has all the
> > lisp/progmodes/eglot.el code of the future Eglot version, advertises
> > itself as Eglot 1.1x, probably doesn't break, but does _not_ provide
> > the same experience as Emacs 28 + package-install
> >
> > Both options are bad, IMO, but 2 is worse.
>
> I don't see why option 2 would be bad, let alone worse.  See below.
>
> > The first reason that both options are bad is that you're discarding
> > whatever value the pretest phase brings to the stability of Eglot's code.
>
> Eglot itself isn't my problem.  My problem is the other packages that
> upgrading Eglot, per option 1 above, will require to update: those
> other packages usually affect much more than just Eglot, and therefore
> bringing their potentially less stable code into emacs-29 might break
> much more than just Eglot.

Yes, as I said option 1 is bad.  Option 2 is worse, but 1 is pretty
bad too, partly for the reasons you state (yes you're totally right
"it's much more than Eglot").

> > Eglot, being a part of Emacs the Emacs code base, benefits from the same
> > testing all its code does.  You're discarding that value, and I think
> > it's bad, because the pretest is supposedly there for a good reason. A bug in
> > Eglot 1.1x will just escape us and there's no time to fix it.
>
> Please leave this consideration to me, it is exactly the kind of
> judgment call I make every several days when someone asks whether a
> particular change is OK for the release branch.  And in the case of
> Eglot I already said that IMO we should include in Emacs 29.1 the
> latest stable version of Eglot, thus my decision about that was
> already made in favor of upgrading Eglot on emacs-29.  But I won't
> insist if you are uncomfortable with that.

There is no one "stable".  You yourself are talking about stability
gradation.   Eglot 1.12.29 is "stablest".  Eglot 1.14 is "stable". Eglot
in master has recently gone two days with a semi-serious live bug (now
fixed) Let's call it "unstable" anyway.  Before I release 1.15, I will
try to make
sure it is as stable as 1.14 (or more).

By the way, the ElDoc change about the non-blinking is "safe for
the release branch" IMHO.  Just in case you didn't get my opinion
on that part of ElDoc of which I am the author and the maintainer
since 2018.

> > The second reason only applies to option 2. It would completely confuse
> > users.  A user running Emacs 28 would see a much better 1.1x than
> > the 1.1x bundled in Emacs 29.
>
> No, users of Emacs 28 who installed the latest stable Eglot from ELPA
> should see almost the same version of Eglot as users of Emacs 29.1 if
> that will come with the same stable Eglot.  Unless you intend to
> somehow degenerate important parts of Eglot in what you call
> "Frankenglot", whatever that means.

Frankenglot 1.1x is just Eglot without the resources that 1.1x would
have in Emacs 28.  Don't you understand?  The "good" Eglot experience
requires things that are not in lisp/progmodes/eglot.el.  It requires,
for example, :echo support in ElDoc's eldoc-documentation-functions
so that echo areas aren't flooded by some LSP servers.
eglot.el can function without it, but users would be surprised.  They would
ask: hey didn't you fix this echo area flooding, back in Eglot 1.14???
And I would have to explain: this is not really Eglot 1.1x, this is
Frankenglot 1.1x:  the content of the eglot.el file is the same but
it's missing a lot of stuff.  And then I wouldn't even know how to
teach them to upgrade.

In so many software packages I know outside of Emacs, bugs and
features are fixed and added just by bumping a dependency.  Surely
you must have seen this, too.

> My interpretation of option 2 is that we get a newer Eglot (1.14 or
> 1.15, whichever you decide is stable enough) with various minor
> fallbacks intended to work around the fact that dependency packages
> are not necessarily at their versions for which Eglot 1.14/1.15 was
> designed to work, if the versions of those dependencies in Emacs 29.1
> are older.

Why put ourselves (mostly myself) through these chores??  Just so that
two weeks later after whatever Emacs 29 is officially released a more
recent,  "stable"  version is already available? And pay the price of
discarding the value of the pretest period?  Just because Eglot is now
much harder to  upgrade?  Then just make it easy to upgrade: it's in your
control.

There there are 0 downsides, as far as I can tell.  As far as anyone
who has commented that patch can tell.   I've seen 0 contestation
of the patch I proposed before, which I've grown almost tired of
linking to.  That patch was engineered to answer precisely your
objections to previous patches.  It was engineered to make _you_
happy and make me happy, and make Eglot users happy.  And you're
ignoring it.

 That is, 1.14/1.15 without the mandatory requirement o
> upgrade any other package already in Emacs.  If that is not what you
> mean, then perhaps consider this as option 3

No, you're just describing option 2, as far as I can see.
But probably no "various minor fallbacks" would be needed, because
I am careful with the interfaces.


-- 
João Távora

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 15:04:30 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Wed, Apr 19, 2023 at 2:35 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > My interpretation of option 2 is that we get a newer Eglot (1.14 or
> > 1.15, whichever you decide is stable enough) with various minor
> > fallbacks intended to work around the fact that dependency packages
> > are not necessarily at their versions for which Eglot 1.14/1.15 was
> > designed to work, if the versions of those dependencies in Emacs 29.1
> > are older.
> 
> Why put ourselves (mostly myself) through these chores??

Because I thought we agreed that requiring newer versions of other
packages where that could be avoided (note: "could be avoided", not
"is nice to have") is a Good Thing, and is well worth these chores.
But if you don't agree, fine; just one more disagreement between us.

> Just so that two weeks later after whatever Emacs 29 is officially
> released a more recent, "stable" version is already available?

No, just so users of Emacs 29 could have a better Eglot without any
complications.

But again, have it your way.  I said I won't argue with your decision
in this matter.  I just wanted you and everyone else to understand my
position on this.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 5:01 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 19 Apr 2023 15:04:30 +0100
> > Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net,
> >       62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> >
> > On Wed, Apr 19, 2023 at 2:35 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > My interpretation of option 2 is that we get a newer Eglot (1.14 or
> > > 1.15, whichever you decide is stable enough) with various minor
> > > fallbacks intended to work around the fact that dependency packages
> > > are not necessarily at their versions for which Eglot 1.14/1.15 was
> > > designed to work, if the versions of those dependencies in Emacs 29.1
> > > are older.
> >
> > Why put ourselves (mostly myself) through these chores??
>
> Because I thought we agreed that requiring newer versions of other
> packages where that could be avoided (note: "could be avoided", not
> "is nice to have") is a Good Thing, and is well worth these chores.
> But if you don't agree, fine; just one more disagreement between us.

I gave plenty of arguments (which you didn't contest) for why doing
this is a very bad thing (in my opinion of course).  If your utmost priority
is to not require newer versions of other packages, then 1.12.29 should be
just fine!  It will work without an internet connection.  And it will be much
more consistent and well tested than Frankenglot 1.1x, because it will
go through the pretest period.

> > Just so that two weeks later after whatever Emacs 29 is officially
> > released a more recent, "stable" version is already available?
>
> No, just so users of Emacs 29 could have a better Eglot without any
> complications.

It wouldn't be better.  It would be a different thing, a maintenance
nightmare for one.  What if the user then _did_ explicitly install of
those dependencies?  Installing a package and its dependencies isn't a
complication, it's how it has worked for years.  _You've_ decided it is
a complication (and made it more complicated) for Eglot Emacs 29. But
people have been doing it for years.


João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 19/04/2023 15:05, Eli Zaretskii wrote:
> It isn't my call in this case, but FWIW: I still have no idea why
> wouldn't we want Eglot 1.14 or 1.15 to be in Emacs 29.1.  I didn't
> hear any serious argument against doing that; every reason that was
> raised was almost immediately explained away as not being a hard
> limitation.

Okay, let me try to answer this: since the goal is (apparently) to have 
a stable version of Eglot in emacs-29, we don't know yet whether 1.14 or 
1.15 is "stable".

> And mind you: Emacs 29.1 will not be released tomorrow or the day
> after.  We still have at least several weeks till then, with at least
> one more pretest.  So the decision whether to import a newer Eglot
> into the release branch doesn't have to be today.  However, the
> argument against updating Eglot on the release branch, such as they
> were, are of some vaguely "fundamental" nature, so I'm not sure a few
> more weeks of time will change the decision.  No one said something
> like "if Emacs 29.1 were to be released in NN weeks or more, it would
> be okay to update Eglot on the release branch."  But then I already
> admitted to not understanding those reasons, so maybe I'm missing
> something here.

Let's imagine I was making this choice.

I would include (or propose for inclusion) Eglot 1.x.y in Emacs 29 only 
N weeks after it has been tagged on master and thus published to ELPA, 
on the condition that no major bugs have been discovered in the meantime 
that required major reworks (because any bugfix would reset the timer to 
L weeks where L < N, but a major change would reset it to N weeks 
again). That would be the general guideline. Add to that the 
maintainer's best judgment, who would be able to reduce or extend these 
periods on case by case basis as well, according to the changes that 
went into every release.

To answer the original question: N weeks still haven't passed (I guess) 
since 1.15 was tagged, so we don't quite know whether it's acceptable 
for emacs-29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 18:48:00 +0300
> Cc: joaotavora@gmail.com, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > And mind you: Emacs 29.1 will not be released tomorrow or the day
> > after.  We still have at least several weeks till then, with at least
> > one more pretest.  So the decision whether to import a newer Eglot
> > into the release branch doesn't have to be today.  However, the
> > argument against updating Eglot on the release branch, such as they
> > were, are of some vaguely "fundamental" nature, so I'm not sure a few
> > more weeks of time will change the decision.  No one said something
> > like "if Emacs 29.1 were to be released in NN weeks or more, it would
> > be okay to update Eglot on the release branch."  But then I already
> > admitted to not understanding those reasons, so maybe I'm missing
> > something here.
> 
> Let's imagine I was making this choice.
> 
> I would include (or propose for inclusion) Eglot 1.x.y in Emacs 29 only 
> N weeks after it has been tagged on master and thus published to ELPA, 
> on the condition that no major bugs have been discovered in the meantime 
> that required major reworks (because any bugfix would reset the timer to 
> L weeks where L < N, but a major change would reset it to N weeks 
> again). That would be the general guideline. Add to that the 
> maintainer's best judgment, who would be able to reduce or extend these 
> periods on case by case basis as well, according to the changes that 
> went into every release.
> 
> To answer the original question: N weeks still haven't passed (I guess) 
> since 1.15 was tagged, so we don't quite know whether it's acceptable 
> for emacs-29.

That is fine with me, assuming N has some reasonable value.  It means
I could ask you again before the next pretest, and then again before
RC, and perhaps you'd then agree to import a newer Eglot.

But note that this is not what João is saying.  He says 1.14 will not
be in Emacs 29.1, period.  No matter how long I will drag the pretest.
He certainly doesn't want to invest the effort of making Eglot 1.14
less dependent on latest changes in other packages, so as to make sure
we could drop Eglot 1.14 into Emacs 29 without risking any problems
elsewhere.  And that more or less seals the issue, effectively setting
your N to infinity.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 5:10 PM Eli Zaretskii <eliz@gnu.org> wrote:

> But note that this is not what João is saying.  He says 1.14 will not
> be in Emacs 29.1, period.  No matter how long I will drag the pretest.
> He certainly doesn't want to invest the effort of making Eglot 1.14
> less dependent on latest changes in other packages, so as to make sure
> we could drop Eglot 1.14 into Emacs 29 without risking any problems
> elsewhere.  And that more or less seals the issue, effectively setting
> your N to infinity.

This is extremely odd for me.  Weren't you the one very, very sternly
asking for almost _no_ changes to go into Elisp code of the Emacs 29
now that it is in pretest.  And now you're liberally and casually
suggesting way-over-last minute changes and work to go into that same
version?

This just doesn't make any sense to me.  Can't you understand that other
maintainers also value stability for their packages?

It's certainly NOT about "not wanting to invest the effort".  That effort
would amount to forking Eglot in its feature set so that effort would be
a disservice to everybody.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 17:23:19 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Wed, Apr 19, 2023 at 5:10 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > But note that this is not what João is saying.  He says 1.14 will not
> > be in Emacs 29.1, period.  No matter how long I will drag the pretest.
> > He certainly doesn't want to invest the effort of making Eglot 1.14
> > less dependent on latest changes in other packages, so as to make sure
> > we could drop Eglot 1.14 into Emacs 29 without risking any problems
> > elsewhere.  And that more or less seals the issue, effectively setting
> > your N to infinity.
> 
> This is extremely odd for me.  Weren't you the one very, very sternly
> asking for almost _no_ changes to go into Elisp code of the Emacs 29
> now that it is in pretest.  And now you're liberally and casually
> suggesting way-over-last minute changes and work to go into that same
> version?

I'm talking only about Eglot, not in general.  I believe I've stressed
that point already.  So what I said or did or asked to do regarding
other packages or other places in Emacs is not necessarily relevant
for this discussion.  The decisions about these issues are always on a
case by case basis, so you cannot compare different cases and expect
them to yield the same decisions.

> This just doesn't make any sense to me.  Can't you understand that other
> maintainers also value stability for their packages?

Of course I can.  But once again: if Eglot 1.14 is not stable enough,
then why do we recommend users of Emacs 29 to update their bundled
Eglot to v1.14?  This is inconsistent: if 1.14 is not stable enough to
be in Emacs 29, we should only recommend it for users of Emacs 30.

I already explained this inconsistency more than once.  Why do you
keep bringing it up time and again?  You might disagree, but why do
you insist that I accept what I perceive as inconsistent logic?  Just
let it go and accept that we disagree about this.

> It's certainly NOT about "not wanting to invest the effort".  That effort
> would amount to forking Eglot in its feature set so that effort would be
> a disservice to everybody.

No, it doesn't require any forks.  It requires more cautious
introduction of new features into Eglot on master.  And yes, it's
extra effort.  But IMNSHO, users will benefit, so in my book it's
worth it.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 5:50 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > This just doesn't make any sense to me.  Can't you understand that other
> > maintainers also value stability for their packages?
>
> Of course I can.  But once again: if Eglot 1.14 is not stable enough,

Stable enough for whom, or for what?  Stability is quantity in a spectrum.

I think Emacs releases should come with the most well tested code.  No
program is perfect.  Eglot 1.14 is stable, but it's not _as_ stable as Eglot
1.12 because the latter has seen more testing?

So different users and different use cases will accept different things.

> > It's certainly NOT about "not wanting to invest the effort".  That effort
> > would amount to forking Eglot in its feature set so that effort would be
> > a disservice to everybody.
>
> No, it doesn't require any forks.  It requires more cautious
> introduction of new features into Eglot on master.  And yes, it's
> extra effort.  But IMNSHO, users will benefit, so in my book it's
> worth it.

AFAICT you suggest a different eglot.el in Emacs 29 which has the
same features as eglot.el in Emacs master, but without needing
the dependencies that the master version can enjoy.  That's a
fork if I've ever seen one.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 18:27:08 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Wed, Apr 19, 2023 at 5:50 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > This just doesn't make any sense to me.  Can't you understand that other
> > > maintainers also value stability for their packages?
> >
> > Of course I can.  But once again: if Eglot 1.14 is not stable enough,
> 
> Stable enough for whom, or for what?  Stability is quantity in a spectrum.

Stable for us.  We are not talking about absolutes here, we are
talking about relative stability.  I'm saying that if some package is
stable enough to be used with Emacs version X.Y, it is by definition
also stable enough to be included in Emacs version X.Y.  The relative
stability levels of these two cases must be the same, or else we are
inconsistent in our own judgment of stability.

> I think Emacs releases should come with the most well tested code.  No
> program is perfect.  Eglot 1.14 is stable, but it's not _as_ stable as Eglot
> 1.12 because the latter has seen more testing?

Then how come we tell users of this same Emacs 29 to update to Eglot
1.14 without too much thought?  And you even insist on making that
automatic when packages are updated at startup.  Won't that
destabilize their Emacs?

> > > It's certainly NOT about "not wanting to invest the effort".  That effort
> > > would amount to forking Eglot in its feature set so that effort would be
> > > a disservice to everybody.
> >
> > No, it doesn't require any forks.  It requires more cautious
> > introduction of new features into Eglot on master.  And yes, it's
> > extra effort.  But IMNSHO, users will benefit, so in my book it's
> > worth it.
> 
> AFAICT you suggest a different eglot.el in Emacs 29 which has the
> same features as eglot.el in Emacs master, but without needing
> the dependencies that the master version can enjoy.  That's a
> fork if I've ever seen one.

No, I suggest that you make changes on master so that these problems
are avoided in the first place.  Changes in a core package on the
Emacs master branch should be done while keeping in mind that this
same version of a package will be on ELPA and users of older Emacsen
will install that newer version.  So the newer version on master
should avoid making changes which would mandate newer versions of
other packages, by providing the necessary compatibility fallbacks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 19, 2023 at 6:59 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 19 Apr 2023 18:27:08 +0100
> > Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net,
> >       62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> >
> > On Wed, Apr 19, 2023 at 5:50 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > > This just doesn't make any sense to me.  Can't you understand that other
> > > > maintainers also value stability for their packages?
> > >
> > > Of course I can.  But once again: if Eglot 1.14 is not stable enough,
> >
> > Stable enough for whom, or for what?  Stability is quantity in a spectrum.
>
> Stable for us.  We are not talking about absolutes here, we are
> talking about relative stability.  I'm saying that if some package is
> stable enough to be used with Emacs version X.Y, it is by definition
> also stable enough to be included in Emacs version X.Y.  The relative
> stability levels of these two cases must be the same, or else we are
> inconsistent in our own judgment of stability.
>
> > I think Emacs releases should come with the most well tested code.  No
> > program is perfect.  Eglot 1.14 is stable, but it's not _as_ stable as Eglot
> > 1.12 because the latter has seen more testing?
>
> Then how come we tell users of this same Emacs 29 to update to Eglot
> 1.14 without too much thought?  And you even insist on making that
> automatic when packages are updated at startup.  Won't that
> destabilize their Emacs?

I don't dictate to users how much to think :-)  I just assume that
they've been thinking about it in Emacs 28, and they are happy
with this amount of thinking, else they would be complaining
about stability, and they aren't.  Maybe some users are holding
on to Emacs 26.3 + Eglot 1.0 for dear life, who knows? It's very
simple to me.

To summarize and to hopefully answer the question that you repeatedly
put before me, this is my recommendation to users:

 "Dear user,

  Emacs 29 will come with the most stable version I can offer, and I
  can tell you, prospective user, that it has been through as much
  of the Emacs 29 pretest period as possible. If you, prospective user,
  are interested in the features listed in etc/EGLOT-NEWS, you may try
  upgrading to the latest version (I hope it will be easy, good luck!)
  In the unlikely event that things go sour for your particular
  LSP server and use case, you have my deepest apologies, and you
  should revert back to the version bundled with Emacs 29.

  Thank you,
  Your maintainer"

The second part of this recommendation is what I've always recommended.
I've never had to phrase it like this, but this has always been my
recommendation.

> No, I suggest that you make changes on master so that these problems
> are avoided in the first place.  Changes in a core package on the
> Emacs master branch should be done while keeping in mind that this
> same version of a package will be on ELPA and users of older Emacsen
> will install that newer version.  So the newer version on master
> should avoid making changes which would mandate newer versions of
> other packages, by providing the necessary compatibility fallbacks.

So, if I want to do a feature on master that depends on some
new infrastructure on package X that appeared on master (say Xref's
upcoming support for "find any type of thing declaration/macro/etc"),
I have to do no less than duplicate that new infrastructure in Eglot.el
and do a runtime check.  That's... not good, to say the least.

But noted.  Now I (think I) understand what you suggest.

AFAICT basically suggesting all Package-Requires are eliminated,
to get rid of package.el dependency system.  You should argue for
that in emacs-devel explicitly so, I think, because that will
probably elicit some interesting reactions from :core package
developers. "Package-Requires" is used liberally now AFAIK

It's the most basic underpinning of Eglot's design philosophy,
if there such a thing.  That's why Eglot is called "minimal".  It's
not minimal in functionality!  It's minimal in size and complexity
because it reuses code and keeps itself simple.  And if I may be a bit less
humble about it, I've gotten a few compliments about it.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 19:27:11 +0100
> Cc: dmitry@gutov.dev, rpluim@gmail.com, philipk@posteo.net, 
> 	62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> On Wed, Apr 19, 2023 at 6:59 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > No, I suggest that you make changes on master so that these problems
> > are avoided in the first place.  Changes in a core package on the
> > Emacs master branch should be done while keeping in mind that this
> > same version of a package will be on ELPA and users of older Emacsen
> > will install that newer version.  So the newer version on master
> > should avoid making changes which would mandate newer versions of
> > other packages, by providing the necessary compatibility fallbacks.
> 
> So, if I want to do a feature on master that depends on some
> new infrastructure on package X that appeared on master (say Xref's
> upcoming support for "find any type of thing declaration/macro/etc"),
> I have to do no less than duplicate that new infrastructure in Eglot.el
> and do a runtime check.

Not necessarily duplicate.  That is only one alternative, and not the
best one, IMO.  Other alternatives could be:

  . decide that users who use this new version of Eglot without that
    new version of Xref will not have this new Eglot feature
  . provide a less functional and simpler replacement for that Xref
    feature
  . find a way of providing the new Eglot feature without relying on
    Xref, but via some alternative solution

Whether each one of the above could be relevant and reasonable depends
on what exactly are those Eglot and Xref features.

I'm sure you don't need the above, you know that better than I do.
It's stuff we do in Emacs every day, and core packages consider these
factors since we began having core packages.  The question is just how
far to go in that direction.  I'm trying to make a point that going as
far as is practical will allow us to bundle at least some packages in
newer versions, which in the end will benefit users.

> AFAICT basically suggesting all Package-Requires are eliminated,
> to get rid of package.el dependency system.

No!  Not "all dependencies eliminated", that's impractical.  I'm
talking only about dependencies on core packages, and I'm asking to
avoid mandatory updates to newer versions.  The dependencies should
still exist, but the minimum supported version of each dependency
should ideally barely change, or at least change as infrequently as
possible.

> You should argue for that in emacs-devel explicitly so

That's what I did.  I'm not to bl;ame that you keep responding here
when I asked a day ago to respond to the discussion I started on
emacs-devel.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 19/04/2023 19:10, Eli Zaretskii wrote:
>> To answer the original question: N weeks still haven't passed (I guess)
>> since 1.15 was tagged, so we don't quite know whether it's acceptable
>> for emacs-29.
> That is fine with me, assuming N has some reasonable value.  It means
> I could ask you again before the next pretest, and then again before
> RC, and perhaps you'd then agree to import a newer Eglot.

More or less.

> But note that this is not what João is saying.  He says 1.14 will not
> be in Emacs 29.1, period.  No matter how long I will drag the pretest.
> He certainly doesn't want to invest the effort of making Eglot 1.14
> less dependent on latest changes in other packages, so as to make sure
> we could drop Eglot 1.14 into Emacs 29 without risking any problems
> elsewhere.  And that more or less seals the issue, effectively setting
> your N to infinity.

I think you're simply talking past each other, and will essentially 
agree on that "N weeks" thing outside of this discussion.

But also note that you added a complication: avoid bumping the required 
dependencies. If the backport is not performed as-is and needs 
additional changes (with extra shims, stubbed new features, etc, instead 
of simply using new features from the latest eldoc.el), then you are not 
just asking whether the maintainer thinks the new version is good and 
stable, but also whether they are willing to expend extra effort 
altering it and maintaining diverging versions of code.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 20:23:05 +0300
> Cc: joaotavora@gmail.com, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > But note that this is not what João is saying.  He says 1.14 will not
> > be in Emacs 29.1, period.  No matter how long I will drag the pretest.
> > He certainly doesn't want to invest the effort of making Eglot 1.14
> > less dependent on latest changes in other packages, so as to make sure
> > we could drop Eglot 1.14 into Emacs 29 without risking any problems
> > elsewhere.  And that more or less seals the issue, effectively setting
> > your N to infinity.
> 
> I think you're simply talking past each other, and will essentially 
> agree on that "N weeks" thing outside of this discussion.

I wish.

> But also note that you added a complication: avoid bumping the required 
> dependencies. If the backport is not performed as-is and needs 
> additional changes (with extra shims, stubbed new features, etc, instead 
> of simply using new features from the latest eldoc.el), then you are not 
> just asking whether the maintainer thinks the new version is good and 
> stable, but also whether they are willing to expend extra effort 
> altering it and maintaining diverging versions of code.

Yes, noted.

Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

This discussion no longer belongs to the bug tracker, so I'm moving it
to emacs-devel and changing its Subject.  Please reply here, not
there.

For those who see this for the first time, and want more background,
here are some relevant messages which discussed this aspect as part of
the otherwise huge thread, which is only loosely related to this
particular issue:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#383
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#398
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#401
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#410
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#413

> Date: Tue, 18 Apr 2023 04:25:01 +0300
> Cc: joaotavora@gmail.com, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 17/04/2023 05:24, Eli Zaretskii wrote:
> >> Date: Sun, 16 Apr 2023 23:46:37 +0300
> >> Cc: rpluim@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
> >>   larsi@gnus.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >>
> >> On 14/04/2023 22:28, Eli Zaretskii wrote:
> >>> If, OTOH,
> >>> you think that it's imperative to allow_all_  users of Eglot with
> >>> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> >>> become available), then we should release Emacs 29 with 1.14.
> >>
> >> Was this question about stability only?
> > 
> > It was about the criteria for which versions of core packages to ship
> > with a release.
> 
> I don't think we can get a single set of criteria across core packages.

We don't need to have just one set.  Packages are different in both
their complexity, their dependence on other packages, and dependence
of other packages on them.  So one set is unlikely to fit the bill,
indeed.

But that doesn't mean we shouldn't have criteria at all.  We should
strive to have a small number of them, and we should know which set is
applicable to which class of packages.

This will be one of the serious issues if we ever move to having some
packages only in elpa.git, and will then bundle them when preparing an
Emacs release tarball.  It will be imperative to know at that time
which version/branch of each such package to take as part of preparing
a release.  We must have a solution by then, so this is as good time
as any to start discussing the issue.

> E.g. Org is developed externally, has its own community of significant 
> size, and does split off release branches (with additional testing, I',m 
> guessing).
> 
> Eglot, OTOH, is developed only here, with no additional release workflow 
> other than what MELPA/GNU ELPA historically provided: collect up some 
> features/fixes, bump the Version header, and push a new release out to 
> the users. The lack of extended testing period is made up for with the 
> capability to push out a new fixed version overnight. That's why the 
> difficulty in upgrading to the latest version (for Emacs 29 users) is 
> going to hurt.

If some core package is not tested enough before it gets a new
version, then why are we okay with telling users of a stable Emacs to
update the package willy-nilly as soon as another version is on ELPA?
Shouldn't we at least warn them, or, better, somehow indicate that
this version is not yet considered stable?

IOW, shouldn't packages have some "stability gradation" that is
visible when users look at the list of packages via package.el?
Shouldn't we allow users to tell package.el which stability they want
to download, so that unstable packages don't inadvertently get
installed and mess up their production environment?

> BTW, if you recall the threads before Eglot was added, I was against 
> that, and one of the things I cited is an LSP client has inherently high 
> development velocity. Maybe the LSP community will settle/mature/stop 
> adding features one day, but it's not there yet.

I don't see what development pace has to do with this issue.  If a
package is being developed at a high pace, it might mean that the
stable version will lag more.  But what does this change in principle?

> >> Because since we've decided in favor of stability of package.el, and
> >> against eglot's easy upgradability, I would suggest to backport Eglot
> >> 1.14 to emacs-29.
> > 
> > I won't object.  In fact, I asked up front why not.
> 
> Note that that suggestion comes with a fix to eldoc which you so far 
> have rejected for emacs-29.

You mean, the change in ElDoc that avoids the "jumping mode line"
issue?  If so, it is unfortunate for Eglot to depend on that, because
it means users of Emacs 29 will be unable to upgrade to Eglot 1.15
without by side effect installing a newer and potentially less stable
ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)

So this again goes back to the main issue: how should the stability
considerations affect development of core packages and their
"stability gradation"?  Developers of core packages should keep these
aspects in mind at all times, and, for example, avoid dependencies on
other packages that aren't absolutely necessary.  Otherwise, we will
have to advise users who value stability not to upgrade their core
packages for fear of destabilizing their environments, and the whole
advantage of having core packages on ELPA will be effectively lost, at
least for those users who want stability.

IOW, having a core package on ELPA comes with some serious strings
attached, and just ignoring them, or leaving that up to the users
(without duly informing them about their new responsibilities) is not
a good solution, IMO.

We should seriously ponder these aspects, and the sooner the better.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 1:56 PM Eli Zaretskii <eliz@gnu.org> wrote:

> without by side effect installing a newer and potentially less stable
> ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)

It's not a "must".  Eglot can work without it.  The problem happens in
ElDoc and doesn't affect its interface.

But Eglot relies on ElDoc as a whole.  In general you can't expect
to have new features in Eglot without some advancing of the
infrastructure that Eglot depends on.  As highlighted in the manual
and elsewhere, Eglot is relatively thin glue between LSP abstractions
and Emacs abstractions.  It relies on this client-agnostic
infrastructure of ElDoc, Flymake, Xref, Project, Jsonrpc (and
soon others) and drives advances to that infrastructure too.
For example, I completely reworked Flymake in 2017 (in
backward-compatible fashion of course) precisely to support
Eglot's use case.  It supports many others of course, and there is
a healthy collection of non-Eglot Flymake clients.  Exactly the
same happened for ElDoc, and similar processes happen with Xref,
for example. These processes have been happening for a while now,
before Eglot was in core, and the model has proven its value IMO.

Anyway, this was just an introduction to what is Eglot's main
characteristic (and why it's sometimes called "minimal" and
well integrated with Emacs's existing features).

So there _isn't_ a way to partially upgrade a package and not its
dependencies.  And there _shouldn't_ ever be this way, for our
collective sanity.  And there's no good reason this should ever
exist.

> We should seriously ponder these aspects, and the sooner the better.

I don't actually think there is any actual problem present if
we all agree (as you seem to) with Dmitry's preposition that
"there shouldn't be a single set of criteria governing core
packages".  Then we can teach Emacs's upgrade mechanisms to
deal with each set differently, carefully examining the
requirements for each set.

I'd like you, if possible, to also respond (here, if you prefer) to the
points I raised in my own reply to Dmitry's message you're replying
to.  This is the message;:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#529

Thank you,
João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 18 Apr 2023 15:02:01 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> 
> On Tue, Apr 18, 2023 at 1:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > without by side effect installing a newer and potentially less stable
> > ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)
> 
> It's not a "must".  Eglot can work without it.  The problem happens in
> ElDoc and doesn't affect its interface.

Then what Dmitry said about Eglot 1.15 being dependent on that change
in ElDoc is not relevant to the issue at hand, which is whether Eglot
1.15 could be bundled with Emacs 29.1.

> But Eglot relies on ElDoc as a whole.  In general you can't expect
> to have new features in Eglot without some advancing of the
> infrastructure that Eglot depends on.

Understood.  My point is that if you want Eglot users to be able to
upgrade to a newer versions, you need to have compatibility layers, to
avoid the need to upgrade too many other packages, which might hamper
stability.  Otherwise, we cannot in good faith recommend that users of
stable Emacs update their core packages without a second thought.

> So there _isn't_ a way to partially upgrade a package and not its
> dependencies.

Updating a package P1 should require update of as few packages P2...Pn
as possible.  Ideally, none at all.  Users should be able to decide
whether they want or don't want to update any single package without
also needing to decide whether they are okay with updating half a
dozen of others.  This should be our goal, because otherwise updating
a package will be unsafe if you use any Emacs except master (and thus
don't care much about stability anyway).

> > We should seriously ponder these aspects, and the sooner the better.
> 
> I don't actually think there is any actual problem present if
> we all agree (as you seem to) with Dmitry's preposition that
> "there shouldn't be a single set of criteria governing core
> packages".  Then we can teach Emacs's upgrade mechanisms to
> deal with each set differently, carefully examining the
> requirements for each set.

Sure, but we need to have these sets, actually.  Right now, we don't,
not for every core package out there anyway.

> I'd like you, if possible, to also respond (here, if you prefer) to the
> points I raised in my own reply to Dmitry's message you're replying
> to.  This is the message;:
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#529

I didn't respond because I had nothing to say to that which I didn't
already say in response to Dmitry's message.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 3:47 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Tue, 18 Apr 2023 15:02:01 +0100
> > Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> >
> > On Tue, Apr 18, 2023 at 1:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > without by side effect installing a newer and potentially less stable
> > > ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)
> >
> > It's not a "must".  Eglot can work without it.  The problem happens in
> > ElDoc and doesn't affect its interface.
>
> Then what Dmitry said about Eglot 1.15 being dependent on that change
> in ElDoc is not relevant to the issue at hand, which is whether Eglot
> 1.15 could be bundled with Emacs 29.1.

It is quite relevant.

Eglot 1.15 depends on many other things in ElDoc.  That particular
bugfix might not -- or might indeed -- included, depending on what
other non-bugfix things  Eglot will require of ElDoc at the time.
ElDoc is now 1.14 but it could be 1.15 at the time motivated by
Eglot 1.15/16/17.

And even in ElDoc 1.14 there are already things (the :echo display
option) that are not in Emacs 29. And Eglot 1.14 directly depends
on those things.  It relies on them to do a good job.

I would think that if you oppose a bugfix backport you would also
oppose a _feature_ backport, which usually is (and is indeed in
this case) a much more complicated, "scary", bug-prone development.

> > But Eglot relies on ElDoc as a whole.  In general you can't expect
> > to have new features in Eglot without some advancing of the
> > infrastructure that Eglot depends on.
>
> Understood.  My point is that if you want Eglot users to be able to
> upgrade to a newer versions, you need to have compatibility layers, to
> avoid the need to upgrade too many other packages, which might hamper
> stability.  Otherwise, we cannot in good faith recommend that users of
> stable Emacs update their core packages without a second thought.

Yes, and this is why each released version of Eglot specifies exactly
the _released_ versions of its dependencies that it depends on. The
dependency language isn't very elaborated (there is no way to say
Eglot 1.1234  depends on ElDoc between 1.23 and 1.56), but it's been
enough.  It's not much different from what is found in numerous
other package systems.

Another matter is what package.el does with this info and the
implied graph. Transitive dependencies are known not to matter.
And if a package requires ElDoc 1.14 but 1.99 is already available
package.el will just go ahead and install the latest.  Always has,
I'm afraid (anyone can correct me on this if I'm mistaken, which
I'd love to be).

I think the experience of the straight.el and elpaca.el package
manager authors could be useful here to us.  If someone knows them
by heart, please do tell me.  Who knows, maybe what we want
is to bring one of these into Emacs and kill package.el. straight.el
is what a lot of people seem to be using these days anyway (and the cooler
kids elpaca.el).

> > So there _isn't_ a way to partially upgrade a package and not its
> > dependencies.
>
> Updating a package P1 should require update of as few packages P2...Pn
> as possible.  Ideally, none at all.

And very often that does happen, I suppose.  Not every Eglot release
_requires_ installation of new versions of its dependencies.  But some
do.

> Users should be able to decide
> whether they want or don't want to update any single package without
> also needing to decide whether they are okay with updating half a
> dozen of others.  This should be our goal, because otherwise updating
> a package will be unsafe if you use any Emacs except master (and thus
> don't care much about stability anyway).

I don't know how you can meet that goal in general.  We should
indeed work to minimize dependencies and do things that don't affect
interfaces and don't require changes other's interfaces.  As much as
possible, I agree.  But in general programs rely on other programs.
Dependencies exist.  Like in many other package managers, users
should be presented with the consequences of their wishes, when
that is feasible and when we can do so without breaking their
configs.

That's my idea of stability anyways.

> > "there shouldn't be a single set of criteria governing core
> > packages".  Then we can teach Emacs's upgrade mechanisms to
> > deal with each set differently, carefully examining the
> > requirements for each set.
>
> Sure, but we need to have these sets, actually.  Right now, we don't,
> not for every core package out there anyway.

[ Just a note that :core packages are not "out there", they're
"in here" -- by definition.  That's their main selling point
and precisely what we should take advantage of :-) ]

I propose two main sets of :core packages to start with.

Set 1 - :core packages that have always been core, i.e. they started
their life in the code

Set 2 - :core packages that started their life somewhere else
(GNU ELPA, Github, etc), were installable through ELPA interfaces
and now are :core (which means Git-versioned in Emacs.git but
still installable via ELPA).  Two known such packages are Eglot
and Use-Package.  Both depend on _other_ :core packages.  Eglot
on many of these, Use-Package only on "bind-key", which is
also new, but didn't seem to be installable independently
before Use-Package appeared.

This isn't the end of the analysis, of course.  I'm just
providing these two sets to see if it rings a bell with participants,
because from the opinions I collected in that very long bug, they
seem to make and map neatly onto the requirements that each
party put forth:

- That members of set 1 shouldn't be upgradable "willy nilly" to
maintain exact backward-compatibility.

- That members of set 2 should be upgraded in much easier fashion
because that's what guarantees that people's configs already
doing so won't break.

> > I'd like you, if possible, to also respond (here, if you prefer) to the
> > points I raised in my own reply to Dmitry's message you're replying
> > to.  This is the message;:
> >
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#529
>
> I didn't respond because I had nothing to say to that which I didn't
> already say in response to Dmitry's message.

I proposed a simple bugfix to bug#62720, based precisely on
the above idea of separation of sets.  A dead-simple 7-line
fix.

That patch was +1'ed by Philip and Dmitry and hasn't been
addressed by you.  Philip, by the way, tried until the end to
give you a patch that also didn't have the non-interactive
package-install/use-package lockout, but you insisted.  Why? Why
keep it for those members of Set 1?  What is there to be gained?
Do you acknowledge what there is to be lost?

But moving on from that minor tragedy, it's nevertheless easy
to note that bug#62720 conflates the two sets.  We should work
to improve that, hopefully in time to avert damage.  The code
required to express these sets in Elisp is, of course, quite trivial.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 18 Apr 2023 16:45:33 +0100
> Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> 
> > > It's not a "must".  Eglot can work without it.  The problem happens in
> > > ElDoc and doesn't affect its interface.
> >
> > Then what Dmitry said about Eglot 1.15 being dependent on that change
> > in ElDoc is not relevant to the issue at hand, which is whether Eglot
> > 1.15 could be bundled with Emacs 29.1.
> 
> It is quite relevant.
> 
> Eglot 1.15 depends on many other things in ElDoc.

It isn't like ElDoc is not available in Emacs 29.

> That particular bugfix might not -- or might indeed -- included,
> depending on what other non-bugfix things Eglot will require of
> ElDoc at the time.  ElDoc is now 1.14 but it could be 1.15 at the
> time motivated by Eglot 1.15/16/17.

My point is that Eglot and any other core package will do its users a
favor if it deliberately and consistently makes a point to depend on
as few _new_ features of other core packages as possible.  If you
disagree with that, then we will have to agree to disagree, because
for me this is a very basic issue with core packages and the future of
Emacs.

> And even in ElDoc 1.14 there are already things (the :echo display
> option) that are not in Emacs 29. And Eglot 1.14 directly depends
> on those things.  It relies on them to do a good job.

Which IMO is not a good thing, not unless Eglot 1.14 has fallbacks for
when these features are not available.

But if that is impossible or impractical for some reason in this
particular case, then it simply means that users of Emacs 29 will be
unable to upgrade Eglot without also risking less stability due to
upgrading the dependencies.  Again, if you don't think this
"dependencies hell" is a Bad Thing in general, we will have to agree
to disagree.

> I would think that if you oppose a bugfix backport you would also
> oppose a _feature_ backport, which usually is (and is indeed in
> this case) a much more complicated, "scary", bug-prone development.

I consider each case of a request to backport on its own merit.  So
conclusions such as the above, which don't consider the specifics of
each change, but instead go by some abstract principles, are not what
I usually make.

> > Understood.  My point is that if you want Eglot users to be able to
> > upgrade to a newer versions, you need to have compatibility layers, to
> > avoid the need to upgrade too many other packages, which might hamper
> > stability.  Otherwise, we cannot in good faith recommend that users of
> > stable Emacs update their core packages without a second thought.
> 
> Yes, and this is why each released version of Eglot specifies exactly
> the _released_ versions of its dependencies that it depends on.

You are missing my point.  I'm not talking about dependencies with
other packages.  My point is not about other core packages, it's about
Emacs itself and its stability as a whole.  Users of a stable release
of Emacs can, and usually do, expect their entire Emacs configurations
to be stable, and telling them to upgrade packages without any clear
indication of their stability goes against that.  Yes, requiring
specific versions of the other N packages will minimize breakage due
to incompatibilities between those packages, but what about unintended
consequences, regressions, etc.?  Suppose Eglot 1.14 brings with it
some package whose updated version has a bug -- why should a user who
wants to have a stable Emacs environment to risk this?

> > Updating a package P1 should require update of as few packages P2...Pn
> > as possible.  Ideally, none at all.
> 
> And very often that does happen, I suppose.  Not every Eglot release
> _requires_ installation of new versions of its dependencies.  But some
> do.

I'm asking whether you make an extra effort to avoid such requirements
whenever possible, even if that would call for extra work on your
part?  I hope you and other package developers do, because otherwise
proliferation of core packages would be a very bad deal for the future
of Emacs, which traditionally is perceived as an extremely stable
platform.

> > Users should be able to decide
> > whether they want or don't want to update any single package without
> > also needing to decide whether they are okay with updating half a
> > dozen of others.  This should be our goal, because otherwise updating
> > a package will be unsafe if you use any Emacs except master (and thus
> > don't care much about stability anyway).
> 
> I don't know how you can meet that goal in general.

If you at least think we should try, then we have something to work
with.  Even if the ideal of having to upgrade no package is
unattainable, bringing the number close to zero is a worthy goal.

> We should indeed work to minimize dependencies and do things that
> don't affect interfaces and don't require changes other's
> interfaces.  As much as possible, I agree.  But in general programs
> rely on other programs.  Dependencies exist.  Like in many other
> package managers, users should be presented with the consequences of
> their wishes, when that is feasible and when we can do so without
> breaking their configs.

I'm saying that we should make an extra effort to avoid that, not
accept dependencies without a "fight".

> I propose two main sets of :core packages to start with.
> 
> Set 1 - :core packages that have always been core, i.e. they started
> their life in the code
> 
> Set 2 - :core packages that started their life somewhere else
> (GNU ELPA, Github, etc), were installable through ELPA interfaces
> and now are :core (which means Git-versioned in Emacs.git but
> still installable via ELPA).  Two known such packages are Eglot
> and Use-Package.  Both depend on _other_ :core packages.  Eglot
> on many of these, Use-Package only on "bind-key", which is
> also new, but didn't seem to be installable independently
> before Use-Package appeared.
> 
> This isn't the end of the analysis, of course.

I'm not sure history is the aspect that distinguishes them.  An
important aspect is how "low' is the functionality provided by the
package.  Some packages provide infrastructure -- those affect many
other places by definition.  Others are applications on which nothing
else depends.  Perhaps these are the important factors, I don't know.

> - That members of set 1 shouldn't be upgradable "willy nilly" to
> maintain exact backward-compatibility.
> 
> - That members of set 2 should be upgraded in much easier fashion
> because that's what guarantees that people's configs already
> doing so won't break.

That is completely irrelevant for this discussion, IMO.  It is up to
the users whether to upgrade and how aggressively.  We don't know
enough about the configurations, the environment, and the usage
patterns of the users, we can only give them information -- in this
case regarding stability of each available version -- and let them
make the conclusions as they see fit.  We will never be able to second
guess them well enough.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 5:19 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Tue, 18 Apr 2023 16:45:33 +0100
> > Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> >
> > > > It's not a "must".  Eglot can work without it.  The problem happens in
> > > > ElDoc and doesn't affect its interface.
> > >
> > > Then what Dmitry said about Eglot 1.15 being dependent on that change
> > > in ElDoc is not relevant to the issue at hand, which is whether Eglot
> > > 1.15 could be bundled with Emacs 29.1.
> >
> > It is quite relevant.
> >
> > Eglot 1.15 depends on many other things in ElDoc.
>
> It isn't like ElDoc is not available in Emacs 29.

It is.  But in Emacs 29, it's not the bleeding edge anymore.  It's
1.13.0 which doesn't have an important feature of 1.14.0

  commit e19994fe8c000b0ed2dbc667cdec26cf54356907
  Author: João Távora Date: Thu Mar 23 09:02:18 2023 +0000

  ElDoc: rework rendering of echo area (bug#62029)

And it doesn't have the fix for bug#62816, which is in master
and will appear in ElDoc 1.14.1.  Unless you want it to, of
course.  In the case of that single backport, ElDoc bundled
with Emacs 29 (and with Emacs 29 _only_) should become known
as 1.13.29 (imperfect versioning scheme, but not terrible).

> > That particular bugfix might not -- or might indeed -- included,
> > depending on what other non-bugfix things Eglot will require of
> > ElDoc at the time.  ElDoc is now 1.14 but it could be 1.15 at the
> > time motivated by Eglot 1.15/16/17.
>
> My point is that Eglot and any other core package will do its users a
> favor if it deliberately and consistently makes a point to depend on
> as few _new_ features of other core packages as possible.  If you
> disagree with that, then we will have to agree to disagree, because
> for me this is a very basic issue with core packages and the future of
> Emacs.

Yes, as few _new_ features as possible, but not fewer than that.
Yes, I agree good design is when there is nothing more to take out.
But sometimes you have to add to make things appear.

> > And even in ElDoc 1.14 there are already things (the :echo display
> > option) that are not in Emacs 29. And Eglot 1.14 directly depends
> > on those things.  It relies on them to do a good job.
>
> Which IMO is not a good thing, not unless Eglot 1.14 has fallbacks for
> when these features are not available.

It has.  It doesn't fail.  I try to make everything forward and backward
compatible.  But certainly your at-point documentation experience in Eglot
if you merge if you backport it to Emacs 29 without also backporting ElDoc
will _not_ be the same.  Users will notice this and if we tell them
"why yes, Eglot 1.15 is in Emacs 29!"  they will be triply befuddled,
and rightfully so, because Eglot 1.15 running on Emacs 28 will have much
better behaviour and fewer bugs and run smoother.

> But if that is impossible or impractical for some reason in this
> particular case, then it simply means that users of Emacs 29 will be
> unable to upgrade Eglot without also risking less stability due to
> upgrading the dependencies.  Again, if you don't think this
> "dependencies hell" is a Bad Thing in general, we will have to agree
> to disagree.

No, it doesn't mean that, at all.  First, dependencies exist.  And here
it's not hell at all, not IME.  Secondly, stability is a matter of
expectations.  Eglot users _expect_ :core dependencies to be upgraded
when they request Eglot to be  upgraded.  That's the way it has always
worked for ~5 years.  And non-Eglot users _also_ expect that, because
installing any non-:core ELPA package that depends on :core packages
has _also_ always produced that behaviour for dependencies.

Thirdly, in practice, I have been following this for some time, I've not
seen many (any?) bugs related to these kinds of "furtive" dependency
hell upgrade by package-install.  Maybe Dmitry has?   Or you have?
If anything, I've seen bugs related to upgrades of dependencies
_not_ happening (because of package.el's inability to understand
transitive dependencies -- Basil spotted that, I think).

> > I would think that if you oppose a bugfix backport you would also
> > oppose a _feature_ backport, which usually is (and is indeed in
> > this case) a much more complicated, "scary", bug-prone development.
>
> I consider each case of a request to backport on its own merit.  So
> conclusions such as the above, which don't consider the specifics of
> each change, but instead go by some abstract principles, are not what
> I usually make.

That makes full sense, btw.

> > > Understood.  My point is that if you want Eglot users to be able to
> > > upgrade to a newer versions, you need to have compatibility layers, to
> > > avoid the need to upgrade too many other packages, which might hamper
> > > stability.  Otherwise, we cannot in good faith recommend that users of
> > > stable Emacs update their core packages without a second thought.
> >
> > Yes, and this is why each released version of Eglot specifies exactly
> > the _released_ versions of its dependencies that it depends on.
>
> You are missing my point.  I'm not talking about dependencies with
> other packages.  My point is not about other core packages, it's about
> Emacs itself and its stability as a whole.  Users of a stable release
> of Emacs can, and usually do, expect their entire Emacs configurations
> to be stable, and telling them to upgrade packages without any clear
> indication of their stability goes against that.  Yes, requiring
> specific versions of the other N packages will minimize breakage due
> to incompatibilities between those packages, but what about unintended
> consequences, regressions, etc.?  Suppose Eglot 1.14 brings with it
> some package whose updated version has a bug -- why should a user who
> wants to have a stable Emacs environment to risk this?

She shouldn't.  She should never M-x package-install RET eglot or
have it in her configuration.  This is not a new problem.  Things that
download code bear risks, at least when compared to the stock Emacs 29
that went through a lenghty pretest period.  This is _precisely_ why
I think backporting Eglot 1.15/16/whatever to Emacs 29 is not a good
idea.  This is why that user should use Eglot bundled with Emacs 29.
It's a good, stable, well-tested Eglot that you can do lots of cool
stuff in.

> > > Updating a package P1 should require update of as few packages P2...Pn
> > > as possible.  Ideally, none at all.
> >
> > And very often that does happen, I suppose.  Not every Eglot release
> > _requires_ installation of new versions of its dependencies.  But some
> > do.
>
> I'm asking whether you make an extra effort to avoid such requirements
> whenever possible, even if that would call for extra work on your
> part?  I hope you and other package developers do, because otherwise
> proliferation of core packages would be a very bad deal for the future
> of Emacs, which traditionally is perceived as an extremely stable
> platform.

Yep, I do those efforts, but I wouldn't go so far as to call them "extra".
I think things should go where they belong.  That's, again, the main
design philosophy of Eglot.  Don't invent (too much) UI in Eglot.  Put
UI and other functionality in client-agnostic libraries and use Eglot
to link up LSP interfaces to those libraries.  Consequently, that
requires new libraries and updates to the libraries.

For example, for the "breadcrumb headerline" feature of bug#58431 I'll
be proposing the introduction of a new lisp/progmodes/breadcrumb.el
:core library that depends solely on Emacs's imenu.el.  It works
with Eglot and without Eglot (and it's turning out quite nice, should
work with the imenu info produced by the tree-sitter modes, too).

So is that "proliferation of :core libraries"?  Yes maybe, but it's
also what we want.  We don't want a from-scratch LSP-specific breadcrumb
implementation in Eglot, I think.  So a library is the way to go.

If you don't want it in core, I can very well put it in GNU ELPA, or
even MELPA.  Then Eglot will "optionally" depend on it, much in the
same way it already depends "optionally" on Markdown.el, Company and
Yasnippet.

> > > Users should be able to decide
> > > whether they want or don't want to update any single package without
> > > also needing to decide whether they are okay with updating half a
> > > dozen of others.  This should be our goal, because otherwise updating
> > > a package will be unsafe if you use any Emacs except master (and thus
> > > don't care much about stability anyway).
> >
> > I don't know how you can meet that goal in general.
>
> If you at least think we should try, then we have something to work
> with.  Even if the ideal of having to upgrade no package is
> unattainable, bringing the number close to zero is a worthy goal.
...
> I'm saying that we should make an extra effort to avoid that, not
> accept dependencies without a "fight".

OK, you have a foot soldier to "fight" against dependency hell :-).
But sometimes I will enter talks with these dependencies who
may not be from hell, rather from some kind of more well behaved
purgatory.

> > This isn't the end of the analysis, of course.
>
> I'm not sure history is the aspect that distinguishes them.  An
> important aspect is how "low' is the functionality provided by the
> package.  Some packages provide infrastructure -- those affect many
> other places by definition.  Others are applications on which nothing
> else depends.  Perhaps these are the important factors, I don't know.

Yes, these are other important factors.  But history is important
as a factor because the _past_ dictates user expectations, which
are by definition based on it.  And stability is about expectations
(as someone wisely wrote here recently, wisely, apropos a security
bug).

> > - That members of set 1 shouldn't be upgradable "willy nilly" to
> > maintain exact backward-compatibility.
> >
> > - That members of set 2 should be upgraded in much easier fashion
> > because that's what guarantees that people's configs already
> > doing so won't break.
>
> That is completely irrelevant for this discussion, IMO.  It is up to
> the users whether to upgrade and how aggressively.  We don't know
> enough about the configurations, the environment, and the usage
> patterns of the users, we can only give them information -- in this
> case regarding stability of each available version -- and let them
> make the conclusions as they see fit.  We will never be able to second
> guess them well enough.

It is not irrelevant.  We can know _exactly_ how certain things worked
in Emacs 26, 27 and 28 and how much of a backward-incompatible change
we're making.  Thus we can act accordingly to minimize that change.
Or even eliminate it for the specific cases where we want to eliminate
it.  Minimizing change is the bread-and-butter of stability, which
seems to be something you're also interested in, but also at the same
time, contradictingly, aren't.  That part I still don't understand.

In this case, the "certain thing" is the behaviour of the very
popular and extremely simple (use-package eglot :ensure t) or the
presence or a (package-install eglot) in a configuration. It does one
thing in Emacs 26/7/8 it will do another different thing in Emacs 29.

How "popular" is this (use-package eglot :ensure t), you ask?  Just check
it out for yourself:
https://github.com/joaotavora/eglot/search?q=%22use-package%22&type=issues

it pops up every other issue.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 18/04/2023 20:49, João Távora wrote:
>> But if that is impossible or impractical for some reason in this
>> particular case, then it simply means that users of Emacs 29 will be
>> unable to upgrade Eglot without also risking less stability due to
>> upgrading the dependencies.  Again, if you don't think this
>> "dependencies hell" is a Bad Thing in general, we will have to agree
>> to disagree.
> No, it doesn't mean that, at all.  First, dependencies exist.  And here
> it's not hell at all, not IME.  Secondly, stability is a matter of
> expectations.  Eglot users_expect_  :core dependencies to be upgraded
> when they request Eglot to be  upgraded.  That's the way it has always
> worked for ~5 years.  And non-Eglot users_also_  expect that, because
> installing any non-:core ELPA package that depends on :core packages
> has_also_  always produced that behaviour for dependencies.

To my recollection, Eglot historically only raised the dependency 
versions when it needed to -- to bring in some required feature. Which 
is quite reasonable from my POV.

Although I see some very recent more bumps for xref and project there, 
those I didn't expect, but they probably follow the same logic.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/18/2023 5:57 AM, Eli Zaretskii wrote:
> So this again goes back to the main issue: how should the stability
> considerations affect development of core packages and their
> "stability gradation"?

It sounds to me like there are 3 or 4 levels (depending on how you count):

* Stable: the version of a package included in the latest Emacs tarball
* Latest: the latest version on GNU ELPA (etc)
* Devel: the latest version on GNU-devel ELPA (etc)

You could possibly add:

* Core(?): the version of a package included in the tarball of the 
user's *current* Emacs installation

I think these 3 or 4 levels should be plenty for 99% of scenarios. From 
this, we can also propose a hard rule: packages at a certain level 
should only depend on other packages at the same level or lower (so a 
"latest" package can't depend on a "devel" package). We could also add 
as soft guidance: a package should do its best to require only stable 
versions of dependencies where feasible.

However, this runs into the problem João saw: if your package (e.g. 
Eglot) would strongly benefit from requiring a newer version of a 
dependency (ElDoc), what should you do? Currently, the only options are 
a) do nothing and let users have a worse experience or b) make the user 
upgrade the dependency too, even if they don't particularly want it. 
Neither solution seems ideal to me.

One alternative would be for packages to be able to *recommend* 
dependencies. Then, Eglot could recommend newer versions of ElDoc, but 
they wouldn't actually be required. We could do this via some extra 
package metadata, or maybe the follow simple solution would be enough: 
when installing a package from a particular archive[1], offer the user 
the option of also installing any of the package's requirements from 
that archive too. So, if I installed Eglot from GNU ELPA, Emacs would 
suggest that I also install ElDoc, etc from GNU ELPA too. (Of course, if 
a package truly *requires* a newer version of a package that's only 
available in ELPA, it wouldn't need to prompt the user.)

[1] You could also think of this as "installing a package of a 
particular stability level".

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Tue, 18 Apr 2023 11:56:58 -0700
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 4/18/2023 5:57 AM, Eli Zaretskii wrote:
> > So this again goes back to the main issue: how should the stability
> > considerations affect development of core packages and their
> > "stability gradation"?
> 
> It sounds to me like there are 3 or 4 levels (depending on how you count):
> 
> * Stable: the version of a package included in the latest Emacs tarball
> * Latest: the latest version on GNU ELPA (etc)
> * Devel: the latest version on GNU-devel ELPA (etc)

I think we need only two.  Stable can move to the next version, since
packages are released more frequently than Emacs.

> However, this runs into the problem João saw: if your package (e.g. 
> Eglot) would strongly benefit from requiring a newer version of a 
> dependency (ElDoc), what should you do? Currently, the only options are 
> a) do nothing and let users have a worse experience or b) make the user 
> upgrade the dependency too, even if they don't particularly want it. 
> Neither solution seems ideal to me.

How is this different from what we have in Emacs?  An exciting new
feature is sometimes deferred to the next major release if the release
branch is close enough to a release.  There's nothing new here, just
the fact that sometimes useful new features could destabilize Emacs,
so one needs to choose which one it wants more.

> One alternative would be for packages to be able to *recommend* 
> dependencies. Then, Eglot could recommend newer versions of ElDoc, but 
> they wouldn't actually be required.

This is probably needed, but it requires non-trivial support from
package.el, to let informed users select the updates that fit their
stability requirements and feature needs.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/18/2023 12:21 PM, Eli Zaretskii wrote:
>> Date: Tue, 18 Apr 2023 11:56:58 -0700
>> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> It sounds to me like there are 3 or 4 levels (depending on how you count):
>>
>> * Stable: the version of a package included in the latest Emacs tarball
>> * Latest: the latest version on GNU ELPA (etc)
>> * Devel: the latest version on GNU-devel ELPA (etc)
> 
> I think we need only two.  Stable can move to the next version, since
> packages are released more frequently than Emacs.

At least from a user POV, I think it makes sense to distinguish among 
all three of these. For example, I might use the version of Eglot in 
Emacs 29, or I might install it from GNU ELPA, or I could even install 
it from GNU-devel ELPA. I might even switch back and forth depending on 
what my needs are (and in fact, that's exactly what I do with Eglot; 
while I usually prefer to stick on the GNU ELPA version, sometimes I 
switch to GNU-devel ELPA if there's a fix for a bug I find very bothersome).

I think this set of three levels also makes it easier - at least for me 
- to reason about what to do with something like ElDoc. If a user 
installs Eglot from GNU ELPA (i.e. the user gets "Eglot latest"), should 
it automatically install ElDoc from GNU ELPA ("ElDoc latest") or should 
it use the ElDoc from the Emacs tarball ("ElDoc stable")?

If there were only two levels - latest and devel - then I think the 
answer to the Eglot/ElDoc problem would simply be: installing Eglot 
latest should pull in ElDoc latest. Since there's no higher "stability 
gradation" than latest, we wouldn't really be able to say that 
ElDoc-from-Emacs is better/stabler than ElDoc-from-ELPA. (Well, we can 
still *say* that, but I think it helps to embed our reasoning for it 
into these stability gradations.)

> How is this different from what we have in Emacs?  An exciting new
> feature is sometimes deferred to the next major release if the release
> branch is close enough to a release.  There's nothing new here, just
> the fact that sometimes useful new features could destabilize Emacs,
> so one needs to choose which one it wants more.

I think the main difference is that Eglot and Emacs (and ElDoc, for that 
matter) all have different release cadences, so it would be helpful to 
have some functionality to help us manage that. With Emacs itself, we 
can ensure that every package that ships in the tarball is works well 
with each other; when we distribute a package on ELPA, this gets trickier.

>> One alternative would be for packages to be able to *recommend*
>> dependencies. Then, Eglot could recommend newer versions of ElDoc, but
>> they wouldn't actually be required.
> 
> This is probably needed, but it requires non-trivial support from
> package.el, to let informed users select the updates that fit their
> stability requirements and feature needs.

Yeah, that's where this gets tricky for Emacs 29: it's probably a bit 
late to add major new features like this to package.el for 29. However, 
we might be able to accept an imperfect solution for Eglot today while 
working to provide a better way for Emacs 30.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Tue, 18 Apr 2023 12:36:12 -0700
> Cc: dmitry@gutov.dev, joaotavora@gmail.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> >> * Stable: the version of a package included in the latest Emacs tarball
> >> * Latest: the latest version on GNU ELPA (etc)
> >> * Devel: the latest version on GNU-devel ELPA (etc)
> > 
> > I think we need only two.  Stable can move to the next version, since
> > packages are released more frequently than Emacs.
> 
> At least from a user POV, I think it makes sense to distinguish among 
> all three of these. For example, I might use the version of Eglot in 
> Emacs 29, or I might install it from GNU ELPA, or I could even install 
> it from GNU-devel ELPA. I might even switch back and forth depending on 
> what my needs are (and in fact, that's exactly what I do with Eglot; 
> while I usually prefer to stick on the GNU ELPA version, sometimes I 
> switch to GNU-devel ELPA if there's a fix for a bug I find very bothersome).

From a user POV, what is the qualitative difference between "Latest"
and "Stable"?  As soon as some newer version on ELPA becomes stable
enough, why should the user assign any significance to the previous
version included in the last released Emacs tarball?  Please keep in
mind that the version you call "Stable" is just one that was ready in
time for the last Emacs release, and is otherwise no different from
what you call "Latest".  IOW, assuming that versions in Emacs and on
ELPA are declared "Stable" using the same criteria and the same
procedures, they are not really different from the stability POV.
Moreover, the version in Emacs could (hopefully, very infrequently)
have some bug that went unnoticed during the pretest, and that bug
could now be fixed in the ELPA version.

Unless, that is, you somehow trust the decision-making process for an
Emacs release much more than you trust the same process for ELPA
releases.  But in my book, those should be the same processes,
especially if we ever get to the situation where such packages are not
in emacs.git anymore.

> I think this set of three levels also makes it easier - at least for me 
> - to reason about what to do with something like ElDoc. If a user 
> installs Eglot from GNU ELPA (i.e. the user gets "Eglot latest"), should 
> it automatically install ElDoc from GNU ELPA ("ElDoc latest") or should 
> it use the ElDoc from the Emacs tarball ("ElDoc stable")?

My answer is NO, but that is a separate issue.

> If there were only two levels - latest and devel - then I think the 
> answer to the Eglot/ElDoc problem would simply be: installing Eglot 
> latest should pull in ElDoc latest.

By default, perhaps.  But the user should still be able to say not to
upgrade any dependencies unless they _must_ be upgrade, i.e. the
package the user wants to upgrade _requires_ a newer version of
another package.

And again: this is a separate issue.

> Since there's no higher "stability 
> gradation" than latest, we wouldn't really be able to say that 
> ElDoc-from-Emacs is better/stabler than ElDoc-from-ELPA. (Well, we can 
> still *say* that, but I think it helps to embed our reasoning for it 
> into these stability gradations.)

It isn't our decision, it is up to the user.  _We_ could recommend to
upgrade to the latest version (if we will indeed get to the position
where we can do that in good faith), but the user could decide he/she
doesn't believe us, and doesn't want any additional risks.

> > How is this different from what we have in Emacs?  An exciting new
> > feature is sometimes deferred to the next major release if the release
> > branch is close enough to a release.  There's nothing new here, just
> > the fact that sometimes useful new features could destabilize Emacs,
> > so one needs to choose which one it wants more.
> 
> I think the main difference is that Eglot and Emacs (and ElDoc, for that 
> matter) all have different release cadences

But this difference, if it is real, should really disappear, right?
the periods of time when an Emacs release branch is closed to unsafe
changes is nowadays relatively short, so there should be no problem
for the core package to adhere to the same basic routine, i.e. keep
the next "candidate for latest" closed to risky changes for a couple
of months, before it actually becomes "latest".  And then the
difference will all but disappear.

> With Emacs itself, we can ensure that every package that ships in
> the tarball is works well with each other; when we distribute a
> package on ELPA, this gets trickier.

This "trickier" part will need to be fully resolved as part of getting
our act together for the brave new world where core packages are
maintained only on ELPA.  So we might as well start now, because I see
no reason why this difference should even exist -- it's not like we do
in Emacs something too complicated and expensive.

Btw, does package.el even allow to "downgrade" back to the version
that came with Emacs?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 7:57 PM Jim Porter <jporterbugs@gmail.com> wrote:
>
> On 4/18/2023 5:57 AM, Eli Zaretskii wrote:
> > So this again goes back to the main issue: how should the stability
> > considerations affect development of core packages and their
> > "stability gradation"?
>
> It sounds to me like there are 3 or 4 levels (depending on how you count):
>
> * Stable: the version of a package included in the latest Emacs tarball
> * Latest: the latest version on GNU ELPA (etc)
> * Devel: the latest version on GNU-devel ELPA (etc)
>
> You could possibly add:
>
> * Core(?): the version of a package included in the tarball of the
> user's *current* Emacs installation

These are interesting levels, but I was under the impression
that the goal is to partition the set of :core packages according
to some kind of gradation, what Eli called "stability gradation".
The set can be found in the variable package--builtins.

In contrast, you seem to be proposing to partition the much larger
set of pairs (CORE-PACKAGE . VERSION) (where VERSION is the version
of CORE-PACKAGE, not Emacs).

These two exercises are potentially interesting, but merely as
academic exercises they are not particularly useful.

For the first exercise, the useful thing that can be extracted
from it is the optimal behaviour w.r.t to stability of our existing
tools when confronted with an argument from that set.

Such existing tools include package-install, use-package,
package-update, etc. For example.

A. The effects of the form (package-install 'eglot) moving from
Emacs 26,27,28 to Emacs 29 are drastically different.
That's bad stability.

B. The effects of, say, the form (package-install 'xref) moving
from Emacs 28 to Emacs 29 are the same.  That's good stability.

What I'd like to know if anyone else thinks, like me, that this is
a problem.  We can discuss how bad of a problem (I think it's
moderately serious).

But, more importantly, am I the only one that sees an easy
fix to this we can fix case A and keep case B untouched?

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2175 bytes --]


João Távora <joaotavora@gmail.com> writes:

> A. The effects of the form (package-install 'eglot) moving from
> Emacs 26,27,28 to Emacs 29 are drastically different.
> That's bad stability.
>
> B. The effects of, say, the form (package-install 'xref) moving
> from Emacs 28 to Emacs 29 are the same.  That's good stability.
>
> What I'd like to know if anyone else thinks, like me, that this is
> a problem.  We can discuss how bad of a problem (I think it's
> moderately serious).

To me any breakage in behavior when updating to a newer version is
serious. I use Emacs for many scenarios, and some of those I only touch
twice a year (when I give a lecture and update slides for it), while
others I might not use for 3 years until I go back into that old Python
package.

Usually when I get in there and someting is broken in my Emacs that I
didn’t see otherwise, that’s very painful, because I’m often under time
pressure (like “the lecture is tomorrow and I just realized I have to
change these slides, but with the new version I’ve been using for half a
year the layout of just exactly these slides is broken and I still have
to make dinner for the family and actually get some sleep afterwards!”).

But there are too many different tasks to actually check them all every
time I update. And I usually don’t choose to do an Emacs update, but
just update when my distro ships the new version.

Emacs has been mostly really good at just working after an update
(except for some rare cases where I used a fringe org-feature or where
electric indent got enabled in org-mode), and to me it is essential that
Emacs with moderate customization (I have around 1900 lines of
custom-set-variables — mostly org-capture-templates, org-latex-classes,
and safe-local-variable-values — and 1000 lines of customization — most
of those via use-package) stays stable and does not become volatile.

To some argumentation why that’s not just my preference but essential:
https://stevelosh.com/blog/2012/04/volatile-software/

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
> Cc: Jim Porter <jporterbugs@gmail.com>, Eli Zaretskii <eliz@gnu.org>, Dmitry
>  Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> Date: Wed, 19 Apr 2023 14:13:40 +0200
> 
> João Távora <joaotavora@gmail.com> writes:
> 
> > A. The effects of the form (package-install 'eglot) moving from
> > Emacs 26,27,28 to Emacs 29 are drastically different.
> > That's bad stability.
> >
> > B. The effects of, say, the form (package-install 'xref) moving
> > from Emacs 28 to Emacs 29 are the same.  That's good stability.
> >
> > What I'd like to know if anyone else thinks, like me, that this is
> > a problem.  We can discuss how bad of a problem (I think it's
> > moderately serious).
> 
> To me any breakage in behavior when updating to a newer version is
> serious. I use Emacs for many scenarios, and some of those I only touch
> twice a year (when I give a lecture and update slides for it), while
> others I might not use for 3 years until I go back into that old Python
> package.
> 
> Usually when I get in there and someting is broken in my Emacs that I
> didn’t see otherwise, that’s very painful, because I’m often under time
> pressure (like “the lecture is tomorrow and I just realized I have to
> change these slides, but with the new version I’ve been using for half a
> year the layout of just exactly these slides is broken and I still have
> to make dinner for the family and actually get some sleep afterwards!”).
> 
> But there are too many different tasks to actually check them all every
> time I update. And I usually don’t choose to do an Emacs update, but
> just update when my distro ships the new version.

I don't think anyone will argue that breaking changes are good.

But that is not what was difficult about the particular case João is
talking about.  In that case, any solution that was proposed had a
potential to break someone's workflow.

Specifically, users of Emacs 28 and older, who had Eglot installed,
and expect Eglot to be automatically updated upon Emacs startup
whenever a new Eglot version is available, will now have their
expectations broken after they upgrade to Emacs 29, because Eglot is
now a built-in package, and package.el won't by default upgrade a
built-in package.

OTOH, the proposal to change package.el so it automatically updates
built-in packages as well would break the workflows of people who
don't expect built-in packages to be updated: they would suddenly see
packages like ElDoc or Xref or Project (and others) being updated
automatically at startup instead of staying at the version shipped
with Emacs.

So there's a dilemma here: which of the two groups of users to break?

And that is the real issue here, at least from my POV.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 6:02 PM Eli Zaretskii <eliz@gnu.org> wrote:

> OTOH, the proposal to change package.el so it automatically updates
> built-in packages as well would break the workflows of people who
> don't expect built-in packages to be updated: they would suddenly see
> packages like ElDoc or Xref or Project (and others) being updated
> automatically at startup instead of staying at the version shipped
> with Emacs.

OK. In my initial evaluation I disagreed that this was a problem,
because that has always happened in Emacs 28.

But I acknowledge that if some user has

  (package-install always-has-been-a-core-package)

in her Emacs 26/27/28 config, then some behaviour would have been
changed with some of my previous solutions.  I give you that, it
doesn't need contesting.

And that is why I later I gave a solution that _doesn't_,
I repeat, _doesn't_ let that happen.  That solution is in

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#467

It's a minimal patch, much smaller than the one that did make
it in.  I've linked to it many times now, but it has always
been ignored by all except Philip and Dmitry, who said it was a
"fine" solution and +1, respectively.

It will _not_ update Xref and ElDoc _unless_ the user specifically
asked for Eglot to be installed.  Which is, of course, exactly what
happened already in Emacs 28. For example, if the user has

  (package-install 'project)

It _won't_ be updated. And neither will its dependency Xref. Again, just like
in Emacs 28.

Why are we ignoring my patch? What's really wrong with it? In
technical terms, if possible.

> So there's a dilemma here: which of the two groups of users to break?

There's no dilemma.  There's no need for the heavy-handed solution
you chose that breaks one of the groups of users.  Presumably it was
the one you judged least numerous/valuable.  I don't even want to
dispute that judgement, because there's no need!  We can have
perfectly backward-compatible behavior, so if we value it, let's
have it.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 18:21:23 +0100
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>, jporterbugs@gmail.com, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> But I acknowledge that if some user has
> 
>   (package-install always-has-been-a-core-package)
> 
> in her Emacs 26/27/28 config, then some behaviour would have been
> changed with some of my previous solutions.  I give you that, it
> doesn't need contesting.
> 
> And that is why I later I gave a solution that _doesn't_,
> I repeat, _doesn't_ let that happen.  That solution is in
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#467
> 
> It's a minimal patch, much smaller than the one that did make
> it in.  I've linked to it many times now, but it has always
> been ignored by all except Philip and Dmitry, who said it was a
> "fine" solution and +1, respectively.
> 
> It will _not_ update Xref and ElDoc _unless_ the user specifically
> asked for Eglot to be installed.  Which is, of course, exactly what
> happened already in Emacs 28. For example, if the user has
> 
>   (package-install 'project)
> 
> It _won't_ be updated. And neither will its dependency Xref. Again, just like
> in Emacs 28.
> 
> Why are we ignoring my patch? What's really wrong with it? In
> technical terms, if possible.

It has similar problems: it will automatically update packages
mentioned in package--safely-upgradeable-builtins, which might not be
what users want for built-in packages.  You assume that everyone will
want Eglot and use-package automatically updated, but this assumption
has no real basis.  I object in general to any feature that
unexpectedly installs some software without the explicit user's
say-so.

Plus, it adds to the maintenance burden of maintaining this
(internal? not a defcustom??) variable for good.

Those issues were enough for me to reject the proposal.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 21:07, Eli Zaretskii wrote:
> It has similar problems: it will automatically update packages
> mentioned in package--safely-upgradeable-builtins, which might not be
> what users want for built-in packages.

IMO that kind of choice could be deferred to the maintainer of each 
individual package. Or make it a defcustom if you're really worried.

BTW, even choosing that patch where this user option is a defcustom 
defaulting to nil would make more sense to me than the patch we 
currently decided to install.

> You assume that everyone will
> want Eglot and use-package automatically updated, but this assumption
> has no real basis.

People don't call 'M-x package-install' automatically, nor do they put 
those calls in their init files automatically. The only cases where 
Eglot would be automatically updated, even with the proposed patch, is 
when the user previously declared their intention to have Eglot updated 
at least once, in some manual fashion (e.g. using 'package-install' 
interactively).

> I object in general to any feature that
> unexpectedly installs some software without the explicit user's
> say-so.

No disagreement here.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 21:14:06 +0300
> Cc: arne_bab@web.de, jporterbugs@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 19/04/2023 21:07, Eli Zaretskii wrote:
> > It has similar problems: it will automatically update packages
> > mentioned in package--safely-upgradeable-builtins, which might not be
> > what users want for built-in packages.
> 
> IMO that kind of choice could be deferred to the maintainer of each 
> individual package.

No, it cannot, and this and the sibling discussions show why: the
package maintainers are biased in favor of their packages.  That
(completely understandable and expected) bias prevents them from
seeing the overall picture objectively.

> Or make it a defcustom if you're really worried.

That doesn't change the picture, unless the default for the defcustom
will be nil.  Which I expect João to object to, because he wants Eglot
to be updated by default and automatically.  Whereas I think the
compromise, whereby the user should say just once that he/she wants
Eglot to be automatically updated, is a good compromise given the
constraints in this case.  Not ideal, but a good-enough compromise.

> BTW, even choosing that patch where this user option is a defcustom 
> defaulting to nil would make more sense to me than the patch we 
> currently decided to install.

See above.

> > You assume that everyone will
> > want Eglot and use-package automatically updated, but this assumption
> > has no real basis.
> 
> People don't call 'M-x package-install' automatically, nor do they put 
> those calls in their init files automatically.

That's factually incorrect, AFAIU.  Moreover, the cases that bothered
João (again, AFAIU) were exactly those which you say don't exist.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 7:32 PM Eli Zaretskii <eliz@gnu.org> wrote:

> No, it cannot, and this and the sibling discussions show why: the
> package maintainers are biased in favor of their packages.  That
> (completely understandable and expected) bias prevents them from
> seeing the overall picture objectively.

What is the part of the picture that I'm not seeing?  Who am I
ignoring, who am I missing? What are these people that I'm
(IYO unwillingly, fortunately) biased against.  What exactly are
do you conjecture or know they are doing right now in Emacs 28 that
my patch will prevent, affect, break, make difficult?  Just one
example, please.

> > Or make it a defcustom if you're really worried.
> That doesn't change the picture, unless the default for the defcustom
> will be nil.  Which I expect João to object to, because he wants Eglot
> to be updated by default and automatically.

And _only_ Eglot.  Because, that's what happened to it in Emacs 28.

> > > You assume that everyone will
> > > want Eglot and use-package automatically updated, but this assumption
> > > has no real basis.
> >
> > People don't call 'M-x package-install' automatically, nor do they put
> > those calls in their init files automatically.
>
> That's factually incorrect, AFAIU.  Moreover, the cases that bothered
> João (again, AFAIU) were exactly those which you say don't exist.

Yes, you're right Eli.  I pointed to numerous issues collected over
the years, where I found proof that users have this kind of form
int heir configurations:

(use-package eglot :ensure t :config() )

Even ChatGPT suggests this. Just ask it "How do I configure the Eglot
Emacs extension for the <Your favourite language> LSP server"?

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 178 bytes --]

On Wed, Apr 19, 2023 at 08:33:44PM +0100, João Távora wrote:

[...]

> Even ChatGPT suggests this [...]

OK. Now I know I don't want an LSP server ;-P

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 21:32, Eli Zaretskii wrote:
>> Date: Wed, 19 Apr 2023 21:14:06 +0300
>> Cc: arne_bab@web.de, jporterbugs@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 19/04/2023 21:07, Eli Zaretskii wrote:
>>> It has similar problems: it will automatically update packages
>>> mentioned in package--safely-upgradeable-builtins, which might not be
>>> what users want for built-in packages.
>>
>> IMO that kind of choice could be deferred to the maintainer of each
>> individual package.
> 
> No, it cannot, and this and the sibling discussions show why: the
> package maintainers are biased in favor of their packages.  That
> (completely understandable and expected) bias prevents them from
> seeing the overall picture objectively.

I don't know, I never really minded that project.el and xref aren't 
upgraded too often.

Though to be honest I would still expect that, when the user types 'M-x 
package-install', they'd be able to upgrade project or xref to the 
latest version. Simply because that looks like an unambiguous request 
for this exact outcome.

>> Or make it a defcustom if you're really worried.
> 
> That doesn't change the picture, unless the default for the defcustom
> will be nil.  Which I expect João to object to, because he wants Eglot
> to be updated by default and automatically.

Only when their init script contains

   (package-install 'eglot)

or

   (use-package 'eglot :force t)

right?

> Whereas I think the
> compromise, whereby the user should say just once that he/she wants
> Eglot to be automatically updated, is a good compromise given the
> constraints in this case.  Not ideal, but a good-enough compromise.

Might be. In saying the above I'm thinking about a different issue: user 
options should be logical (as much as we possibly manage). When they 
are, they are easier to find and enable.

So if we absolutely can't resolve the problem using the solution 
previously recommended by Philip, Joao and I, we should at least choose 
the most logical among approaches containing a new user option.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

Wed, Apr 19, 2023 at 8:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> >> Or make it a defcustom if you're really worried.
> >
> > That doesn't change the picture, unless the default for the defcustom
> > will be nil.  Which I expect João to object to, because he wants Eglot
> > to be updated by default and automatically.
>
> Only when their init script contains
>
>    (package-install 'eglot)
>
> or
>
>    (use-package 'eglot :force t)
>
> right?

Yes, at least that.  I've given up on interactive M
-x package-install offering Eglot in the completions.  It's also a
backward incompatibility but at least that will alarm users and they
will get some notice that something changed from Emacs 28 to
Emacs 29.

Whereas the silent noop of the non-interactive case is much
worse.

And it's ":ensure", not ":force" by the way. Just ask ChatGPT,
it'll tell you ;-)


Curiously, after many many iterations Philip's patch up to the
very end had this minimally backward compatible behaviour for
the non-interactive calls.  But alas, the final version forbade it too.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 22:46, João Távora wrote:
> Wed, Apr 19, 2023 at 8:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>>>> Or make it a defcustom if you're really worried.
>>>
>>> That doesn't change the picture, unless the default for the defcustom
>>> will be nil.  Which I expect João to object to, because he wants Eglot
>>> to be updated by default and automatically.
>>
>> Only when their init script contains
>>
>>     (package-install 'eglot)
>>
>> or
>>
>>     (use-package 'eglot :force t)
>>
>> right?
> 
> Yes, at least that.  I've given up on interactive M
> -x package-install offering Eglot in the completions.  It's also a
> backward incompatibility but at least that will alarm users and they
> will get some notice that something changed from Emacs 28 to
> Emacs 29.
> 
> Whereas the silent noop of the non-interactive case is much
> worse.
> 
> And it's ":ensure", not ":force" by the way. Just ask ChatGPT,
> it'll tell you ;-)

I try not to talk to robots in my spare time ;-)

So... are you sure that (use-package 'eglot :ensure t) upgrades/upgraded 
the package?

 From what I'm reading in use-package-ensure-elpa, all 'package-install' 
calls are wrapped in

   (unless (package-installed-p package)
     ...

That seems to mean that, as long as the package is installed already 
(whether by being bundled, or because of a previous manual 
installation), it shouldn't get upgraded when this form is executed.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 9:50 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> So... are you sure that (use-package 'eglot :ensure t) upgrades/upgraded
> the package?
>
>  From what I'm reading in use-package-ensure-elpa, all 'package-install'
> calls are wrapped in
>
>    (unless (package-installed-p package)
>      ...
>
> That seems to mean that, as long as the package is installed already
> (whether by being bundled, or because of a previous manual
> installation), it shouldn't get upgraded when this form is executed.

That's true.  Exactly the same as package-install. It will not
reinstall over another one.

But that doesn't change anything.  In Emacs 26/27/28 from scratch
package-install and use-package _will_ rev up Eglot to whatever is the
newest version.  In Emacs 29 it won't.

In fact deleting Eglot and restarting Emacs the config run again is what
I suppose the most common upgrade method (since there is no
package-update in Emacs 28).

Other than that, think CI scripts, dockerfiles, VMs, or just the
casual user who trashes the packages dir to get a fresh set when
looking for a bug (like I do, and multiple people I've interacted with).

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 1:57 PM, João Távora wrote:
> But that doesn't change anything.  In Emacs 26/27/28 from scratch
> package-install and use-package _will_ rev up Eglot to whatever is the
> newest version.  In Emacs 29 it won't.

But in that case, the user simply ends up with, at best, whatever 
version of Eglot was current at the time they ran Emacs with their 
config for the first time on that machine. The status quo is already 
that users are likely on an older version of Eglot, unless they wipe 
their elpa/ directory regularly (which seems an awkward way to update 
your packages to me).

> In fact deleting Eglot and restarting Emacs the config run again is what
> I suppose the most common upgrade method (since there is no
> package-update in Emacs 28).

As mentioned elsewhere in this thread, I just use the *Packages* buffer 
and 'package-menu-mark-upgrades' (bound to "U"). That works on older 
Emacsen too. In 29, 'package-update' would make it easier to update a 
single package though, and 'package-update-all' provides a programmatic 
way to do what I do interactively in 28.

(Though now I wonder why we use both "update" and "upgrade"...)

> Other than that, think CI scripts, dockerfiles, VMs, or just the
> casual user who trashes the packages dir to get a fresh set when
> looking for a bug (like I do, and multiple people I've interacted with).

If a user hasn't done anything to specify the release channel they want, 
then I think the current behavior is correct. "(package-install 'eglot)" 
just means "make sure I have Eglot". However, if the user pinned Eglot 
to GNU ELPA, I think it would make some sense for 'package-install' to 
install Eglot from GNU ELPA (i.e. it now means "make sure I have Eglot 
*from GNU ELPA*").

I know this isn't automatic; a user would need to pin Eglot to GNU ELPA 
to get this behavior, but it's fairly easy to do. use-package already 
supports pinning as well. We'd just need to tweak 'package-install' and 
'use-package :ensure' to handle pinning as described above.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 10:58 PM Jim Porter <jporterbugs@gmail.com> wrote:
>

> As mentioned elsewhere in this thread, I just use the *Packages* buffer
> and 'package-menu-mark-upgrades' (bound to "U"). That works on older
> Emacsen too. In 29, 'package-update' would make it easier to update a
> single package though, and 'package-update-all' provides a programmatic
> way to do what I do interactively in 28.

No. Both are absent in Emacs 28.  I can't say what the the most
common method is, but M-x package-delete + restart + letting your
config run again is the method I use.

I didn't know about "U" until recently.  And that's because
the package menu is a very slow and unreliable command IME.

And besides, "U" _also does different things in Emacs 28 and Emacs 29.
 In 28 it _will_ upgrade Eglot to the latest (so they tell me). But
not in Emacs 29.

This _also_ deserved a fix, but is IMO, not as important as the
simple non-interactice package-install case.

And, for good measure, the new M-x package-update and M-x
package-update-all in Emacs 29 _also_ won't update Eglot.

> (Though now I wonder why we use both "update" and "upgrade"...)

There's a separate bug for that.

> > Other than that, think CI scripts, dockerfiles, VMs, or just the
> > casual user who trashes the packages dir to get a fresh set when
> > looking for a bug (like I do, and multiple people I've interacted with).
>
> If a user hasn't done anything to specify the release channel they want,
> then I think the current behavior is correct. "(package-install 'eglot)"
> just means "make sure I have Eglot". However, if the user pinned Eglot
> to GNU ELPA, I think it would make some sense for 'package-install' to
> install Eglot from GNU ELPA (i.e. it now means "make sure I have Eglot
> *from GNU ELPA*").

By default, Emacs comes with GNU ELPA configured only, I think.
And Eglot has always existed there.  I don't know what you mean
by "pin".  I do know some users _also_ had Eglot on MELPA to get
an even newer version, but I've never really recommended that.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 3:29 PM, João Távora wrote:
> On Wed, Apr 19, 2023 at 10:58 PM Jim Porter <jporterbugs@gmail.com> wrote:
>> As mentioned elsewhere in this thread, I just use the *Packages* buffer
>> and 'package-menu-mark-upgrades' (bound to "U"). That works on older
>> Emacsen too. In 29, 'package-update' would make it easier to update a
>> single package though, and 'package-update-all' provides a programmatic
>> way to do what I do interactively in 28.
> 
> No. Both are absent in Emacs 28.

Yes, but the "U" keybinding is in Emacs 28, which is what I meant there.

> And besides, "U" _also does different things in Emacs 28 and Emacs 29.
>   In 28 it _will_ upgrade Eglot to the latest (so they tell me). But
> not in Emacs 29.

That depends. If I had previously installed Eglot in 28, and then 
download Emacs 29 and upgrade my packages (without deleting my elpa/ 
dir), I'll still get new versions of Eglot from GNU ELPA. I think that 
part works as expected.

> By default, Emacs comes with GNU ELPA configured only, I think.
> And Eglot has always existed there.  I don't know what you mean
> by "pin".  I do know some users _also_ had Eglot on MELPA to get
> an even newer version, but I've never really recommended that.

You can "pin" a package (see 'package-pinned-packages' or the ':pin' 
keyword in use-package) so that you always get updates from a particular 
archive, which is useful if you use a package that rarely gets releases, 
among other things. Then you can set the devel archives to a lower 
priority (see 'package-archive-priorities') and pin that package to the 
devel archive without having to live on the edge for every other 
package. This also means that if I wipe out my elpa/ dir, I get my 
packages from the archive I want.

I think this should also work as a way of declaring that you want Eglot 
from GNU ELPA even if it's built-in, but that part doesn't work right now.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 11:42 PM Jim Porter <jporterbugs@gmail.com> wrote:

> > No. Both are absent in Emacs 28.
>
> Yes, but the "U" keybinding is in Emacs 28, which is what I meant there.
>
> > And besides, "U" _also does different things in Emacs 28 and Emacs 29.
> >   In 28 it _will_ upgrade Eglot to the latest (so they tell me). But
> > not in Emacs 29.
>
> That depends. If I had previously installed Eglot in 28, and then
> download Emacs 29 and upgrade my packages (without deleting my elpa/
> dir), I'll still get new versions of Eglot from GNU ELPA. I think that
> part works as expected.

Yes it does.  Come on, not _everything_ is broken :-)

The bug#62720, reported by me, listed the only workaround
that works identically in Emacs 2*.  Just go to the package
menu and press 'I' on the package you want to install.  Boom, there
go the ancient safeguards against updating builtin packages.

If you look a bit at the package.el code, you'll see that
all this code goes through different flows etc.  All these
safeguards and existing behaviour are more likely to be
be accidental than intentional.

Nevertheless,  it is possible to pick one of them to
maintain the same behaviour.


João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 23:57, João Távora wrote:
> On Wed, Apr 19, 2023 at 9:50 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> 
>> So... are you sure that (use-package 'eglot :ensure t) upgrades/upgraded
>> the package?
>>
>>   From what I'm reading in use-package-ensure-elpa, all 'package-install'
>> calls are wrapped in
>>
>>     (unless (package-installed-p package)
>>       ...
>>
>> That seems to mean that, as long as the package is installed already
>> (whether by being bundled, or because of a previous manual
>> installation), it shouldn't get upgraded when this form is executed.
> That's true.  Exactly the same as package-install. It will not
> reinstall over another one.

OK then, I think have to re-evaluate my position on this. Previously, I 
guess, I made some hasty conclusions from how the discussion went on 
without refreshing the exact details about package.el and use-package 
(the latter I never knew to begin with). Apologies.

Eli, let me know if we should take this back to the bug tracker instead.

> But that doesn't change anything.  In Emacs 26/27/28 from scratch
> package-install and use-package_will_  rev up Eglot to whatever is the
> newest version.  In Emacs 29 it won't.

I think one of the conclusions to be made here is that even if 
(package-install 'eglot) now installs the newest version of Eglot in 
Emacs 29,

   (use-package 'eglot :ensure t)

still won't do that in Emacs 29 because (package-installed-p 'eglot) 
returns t still. So the commit 580d8278c5f48 doesn't help with your 
"most common upgrade method" cited below, if they rely on use-package 
instead of calling 'package-install' directly.

The patch I +1'd here https://debbugs.gnu.org/62720#467 wouldn't help 
with (use-package 'eglot :ensure t) either, IIUC.

But it would help in case the user put (package-install 'eglot) inside 
their init script, or if the CI script contains that. But the CI scripts 
could be fixed anyway (there must be a lot fewer of them out there than 
there are user installations).

Next: 'package-install' isn't supposed to install a new version when 
some version is already installed. The docstring says so, and I just 
tried this out: had older 'vertico' installed (with a newer one 
available in the repo), evaluated (package-install 'vertico), got back 
the message "already installed".

Do we want to change the semantics of 'package-install' just so the 
(minor) fraction of users who call this function directly will get Eglot 
upgraded when starting over from an empty config with Emacs 29? And/or 
change (package-installed-p 'eglot) to return nil when the appropriate 
user option is set? I'm not sure that's wise, certainly not the latter.

> In fact deleting Eglot and restarting Emacs the config run again is what
> I suppose the most common upgrade method (since there is no
> package-update in Emacs 28).

I've never considered this particular scenario, but as explained above, 
it's still not "fixed" by either of the proposed approaches. Not 
entirely, at least.

The most common upgrade method, though, is hopefully 
'package-menu-mark-upgrades'. Or:

   M-x list-packages
   U
   x

And that one also doesn't work for Eglot and other builtin packages. 
Unlike 'package-install', this behavior is not obviously correct, 
although it's "always been like this" so we're probably not going to 
change what it does by default at this point in time.

Another function which is supposed to upgrade packages in 'M-x 
package-update' (why aren't people commenting on bug#62750? the naming's 
gonna drive me crazy). This one is obviously incorrect because upgrades 
exist for Eglot, and yet it doesn't suggest any.

So I would suggest to focus on functions that don't work as intended. 
Namely:

- Add a user option for the list of builtin packages which would be 
upgraded automatically by 'package-menu-mark-upgrades' and 'M-x 
package-upgrade-all' (nee package-update-all). Maybe make it nil by 
default, or maybe add 'eglot' to it. I don't have a strong opinion.

- Fix 'M-x package-upgrade' (nee package-update) to suggest Eglot as one 
of the options and actually perform the upgrade. That shouldn't require 
changes to 'package-install' because, as we already know, the user can 
already install a newer version of Eglot using the 'list-packages' menu 
(and picking the exact version manually). That execution path is going 
to go through 'package-install' as well, so it must be suitable already.

- Revert 580d8278c5f48 because it creates odd semantics (upgrading 
certain packages that are already installed but not others) and it 
doesn't solve the issue with (use-package 'eglot :ensure t) anyway. We 
could keep it, but seems like a half-measure that didn't make anyone 
happy anyway. OTOH, it could minimize the rewrites of CI scripts.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 3:06 PM, Dmitry Gutov wrote:
> - Add a user option for the list of builtin packages which would be 
> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x 
> package-upgrade-all' (nee package-update-all). Maybe make it nil by 
> default, or maybe add 'eglot' to it. I don't have a strong opinion.

I think (with some fixes), 'package-pinned-packages' should work for 
this. If you want to get Eglot from GNU ELPA, then you can pin it as 
such, and it should do the right thing.

Unfortunately, it doesn't *actually* do the right thing, but I think 
that's pretty clearly a bug.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 01:21, Jim Porter wrote:
> On 4/19/2023 3:06 PM, Dmitry Gutov wrote:
>> - Add a user option for the list of builtin packages which would be 
>> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x 
>> package-upgrade-all' (nee package-update-all). Maybe make it nil by 
>> default, or maybe add 'eglot' to it. I don't have a strong opinion.
> 
> I think (with some fixes), 'package-pinned-packages' should work for 
> this. If you want to get Eglot from GNU ELPA, then you can pin it as 
> such, and it should do the right thing.
> 
> Unfortunately, it doesn't *actually* do the right thing, but I think 
> that's pretty clearly a bug.

It might work, too (though I suppose altering this var might be a bit 
harder for an average use than changing a new option to t or adding 
'eglot to it).

Depends on how complex the fix will be, for the "clear bug" you mention.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 3:27 PM, Dmitry Gutov wrote:
> It might work, too (though I suppose altering this var might be a bit 
> harder for an average use than changing a new option to t or adding 
> 'eglot to it).

use-package users should have an easy time of it, at least. You can just 
add ":pin gnu" to your use-package declaration.

That said, I think there should be a way to do it from the *Packages* 
buffer, just like installing a package.

> Depends on how complex the fix will be, for the "clear bug" you mention.

Yeah, we'll have to see. I'll take a look if/when we agree that this is 
a solution to the problem.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 01:06:10 +0300
> Cc: Eli Zaretskii <eliz@gnu.org>, arne_bab@web.de, jporterbugs@gmail.com,
>  emacs-devel <emacs-devel@gnu.org>
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> OK then, I think have to re-evaluate my position on this. Previously, I 
> guess, I made some hasty conclusions from how the discussion went on 
> without refreshing the exact details about package.el and use-package 
> (the latter I never knew to begin with). Apologies.
> 
> Eli, let me know if we should take this back to the bug tracker instead.

I've moved this back to the bug-tracker.  Please post all further
replies about this particular issue, i.e. updating of built-in
packages with package.el, to this bug and not to emacs-devel.

> So I would suggest to focus on functions that don't work as intended. 
> Namely:
> 
> - Add a user option for the list of builtin packages which would be 
> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x 
> package-upgrade-all' (nee package-update-all). Maybe make it nil by 
> default, or maybe add 'eglot' to it. I don't have a strong opinion.
> 
> - Fix 'M-x package-upgrade' (nee package-update) to suggest Eglot as one 
> of the options and actually perform the upgrade. That shouldn't require 
> changes to 'package-install' because, as we already know, the user can 
> already install a newer version of Eglot using the 'list-packages' menu 
> (and picking the exact version manually). That execution path is going 
> to go through 'package-install' as well, so it must be suitable already.
> 
> - Revert 580d8278c5f48 because it creates odd semantics (upgrading 
> certain packages that are already installed but not others) and it 
> doesn't solve the issue with (use-package 'eglot :ensure t) anyway. We 
> could keep it, but seems like a half-measure that didn't make anyone 
> happy anyway. OTOH, it could minimize the rewrites of CI scripts.

I don't think we are ready to make any new decisions in this matter.
I think we don't even have a comprehensive and detailed picture of the
problems with updating/upgrading built-in packages in Emacs 29.
People are still discovering facts and subtleties of various
package.el commands and features, and are still arguing what exactly
happens in this or that scenario.

So before we discuss solutions, we need a full and detailed
description of the problems to solve.  If someone can do the footwork
of collecting this information and posting it here, please do, and
TIA.

I will say up front that, given what I already read here and in the
related thread on emacs-devel, there seem to be too many
inconsistencies and dark corners in this, in particular when built-in
packages are involved.  We will probably be unable to solve them in
time for Emacs 29.1, certainly not all of them.  So don't raise your
expectations too high.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 11:02 AM Eli Zaretskii <eliz@gnu.org> wrote:

> I will say up front that, given what I already read here and in the
> related thread on emacs-devel, there seem to be too many
> inconsistencies and dark corners in this, in particular when built-in
> packages are involved.  We will probably be unable to solve them in
> time for Emacs 29.1, certainly not all of them.  So don't raise your
> expectations too high.

Yes, I don't have these expectations.  Please in the meantime, allow
me add to eglot.el:

 (defun eglot-update ()  "Update Eglot regardless of package.el policy."
    (interactive)
    (unless package-archive-contents (package-refresh-contents))
    (package-install (cadr (assoc 'eglot package-archive-contents))

This will allow at least Eglot users to weather the storm until Emacs
30.  I know you forbade this in the past, but now that more "facts and
subtleties" have come to light, maybe you can reconsider?  I'll readily
deprecate this function when a solution has been found for the next
Emacs.  It doesn't really affect anyone but Eglot users and these
are, in your own admission, already negatively affected anyway, so
what is there to lose?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 11:31:52 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, jporterbugs@gmail.com, 62720@debbugs.gnu.org, 
> 	philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> 
> On Thu, Apr 20, 2023 at 11:02 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Please in the meantime, allow me add to eglot.el:
> 
>  (defun eglot-update ()  "Update Eglot regardless of package.el policy."
>     (interactive)
>     (unless package-archive-contents (package-refresh-contents))
>     (package-install (cadr (assoc 'eglot package-archive-contents))
> 
> This will allow at least Eglot users to weather the storm until Emacs
> 30.  I know you forbade this in the past, but now that more "facts and
> subtleties" have come to light, maybe you can reconsider?  I'll readily
> deprecate this function when a solution has been found for the next
> Emacs.  It doesn't really affect anyone but Eglot users and these
> are, in your own admission, already negatively affected anyway, so
> what is there to lose?

I didn't forbid it.  I said I'd prefer not to have it in Emacs, and I
still do.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 12:49 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > Please in the meantime, allow me add to eglot.el:
> >
> >  (defun eglot-update ()  "Update Eglot regardless of package.el policy."
> >     (interactive)
> >     (unless package-archive-contents (package-refresh-contents))
> >     (package-install (cadr (assoc 'eglot package-archive-contents))
> >
> > This will allow at least Eglot users to weather the storm until Emacs
> > 30.  I know you forbade this in the past, but now that more "facts and
> > subtleties" have come to light, maybe you can reconsider?  I'll readily
> > deprecate this function when a solution has been found for the next
> > Emacs.  It doesn't really affect anyone but Eglot users and these
> > are, in your own admission, already negatively affected anyway, so
> > what is there to lose?
>
> I didn't forbid it.  I said I'd prefer not to have it in Emacs, and I
> still do.


OK.  Thanks for clarifying.  I would _also_ prefer not to.  And
if a good solution is found for Emacs 29, I will take it out or just
deprecate it.  But given this situation, I'm just going to
add it.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 12:53:35 +0100
> Cc: dmitry@gutov.dev, jporterbugs@gmail.com, 62720@debbugs.gnu.org, 
> 	philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> 
> On Thu, Apr 20, 2023 at 12:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I didn't forbid it.  I said I'd prefer not to have it in Emacs, and I
> > still do.
> 
> 
> OK.  Thanks for clarifying.  I would _also_ prefer not to.  And
> if a good solution is found for Emacs 29, I will take it out or just
> deprecate it.  But given this situation, I'm just going to
> add it.

Sigh...

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 13:02, Eli Zaretskii wrote:
>> Date: Thu, 20 Apr 2023 01:06:10 +0300
>> Cc: Eli Zaretskii <eliz@gnu.org>, arne_bab@web.de, jporterbugs@gmail.com,
>>   emacs-devel <emacs-devel@gnu.org>
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> OK then, I think have to re-evaluate my position on this. Previously, I
>> guess, I made some hasty conclusions from how the discussion went on
>> without refreshing the exact details about package.el and use-package
>> (the latter I never knew to begin with). Apologies.
>>
>> Eli, let me know if we should take this back to the bug tracker instead.
> 
> I've moved this back to the bug-tracker.  Please post all further
> replies about this particular issue, i.e. updating of built-in
> packages with package.el, to this bug and not to emacs-devel.

Noted.

>> So I would suggest to focus on functions that don't work as intended.
>> Namely:
>>
>> - Add a user option for the list of builtin packages which would be
>> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x
>> package-upgrade-all' (nee package-update-all). Maybe make it nil by
>> default, or maybe add 'eglot' to it. I don't have a strong opinion.
>>
>> - Fix 'M-x package-upgrade' (nee package-update) to suggest Eglot as one
>> of the options and actually perform the upgrade. That shouldn't require
>> changes to 'package-install' because, as we already know, the user can
>> already install a newer version of Eglot using the 'list-packages' menu
>> (and picking the exact version manually). That execution path is going
>> to go through 'package-install' as well, so it must be suitable already.
>>
>> - Revert 580d8278c5f48 because it creates odd semantics (upgrading
>> certain packages that are already installed but not others) and it
>> doesn't solve the issue with (use-package 'eglot :ensure t) anyway. We
>> could keep it, but seems like a half-measure that didn't make anyone
>> happy anyway. OTOH, it could minimize the rewrites of CI scripts.
> 
> I don't think we are ready to make any new decisions in this matter.
> I think we don't even have a comprehensive and detailed picture of the
> problems with updating/upgrading built-in packages in Emacs 29.
> People are still discovering facts and subtleties of various
> package.el commands and features, and are still arguing what exactly
> happens in this or that scenario.

True.

> So before we discuss solutions, we need a full and detailed
> description of the problems to solve.  If someone can do the footwork
> of collecting this information and posting it here, please do, and
> TIA.

I think I have made a fair attempt at this, though. Here's an update:

- package-upgrade (nee package-update) doesn't upgrade builtin packages 
that never been upgraded before. It's a bug. Hopefully not too hard to fix.

- package-menu-mark-upgrades and package-update-all don't upgrade them 
either. That's not necessarily a bug, but a problem nevertheless. A new 
user option could help.

- Fixing package-update should also obviate the need for eglot-update. 
Though perhaps the latter could still be useful as a single entry point 
to recommend to both users of Emacs 28 and 29+.

- The current fix (commit 580d8278c5f48) is not comprehensive WRT to 
Joao's scenario because use-package-ensure-elpa short-circuits when it 
find that the package is installed ('package-installed-p' returns t). So 
(use-package eglot :ensure t) does not upgrade Eglot even when 
package-install-upgrade-built-in is t.

- package-install-upgrade-built-in is not nuanced: if we suggest the 
users to set it to t, that can result in making _all_ builtin packages 
upgradable with 'package-install'. Whereas I think we originally only 
wanted that for Eglot and maybe for use-package. For this and other 
minor reasons I would suggest reverting 580d8278c5f48. But I suppose we 
could also try to make it more granular (e.g. turn the boolean option 
into a list). I'm not sure it's a good direction overall, however.

- According to Jim P., package pinning doesn't work for builtin packages 
either. Which could be a decent solution as well, e.g. putting something 
like this in the docs: (use-package eglot :ensure t :pin gnu) if the 
users want the Eglot version from ELPA -- and this form might even be 
compatible with Emacs 28. I'm not sure how difficult it would be to fix 
package pinning, however.

I haven't spent as much time on this bug as some others here, though, so 
corrections and additions are welcome.

> I will say up front that, given what I already read here and in the
> related thread on emacs-devel, there seem to be too many
> inconsistencies and dark corners in this, in particular when built-in
> packages are involved.  We will probably be unable to solve them in
> time for Emacs 29.1, certainly not all of them.  So don't raise your
> expectations too high.

The reasons for me to be hopeful are:

- The functions are propose to fix are "leaves": those that are supposed 
to be used interactively, without (many) known callers in-tree. E.g. 
fixing package-update-all shouldn't affect some other part of Emacs as 
much as changing package-install could.

- I imagine the diffs will be rather short. I haven't tried to take a 
stab at this yet, though, and if somebody wants to take the initiative, 
they're very welcome.

The reasons for me to be less hopeful:

- We seem to be unable to come to any agreement here. :-D

Joao's approval for the above list would go a long way.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 2:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> The reasons for me to be less hopeful:
>
> - We seem to be unable to come to any agreement here. :-D
>
> Joao's approval for the above list would go a long way.

First of all, thank you Dmitry for making such an extensive
and detailed summary of the simualtion.  It's really
impressive.

What you say about a fixed package-update obviating the need
for eglot-update is totally true.  In fact I proposed a fixed
package-update much much earlier in this thread.  If you
manage to do that, then I'm more than happy to remove
eglot-update.

What more approval do you want? :-) I mean, I've resigned
myself to a bad situation already (but if you can make
package-install DTRT, I'm of course even happier).  So
package-update or eglot-update -- anything that I can put
on the README .

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 2:56 PM João Távora <joaotavora@gmail.com> wrote:
>
> On Thu, Apr 20, 2023 at 2:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> > The reasons for me to be less hopeful:
> >
> > - We seem to be unable to come to any agreement here. :-D
> >
> > Joao's approval for the above list would go a long way.
>
> First of all, thank you Dmitry for making such an extensive
> and detailed summary of the simualtion.  It's really
> impressive.
>
> What you say about a fixed package-update obviating the need
> for eglot-update is totally true.  In fact I proposed a fixed
> package-update much much earlier in this thread.  If you
> manage to do that, then I'm more than happy to remove
> eglot-update.

Whoops, sorry, made a thinko.  package-update _doesn't_ really
obviate the need for eglot-update, because it won't be callable
in Emacs 26/27/28, and eglot-update will.

...unless package.el itself becomes a :core package and
self-updates through ELPA, which I think is being proposed
around here somewhere.  Could happen in master (and leave
Emacs's 29 version untouched).

So for the moment eglot-update is my best out.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 17:25, João Távora wrote:
> Whoops, sorry, made a thinko.  package-update_doesn't_  really
> obviate the need for eglot-update, because it won't be callable
> in Emacs 26/27/28, and eglot-update will.
> 
> ...unless package.el itself becomes a :core package and
> self-updates through ELPA, which I think is being proposed
> around here somewhere.  Could happen in master (and leave
> Emacs's 29 version untouched).
> 
> So for the moment eglot-update is my best out.

We could add it and then make a tentative plan (with a TODO comment) to 
remove it as soon as a version of Emacs with package-upgrade that 
supports upgrading builtins becomes the older supported version.

That would be in line with our general policy of developing :core 
packages anyway.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 3:31 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 20/04/2023 17:25, João Távora wrote:
> > Whoops, sorry, made a thinko.  package-update_doesn't_  really
> > obviate the need for eglot-update, because it won't be callable
> > in Emacs 26/27/28, and eglot-update will.
> >
> > ...unless package.el itself becomes a :core package and
> > self-updates through ELPA, which I think is being proposed
> > around here somewhere.  Could happen in master (and leave
> > Emacs's 29 version untouched).
> >
> > So for the moment eglot-update is my best out.
>
> We could add it and then make a tentative plan (with a TODO comment) to
> remove it as soon as a version of Emacs with package-upgrade that
> supports upgrading builtins becomes the older supported version.
>
> That would be in line with our general policy of developing :core
> packages anyway.

Yes, for sure, we'll deprecate it and make it re-route to
package-upgrade.  And what do you think of the idea of package.el
becoming :core itself.  It doesn't seem to have many dependencies.
If that were to happen and the fixed package.el you (or someone else)
is going to eventually propose to Emacs 29 was out in the open,
eglot-update wouldn't be needed.  And we would never have these
discussions under the shadow of the no-more-changes and
the pretest's pressure.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 17:40, João Távora wrote:
> Yes, for sure, we'll deprecate it and make it re-route to
> package-upgrade.  And what do you think of the idea of package.el
> becoming :core itself.  It doesn't seem to have many dependencies.
> If that were to happen and the fixed package.el you (or someone else)
> is going to eventually propose to Emacs 29 was out in the open,
> eglot-update wouldn't be needed.  And we would never have these
> discussions under the shadow of the no-more-changes and
> the pretest's pressure.

I don't know, someone should really test that idea, hard.

It could exhibit the same problem that some package upgrades do: the 
installed packages is not properly reloaded, and the new version is not 
quite usable until Emacs' restart (I recall you saw something like this 
with project.el).

For such a central piece of infrastructure this could be a bigger 
problem by itself, and could even be made worse by the fact that the 
code which installs the new version of package.el also belongs to it, so 
it'll be hotswapped (right?) sometime during its execution.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 15:25:00 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, jporterbugs@gmail.com, 62720@debbugs.gnu.org, 
> 	philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> 
> Whoops, sorry, made a thinko.  package-update _doesn't_ really
> obviate the need for eglot-update, because it won't be callable
> in Emacs 26/27/28, and eglot-update will.
> 
> ...unless package.el itself becomes a :core package and
> self-updates through ELPA, which I think is being proposed
> around here somewhere.  Could happen in master (and leave
> Emacs's 29 version untouched).
> 
> So for the moment eglot-update is my best out.

Of course, introducing and advertising eglot-update will mean users of
Emacs 29+ will need to use a different command to upgrade Eglot than
users of previous Emacs versions -- the same problem that you consider
so awful in the change we made in package-update, where these users
will now have to specify a prefix argument (and do it just once,
AFAIU).  That is, of course, a very consistent logic on your part.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 3:49 PM Eli Zaretskii <eliz@gnu.org> wrote:

> Of course, introducing and advertising eglot-update will mean users of
> Emacs 29+ will need to use a different command to upgrade Eglot than
> users of previous Emacs versions -- the same problem that you consider
> so awful in the change we made in package-update, where these users
> will now have to specify a prefix argument (and do it just once,
> AFAIU).  That is, of course, a very consistent logic on your part.

Sarcasm?  It's my best out.  eglot-update will eventually be available
to users of previous Emacs versions.  And I can precisely control
its behaviour to "always update Eglot, no matter what" (which
package-install didn't do -- BTW you mean package-install I think).
And it will work non-interactively in a config. And the change you're
referring to has been proposed for reversion anyway, because problems.
I don't appreciate eglot-update especially -- I've said that.  But
it's the best out and won't harm anyone.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

(I haven't had the time to follow the thread in detail, so I might have
missed some context)

João Távora <joaotavora@gmail.com> writes:

> On Thu, Apr 20, 2023 at 2:56 PM João Távora <joaotavora@gmail.com> wrote:
>>
>> On Thu, Apr 20, 2023 at 2:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>>
>> > The reasons for me to be less hopeful:
>> >
>> > - We seem to be unable to come to any agreement here. :-D
>> >
>> > Joao's approval for the above list would go a long way.
>>
>> First of all, thank you Dmitry for making such an extensive
>> and detailed summary of the simualtion.  It's really
>> impressive.
>>
>> What you say about a fixed package-update obviating the need
>> for eglot-update is totally true.  In fact I proposed a fixed
>> package-update much much earlier in this thread.  If you
>> manage to do that, then I'm more than happy to remove
>> eglot-update.
>
> Whoops, sorry, made a thinko.  package-update _doesn't_ really
> obviate the need for eglot-update, because it won't be callable
> in Emacs 26/27/28, and eglot-update will.
>
> ...unless package.el itself becomes a :core package and
> self-updates through ELPA, which I think is being proposed
> around here somewhere.  Could happen in master (and leave
> Emacs's 29 version untouched).

BTW, everything that this would require has been done so this is not a
far-off solution.

> So for the moment eglot-update is my best out.
>
> João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 16:56, João Távora wrote:
> First of all, thank you Dmitry for making such an extensive
> and detailed summary of the simualtion.  It's really
> impressive.
> 
> What you say about a fixed package-update obviating the need
> for eglot-update is totally true.  In fact I proposed a fixed
> package-update much much earlier in this thread.  If you
> manage to do that, then I'm more than happy to remove
> eglot-update.
> 
> What more approval do you want? 😄

An explicit approval always helps anyway. Thanks!

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 16:39:20 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > So before we discuss solutions, we need a full and detailed
> > description of the problems to solve.  If someone can do the footwork
> > of collecting this information and posting it here, please do, and
> > TIA.
> 
> I think I have made a fair attempt at this, though. Here's an update:

Thanks.

What I miss in this description is something a bit higher-level: the
list of all the ways/commands/methods people use to install and
upgrade the packages.  We must have this list before our eyes to make
sure we review all the behaviors related to these activities, if we
want them all to eventually behave consistently.

E.g., you don't mention any commands invoked from the package menu.

> - package-upgrade (nee package-update) doesn't upgrade builtin packages 
> that never been upgraded before. It's a bug. Hopefully not too hard to fix.

I'm okay with adding the same prefix argument to package-upgrade,
which would then allow upgrading a built-in package.  IOW, a change
similar to what we did in package-install -- provided that the change
is safe enough to go into Emacs 29.

But note that AFAIU there's a relatively easy workaround for this:
install the later version of the package manually, just once.

> - package-menu-mark-upgrades and package-update-all don't upgrade them 
> either. That's not necessarily a bug, but a problem nevertheless. A new 
> user option could help.

A very relevant question is: can this wait till Emacs 30?

> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to 
> Joao's scenario because use-package-ensure-elpa short-circuits when it 
> find that the package is installed ('package-installed-p' returns t). So 
> (use-package eglot :ensure t) does not upgrade Eglot even when 
> package-install-upgrade-built-in is t.

I don't think (use-package eglot :ensure t) should automatically
upgrade built-in packages.  We could make it do that if
package-install-upgrade-built-in is non-nil -- again, if such a change
could be safe enough.  If not, then the same workaround as for
package-upgrade would do here, I suppose?

> - package-install-upgrade-built-in is not nuanced: if we suggest the 
> users to set it to t, that can result in making _all_ builtin packages 
> upgradable with 'package-install'.

It should be set to t only by users who indeed want all built-in
packages to be updated.  That's why the default is nil.

> Whereas I think we originally only wanted that for Eglot and maybe
> for use-package.

"We" never did want that.  João did, for obvious reasons, but that was
never my intent.  The issue is indeed more general: what should
package-install and package.el in general do with built-in packages
for which a newer version is on ELPA?  We don't yet know the answer to
that, and no hope of obtaining one before Emacs 29.1 is released
(barring any calamities).

> For this and other minor reasons I would suggest reverting
> 580d8278c5f48.

Not going to happen, not unless someone comes up with a better
solution that is much better and still safe enough.  Personally, I
don't believe such a solution exists, since we don't really know the
answer to the above question.

> But I suppose we could also try to make it more granular (e.g. turn
> the boolean option into a list). I'm not sure it's a good direction
> overall, however.

I don't think we should go in this direction, precisely because we
don't know it's a good one.

> - According to Jim P., package pinning doesn't work for builtin packages 
> either. Which could be a decent solution as well, e.g. putting something 
> like this in the docs: (use-package eglot :ensure t :pin gnu) if the 
> users want the Eglot version from ELPA -- and this form might even be 
> compatible with Emacs 28. I'm not sure how difficult it would be to fix 
> package pinning, however.

As long as this is optional behavior (and AFAIU it is), I won't
object, but again, provided that the addition of this will not touch
any other code in unsafe ways.

> The reasons for me to be hopeful are:
> 
> - The functions are propose to fix are "leaves": those that are supposed 
> to be used interactively, without (many) known callers in-tree. E.g. 
> fixing package-update-all shouldn't affect some other part of Emacs as 
> much as changing package-install could.

Experience until now in this thread indicates otherwise: almost every
suggested change touched low-level code with many callers.  It isn't
an accident that arriving at what was finally installed took so many
iterations for such a simple change.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Thu, 20 Apr 2023 17:25:49 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> "We" never did want that.  João did, for obvious reasons, but that was
    Eli> never my intent.  The issue is indeed more general: what should
    Eli> package-install and package.el in general do with built-in packages
    Eli> for which a newer version is on ELPA?  We don't yet know the answer to
    Eli> that, and no hope of obtaining one before Emacs 29.1 is released
    Eli> (barring any calamities).

"We" may not know that, but by the principle of not giving me things *I*
didnʼt ask for, *I* donʼt want ':core' packages being automatically
upgraded, unless I either

1. explicitly ask for such an upgrade
2. have myself somehow installed a newer version already, in which case
  itʼs no longer a ':core' package

Iʼve not checked, but does whatʼs currently in emacs-29 not give us at
least [1]?

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Thu, 20 Apr 2023 17:25:49 +0300, Eli Zaretskii <eliz@gnu.org> said:
>
>     Eli> "We" never did want that.  João did, for obvious reasons, but that was
>     Eli> never my intent.  The issue is indeed more general: what should
>     Eli> package-install and package.el in general do with built-in packages
>     Eli> for which a newer version is on ELPA?  We don't yet know the answer to
>     Eli> that, and no hope of obtaining one before Emacs 29.1 is released
>     Eli> (barring any calamities).
>
> "We" may not know that, but by the principle of not giving me things *I*
> didnʼt ask for, *I* donʼt want ':core' packages being automatically
> upgraded, unless I either
>
> 1. explicitly ask for such an upgrade
> 2. have myself somehow installed a newer version already, in which case
>   itʼs no longer a ':core' package
>
> Iʼve not checked, but does whatʼs currently in emacs-29 not give us at
> least [1]?

Yes, this is currently provided by C-u M-x package-install.  2. should
(?) have always been the case.

> Robert

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Thu, Apr 20, 2023 at 7:08 PM Robert Pluim <rpluim@gmail.com> wrote:

> "We" may not know that, but by the principle of not giving me things *I*
> didnʼt ask for, *I* donʼt want ':core' packages being automatically
> upgraded, unless I either
>
> 1. explicitly ask for such an upgrade
> 2. have myself somehow installed a newer version already, in which case
>   itʼs no longer a ':core' package
>
> Iʼve not checked, but does whatʼs currently in emacs-29 not give us at
> least [1]?

Not when package dependencies are involved.  If you weren't aware,
in Emacs 26 (including 29), if you explicitly ask to install package A
and it depends on :core package B, which you didn't ask to install,
package B gets upgraded.

In another data point, the last two patches I proposed to package-install
are similar and only extend the "upgrade-even-if-:core" behaviour
to two packages that weren't core are now :core.  Those two packages
are Eglot and Use-Package.  The rationale I used to develop these patches
was to protect users like you (inasmuch as Emacs 28 already protects
you) _and_ to protect Eglot users.  I am of course presuming that
you aren't/weren't also an Eglot user, otherwise you would have been
hit by these package upgrades related to dependencies in Emacs 28
and would have noticed them.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Thu, 20 Apr 2023 19:53:16 +0100, João Távora <joaotavora@gmail.com> said:

    João> On Thu, Apr 20, 2023 at 7:08 PM Robert Pluim <rpluim@gmail.com> wrote:
    >> "We" may not know that, but by the principle of not giving me things *I*
    >> didnʼt ask for, *I* donʼt want ':core' packages being automatically
    >> upgraded, unless I either
    >> 
    >> 1. explicitly ask for such an upgrade
    >> 2. have myself somehow installed a newer version already, in which case
    >> itʼs no longer a ':core' package
    >> 
    >> Iʼve not checked, but does whatʼs currently in emacs-29 not give us at
    >> least [1]?

    João> Not when package dependencies are involved.  If you weren't aware,
    João> in Emacs 26 (including 29), if you explicitly ask to install package A
    João> and it depends on :core package B, which you didn't ask to install,
    João> package B gets upgraded.

I think thatʼs OK as a behaviour

    João> In another data point, the last two patches I proposed to package-install
    João> are similar and only extend the "upgrade-even-if-:core" behaviour
    João> to two packages that weren't core are now :core.  Those two packages
    João> are Eglot and Use-Package.  The rationale I used to develop these patches
    João> was to protect users like you (inasmuch as Emacs 28 already protects
    João> you) _and_ to protect Eglot users.  I am of course presuming that
    João> you aren't/weren't also an Eglot user, otherwise you would have been
    João> hit by these package upgrades related to dependencies in Emacs 28
    João> and would have noticed them.

I use eglot on emacs-29, but have never installed it (and I never run
emacs-28 any more :-))

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

On Mon, Apr 24, 2023, 08:48 Robert Pluim <rpluim@gmail.com> wrote:

>
>
>     João> Not when package dependencies are involved.  If you weren't
> aware,
>     João> in Emacs 26 (including 29), if you explicitly ask to install
> package A
>     João> and it depends on :core package B, which you didn't ask to
> install,
>     João> package B gets upgraded.
>
> I think thatʼs OK as a behaviour


Noted :)

  .  I am of course presuming that
>     João> you aren't/weren't also an Eglot user, otherwise you would have
> been
>     João> hit by these package upgrades related to dependencies in Emacs 28
>     João> and would have noticed them.
>
> I use eglot on emacs-29, but have never installed it (and I never run
> emacs-28 any more :-))
>

Also noted. Eglot on Emacs 29 is generally the most stable version of it
you can get. However is has less features. Expect the gap to grow with
time. Some fringe LSP servers are also better supported on later versions.

If you never use package-install on Eglot, then there's even less reason to
worry about my patch, which only affects Eglot.

(Meaning, FTR, that we're still missing an account from at least one user
who would be perturbed by my patch.)

Robert, do you use package-install at all? On what packages?

João

[-- Attachment #2: Type: text/html, Size: 2332 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Mon, 24 Apr 2023 09:57:43 +0100, João Távora <joaotavora@gmail.com> said:
    >> I use eglot on emacs-29, but have never installed it (and I never run
    >> emacs-28 any more :-))
    >> 

    João> Also noted. Eglot on Emacs 29 is generally the most stable version of it
    João> you can get. However is has less features. Expect the gap to grow with
    João> time. Some fringe LSP servers are also better supported on later versions.

There are too many things going on to attempt to keep up to date with
them all :-)

    João> If you never use package-install on Eglot, then there's even less reason to
    João> worry about my patch, which only affects Eglot.

    João> (Meaning, FTR, that we're still missing an account from at least one user
    João> who would be perturbed by my patch.)

    João> Robert, do you use package-install at all? On what packages?

Explicitly in my init file, no. The only package installation and
upgrading I do is via `list-packages', for the usual suspects like
magit and helm. And even that I do reluctantly, since something always
seems to break.

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Mon, Apr 24, 2023 at 10:38 AM Robert Pluim <rpluim@gmail.com> wrote:
>
> >>>>> On Mon, 24 Apr 2023 09:57:43 +0100, João Távora <joaotavora@gmail.com> said:
>     >> I use eglot on emacs-29, but have never installed it (and I never run
>     >> emacs-28 any more :-))
>     >>
>
>     João> Also noted. Eglot on Emacs 29 is generally the most stable version of it
>     João> you can get. However is has less features. Expect the gap to grow with
>     João> time. Some fringe LSP servers are also better supported on later versions.
>
> There are too many things going on to attempt to keep up to date with
> them all :-)

Perfectly reasonable.  It would be rather unwise to force a newer
version of Eglot on you.  Do let me know if you find any bugs in
the Emacs 29 eglot.el, which is version 1.12.29. The pretest is here
for that.

>     João> If you never use package-install on Eglot, then there's even less reason to
>     João> worry about my patch, which only affects Eglot.
>
>     João> (Meaning, FTR, that we're still missing an account from at least one user
>     João> who would be perturbed by my patch.)
>
>     João> Robert, do you use package-install at all? On what packages?
>
> Explicitly in my init file, no. The only package installation and
> upgrading I do is via `list-packages', for the usual suspects like
> magit and helm. And even that I do reluctantly, since something always
> seems to break.

That's likely something to bring up with the maintainers of
said packages (which aren't GNU ELPA btw).  I don't even use
them (I use stock VC and stock M-x fido-vertical-mode).

But do you have any suspicion if that is related to upgrade of
dependencies, or -- more to the point of this bug report --
to upgrade of :core dependencies?

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Mon, 24 Apr 2023 12:43:21 +0100, João Távora <joaotavora@gmail.com> said:

    João> On Mon, Apr 24, 2023 at 10:38 AM Robert Pluim <rpluim@gmail.com> wrote:
    >> 
    >> >>>>> On Mon, 24 Apr 2023 09:57:43 +0100, João Távora <joaotavora@gmail.com> said:
    >> >> I use eglot on emacs-29, but have never installed it (and I never run
    >> >> emacs-28 any more :-))
    >> >>
    >> 
    João> Also noted. Eglot on Emacs 29 is generally the most stable version of it
    João> you can get. However is has less features. Expect the gap to grow with
    João> time. Some fringe LSP servers are also better supported on later versions.
    >> 
    >> There are too many things going on to attempt to keep up to date with
    >> them all :-)

    João> Perfectly reasonable.  It would be rather unwise to force a newer
    João> version of Eglot on you.  Do let me know if you find any bugs in
    João> the Emacs 29 eglot.el, which is version 1.12.29. The pretest is here
    João> for that.

I will, although I use clangd which I suspect is pretty heavily tested
anyway.

    >> Explicitly in my init file, no. The only package installation and
    >> upgrading I do is via `list-packages', for the usual suspects like
    >> magit and helm. And even that I do reluctantly, since something always
    >> seems to break.

    João> That's likely something to bring up with the maintainers of
    João> said packages (which aren't GNU ELPA btw).  I don't even use
    João> them (I use stock VC and stock M-x fido-vertical-mode).

I donʼt think magit or helm are to blame here.

    João> But do you have any suspicion if that is related to upgrade of
    João> dependencies, or -- more to the point of this bug report --
    João> to upgrade of :core dependencies?

`list-packages' seems to get confused about which dependent packages
need to be uninstalled, and sometimes uninstalls too many (or maybe it
fails to upgrade them, Iʼm not sure). I then get errors about 'package
bar requires package foo version 1.12, only found 1.11' or similar.

Every time it happens I manually install the packages that should not
have been uninstalled, and get on with other things, since I donʼt
normally have time to spend on creating a minimal test case.

Now you understand why I only upgrade my packages every couple of
months.

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Jim Porter <jporterbugs@gmail.com>,  62720@debbugs.gnu.org,  "Philip K."
>  <philipk@posteo.net>,  Dmitry Gutov <dmitry@gutov.dev>,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  Eli Zaretskii <eliz@gnu.org>,  Lars
>  Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 24 Apr 2023 15:01:59 +0200
> 
>     João> But do you have any suspicion if that is related to upgrade of
>     João> dependencies, or -- more to the point of this bug report --
>     João> to upgrade of :core dependencies?
> 
> `list-packages' seems to get confused about which dependent packages
> need to be uninstalled, and sometimes uninstalls too many (or maybe it
> fails to upgrade them, Iʼm not sure). I then get errors about 'package
> bar requires package foo version 1.12, only found 1.11' or similar.

This could be related to the fact that I see here some packages as
"incompat".  Eglot, Flymake, Project, and others are in that category.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Robert Pluim]

>>>>> On Mon, 24 Apr 2023 16:08:57 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: Jim Porter <jporterbugs@gmail.com>,  62720@debbugs.gnu.org,  "Philip K."
    >> <philipk@posteo.net>,  Dmitry Gutov <dmitry@gutov.dev>,  Stefan Monnier
    >> <monnier@iro.umontreal.ca>,  Eli Zaretskii <eliz@gnu.org>,  Lars
    >> Ingebrigtsen <larsi@gnus.org>
    >> Date: Mon, 24 Apr 2023 15:01:59 +0200
    >> 
    João> But do you have any suspicion if that is related to upgrade of
    João> dependencies, or -- more to the point of this bug report --
    João> to upgrade of :core dependencies?
    >> 
    >> `list-packages' seems to get confused about which dependent packages
    >> need to be uninstalled, and sometimes uninstalls too many (or maybe it
    >> fails to upgrade them, Iʼm not sure). I then get errors about 'package
    >> bar requires package foo version 1.12, only found 1.11' or similar.

    Eli> This could be related to the fact that I see here some packages as
    Eli> "incompat".  Eglot, Flymake, Project, and others are in that category.

I donʼt see that, but I do have eg:

  helm-core                      20230317.1729  available    melpa    Development files for Helm
  helm-core                      20220425.1625  dependency            Development files for Helm
  helm-core                      20220824.1925  obsolete              Development files for Helm

Maybe I should trim my `package-archives'

Robert
-- 

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 24/04/2023 16:01, Robert Pluim wrote:
> Every time it happens I manually install the packages that should not
> have been uninstalled, and get on with other things, since I donʼt
> normally have time to spend on creating a minimal test case.

We might have some more bugs with package being removed from 
package-selected-packages during installation/upgrade. I just found one 
in 'M-x package-update' (discussed within the context of the patch for it).

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Mon, Apr 24, 2023 at 2:02 PM Robert Pluim <rpluim@gmail.com> wrote:

> Now you understand why I only upgrade my packages every couple of
> months.

It doesn't seem like you have any problems with the packages
themselves but rather with a significantly buggy idiosyncratic
package management system(s).

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 20/04/2023 17:25, Eli Zaretskii wrote:

> What I miss in this description is something a bit higher-level: the
> list of all the ways/commands/methods people use to install and
> upgrade the packages.  We must have this list before our eyes to make
> sure we review all the behaviors related to these activities, if we
> want them all to eventually behave consistently.

Why list the commands people use to install packages if we're talking 
about upgrading them? Hopefully we'll decide that 'package-install' 
won't upgrade packages because it hasn't done that in the past. Or if 
it's still going to do that because we don't want to revert commits, 
we'll just avoid touching that.

> E.g., you don't mention any commands invoked from the package menu.

The command which people use to upgrade packages from the package menu 
is called package-menu-mark-upgrades (bound to 'U'), mentioned here a 
few times. I also said that I hope it's *the* main way people use to 
upgrade packages.

The others would be package-update and package-update-all (can we please 
do the rename now? I know they've been in for a year, but just as this 
very discussion has shown, most people weren't even aware of their 
existence).

>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
> 
> I'm okay with adding the same prefix argument to package-upgrade,
> which would then allow upgrading a built-in package.  IOW, a change
> similar to what we did in package-install -- provided that the change
> is safe enough to go into Emacs 29.

If we agree it's a bug, why don't we just fix it? 'package-update' is an 
interactive function which itself it called from only one place: 
'package-update-all', and since the plan is to improve both, we can make 
sure they only do what we ask of them: package-update will upgrade 
builtins when invoked directly, and package-update-all will upgrade them 
only when the builtin has been upgraded before (making it not a builtin 
anymore), or a new user option is set.

> But note that AFAIU there's a relatively easy workaround for this:
> install the later version of the package manually, just once.

It's indeed a workaround that exists, and if we were satisfied with it, 
we would have closed this bug as "wontfix". But it's not a very friendly 
way to do that (the user must hunt for the appropriate version in the 
packages menu), and it's very poorly scriptable (you won't be able to 
provide an easy and readable snippet for the user to evaluate or put in 
their init script).

>> - package-menu-mark-upgrades and package-update-all don't upgrade them
>> either. That's not necessarily a bug, but a problem nevertheless. A new
>> user option could help.
> 
> A very relevant question is: can this wait till Emacs 30?

If you ask my opinion, then I already wrote it. The only thing that 
looks potentially complex is fixing the pinning of packages to archives. 
The rest should be fairly trivial.

Should we leave bugfixing and easy upgrading of builtin packages to 
Emacs 30? I suppose we could. Let me know if it's your foregone 
conclusion, then I'll just walk away now. This issue doesn't solve any 
problems for me personally, to be clear.

>> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
>> Joao's scenario because use-package-ensure-elpa short-circuits when it
>> find that the package is installed ('package-installed-p' returns t). So
>> (use-package eglot :ensure t) does not upgrade Eglot even when
>> package-install-upgrade-built-in is t.
> 
> I don't think (use-package eglot :ensure t) should automatically
> upgrade built-in packages.

I don't think it should, either.

But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.

> We could make it do that if
> package-install-upgrade-built-in is non-nil -- again, if such a change
> could be safe enough.  If not, then the same workaround as for
> package-upgrade would do here, I suppose?

What workaround would that be? use-package is not invoked interactively 
-- there is no prefix argument to pass.

>> - package-install-upgrade-built-in is not nuanced: if we suggest the
>> users to set it to t, that can result in making _all_ builtin packages
>> upgradable with 'package-install'.
> 
> It should be set to t only by users who indeed want all built-in
> packages to be updated.  That's why the default is nil.

Sounds unnecessarily dangerous.

>> Whereas I think we originally only wanted that for Eglot and maybe
>> for use-package.
> 
> "We" never did want that.  João did, for obvious reasons, but that was
> never my intent.  The issue is indeed more general: what should
> package-install and package.el in general do with built-in packages
> for which a newer version is on ELPA?

It could continue doing what it's done before: when a package is already 
installed, abort. For upgrading, we should recommend commands with 
"upgrade" in their names.

> We don't yet know the answer to
> that, and no hope of obtaining one before Emacs 29.1 is released
> (barring any calamities).

I'm not sure how we're going to get the answer to that question after 
29.1 is released, too. Similar to that thing with treesit modes and 
auto-mode-alist.

>> For this and other minor reasons I would suggest reverting
>> 580d8278c5f48.
> 
> Not going to happen, not unless someone comes up with a better
> solution that is much better and still safe enough.  Personally, I
> don't believe such a solution exists, since we don't really know the
> answer to the above question.

Could you specify which problem it's currently solving? Some particular 
scenario.

Then I could confidently propose an alternative (or not).

>> But I suppose we could also try to make it more granular (e.g. turn
>> the boolean option into a list). I'm not sure it's a good direction
>> overall, however.
> 
> I don't think we should go in this direction, precisely because we
> don't know it's a good one.

If we made it more granular, its use could become a reasonable option, 
then we'd be able to get some feedback from people who end up using it. 
I don't know what kind of people that will be, though.

>> - According to Jim P., package pinning doesn't work for builtin packages
>> either. Which could be a decent solution as well, e.g. putting something
>> like this in the docs: (use-package eglot :ensure t :pin gnu) if the
>> users want the Eglot version from ELPA -- and this form might even be
>> compatible with Emacs 28. I'm not sure how difficult it would be to fix
>> package pinning, however.
> 
> As long as this is optional behavior (and AFAIU it is), I won't
> object, but again, provided that the addition of this will not touch
> any other code in unsafe ways.

Yes, the use of pinning is and will remain optional. I'm not sure what 
code it will touch and how much, however. There is not a single function 
called "pin that" which needs to be fixed, the fix(es) will be somewhere 
on the common path(s) of package installation and upgrades, I think.

>> The reasons for me to be hopeful are:
>>
>> - The functions are propose to fix are "leaves": those that are supposed
>> to be used interactively, without (many) known callers in-tree. E.g.
>> fixing package-update-all shouldn't affect some other part of Emacs as
>> much as changing package-install could.
> 
> Experience until now in this thread indicates otherwise: almost every
> suggested change touched low-level code with many callers.  It isn't
> an accident that arriving at what was finally installed took so many
> iterations for such a simple change.

You might be correct at that, but note that I'm suggesting a slightly 
different set of goals than what had been discussed here before. So the 
changes will likewise need to be different.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 21 Apr 2023 03:50:52 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/04/2023 17:25, Eli Zaretskii wrote:
> 
> > What I miss in this description is something a bit higher-level: the
> > list of all the ways/commands/methods people use to install and
> > upgrade the packages.  We must have this list before our eyes to make
> > sure we review all the behaviors related to these activities, if we
> > want them all to eventually behave consistently.
> 
> Why list the commands people use to install packages if we're talking 
> about upgrading them?

I said "to install and upgrade", not just "install".

And I do think we should also consider commands that install packages
because they are part of this issue, and in particular, should be
consistent with the upgrade commands.  Also, package-install was the
original cause of this thread.

> Hopefully we'll decide that 'package-install' 
> won't upgrade packages because it hasn't done that in the past.

But it does upgrade non-built-in packages, doesn't it?  And at least
João (and I think others as well) expected it to upgrade Eglot even
though it is now built in.

> > E.g., you don't mention any commands invoked from the package menu.
> 
> The command which people use to upgrade packages from the package menu 
> is called package-menu-mark-upgrades (bound to 'U'), mentioned here a 
> few times. I also said that I hope it's *the* main way people use to 
> upgrade packages.
> 
> The others would be package-update and package-update-all (can we please 
> do the rename now? I know they've been in for a year, but just as this 
> very discussion has shown, most people weren't even aware of their 
> existence).

If the list you provided covers all of the relevant commands and
functions, fine.  Still, for completeness' sake, we should have the
higher-level description as well, as a framework against which to
examine the proposed solutions.  And I'm still not convinced that we
covered all the relevant package-menu commands.

> >> - package-upgrade (nee package-update) doesn't upgrade builtin packages
> >> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
> > 
> > I'm okay with adding the same prefix argument to package-upgrade,
> > which would then allow upgrading a built-in package.  IOW, a change
> > similar to what we did in package-install -- provided that the change
> > is safe enough to go into Emacs 29.
> 
> If we agree it's a bug, why don't we just fix it?

Precisely because, as with package-install, this is a bug for some and
a feature for others, depending on whether people do or don't want the
built-in packages to be upgraded by default.

> 'package-update' is an 
> interactive function which itself it called from only one place: 
> 'package-update-all', and since the plan is to improve both, we can make 
> sure they only do what we ask of them: package-update will upgrade 
> builtins when invoked directly, and package-update-all will upgrade them 
> only when the builtin has been upgraded before (making it not a builtin 
> anymore), or a new user option is set.

This is one possibility, and it might make sense to some users.  But I
don't think we can be sure it will make sense to an overwhelming
majority of the users.

> > But note that AFAIU there's a relatively easy workaround for this:
> > install the later version of the package manually, just once.
> 
> It's indeed a workaround that exists, and if we were satisfied with it, 
> we would have closed this bug as "wontfix". But it's not a very friendly 
> way to do that (the user must hunt for the appropriate version in the 
> packages menu), and it's very poorly scriptable (you won't be able to 
> provide an easy and readable snippet for the user to evaluate or put in 
> their init script).

For Emacs 29, we don't have the luxury of rejecting slightly awkward
workarounds.

> >> - package-menu-mark-upgrades and package-update-all don't upgrade them
> >> either. That's not necessarily a bug, but a problem nevertheless. A new
> >> user option could help.
> > 
> > A very relevant question is: can this wait till Emacs 30?
> 
> If you ask my opinion, then I already wrote it.

I'm asking everyone who reads this.

> Should we leave bugfixing and easy upgrading of builtin packages to 
> Emacs 30? I suppose we could. Let me know if it's your foregone 
> conclusion, then I'll just walk away now.

It isn't a forgone conclusion, because no one has yet shown the code
to fix the issues you mention.  If the code is safe enough and doesn't
risk breaking some reasonable workflows, it could be considered for
Emacs 29.

> >> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
> >> Joao's scenario because use-package-ensure-elpa short-circuits when it
> >> find that the package is installed ('package-installed-p' returns t). So
> >> (use-package eglot :ensure t) does not upgrade Eglot even when
> >> package-install-upgrade-built-in is t.
> > 
> > I don't think (use-package eglot :ensure t) should automatically
> > upgrade built-in packages.
> 
> I don't think it should, either.
> 
> But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.

No, it was supposed to allow the user to invoke package-install in a
way that upgrades built-in packages, something that doesn't happen by
default.  IOW, it was a partial solution to the original problem which
started this discussion, see

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5

> > We could make it do that if
> > package-install-upgrade-built-in is non-nil -- again, if such a change
> > could be safe enough.  If not, then the same workaround as for
> > package-upgrade would do here, I suppose?
> 
> What workaround would that be? use-package is not invoked interactively 
> -- there is no prefix argument to pass.

The workaround is to manually install the package from ELPA, once,
using "C-u M-x package-install RET".

> >> - package-install-upgrade-built-in is not nuanced: if we suggest the
> >> users to set it to t, that can result in making _all_ builtin packages
> >> upgradable with 'package-install'.
> > 
> > It should be set to t only by users who indeed want all built-in
> > packages to be updated.  That's why the default is nil.
> 
> Sounds unnecessarily dangerous.

Which is why it shouldn't be the default.

> >> Whereas I think we originally only wanted that for Eglot and maybe
> >> for use-package.
> > 
> > "We" never did want that.  João did, for obvious reasons, but that was
> > never my intent.  The issue is indeed more general: what should
> > package-install and package.el in general do with built-in packages
> > for which a newer version is on ELPA?
> 
> It could continue doing what it's done before: when a package is already 
> installed, abort. For upgrading, we should recommend commands with 
> "upgrade" in their names.

If people agree with that, I don't think I'll object.  But this is in
a sense a breaking change: package-install will only install, and
thereafter users will need to use package-upgrade.  Some might dislike
such behavior changes.  And we will need to make sure that all the
available methods of "installing" do not "upgrade", for consistency.

> > We don't yet know the answer to
> > that, and no hope of obtaining one before Emacs 29.1 is released
> > (barring any calamities).
> 
> I'm not sure how we're going to get the answer to that question after 
> 29.1 is released, too. Similar to that thing with treesit modes and 
> auto-mode-alist.

Time will tell.  If we have no significant new information and data
points by the time the release of Emacs 30 is on the table, we will
have to use our best judgment at that time.

> >> For this and other minor reasons I would suggest reverting
> >> 580d8278c5f48.
> > 
> > Not going to happen, not unless someone comes up with a better
> > solution that is much better and still safe enough.  Personally, I
> > don't believe such a solution exists, since we don't really know the
> > answer to the above question.
> 
> Could you specify which problem it's currently solving? Some particular 
> scenario.

The scenario which started this bug report, see the message whose URL
I mentioned above.  IOW, we now allow users to explicitly request that
package-install includes built-in packages in the list of candidates,
and will therefore allow to upgrade them.

> >> But I suppose we could also try to make it more granular (e.g. turn
> >> the boolean option into a list). I'm not sure it's a good direction
> >> overall, however.
> > 
> > I don't think we should go in this direction, precisely because we
> > don't know it's a good one.
> 
> If we made it more granular, its use could become a reasonable option, 
> then we'd be able to get some feedback from people who end up using it. 
> I don't know what kind of people that will be, though.

It could be a reasonable option, or it could be a nuisance because of
a gratuitous proliferation of too-granular options.  To make a good
decision, we need data and experience we don't have yet.  So let's not
rush into solutions about which we are uncertain.  We can always make
a boolean option have more values in the future, this kind of change
is usually easy and backward-compatible.

> >> - The functions are propose to fix are "leaves": those that are supposed
> >> to be used interactively, without (many) known callers in-tree. E.g.
> >> fixing package-update-all shouldn't affect some other part of Emacs as
> >> much as changing package-install could.
> > 
> > Experience until now in this thread indicates otherwise: almost every
> > suggested change touched low-level code with many callers.  It isn't
> > an accident that arriving at what was finally installed took so many
> > iterations for such a simple change.
> 
> You might be correct at that, but note that I'm suggesting a slightly 
> different set of goals than what had been discussed here before. So the 
> changes will likewise need to be different.

It's possible.  I said what I said not as an up-front decision, but as
a warning: we should not raise our expectations too high.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 21/04/2023 09:37, Eli Zaretskii wrote:

>> Why list the commands people use to install packages if we're talking
>> about upgrading them?
> 
> I said "to install and upgrade", not just "install".

I listed the upgrade commands.

> And I do think we should also consider commands that install packages
> because they are part of this issue, and in particular, should be
> consistent with the upgrade commands.  Also, package-install was the
> original cause of this thread.

Okay, we also have package-menu-mark-install and package-menu-execute.

>> Hopefully we'll decide that 'package-install'
>> won't upgrade packages because it hasn't done that in the past.
> 
> But it does upgrade non-built-in packages, doesn't it?

Apparently not. I didn't remember whether it does, and I deduced that it 
does just from reading this discussion previously, but it does not.

E.g. just now pressing 'U' after 'M-x list-packages' showed me that I 
have available upgrades for a lot of packages. But still if I evaluate

   (package-install 'sml-mode)

where sml-mode is one of said packages, I just get the message:

   ‘sml-mode’ is already installed

> And at least
> João (and I think others as well) expected it to upgrade Eglot even
> though it is now built in.

I think he wants that because this way (package-install 'eglot) and 
(use-package eglot :ensure t) could match the behavior of Emacs 28 with 
an empty init directory. Backward compatibility and all that.

But I think that's questionable, semantically. Given that Eglot is 
already "installed". Though, of course, one could argue that a bundled 
package is not exactly installed, but then we should change what 
'package-installed-p' does as well. And think hard before doing that.

>> The others would be package-update and package-update-all (can we please
>> do the rename now? I know they've been in for a year, but just as this
>> very discussion has shown, most people weren't even aware of their
>> existence).
> 
> If the list you provided covers all of the relevant commands and
> functions, fine.  Still, for completeness' sake, we should have the
> higher-level description as well, as a framework against which to
> examine the proposed solutions.  And I'm still not convinced that we
> covered all the relevant package-menu commands.

It might be easier if you just ask additional question during review.

>>>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
>>>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
>>>
>>> I'm okay with adding the same prefix argument to package-upgrade,
>>> which would then allow upgrading a built-in package.  IOW, a change
>>> similar to what we did in package-install -- provided that the change
>>> is safe enough to go into Emacs 29.
>>
>> If we agree it's a bug, why don't we just fix it?
> 
> Precisely because, as with package-install, this is a bug for some and
> a feature for others, depending on whether people do or don't want the
> built-in packages to be upgraded by default.

I'm having a hard time imagining someone evaluating (package-upgrade 
'eglot) without intention to upgrade it to the latest version. Or 
invoking it interactively with same argument, expecting a different result.

I don't think anyone will put it straight this way into their init 
script either.

>> 'package-update' is an
>> interactive function which itself it called from only one place:
>> 'package-update-all', and since the plan is to improve both, we can make
>> sure they only do what we ask of them: package-update will upgrade
>> builtins when invoked directly, and package-update-all will upgrade them
>> only when the builtin has been upgraded before (making it not a builtin
>> anymore), or a new user option is set.
> 
> This is one possibility, and it might make sense to some users.  But I
> don't think we can be sure it will make sense to an overwhelming
> majority of the users.

Hence the user option?

But okay, this particular addition, though trivial, we could probably 
postpone until Emacs 30, or even avoid adding at all. It is indeed not 
obvious that people will really need it, although

   (setq package-upgradable-builtins '(eglot use-package))
   (package-upgrade-all)
   ;; or M-x package-upgrade-all
   ;; or 'U' in the list-packages menu

seems like a plausible scenario for a certain kind of user. Because 
package-upgrade does not have a menu entry, or a button anywhere, 
whereas package-upgradable-builtins can be altered from the Customize UI.

>>>> - The current fix (commit 580d8278c5f48) is not comprehensive WRT to
>>>> Joao's scenario because use-package-ensure-elpa short-circuits when it
>>>> find that the package is installed ('package-installed-p' returns t). So
>>>> (use-package eglot :ensure t) does not upgrade Eglot even when
>>>> package-install-upgrade-built-in is t.
>>>
>>> I don't think (use-package eglot :ensure t) should automatically
>>> upgrade built-in packages.
>>
>> I don't think it should, either.
>>
>> But IIUC that's the scenario 580d8278c5f48 was supposed to make possible.
> 
> No, it was supposed to allow the user to invoke package-install in a
> way that upgrades built-in packages, something that doesn't happen by
> default.  IOW, it was a partial solution to the original problem which
> started this discussion, see
> 
>    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#5

Fair enough.

>>> We could make it do that if
>>> package-install-upgrade-built-in is non-nil -- again, if such a change
>>> could be safe enough.  If not, then the same workaround as for
>>> package-upgrade would do here, I suppose?
>>
>> What workaround would that be? use-package is not invoked interactively
>> -- there is no prefix argument to pass.
> 
> The workaround is to manually install the package from ELPA, once,
> using "C-u M-x package-install RET".

That's not the use-package workflow.

>>>> Whereas I think we originally only wanted that for Eglot and maybe
>>>> for use-package.
>>>
>>> "We" never did want that.  João did, for obvious reasons, but that was
>>> never my intent.  The issue is indeed more general: what should
>>> package-install and package.el in general do with built-in packages
>>> for which a newer version is on ELPA?
>>
>> It could continue doing what it's done before: when a package is already
>> installed, abort. For upgrading, we should recommend commands with
>> "upgrade" in their names.
> 
> If people agree with that, I don't think I'll object.  But this is in
> a sense a breaking change: package-install will only install, and
> thereafter users will need to use package-upgrade.  Some might dislike
> such behavior changes.  And we will need to make sure that all the
> available methods of "installing" do not "upgrade", for consistency.

Yeah, apparently it won't be a breaking change, or a change at all.

>>>> For this and other minor reasons I would suggest reverting
>>>> 580d8278c5f48.
>>>
>>> Not going to happen, not unless someone comes up with a better
>>> solution that is much better and still safe enough.  Personally, I
>>> don't believe such a solution exists, since we don't really know the
>>> answer to the above question.
>>
>> Could you specify which problem it's currently solving? Some particular
>> scenario.
> 
> The scenario which started this bug report, see the message whose URL
> I mentioned above.  IOW, we now allow users to explicitly request that
> package-install includes built-in packages in the list of candidates,
> and will therefore allow to upgrade them.

After we fix 'package-upgrade', users will be able to 'M-x 
package-upgrade RET eglot RET'.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 21 Apr 2023 13:19:39 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 21/04/2023 09:37, Eli Zaretskii wrote:
> 
> >> Why list the commands people use to install packages if we're talking
> >> about upgrading them?
> > 
> > I said "to install and upgrade", not just "install".
> 
> I listed the upgrade commands.

Yes, and I think we should consider both install and upgrade commands.

> >> Hopefully we'll decide that 'package-install'
> >> won't upgrade packages because it hasn't done that in the past.
> > 
> > But it does upgrade non-built-in packages, doesn't it?
> 
> Apparently not. I didn't remember whether it does, and I deduced that it 
> does just from reading this discussion previously, but it does not.
> 
> E.g. just now pressing 'U' after 'M-x list-packages' showed me that I 
> have available upgrades for a lot of packages. But still if I evaluate
> 
>    (package-install 'sml-mode)
> 
> where sml-mode is one of said packages, I just get the message:
> 
>    ‘sml-mode’ is already installed
> 
> > And at least
> > João (and I think others as well) expected it to upgrade Eglot even
> > though it is now built in.
> 
> I think he wants that because this way (package-install 'eglot) and 
> (use-package eglot :ensure t) could match the behavior of Emacs 28 with 
> an empty init directory. Backward compatibility and all that.

But if, with older Emacsen, package-install would refuse to update to
a newer version of Eglot if _some_ older version of Eglot is already
installed, then where's the problem with the default behavior of
package-install? it behaves exactly like in previous versions of
Emacs.  And why is this a problem for users of Eglot, if they couldn't
use package-install more than once for Eglot anyway?

Something is amiss here.

> But I think that's questionable, semantically. Given that Eglot is 
> already "installed". Though, of course, one could argue that a bundled 
> package is not exactly installed, but then we should change what 
> 'package-installed-p' does as well. And think hard before doing that.

I'd question why we have two commands instead of just one, but that's
probably water under the bridge at this point.

> >>>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
> >>>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
> >>>
> >>> I'm okay with adding the same prefix argument to package-upgrade,
> >>> which would then allow upgrading a built-in package.  IOW, a change
> >>> similar to what we did in package-install -- provided that the change
> >>> is safe enough to go into Emacs 29.
> >>
> >> If we agree it's a bug, why don't we just fix it?
> > 
> > Precisely because, as with package-install, this is a bug for some and
> > a feature for others, depending on whether people do or don't want the
> > built-in packages to be upgraded by default.
> 
> I'm having a hard time imagining someone evaluating (package-upgrade 
> 'eglot) without intention to upgrade it to the latest version. Or 
> invoking it interactively with same argument, expecting a different result.

In interactive invocation, package-upgrade calls completing-read with
its 4th argument non-nil, so you cannot select a package which is not
in the collection returned by package--updateable-packages.  What I
meant above is to allow that collection to include built-in packages
as optional behavior.  I just tried invoking package-update for ElDoc,
and I get "No match" after typing "eldoc" to its prompt, although
eldoc version 1.14.0 is in the list presented by list-packages as
"available".

> >> 'package-update' is an
> >> interactive function which itself it called from only one place:
> >> 'package-update-all', and since the plan is to improve both, we can make
> >> sure they only do what we ask of them: package-update will upgrade
> >> builtins when invoked directly, and package-update-all will upgrade them
> >> only when the builtin has been upgraded before (making it not a builtin
> >> anymore), or a new user option is set.
> > 
> > This is one possibility, and it might make sense to some users.  But I
> > don't think we can be sure it will make sense to an overwhelming
> > majority of the users.
> 
> Hence the user option?

Which one?  Are you suggesting to add a new one?  If so, why not use
the one we already added, package-install-upgrade-built-in?

> But okay, this particular addition, though trivial, we could probably 
> postpone until Emacs 30, or even avoid adding at all. It is indeed not 
> obvious that people will really need it, although

If by "this particular addition" you mean to allow package-update to
update built-in packages, then I thought adding that for consistency
with package-install was one of your main bothers?  Or what am I
missing?

>    (setq package-upgradable-builtins '(eglot use-package))
>    (package-upgrade-all)
>    ;; or M-x package-upgrade-all
>    ;; or 'U' in the list-packages menu
> 
> seems like a plausible scenario for a certain kind of user.

Why not treat the fact that some version was already installed from
ELPA as an indication that the user wants this?

> Because 
> package-upgrade does not have a menu entry, or a button anywhere, 
> whereas package-upgradable-builtins can be altered from the Customize UI.

Maybe marking a package in the list for update could be interpreted as
"upgrade that, no questions asked", and we will need no user options?

> >>> We could make it do that if
> >>> package-install-upgrade-built-in is non-nil -- again, if such a change
> >>> could be safe enough.  If not, then the same workaround as for
> >>> package-upgrade would do here, I suppose?
> >>
> >> What workaround would that be? use-package is not invoked interactively
> >> -- there is no prefix argument to pass.
> > 
> > The workaround is to manually install the package from ELPA, once,
> > using "C-u M-x package-install RET".
> 
> That's not the use-package workflow.

The use-package workflow should perhaps get a separate and different
solution.

> >>>> Whereas I think we originally only wanted that for Eglot and maybe
> >>>> for use-package.
> >>>
> >>> "We" never did want that.  João did, for obvious reasons, but that was
> >>> never my intent.  The issue is indeed more general: what should
> >>> package-install and package.el in general do with built-in packages
> >>> for which a newer version is on ELPA?
> >>
> >> It could continue doing what it's done before: when a package is already
> >> installed, abort. For upgrading, we should recommend commands with
> >> "upgrade" in their names.
> > 
> > If people agree with that, I don't think I'll object.  But this is in
> > a sense a breaking change: package-install will only install, and
> > thereafter users will need to use package-upgrade.  Some might dislike
> > such behavior changes.  And we will need to make sure that all the
> > available methods of "installing" do not "upgrade", for consistency.
> 
> Yeah, apparently it won't be a breaking change, or a change at all.

I'm not sure, see above.

Also, when you mark packages for update from the list presented by
list-packages, the menu entry says

  i    Mark for Install

and its help-echo says "Mark a package for installation and move to
the next line", so we already confuse "install" and "upgrade".

> >>>> For this and other minor reasons I would suggest reverting
> >>>> 580d8278c5f48.
> >>>
> >>> Not going to happen, not unless someone comes up with a better
> >>> solution that is much better and still safe enough.  Personally, I
> >>> don't believe such a solution exists, since we don't really know the
> >>> answer to the above question.
> >>
> >> Could you specify which problem it's currently solving? Some particular
> >> scenario.
> > 
> > The scenario which started this bug report, see the message whose URL
> > I mentioned above.  IOW, we now allow users to explicitly request that
> > package-install includes built-in packages in the list of candidates,
> > and will therefore allow to upgrade them.
> 
> After we fix 'package-upgrade', users will be able to 'M-x 
> package-upgrade RET eglot RET'.

This goes back to the issue of having two confusingly-similar but
different commands, as mentioned above.  I guess we should first make
up our minds what, if anything, we want to do about this.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 21/04/2023 14:05, Eli Zaretskii wrote:

>>> And at least
>>> João (and I think others as well) expected it to upgrade Eglot even
>>> though it is now built in.
>>
>> I think he wants that because this way (package-install 'eglot) and
>> (use-package eglot :ensure t) could match the behavior of Emacs 28 with
>> an empty init directory. Backward compatibility and all that.
> 
> But if, with older Emacsen, package-install would refuse to update to
> a newer version of Eglot if _some_ older version of Eglot is already
> installed, then where's the problem with the default behavior of
> package-install? it behaves exactly like in previous versions of
> Emacs.  And why is this a problem for users of Eglot, if they couldn't
> use package-install more than once for Eglot anyway?
> 
> Something is amiss here.

The above scenarios (package-install or (use-package eglot :ensure t)) 
when ~/.emacs.d/elpa is empty, result in the very latest Eglot being 
installed when using Emacs 28. And will result in something different 
with Emacs 29 (not the latest version). That's not nothing: the CI 
scripts will have to be updated, and the user instructions for reporting 
problems will have to be made more complicated as well. Some 
possibilities will simply be gone (the user won't be able to upgrade 
Eglot to the very latest by deleting it from ~/.emacs.d/elpa and calling 
package-install, for example). This *is* a non-backward compatible 
change from the perspective of Eglot's maintainer.

I, personally, don't really buy this kind of argument, but I figured you 
might. After all, it's rather in line with reasoning we've seen voiced 
around these parts many times ("X has worked this way for Y years, let's 
never change it from now on"). Classic https://xkcd.com/1172/.

>> But I think that's questionable, semantically. Given that Eglot is
>> already "installed". Though, of course, one could argue that a bundled
>> package is not exactly installed, but then we should change what
>> 'package-installed-p' does as well. And think hard before doing that.
> 
> I'd question why we have two commands instead of just one, but that's
> probably water under the bridge at this point.

Either way would be fine, IMO, as long as the behavior is logical and 
matches documentation. But having a separate command to upgrade now lets 
us fix it separately without worry of breaking something more globally.

>>>>>> - package-upgrade (nee package-update) doesn't upgrade builtin packages
>>>>>> that never been upgraded before. It's a bug. Hopefully not too hard to fix.
>>>>>
>>>>> I'm okay with adding the same prefix argument to package-upgrade,
>>>>> which would then allow upgrading a built-in package.  IOW, a change
>>>>> similar to what we did in package-install -- provided that the change
>>>>> is safe enough to go into Emacs 29.
>>>>
>>>> If we agree it's a bug, why don't we just fix it?
>>>
>>> Precisely because, as with package-install, this is a bug for some and
>>> a feature for others, depending on whether people do or don't want the
>>> built-in packages to be upgraded by default.
>>
>> I'm having a hard time imagining someone evaluating (package-upgrade
>> 'eglot) without intention to upgrade it to the latest version. Or
>> invoking it interactively with same argument, expecting a different result.
> 
> In interactive invocation, package-upgrade calls completing-read with
> its 4th argument non-nil, so you cannot select a package which is not
> in the collection returned by package--updateable-packages.  What I
> meant above is to allow that collection to include built-in packages
> as optional behavior.  I just tried invoking package-update for ElDoc,
> and I get "No match" after typing "eldoc" to its prompt, although
> eldoc version 1.14.0 is in the list presented by list-packages as
> "available".

That's what I imagined: adding a new optional argument to 
package--updateable-packages which would include builtins in the result.

And only pass it when called from package-upgrade.

Hopefully that's the kind of optional that you meant.

>>>> 'package-update' is an
>>>> interactive function which itself it called from only one place:
>>>> 'package-update-all', and since the plan is to improve both, we can make
>>>> sure they only do what we ask of them: package-update will upgrade
>>>> builtins when invoked directly, and package-update-all will upgrade them
>>>> only when the builtin has been upgraded before (making it not a builtin
>>>> anymore), or a new user option is set.
>>>
>>> This is one possibility, and it might make sense to some users.  But I
>>> don't think we can be sure it will make sense to an overwhelming
>>> majority of the users.
>>
>> Hence the user option?
> 
> Which one?  Are you suggesting to add a new one?  If so, why not use
> the one we already added, package-install-upgrade-built-in?

The user option I was thinking of would probably be called a little 
shorter: package-upgrade-built-in. And it would only affect the upgrader 
commands.

I'm not sure tying that optional behavior to the new optional behavior 
of package-install is a good idea (one upgrades, another installs). But 
I'm also not sure keeping the latter around is a good idea, as you know.

>> But okay, this particular addition, though trivial, we could probably
>> postpone until Emacs 30, or even avoid adding at all. It is indeed not
>> obvious that people will really need it, although
> 
> If by "this particular addition" you mean to allow package-update to
> update built-in packages, then I thought adding that for consistency
> with package-install was one of your main bothers?  Or what am I
> missing?

That particular addition would be the option package-upgrade-built-in 
proposed above. One that would affect package-upgrade-all and 
package-menu-mark-upgrades only.

So the plan minimum is to fix package-upgrade.

>>     (setq package-upgradable-builtins '(eglot use-package))
>>     (package-upgrade-all)
>>     ;; or M-x package-upgrade-all
>>     ;; or 'U' in the list-packages menu
>>
>> seems like a plausible scenario for a certain kind of user.
> 
> Why not treat the fact that some version was already installed from
> ELPA as an indication that the user wants this?

That's how it works now, indeed. And we _could_ leave it at that.

>> Because
>> package-upgrade does not have a menu entry, or a button anywhere,
>> whereas package-upgradable-builtins can be altered from the Customize UI.
> 
> Maybe marking a package in the list for update could be interpreted as
> "upgrade that, no questions asked", and we will need no user options?

There is no handy "upgrade that" binding in the packages menu. The only 
command that's available there related to upgrading is 
package-menu-mark-upgrades, which does that to all packages (except 
builtins).

To manually execute an upgrade of one package, one needs to both mark 
the new version for installation (after first scrolling down the list to 
find it), and mark the current version for deletion. This is what 
currently an upgrade consists of. If they just do the first step, they 
end up with two installed versions, one of them "obsolete". package.el 
will hopefully handle this fine during activation after restart 
(ignoring the older one), but it's not a great configuration to leave 
the user in.

All this is to say, the first step (upgrading Eglot to the version from 
ELPA) will be less user-friendly compared to the other UIs we have. But 
it's probably manageable, especially if documented well.

>>>>> We could make it do that if
>>>>> package-install-upgrade-built-in is non-nil -- again, if such a change
>>>>> could be safe enough.  If not, then the same workaround as for
>>>>> package-upgrade would do here, I suppose?
>>>>
>>>> What workaround would that be? use-package is not invoked interactively
>>>> -- there is no prefix argument to pass.
>>>
>>> The workaround is to manually install the package from ELPA, once,
>>> using "C-u M-x package-install RET".
>>
>> That's not the use-package workflow.
> 
> The use-package workflow should perhaps get a separate and different
> solution.

Possibly (I suppose people are welcome to submit related proposals -- 
the only reasonable one I heard is related to pinning, see previous msgs).

My point here, however, is that commit 580d8278c5f48 improved the 
situation to a lesser degree that some people here might have expected.

>>>> It could continue doing what it's done before: when a package is already
>>>> installed, abort. For upgrading, we should recommend commands with
>>>> "upgrade" in their names.
>>>
>>> If people agree with that, I don't think I'll object.  But this is in
>>> a sense a breaking change: package-install will only install, and
>>> thereafter users will need to use package-upgrade.  Some might dislike
>>> such behavior changes.  And we will need to make sure that all the
>>> available methods of "installing" do not "upgrade", for consistency.
>>
>> Yeah, apparently it won't be a breaking change, or a change at all.
> 
> I'm not sure, see above.
> 
> Also, when you mark packages for update from the list presented by
> list-packages, the menu entry says
> 
>    i    Mark for Install
> 
> and its help-echo says "Mark a package for installation and move to
> the next line", so we already confuse "install" and "upgrade".

But that's the thing: we don't have a separate action in the packages 
list to "upgrade" a single package. The bindings that you see in the 
menu are "Mark for [i]nstall" and "Mark for [d]eletion". The first marks 
a specific (usually newer) version of a package for installation without 
deleting an existing one. The second marks a version for deletion. 
Neither means "upgrade".

The fact that having multiple versions installed will (probably) result 
in the newest one being used after a restart is great, of course, but 
that's still not a proper upgrade.

But we can conclude that we do have two different installation actions:

  - package-install which will install the latest version of a package, 
but only if it's not installed;
  - package-menu-mark-install, which will mark a specific version for 
installation, disregarding whether some earlier version is already 
installed; the previous version will remain installed still.

>>>>>> For this and other minor reasons I would suggest reverting
>>>>>> 580d8278c5f48.
>>>>>
>>>>> Not going to happen, not unless someone comes up with a better
>>>>> solution that is much better and still safe enough.  Personally, I
>>>>> don't believe such a solution exists, since we don't really know the
>>>>> answer to the above question.
>>>>
>>>> Could you specify which problem it's currently solving? Some particular
>>>> scenario.
>>>
>>> The scenario which started this bug report, see the message whose URL
>>> I mentioned above.  IOW, we now allow users to explicitly request that
>>> package-install includes built-in packages in the list of candidates,
>>> and will therefore allow to upgrade them.
>>
>> After we fix 'package-upgrade', users will be able to 'M-x
>> package-upgrade RET eglot RET'.
> 
> This goes back to the issue of having two confusingly-similar but
> different commands, as mentioned above.  I guess we should first make
> up our minds what, if anything, we want to do about this.

Sure.

But even if we decide that we want to eliminate that split, doing *that* 
would really be a breaking change. I don't have a reasonable plan to 
present for doing that in Emacs 29, so far.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 02:12, Dmitry Gutov wrote:
>> In interactive invocation, package-upgrade calls completing-read with
>> its 4th argument non-nil, so you cannot select a package which is not
>> in the collection returned by package--updateable-packages.  What I
>> meant above is to allow that collection to include built-in packages
>> as optional behavior.  I just tried invoking package-update for ElDoc,
>> and I get "No match" after typing "eldoc" to its prompt, although
>> eldoc version 1.14.0 is in the list presented by list-packages as
>> "available".
> 
> That's what I imagined: adding a new optional argument to 
> package--updateable-packages which would include builtins in the result.
> 
> And only pass it when called from package-upgrade.
> 
> Hopefully that's the kind of optional that you meant.

Here's a patch which does that. The diff could be reduced (the 
package-update part) by binding the new option 
(package-install-upgrade-built-in), but I figured it's better to avoid 
interdependency while we're still deciding what to keep.

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ffa6272dd1f..1f0a47f6b6a 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2270,17 +2270,21 @@ package-update
    "Update package NAME if a newer version exists."
    (interactive
     (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: " (package--updateable-packages t) nil t)))
    (let* ((package (if (symbolp name)
                        name
                      (intern name)))
           (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
+    (if (and pkg-desc (package-vc-p pkg-desc))
          (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
-
-(defun package--updateable-packages ()
+      (when pkg-desc
+        (package-delete pkg-desc 'force))
+      (package-install-from-archive
+       (car (last (seq-sort-by #'package-desc-priority-version
+                               #'version-list-<
+                               (cdr (assq package 
package-archive-contents)))))))))
+
+(defun package--updateable-packages (&optional allow-builtins)
    ;; Initialize the package system to get the list of package
    ;; symbols for completion.
    (package--archives-initialize)
@@ -2291,11 +2295,21 @@ package--updateable-packages
        (or (let ((available
                   (assq (car elt) package-archive-contents)))
              (and available
-                 (version-list-<
-                  (package-desc-version (cadr elt))
-                  (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+                 (or (and
+                      allow-builtins
+                      (not (package-desc-version (cadr elt))))
+                     (version-list-<
+                      (package-desc-version (cadr elt))
+                      (package-desc-version (cadr available))))))
+          (package-vc-p (cadr elt))))
+    (if allow-builtins
+        (append package-alist
+                (mapcan
+                 (lambda (elt)
+                   (when (not (assq (car elt) package-alist))
+                     (list (list (car elt) (package--from-builtin elt)))))
+                 package--builtins))
+      package-alist))))

  ;;;###autoload
  (defun package-update-all (&optional query)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 03:57:03 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> > That's what I imagined: adding a new optional argument to 
> > package--updateable-packages which would include builtins in the result.
> > 
> > And only pass it when called from package-upgrade.
> > 
> > Hopefully that's the kind of optional that you meant.
> 
> Here's a patch which does that. The diff could be reduced (the 
> package-update part) by binding the new option 
> (package-install-upgrade-built-in), but I figured it's better to avoid 
> interdependency while we're still deciding what to keep.

Thanks, but this is not what was being discussed, AFAIU.  What I said
I'd agree to is to have package-update accept a prefix argument and
heed package-install-upgrade-built-in (perhaps renamed), and only then
update built-in packages.

I also don't think I like the significant changes in package-update,
nor understand why they are needed.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 11:33, Eli Zaretskii wrote:
>> Date: Sat, 22 Apr 2023 03:57:03 +0300
>> From: Dmitry Gutov<dmitry@gutov.dev>
>> Cc:jporterbugs@gmail.com,philipk@posteo.net,62720@debbugs.gnu.org,
>>   joaotavora@gmail.com,larsi@gnus.org,monnier@iro.umontreal.ca
>>
>>> That's what I imagined: adding a new optional argument to
>>> package--updateable-packages which would include builtins in the result.
>>>
>>> And only pass it when called from package-upgrade.
>>>
>>> Hopefully that's the kind of optional that you meant.
>> Here's a patch which does that. The diff could be reduced (the
>> package-update part) by binding the new option
>> (package-install-upgrade-built-in), but I figured it's better to avoid
>> interdependency while we're still deciding what to keep.
> Thanks, but this is not what was being discussed, AFAIU.  What I said
> I'd agree to is to have package-update accept a prefix argument and
> heed package-install-upgrade-built-in (perhaps renamed),

I think I explained in the previous email why reusing 
package-install-upgrade-built-in doesn't seem like a good idea.

> and only then
> update built-in packages.

I asked what plausible scenario you think might be broken by having 
package-update upgrade builtin package by default.

Do you want to answer that?

> I also don't think I like the significant changes in package-update,
> nor understand why they are needed.

Like I said: the changes are to avoid relying on package-install being 
able to install a package that's already installed. Which currently 
works only for builtins and when only a user option is set. It's a mess.

And to "avoid interdependency".

Just to be clear, we are talking about the 4 lines at the end, right?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 13:30:41 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > Thanks, but this is not what was being discussed, AFAIU.  What I said
> > I'd agree to is to have package-update accept a prefix argument and
> > heed package-install-upgrade-built-in (perhaps renamed),
> 
> I think I explained in the previous email why reusing 
> package-install-upgrade-built-in doesn't seem like a good idea.

And I thought I've explained why I didn't see a need for another
option.

> > and only then
> > update built-in packages.
> 
> I asked what plausible scenario you think might be broken by having 
> package-update upgrade builtin package by default.

That's obvious: this is how package-update behaved until now.

> > I also don't think I like the significant changes in package-update,
> > nor understand why they are needed.
> 
> Like I said: the changes are to avoid relying on package-install being 
> able to install a package that's already installed. Which currently 
> works only for builtins and when only a user option is set. It's a mess.
> 
> And to "avoid interdependency".

Why does this have to be in Emacs 29?  It's a cleanup, right?

> Just to be clear, we are talking about the 4 lines at the end, right?

Yes, and also the (somewhat mysterious) additions of tests for
pkg-desc.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 14:11, Eli Zaretskii wrote:
>> Date: Sat, 22 Apr 2023 13:30:41 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> Thanks, but this is not what was being discussed, AFAIU.  What I said
>>> I'd agree to is to have package-update accept a prefix argument and
>>> heed package-install-upgrade-built-in (perhaps renamed),
>>
>> I think I explained in the previous email why reusing
>> package-install-upgrade-built-in doesn't seem like a good idea.
> 
> And I thought I've explained why I didn't see a need for another
> option.

I also don't see the point of using an option here.

Also think forward to Emacs 30: I think the most reasonable choice would 
be to have package-update upgrade builtins by default, whereas 
package-update-all and package-menu-mark-for-upgrades probably still 
need to be preffed off (not sure, but we won't be able to make the 
choice until later, I think).

But if to make package-update behave properly we need to flip the 
default of the said option, it will flip the behavior of 
package-update-all and package-menu-mark-for-upgrades as well.

>>> and only then
>>> update built-in packages.
>>
>> I asked what plausible scenario you think might be broken by having
>> package-update upgrade builtin package by default.
> 
> That's obvious: this is how package-update behaved until now.

That's not an answer to the question.

>>> I also don't think I like the significant changes in package-update,
>>> nor understand why they are needed.
>>
>> Like I said: the changes are to avoid relying on package-install being
>> able to install a package that's already installed. Which currently
>> works only for builtins and when only a user option is set. It's a mess.
>>
>> And to "avoid interdependency".
> 
> Why does this have to be in Emacs 29?  It's a cleanup, right?

Not a cleanup, no. If I just keep the previous version of the code, I 
get "package xxx is already installed". Because when upgrading a builtin 
package, the "current" version is not deleted.

So we need to compute the exact version to install (then package-install 
does not say "it's already installed" because the installed version is 
different). The use of package-install-from-archive might have been a 
mistake, though, (in case dependencies need to be updated too) I'm 
looking into that now.

Alternatively, we could add an optional argument to package-install 
which would mean "install the latest version anyway".

>> Just to be clear, we are talking about the 4 lines at the end, right?
> 
> Yes, and also the (somewhat mysterious) additions of tests for
> pkg-desc.

pkg-desc is nil for builtin packages in this case (they are not in 
package-alist, so (assq package package-alist) returns nil).

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

On 22/04/2023 14:24, Dmitry Gutov wrote:
>>>> I also don't think I like the significant changes in package-update,
>>>> nor understand why they are needed.
>>>
>>> Like I said: the changes are to avoid relying on package-install being
>>> able to install a package that's already installed. Which currently
>>> works only for builtins and when only a user option is set. It's a mess.
>>>
>>> And to "avoid interdependency".
>>
>> Why does this have to be in Emacs 29?  It's a cleanup, right?
> 
> Not a cleanup, no. If I just keep the previous version of the code, I 
> get "package xxx is already installed". Because when upgrading a builtin 
> package, the "current" version is not deleted.
> 
> So we need to compute the exact version to install (then package-install 
> does not say "it's already installed" because the installed version is 
> different). The use of package-install-from-archive might have been a 
> mistake, though, (in case dependencies need to be updated too) I'm 
> looking into that now.

Here's an updated patch that's a little closer to what's been there before.

> Alternatively, we could add an optional argument to package-install 
> which would mean "install the latest version anyway".

[-- Attachment #2: package-update-fix.diff --]
[-- Type: text/x-patch, Size: 2476 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ffa6272dd1f..f9ad83d7650 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2270,17 +2270,22 @@ package-update
   "Update package NAME if a newer version exists."
   (interactive
    (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: " (package--updateable-packages t) nil t)))
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
          (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
+    (if (and pkg-desc (package-vc-p pkg-desc))
         (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
-
-(defun package--updateable-packages ()
+      (when pkg-desc
+        (package-delete pkg-desc 'force))
+      (package-install
+       (car (last (seq-sort-by #'package-desc-priority-version
+                               #'version-list-<
+                               (cdr (assq package package-archive-contents)))))
+       'dont-select))))
+
+(defun package--updateable-packages (&optional allow-builtins)
   ;; Initialize the package system to get the list of package
   ;; symbols for completion.
   (package--archives-initialize)
@@ -2291,11 +2296,21 @@ package--updateable-packages
       (or (let ((available
                  (assq (car elt) package-archive-contents)))
             (and available
-                 (version-list-<
-                  (package-desc-version (cadr elt))
-                  (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+                 (or (and
+                      allow-builtins
+                      (not (package-desc-version (cadr elt))))
+                     (version-list-<
+                      (package-desc-version (cadr elt))
+                      (package-desc-version (cadr available))))))
+          (package-vc-p (cadr elt))))
+    (if allow-builtins
+        (append package-alist
+                (mapcan
+                 (lambda (elt)
+                   (when (not (assq (car elt) package-alist))
+                     (list (list (car elt) (package--from-builtin elt)))))
+                 package--builtins))
+      package-alist))))
 
 ;;;###autoload
 (defun package-update-all (&optional query)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 14:29:46 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> 
> Here's an updated patch that's a little closer to what's been there before.

I suggest not to rush with posting patches until we agree about what
the modified code should do.  I would like to minimize the waste of
your time.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 14:24:47 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 22/04/2023 14:11, Eli Zaretskii wrote:
> >> Date: Sat, 22 Apr 2023 13:30:41 +0300
> >> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
> >>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >>
> >>> Thanks, but this is not what was being discussed, AFAIU.  What I said
> >>> I'd agree to is to have package-update accept a prefix argument and
> >>> heed package-install-upgrade-built-in (perhaps renamed),
> >>
> >> I think I explained in the previous email why reusing
> >> package-install-upgrade-built-in doesn't seem like a good idea.
> > 
> > And I thought I've explained why I didn't see a need for another
> > option.
> 
> I also don't see the point of using an option here.

We must not change past behavior unconditionally and by default, not
this close to the release.

> Also think forward to Emacs 30: I think the most reasonable choice would 
> be to have package-update upgrade builtins by default, whereas 
> package-update-all and package-menu-mark-for-upgrades probably still 
> need to be preffed off (not sure, but we won't be able to make the 
> choice until later, I think).

I don't see why package-update and package-update-all should behave
differently wrt core packages.  If the user expresses his/her will to
update core packages, then package-update-all should do this for all
of them.

> But if to make package-update behave properly we need to flip the 
> default of the said option, it will flip the behavior of 
> package-update-all and package-menu-mark-for-upgrades as well.

Which is how it should be, IMO.

> >>> and only then
> >>> update built-in packages.
> >>
> >> I asked what plausible scenario you think might be broken by having
> >> package-update upgrade builtin package by default.
> > 
> > That's obvious: this is how package-update behaved until now.
> 
> That's not an answer to the question.

It is for me (and I'm quite surprised that it is not for you).

> >>> I also don't think I like the significant changes in package-update,
> >>> nor understand why they are needed.
> >>
> >> Like I said: the changes are to avoid relying on package-install being
> >> able to install a package that's already installed. Which currently
> >> works only for builtins and when only a user option is set. It's a mess.
> >>
> >> And to "avoid interdependency".
> > 
> > Why does this have to be in Emacs 29?  It's a cleanup, right?
> 
> Not a cleanup, no. If I just keep the previous version of the code, I 
> get "package xxx is already installed". Because when upgrading a builtin 
> package, the "current" version is not deleted.
> 
> So we need to compute the exact version to install (then package-install 
> does not say "it's already installed" because the installed version is 
> different). The use of package-install-from-archive might have been a 
> mistake, though, (in case dependencies need to be updated too) I'm 
> looking into that now.

With prefix argument, or when package-update-built-in is non-nil, the
behavior of package-install is different.  So just use this, instead
of changing the code of package-update.

> Alternatively, we could add an optional argument to package-install 
> which would mean "install the latest version anyway".

There is already such an option, added as part of fixing this bug.

> >> Just to be clear, we are talking about the 4 lines at the end, right?
> > 
> > Yes, and also the (somewhat mysterious) additions of tests for
> > pkg-desc.
> 
> pkg-desc is nil for builtin packages in this case (they are not in 
> package-alist, so (assq package package-alist) returns nil).

This should either be commented, or a variable with a telltale name
added to reflect this subtlety.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 15:00, Eli Zaretskii wrote:
>> Also think forward to Emacs 30: I think the most reasonable choice would
>> be to have package-update upgrade builtins by default, whereas
>> package-update-all and package-menu-mark-for-upgrades probably still
>> need to be preffed off (not sure, but we won't be able to make the
>> choice until later, I think).
> I don't see why package-update and package-update-all should behave
> differently wrt core packages.  If the user expresses his/her will to
> update core packages, then package-update-all should do this for all
> of them.

The idea is that if the user invokes 'M-x package-update' interactively 
and inputs the name of the package, they express their intention to 
update said package this way. Whether it's core or not.

With package-update-all or package-menu-mark-for-update, the user does 
not have a chance to specify which of the core packages they want to 
upgrade, if any.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 15:14:01 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 22/04/2023 15:00, Eli Zaretskii wrote:
> >> Also think forward to Emacs 30: I think the most reasonable choice would
> >> be to have package-update upgrade builtins by default, whereas
> >> package-update-all and package-menu-mark-for-upgrades probably still
> >> need to be preffed off (not sure, but we won't be able to make the
> >> choice until later, I think).
> > I don't see why package-update and package-update-all should behave
> > differently wrt core packages.  If the user expresses his/her will to
> > update core packages, then package-update-all should do this for all
> > of them.
> 
> The idea is that if the user invokes 'M-x package-update' interactively 
> and inputs the name of the package, they express their intention to 
> update said package this way. Whether it's core or not.
> 
> With package-update-all or package-menu-mark-for-update, the user does 
> not have a chance to specify which of the core packages they want to 
> upgrade, if any.

That's why we should have both the prefix-arg and the user option: one
is for one-off update of a single package, the other for updating many
of them.  It's the same logic as for package-install in those cases.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2245 bytes --]

On 22/04/2023 15:00, Eli Zaretskii wrote:

>>>> I think I explained in the previous email why reusing
>>>> package-install-upgrade-built-in doesn't seem like a good idea.
>>>
>>> And I thought I've explained why I didn't see a need for another
>>> option.
>>
>> I also don't see the point of using an option here.
> 
> We must not change past behavior unconditionally and by default, not
> this close to the release.

We're still allowed bugfixing, I believe.

>>>>> and only then
>>>>> update built-in packages.
>>>>
>>>> I asked what plausible scenario you think might be broken by having
>>>> package-update upgrade builtin package by default.
>>>
>>> That's obvious: this is how package-update behaved until now.
>>
>> That's not an answer to the question.
> 
> It is for me (and I'm quite surprised that it is not for you).

Not sure why you're surprised, you know my approach toward backward 
compatibility. Never a fan of enshrining problems in amber.

But in this case it's also a function that's never been in a released 
Emacs, so the formal conditions are lacking as well.

And it's okay to use the time since the code was added as a rough proxy 
for stability, but when it's pretty clear (just from the comments in 
this very thread) that most people never noticed or forgot that it's 
there. So it's obviously not very well tested.

Just from reading it code and testing, I see another bug: it removes the 
updated package from package-selected-packages because it doesn't pass 
NOSAVE to package-delete. Meaning, 'M-x package-autoremove' at any time 
later will delete it.

>> Alternatively, we could add an optional argument to package-install
>> which would mean "install the latest version anyway".
> 
> There is already such an option, added as part of fixing this bug.

Ok, since you insist. See attached.

>>>> Just to be clear, we are talking about the 4 lines at the end, right?
>>>
>>> Yes, and also the (somewhat mysterious) additions of tests for
>>> pkg-desc.
>>
>> pkg-desc is nil for builtin packages in this case (they are not in
>> package-alist, so (assq package package-alist) returns nil).
> 
> This should either be commented, or a variable with a telltale name
> added to reflect this subtlety.

Not a problem.

[-- Attachment #2: package-update-fix.diff --]
[-- Type: text/x-patch, Size: 2661 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ffa6272dd1f..2c8eec998c1 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2270,17 +2270,23 @@ package-update
   "Update package NAME if a newer version exists."
   (interactive
    (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: " (package--updateable-packages t) nil t)))
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
+         (pkg-desc (cadr (assq package package-alist)))
+         (package-install-upgrade-built-in (not pkg-desc)))
+    ;; `pkg-desc' will be nil when the package is an "active built-in".
+    (if (and pkg-desc (package-vc-p pkg-desc))
         (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
-
-(defun package--updateable-packages ()
+      (when pkg-desc
+        (package-delete pkg-desc 'force 'nosave))
+      (package-install package
+                       ;; An active built-in has never been "selected"
+                       ;; before.  Mark it as installed explicitly.
+                       (and pkg-desc 'dont-select)))))
+
+(defun package--updateable-packages (&optional allow-builtins)
   ;; Initialize the package system to get the list of package
   ;; symbols for completion.
   (package--archives-initialize)
@@ -2291,11 +2297,21 @@ package--updateable-packages
       (or (let ((available
                  (assq (car elt) package-archive-contents)))
             (and available
-                 (version-list-<
-                  (package-desc-version (cadr elt))
-                  (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+                 (or (and
+                      allow-builtins
+                      (not (package-desc-version (cadr elt))))
+                     (version-list-<
+                      (package-desc-version (cadr elt))
+                      (package-desc-version (cadr available))))))
+          (package-vc-p (cadr elt))))
+    (if allow-builtins
+        (append package-alist
+                (mapcan
+                 (lambda (elt)
+                   (when (not (assq (car elt) package-alist))
+                     (list (list (car elt) (package--from-builtin elt)))))
+                 package--builtins))
+      package-alist))))
 
 ;;;###autoload
 (defun package-update-all (&optional query)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sun, 23 Apr 2023 02:46:05 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >>> And I thought I've explained why I didn't see a need for another
> >>> option.
> >>
> >> I also don't see the point of using an option here.
> > 
> > We must not change past behavior unconditionally and by default, not
> > this close to the release.
> 
> We're still allowed bugfixing, I believe.

Not when fixing it might introduce another bug or break someone's
workflow.

> >>>>> and only then
> >>>>> update built-in packages.
> >>>>
> >>>> I asked what plausible scenario you think might be broken by having
> >>>> package-update upgrade builtin package by default.
> >>>
> >>> That's obvious: this is how package-update behaved until now.
> >>
> >> That's not an answer to the question.
> > 
> > It is for me (and I'm quite surprised that it is not for you).
> 
> Not sure why you're surprised, you know my approach toward backward 
> compatibility. Never a fan of enshrining problems in amber.
> 
> But in this case it's also a function that's never been in a released 
> Emacs, so the formal conditions are lacking as well.
> 
> And it's okay to use the time since the code was added as a rough proxy 
> for stability, but when it's pretty clear (just from the comments in 
> this very thread) that most people never noticed or forgot that it's 
> there. So it's obviously not very well tested.

What is obvious to you is not obvious to me.

> Just from reading it code and testing, I see another bug: it removes the 
> updated package from package-selected-packages because it doesn't pass 
> NOSAVE to package-delete. Meaning, 'M-x package-autoremove' at any time 
> later will delete it.

If there are bugs, we need to fix them.

Philip, any comments regarding this particular issue with
package-autoremove after package-update?

> >> Alternatively, we could add an optional argument to package-install
> >> which would mean "install the latest version anyway".
> > 
> > There is already such an option, added as part of fixing this bug.
> 
> Ok, since you insist. See attached.

I don't understand how this patch is supposed to be any progress in
this discussion.  I see no prefix argument handling in package-update
and no change in behavior when package-install-upgrade-built-in is
non-nil.  So why did you think this brings the code closer to what I
suggested?

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 09:39, Eli Zaretskii wrote:
> I don't understand how this patch is supposed to be any progress in
> this discussion.  I see no prefix argument handling in package-update
> and no change in behavior when package-install-upgrade-built-in is
> non-nil.  So why did you think this brings the code closer to what I
> suggested?

It addressed two last points from your previous email, obviously.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sun, 23 Apr 2023 14:58:33 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 23/04/2023 09:39, Eli Zaretskii wrote:
> > I don't understand how this patch is supposed to be any progress in
> > this discussion.  I see no prefix argument handling in package-update
> > and no change in behavior when package-install-upgrade-built-in is
> > non-nil.  So why did you think this brings the code closer to what I
> > suggested?
> 
> It addressed two last points from your previous email, obviously.

Which points are those?  Please help me identify them.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 16:02, Eli Zaretskii wrote:
>> Date: Sun, 23 Apr 2023 14:58:33 +0300
>> Cc:jporterbugs@gmail.com,philipk@posteo.net,62720@debbugs.gnu.org,
>>   joaotavora@gmail.com,larsi@gnus.org,monnier@iro.umontreal.ca
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 23/04/2023 09:39, Eli Zaretskii wrote:
>>> I don't understand how this patch is supposed to be any progress in
>>> this discussion.  I see no prefix argument handling in package-update
>>> and no change in behavior when package-install-upgrade-built-in is
>>> non-nil.  So why did you think this brings the code closer to what I
>>> suggested?
>> It addressed two last points from your previous email, obviously.
> Which points are those?  Please help me identify them.

These two:

 > There is already such an option, added as part of fixing this bug.

The binding was added, so now we straight away delegate to package-install.

 > This should either be commented, or a variable with a telltale name
added to reflect this subtlety.

A comment was added as well.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sun, 23 Apr 2023 16:11:44 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> It addressed two last points from your previous email, obviously.
> > Which points are those?  Please help me identify them.
> 
> These two:
> 
>  > There is already such an option, added as part of fixing this bug.
> 
> The binding was added, so now we straight away delegate to package-install.

I meant to give the user the control on whether package-update will
update built-in packages, like we did with package-install: either via
prefix argument or by customizing the user option.

>  > This should either be commented, or a variable with a telltale name
> added to reflect this subtlety.
> 
> A comment was added as well.

This one I did see, thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 17:24, Eli Zaretskii wrote:
>> Date: Sun, 23 Apr 2023 16:11:44 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> It addressed two last points from your previous email, obviously.
>>> Which points are those?  Please help me identify them.
>>
>> These two:
>>
>>   > There is already such an option, added as part of fixing this bug.
>>
>> The binding was added, so now we straight away delegate to package-install.
> 
> I meant to give the user the control on whether package-update will
> update built-in packages, like we did with package-install: either via
> prefix argument or by customizing the user option.

That would be a different change, though.

So what are we guarding against here? That the user will choose a 
built-in package to upgrade by accident?

And we'll show her an error, saying "use the prefix argument"?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Mon, 24 Apr 2023 00:53:30 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 23/04/2023 17:24, Eli Zaretskii wrote:
> >> Date: Sun, 23 Apr 2023 16:11:44 +0300
> >> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
> >>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >>
> >>>> It addressed two last points from your previous email, obviously.
> >>> Which points are those?  Please help me identify them.
> >>
> >> These two:
> >>
> >>   > There is already such an option, added as part of fixing this bug.
> >>
> >> The binding was added, so now we straight away delegate to package-install.
> > 
> > I meant to give the user the control on whether package-update will
> > update built-in packages, like we did with package-install: either via
> > prefix argument or by customizing the user option.
> 
> That would be a different change, though.
> 
> So what are we guarding against here? That the user will choose a 
> built-in package to upgrade by accident?

Yes.  Also, against invocations of this command from other commands
and from the menu.

> And we'll show her an error, saying "use the prefix argument"?

No, I think we'll do whatever the code does today when passed a
built-in package.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 2076 bytes --]

On 24/04/2023 14:58, Eli Zaretskii wrote:
>> Date: Mon, 24 Apr 2023 00:53:30 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 23/04/2023 17:24, Eli Zaretskii wrote:
>>>> Date: Sun, 23 Apr 2023 16:11:44 +0300
>>>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>>>    joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>>>> From: Dmitry Gutov <dmitry@gutov.dev>
>>>>
>>>>>> It addressed two last points from your previous email, obviously.
>>>>> Which points are those?  Please help me identify them.
>>>>
>>>> These two:
>>>>
>>>>    > There is already such an option, added as part of fixing this bug.
>>>>
>>>> The binding was added, so now we straight away delegate to package-install.
>>>
>>> I meant to give the user the control on whether package-update will
>>> update built-in packages, like we did with package-install: either via
>>> prefix argument or by customizing the user option.
>>
>> That would be a different change, though.
>>
>> So what are we guarding against here? That the user will choose a
>> built-in package to upgrade by accident?
> 
> Yes.  Also, against invocations of this command from other commands
> and from the menu.

It's not in the menu. There are also no known callers aside from 
package-update-all.

>> And we'll show her an error, saying "use the prefix argument"?
> 
> No, I think we'll do whatever the code does today when passed a
> built-in package.

Very well, here's the next version. It adds a new optional argument to 
the function (so that people can evaluate e.g. (package-update 'eglot 
t)). When called interactively, it is determined by current-prefix-argument.

Also please review the docstring change.

Regarding obeying package-install-upgrade-built-in, I think it would 
need to be renamed, and both package-update-all and 
package-menu-mark-upgrades would need to be made obey it too. All that 
could be done in a subsequent change.

[-- Attachment #2: package-update-fix.diff --]
[-- Type: text/x-patch, Size: 3312 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index ffa6272dd1f..b72bf86c773 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2266,21 +2266,39 @@ package-install
 (declare-function package-vc-update "package-vc" (pkg))
 
 ;;;###autoload
-(defun package-update (name)
-  "Update package NAME if a newer version exists."
+(defun package-update (name &optional update-built-ins)
+  "Update package NAME if a newer version exists.
+
+Only packages installed from ELPA are allowed to be updated this
+way.
+
+But if UPDATE-BUILT-INS is non-nil, or if the command is invoked
+interactively with a prefix argument, it will allow upgrading of
+active built-in packages."
   (interactive
    (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: " (package--updateable-packages
+                              current-prefix-arg)
+          nil t)
+         current-prefix-arg))
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
+         (pkg-desc (cadr (assq package package-alist)))
+         (package-install-upgrade-built-in (and
+                                            update-built-ins
+                                            (not pkg-desc))))
+    ;; `pkg-desc' will be nil when the package is an "active built-in".
+    (if (and pkg-desc (package-vc-p pkg-desc))
         (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
-
-(defun package--updateable-packages ()
+      (when pkg-desc
+        (package-delete pkg-desc 'force 'nosave))
+      (package-install package
+                       ;; An active built-in has never been "selected"
+                       ;; before.  Mark it as installed explicitly.
+                       (and pkg-desc 'dont-select)))))
+
+(defun package--updateable-packages (&optional allow-builtins)
   ;; Initialize the package system to get the list of package
   ;; symbols for completion.
   (package--archives-initialize)
@@ -2291,11 +2309,21 @@ package--updateable-packages
       (or (let ((available
                  (assq (car elt) package-archive-contents)))
             (and available
-                 (version-list-<
-                  (package-desc-version (cadr elt))
-                  (package-desc-version (cadr available)))))
-          (package-vc-p (cadr (assq (car elt) package-alist)))))
-    package-alist)))
+                 (or (and
+                      allow-builtins
+                      (not (package-desc-version (cadr elt))))
+                     (version-list-<
+                      (package-desc-version (cadr elt))
+                      (package-desc-version (cadr available))))))
+          (package-vc-p (cadr elt))))
+    (if allow-builtins
+        (append package-alist
+                (mapcan
+                 (lambda (elt)
+                   (when (not (assq (car elt) package-alist))
+                     (list (list (car elt) (package--from-builtin elt)))))
+                 package--builtins))
+      package-alist))))
 
 ;;;###autoload
 (defun package-update-all (&optional query)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Tue, 25 Apr 2023 02:45:46 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> So what are we guarding against here? That the user will choose a
> >> built-in package to upgrade by accident?
> > 
> > Yes.  Also, against invocations of this command from other commands
> > and from the menu.
> 
> It's not in the menu.

There's this:

  <menu-bar> <package> <Mark All Available Upgrades> runs the command
  package-menu-mark-upgrades (found in package-menu-mode-map), which is
  an interactive byte-compiled Lisp function in ‘package.el’.

  It is bound to U.

  (package-menu-mark-upgrades)

  Mark all upgradable packages in the Package Menu.
  For each installed package with a newer version available, place
  an (I)nstall flag on the available version and a (D)elete flag on
  the installed version.  A subsequent x
  call will upgrade the package.

I also envision that we will at some point have an "upgrade" menu
item, because it make little sense to have this command, but not to be
able to invoke it from the menu.

> There are also no known callers aside from package-update-all.

One caller is enough, IMO.

> Very well, here's the next version. It adds a new optional argument to 
> the function (so that people can evaluate e.g. (package-update 'eglot 
> t)). When called interactively, it is determined by current-prefix-argument.

Thanks, this is very close to what I had in mind.  The only thing that
is missing is the support for user option, which should then avoid the
need to invoke the command with a prefix argument.

> Also please review the docstring change.

It looks OK to me.

> Regarding obeying package-install-upgrade-built-in, I think it would 
> need to be renamed, and both package-update-all and 
> package-menu-mark-upgrades would need to be made obey it too. All that 
> could be done in a subsequent change.

If the option will affect more than just package-install, it should
indeed be renamed.

> -(defun package-update (name)
> -  "Update package NAME if a newer version exists."
> +(defun package-update (name &optional update-built-ins)
> +  "Update package NAME if a newer version exists.
> +
> +Only packages installed from ELPA are allowed to be updated this
> +way.

I'm not sure I understand where this restriction comes from.  Did the
original code enforce it?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 25/04/2023 10:47, Eli Zaretskii wrote:
>> Date: Tue, 25 Apr 2023 02:45:46 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> So what are we guarding against here? That the user will choose a
>>>> built-in package to upgrade by accident?
>>>
>>> Yes.  Also, against invocations of this command from other commands
>>> and from the menu.
>>
>> It's not in the menu.
> 
> There's this:
> 
>    <menu-bar> <package> <Mark All Available Upgrades> runs the command
>    package-menu-mark-upgrades (found in package-menu-mode-map), which is
>    an interactive byte-compiled Lisp function in ‘package.el’.
> 
>    It is bound to U.
> 
>    (package-menu-mark-upgrades)
> 
>    Mark all upgradable packages in the Package Menu.
>    For each installed package with a newer version available, place
>    an (I)nstall flag on the available version and a (D)elete flag on
>    the installed version.  A subsequent x
>    call will upgrade the package.

package-menu-mark-upgrades does not delegate to package-update, for 
better or worse.

As discussed previously, if we're going to change its behavior (of 
package-menu-mark-upgrades), we *will* hide that behind the pref.

> I also envision that we will at some point have an "upgrade" menu
> item, because it make little sense to have this command, but not to be
> able to invoke it from the menu.

We might. But since that will only happen in the future, by that time 
there won't be any users who are accustomed to using that menu item and 
thus possibly inconvenienced by the change in behavior.

>> Very well, here's the next version. It adds a new optional argument to
>> the function (so that people can evaluate e.g. (package-update 'eglot
>> t)). When called interactively, it is determined by current-prefix-argument.
> 
> Thanks, this is very close to what I had in mind.  The only thing that
> is missing is the support for user option, which should then avoid the
> need to invoke the command with a prefix argument.
> 
>> Also please review the docstring change.
> 
> It looks OK to me.
> 
>> -(defun package-update (name)
>> -  "Update package NAME if a newer version exists."
>> +(defun package-update (name &optional update-built-ins)
>> +  "Update package NAME if a newer version exists.
>> +
>> +Only packages installed from ELPA are allowed to be updated this
>> +way.
> 
> I'm not sure I understand where this restriction comes from.  Did the
> original code enforce it?

I'm not sure what you mean about "enforce it". That's the essence of the 
bug here: this function's inability to upgrade built-in packages 
(packages installed not from ELPA). Since you are asking to keep that 
behavior by default, it now needs to be documented.

 >> Regarding obeying package-install-upgrade-built-in, I think it would
 >> need to be renamed, and both package-update-all and
 >> package-menu-mark-upgrades would need to be made obey it too. All that
 >> could be done in a subsequent change.
 >
 > If the option will affect more than just package-install, it should
 > indeed be renamed.

That will require some more work. On package-menu-mark-upgrades in 
particular.

TBH, I'm getting more doubts about this change now.

What will we do in Emacs 30? If we add the new argument, it will be hard 
to back out of it, to default to the proper behavior.

Perhaps we should just wait and then fix it on master properly. 
Workarounds exist, after all.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

> Perhaps we should just wait and then fix it on master properly.
> Workarounds exist, after all.

+1

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Tue, 25 Apr 2023 15:08:15 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> -(defun package-update (name)
> >> -  "Update package NAME if a newer version exists."
> >> +(defun package-update (name &optional update-built-ins)
> >> +  "Update package NAME if a newer version exists.
> >> +
> >> +Only packages installed from ELPA are allowed to be updated this
> >> +way.
> > 
> > I'm not sure I understand where this restriction comes from.  Did the
> > original code enforce it?
> 
> I'm not sure what you mean about "enforce it". That's the essence of the 
> bug here: this function's inability to upgrade built-in packages 
> (packages installed not from ELPA). Since you are asking to keep that 
> behavior by default, it now needs to be documented.

Oh, then I misunderstood what that says.  I thought is says ELPA as
opposed to, say, MELPA.

So I think we need to rephrase that.  Something like

  Packages which are part of the Emacs distribution cannot be updated
  that way.

>  >> Regarding obeying package-install-upgrade-built-in, I think it would
>  >> need to be renamed, and both package-update-all and
>  >> package-menu-mark-upgrades would need to be made obey it too. All that
>  >> could be done in a subsequent change.
>  >
>  > If the option will affect more than just package-install, it should
>  > indeed be renamed.
> 
> That will require some more work. On package-menu-mark-upgrades in 
> particular.
> 
> TBH, I'm getting more doubts about this change now.
> 
> What will we do in Emacs 30? If we add the new argument, it will be hard 
> to back out of it, to default to the proper behavior.

I thought that in Emacs 30 we could make the user option be non-nil by
default, assuming we will decide not to treat built-in packages
specially in this regard.  Then the additional argument will become
much less important, perhaps for some rare situations or something.

> Perhaps we should just wait and then fix it on master properly. 
> Workarounds exist, after all.

I won't object, but I thought you and others wanted to have
package-install and package-update to behave consistently in this
respect.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 25/04/2023 15:43, Eli Zaretskii wrote:
>> Date: Tue, 25 Apr 2023 15:08:15 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   joaotavora@gmail.com, larsi@gnus.org, monnier@iro.umontreal.ca
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> -(defun package-update (name)
>>>> -  "Update package NAME if a newer version exists."
>>>> +(defun package-update (name &optional update-built-ins)
>>>> +  "Update package NAME if a newer version exists.
>>>> +
>>>> +Only packages installed from ELPA are allowed to be updated this
>>>> +way.
>>>
>>> I'm not sure I understand where this restriction comes from.  Did the
>>> original code enforce it?
>>
>> I'm not sure what you mean about "enforce it". That's the essence of the
>> bug here: this function's inability to upgrade built-in packages
>> (packages installed not from ELPA). Since you are asking to keep that
>> behavior by default, it now needs to be documented.
> 
> Oh, then I misunderstood what that says.  I thought is says ELPA as
> opposed to, say, MELPA.

No, just ELPA in general.

> So I think we need to rephrase that.  Something like
> 
>    Packages which are part of the Emacs distribution cannot be updated
>    that way.

This is probably better. As long as we understand it to read "packages 
which are part ... and were never upgraded to a version from ELPA".

>>   >> Regarding obeying package-install-upgrade-built-in, I think it would
>>   >> need to be renamed, and both package-update-all and
>>   >> package-menu-mark-upgrades would need to be made obey it too. All that
>>   >> could be done in a subsequent change.
>>   >
>>   > If the option will affect more than just package-install, it should
>>   > indeed be renamed.
>>
>> That will require some more work. On package-menu-mark-upgrades in
>> particular.
>>
>> TBH, I'm getting more doubts about this change now.
>>
>> What will we do in Emacs 30? If we add the new argument, it will be hard
>> to back out of it, to default to the proper behavior.
> 
> I thought that in Emacs 30 we could make the user option be non-nil by
> default, assuming we will decide not to treat built-in packages
> specially in this regard.  Then the additional argument will become
> much less important, perhaps for some rare situations or something.

It would remain an odd vestige, and it might be difficult to repurpose 
for something more useful (e.g. being able to pick a specific version to 
upgrade/downgrade to?)

>> Perhaps we should just wait and then fix it on master properly.
>> Workarounds exist, after all.
> 
> I won't object, but I thought you and others wanted to have
> package-install and package-update to behave consistently in this
> respect.

Having package-install and package-update behave the same was never the 
goal, at least not for me.

Quite the opposite: package-install doesn't install anything when the 
package is already installed (and the argument is a package name symbol, 
which is the case for interactive invocations).

package-update/upgrade, OTOH, is supposed to upgrade already installed 
packages. Which I'm assuming is the category we are going to assign the 
built-ins to, after all.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 25/04/2023 21:35, Dmitry Gutov wrote:
>> So I think we need to rephrase that.  Something like
>>
>>    Packages which are part of the Emacs distribution cannot be updated
>>    that way.
> 
> This is probably better. As long as we understand it to read "packages 
> which are part ... and were never upgraded to a version from ELPA".

Anyway, if we agree to keep this unfixed, we should probably add this 
text somewhere: either into the docstring anyway, or to etc/PROBLEMS.

Not sure how to avoid the impression that this is the intended behavior 
and not something we're looking to change in the future, though.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Thu, 27 Apr 2023 02:05:21 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
> 
> On 25/04/2023 21:35, Dmitry Gutov wrote:
> >> So I think we need to rephrase that.  Something like
> >>
> >>    Packages which are part of the Emacs distribution cannot be updated
> >>    that way.
> > 
> > This is probably better. As long as we understand it to read "packages 
> > which are part ... and were never upgraded to a version from ELPA".
> 
> Anyway, if we agree to keep this unfixed, we should probably add this 
> text somewhere: either into the docstring anyway, or to etc/PROBLEMS.

Do we agree to leave package-update unchanged?

And what about the renaming you suggested in another bug report?

I'd like to make a new pretest soon, with all these issues finalized
as best as we can.

As for PROBLEMS, I'm not sure.  We usually put there only something
that people actually complain about, not proactively.  How about
mentioning this in the doc string instead?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 27/04/2023 08:41, Eli Zaretskii wrote:
>> Date: Thu, 27 Apr 2023 02:05:21 +0300
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
>>
>> On 25/04/2023 21:35, Dmitry Gutov wrote:
>>>> So I think we need to rephrase that.  Something like
>>>>
>>>>     Packages which are part of the Emacs distribution cannot be updated
>>>>     that way.
>>>
>>> This is probably better. As long as we understand it to read "packages
>>> which are part ... and were never upgraded to a version from ELPA".
>>
>> Anyway, if we agree to keep this unfixed, we should probably add this
>> text somewhere: either into the docstring anyway, or to etc/PROBLEMS.
> 
> Do we agree to leave package-update unchanged?

Apparently so.

> And what about the renaming you suggested in another bug report?

Still waiting for your go-ahead. Although it sounds like I have it.

> I'd like to make a new pretest soon, with all these issues finalized
> as best as we can.

Very good.

> As for PROBLEMS, I'm not sure.  We usually put there only something
> that people actually complain about, not proactively.  How about
> mentioning this in the doc string instead?

Sure. Hopefully it won't create an impression that the described 
behavior is _intended_.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Thu, 27 Apr 2023 12:00:51 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > And what about the renaming you suggested in another bug report?
> 
> Still waiting for your go-ahead. Although it sounds like I have it.

No one objected, AFAIU.  So I think you can install that.

> > As for PROBLEMS, I'm not sure.  We usually put there only something
> > that people actually complain about, not proactively.  How about
> > mentioning this in the doc string instead?
> 
> Sure. Hopefully it won't create an impression that the described 
> behavior is _intended_.

We can say something like "Currently, ..." to make that clear.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 27/04/2023 13:44, Eli Zaretskii wrote:
>> Date: Thu, 27 Apr 2023 12:00:51 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> And what about the renaming you suggested in another bug report?
>>
>> Still waiting for your go-ahead. Although it sounds like I have it.
> 
> No one objected, AFAIU.  So I think you can install that.

Thank you, done.

>>> As for PROBLEMS, I'm not sure.  We usually put there only something
>>> that people actually complain about, not proactively.  How about
>>> mentioning this in the doc string instead?
>>
>> Sure. Hopefully it won't create an impression that the described
>> behavior is _intended_.
> 
> We can say something like "Currently, ..." to make that clear.

Very good, I've used that and an additional clarification for a 
recommended alternative. Please see how you like it, or whether it needs 
improving.

Before the next pretest is cut, I'd also like to bring up the question 
again of backporting the patch for bug#62816 (with Joao's support: 
previously mentioned in 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#517, for example).

It improves the part of Eldoc that is not used by default 
(eldoc-documentation-strategy has a different value) but which is used 
by Eglot since it changes that variable in its managed buffers.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 28 Apr 2023 02:51:19 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > We can say something like "Currently, ..." to make that clear.
> 
> Very good, I've used that and an additional clarification for a 
> recommended alternative. Please see how you like it, or whether it needs 
> improving.

I've made a few minor improvements, thanks.

> Before the next pretest is cut, I'd also like to bring up the question 
> again of backporting the patch for bug#62816 (with Joao's support: 
> previously mentioned in 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#517, for example).
> 
> It improves the part of Eldoc that is not used by default 
> (eldoc-documentation-strategy has a different value) but which is used 
> by Eglot since it changes that variable in its managed buffers.

That patch didn't yet accumulate enough time for me to consider it
safe for the release branch.  Depending on when Emacs 29.1 is released
and whether we hear some downsides of the change, it could need to
wait for Emacs 29.2 or for Emacs 30.1.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 28/04/2023 08:19, Eli Zaretskii wrote:
>> Date: Fri, 28 Apr 2023 02:51:19 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>   monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> We can say something like "Currently, ..." to make that clear.
>>
>> Very good, I've used that and an additional clarification for a
>> recommended alternative. Please see how you like it, or whether it needs
>> improving.
> 
> I've made a few minor improvements, thanks.

Thank you.

>> Before the next pretest is cut, I'd also like to bring up the question
>> again of backporting the patch for bug#62816 (with Joao's support:
>> previously mentioned in
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#517, for example).
>>
>> It improves the part of Eldoc that is not used by default
>> (eldoc-documentation-strategy has a different value) but which is used
>> by Eglot since it changes that variable in its managed buffers.
> 
> That patch didn't yet accumulate enough time for me to consider it
> safe for the release branch.  Depending on when Emacs 29.1 is released
> and whether we hear some downsides of the change, it could need to
> wait for Emacs 29.2 or for Emacs 30.1.

Okay.

Let's get back to the previous topic. What about the previous fix for 
package-upgrade that I posted, one that makes it unconditionally upgrade 
built-in packages when invoked?

The one attached here: https://debbugs.gnu.org/62720#718

Can we put it on master now, or do we have to wait for some feedback 
from Emacs 29 first?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 5 May 2023 02:58:25 +0300
> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> Let's get back to the previous topic. What about the previous fix for 
> package-upgrade that I posted, one that makes it unconditionally upgrade 
> built-in packages when invoked?
> 
> The one attached here: https://debbugs.gnu.org/62720#718
> 
> Can we put it on master now, or do we have to wait for some feedback 
> from Emacs 29 first?

I'd prefer the latter.  I'd prefer even more to have same behavior in
Emacs 20 and Emacs 30, which could be possible if we decide to make
this change in Emacs 29.2, based on feedback.  Because is it really a
good idea to have the master and the release branch behave so
differently in this regard?  People who use both branches, or switch
from one to the other, will become confused.

Philip, Stefan: WDYT about this?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 5 May 2023 02:58:25 +0300
>> Cc: jporterbugs@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
>>  monnier@iro.umontreal.ca, larsi@gnus.org, joaotavora@gmail.com
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> Let's get back to the previous topic. What about the previous fix for 
>> package-upgrade that I posted, one that makes it unconditionally upgrade 
>> built-in packages when invoked?
>> 
>> The one attached here: https://debbugs.gnu.org/62720#718
>> 
>> Can we put it on master now, or do we have to wait for some feedback 
>> from Emacs 29 first?
>
> I'd prefer the latter.  I'd prefer even more to have same behavior in
> Emacs 20 and Emacs 30, which could be possible if we decide to make
> this change in Emacs 29.2, based on feedback.  Because is it really a
> good idea to have the master and the release branch behave so
> differently in this regard?  People who use both branches, or switch
> from one to the other, will become confused.
>
> Philip, Stefan: WDYT about this?

I am fine with any change, as long as package-upgrade-all does not
automatically switch from built-in packages to a different version from ELPA.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> People who use both branches, or switch from one to the other, will
> become confused.

I'm not terribly worried about that.

> Philip, Stefan: WDYT about this?

I'll second Philip's reply here.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dmitry@gutov.dev>,  jporterbugs@gmail.com,
>   philipk@posteo.net,  62720@debbugs.gnu.org,  larsi@gnus.org,
>   joaotavora@gmail.com
> Date: Fri, 05 May 2023 09:58:44 -0400
> 
> > People who use both branches, or switch from one to the other, will
> > become confused.
> 
> I'm not terribly worried about that.
> 
> > Philip, Stefan: WDYT about this?
> 
> I'll second Philip's reply here.

So I guess the consensus here is that this change is fine for master,
so let's install it there.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 05/05/2023 17:05, Eli Zaretskii wrote:
>> From: Stefan Monnier<monnier@iro.umontreal.ca>
>> Cc: Dmitry Gutov<dmitry@gutov.dev>,jporterbugs@gmail.com,
>>    philipk@posteo.net,62720@debbugs.gnu.org,larsi@gnus.org,
>>    joaotavora@gmail.com
>> Date: Fri, 05 May 2023 09:58:44 -0400
>>
>>> People who use both branches, or switch from one to the other, will
>>> become confused.
>> I'm not terribly worried about that.
>>
>>> Philip, Stefan: WDYT about this?
>> I'll second Philip's reply here.
> So I guess the consensus here is that this change is fine for master,
> so let's install it there.

Thanks all, done.

I've also installed the minor fix discussed previously on emacs-29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Just from reading it code and testing, I see another bug: it removes the 
>> updated package from package-selected-packages because it doesn't pass 
>> NOSAVE to package-delete. Meaning, 'M-x package-autoremove' at any time 
>> later will delete it.
>
> If there are bugs, we need to fix them.
>
> Philip, any comments regarding this particular issue with
> package-autoremove after package-update?

If this is an issue, I think it would be nice if it could be reported in
a separate issue to ease tracking, so that we can solve it in time for
Emacs 29.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 16:05, Philip Kaludercic wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>>> Just from reading it code and testing, I see another bug: it removes the
>>> updated package from package-selected-packages because it doesn't pass
>>> NOSAVE to package-delete. Meaning, 'M-x package-autoremove' at any time
>>> later will delete it.
>> If there are bugs, we need to fix them.
>>
>> Philip, any comments regarding this particular issue with
>> package-autoremove after package-update?
> If this is an issue, I think it would be nice if it could be reported in
> a separate issue to ease tracking, so that we can solve it in time for
> Emacs 29.

I've included the fix in the patch.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 22, 2023 at 12:12 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> I, personally, don't really buy this kind of argument, but I figured you
> might. After all, it's rather in line with reasoning we've seen voiced
> around these parts many times ("X has worked this way for Y years, let's
> never change it from now on"). Classic https://xkcd.com/1172/.

The CI + user instructions having to be updated is unfortunate, but I can
change those (and M-x eglot-update is a simple way to give predictable
consistent semantics for "bringing Eglot to latest").

Then there are the things I can't change, like users trying out
a new init.el file, a very common operation -- be it for bug
reproduction or just to try something out.  Also note that
package-delete + package-install interactively is a  pretty good way
to update packages in Emacs 28: no need to nuke the home directory.

In general, it's hard to predict the damage of hard-to-explain different
behaviour of these forms depending on whether you're on 28 or 29.
Or, the way this seems to be going, 30.

If 29 ships with this bug, it's going to linger for a good while,
and get worse as time passes.

If you "dont buy this", that's OK :-) I'm not trying to sell it to you
specifically.  I don't think we'll have torches and pitchforks either,
but we're not really talking about "spacebar overheating" here.

Note that I also don't really buy, personally, the "furtive update of
:core packages" argument either.  My reasoning is that that cat has
been out of the bag for a long long, just because of dependencies which
are absolutely liberally by package.el and the fact tha I've never seen
a bug reported about this.

I buy even less that people using M-x package-update in Emacs 29
don't want to update :core packages just because it doesn't do
that right now.  For me, it's obvious people weren't using to
update :core packages because that command lived in master
for the large part of its short life and in master :core packages
don't need any updating, by definition. But of course fact that I
don't buy it  shouldn't mean that it should be disregarded (and
noone is advocating for that).

The last two patches I provided aim to aid Eglot users (lightly,
as you've discovered, since the use-package use case is not well
covered in the first one, and there's still the error behaviour
to ponde).  But, more importantly they are designed so that no
cats that weren't previously out of the bag make it outside
the bag

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 03:57, João Távora wrote:

> Then there are the things I can't change, like users trying out
> a new init.el file, a very common operation -- be it for bug
> reproduction or just to try something out.  Also note that
> package-delete + package-install interactively is a  pretty good way
> to update packages in Emacs 28: no need to nuke the home directory.

It's as good a workaround for ignorance as any, but if we think it's 
okay for a user to do that (a relatively large manual operation), 
perhaps we can guide them to do a 'package-install' from list-packages 
instead.

But the difference from Emacs 28 is indeed unfortunate, no argument there.

Or if the new option stays around, I guess you could be recommending 
they customize it first?

> If you "dont buy this", that's OK :-) I'm not trying to sell it to you
> specifically.  I don't think we'll have torches and pitchforks either,
> but we're not really talking about "spacebar overheating" here.
> 
> Note that I also don't really buy, personally, the "furtive update of
> :core packages" argument either.  My reasoning is that that cat has
> been out of the bag for a long long, just because of dependencies which
> are absolutely liberally by package.el and the fact tha I've never seen
> a bug reported about this.

I don't have a strong opinion one way or the other, but this part seems 
to be worth treating conservatively, I think. This close to release, etc.

We might as well switch to the "update all the cores!" model later, but 
hopefully by that time we have a good CI setup that tests the 
compatibility of all core packages with all Emacs versions they are 
supposed to support.

> I buy even less that people using M-x package-update in Emacs 29
> don't want to update :core packages just because it doesn't do
> that right now.  For me, it's obvious people weren't using to
> update :core packages because that command lived in master
> for the large part of its short life and in master :core packages
> don't need any updating, by definition. But of course fact that I
> don't buy it  shouldn't mean that it should be disregarded (and
> noone is advocating for that).

This one I'm trying to fix, currently.

> The last two patches I provided aim to aid Eglot users (lightly,
> as you've discovered, since the use-package use case is not well
> covered in the first one, and there's still the error behaviour
> to ponde).  But, more importantly they are designed so that no
> cats that weren't previously out of the bag make it outside
> the bag

Do you prefer that package-install-upgrade-built-in (together with the 
behavior of package-install it enables) stays around?

One possible alternative I mentioned, is package-install could grow 
another programmatic parameter which would mean "install the very latest 
version". The users will then need to evaluate (package-install 'eglot 
nil t) for that to happen. Also a breaking change in instructions, 
though, since in Emacs 28 this will error.

Another gotcha with the latter, is that if some version from ELPA is 
already installed, they will get two versions installed as a result.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 22, 2023 at 12:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> But the difference from Emacs 28 is indeed unfortunate, no argument there.
>
> Or if the new option stays around, I guess you could be recommending
> they customize it first?

No idea yet. I'm leaning eglot-update as it guaranteedly the most
consistent.  That or straight.el or elpaca.el.  Or no recommendation
whatsoever.

I just got my first bug report about a user where I can't tell
the version of Eglot being used but which is consistent with an
older version being used, perhaps unknowingly. This is, of course,
not unheard of,  so let's see.

> We might as well switch to the "update all the cores!" model later, but
> hopefully by that time we have a good CI setup that tests the
> compatibility of all core packages with all Emacs versions they are
> supposed to support.

Right, that makes sense.  If it helps, Eglot's CI has been doing that
for year now.  It tests Eglot -- and its up-to-date dependencies --
in Emacs 26, 27 and 28.  It's not very complex to set up.

> Do you prefer that package-install-upgrade-built-in (together with the
> behavior of package-install it enables) stays around?

No, my opinion is that I find this user unfriendly, and of course
even more so with the default value.  I've already answered this
early on in the thread.

Creating customization points as a "cure" for haphazard design is
not a good practice.  It's like there are two package managers
in Emacs. The package menu and the package-install.  The
"upgrade :core" cat has always been out of the bag in the
first one fully, and in the second almost fully.  And then
there's the third "package manager", i.e. use-package's specific
package-install behaviour.   And nwo there is package-update which
is almost a newborn, but is already inexplicably broken. Putting
more customization variables into the mix is just adding variations
to a very crowded field, more questions to ask, more
fear-uncertainty-doubt.

The "update all cores" is frankly the safest model IMO.  Of course,
I understand that others disagree -- and I even see their well.
So knowing emacs-devel, I don't have any illusions.

But I suspect they see this "update all cores" idea as some kind of
gargantuan jump from what is in Emacs 28, when in reality it's not
(again, because dependencies).  And I also tend to think they
disagree because they haven't been working with this and with users
for as long as I have (and you have), so from afar it seems dangerous.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 02:12:23 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 21/04/2023 14:05, Eli Zaretskii wrote:
> 
> >>> And at least
> >>> João (and I think others as well) expected it to upgrade Eglot even
> >>> though it is now built in.
> >>
> >> I think he wants that because this way (package-install 'eglot) and
> >> (use-package eglot :ensure t) could match the behavior of Emacs 28 with
> >> an empty init directory. Backward compatibility and all that.
> > 
> > But if, with older Emacsen, package-install would refuse to update to
> > a newer version of Eglot if _some_ older version of Eglot is already
> > installed, then where's the problem with the default behavior of
> > package-install? it behaves exactly like in previous versions of
> > Emacs.  And why is this a problem for users of Eglot, if they couldn't
> > use package-install more than once for Eglot anyway?
> > 
> > Something is amiss here.
> 
> The above scenarios (package-install or (use-package eglot :ensure t)) 
> when ~/.emacs.d/elpa is empty, result in the very latest Eglot being 
> installed when using Emacs 28. And will result in something different 
> with Emacs 29 (not the latest version). That's not nothing: the CI 
> scripts will have to be updated, and the user instructions for reporting 
> problems will have to be made more complicated as well. Some 
> possibilities will simply be gone (the user won't be able to upgrade 
> Eglot to the very latest by deleting it from ~/.emacs.d/elpa and calling 
> package-install, for example). This *is* a non-backward compatible 
> change from the perspective of Eglot's maintainer.

We were talking about users installing and updating packages.  The CI
scenario doesn't belong here.  It is also much less important one
(test suites are always required to chase the changes in development).

Let's not complicate an already complicated set of issues by bringing
up unimportant secondary use cases.

> I, personally, don't really buy this kind of argument, but I figured you 
> might. After all, it's rather in line with reasoning we've seen voiced 
> around these parts many times ("X has worked this way for Y years, let's 
> never change it from now on"). Classic https://xkcd.com/1172/.

If you allude to my reasoning, then it is never that simplistic, and
always considers each case separately, not "by analogy".

> >> But I think that's questionable, semantically. Given that Eglot is
> >> already "installed". Though, of course, one could argue that a bundled
> >> package is not exactly installed, but then we should change what
> >> 'package-installed-p' does as well. And think hard before doing that.
> > 
> > I'd question why we have two commands instead of just one, but that's
> > probably water under the bridge at this point.
> 
> Either way would be fine, IMO, as long as the behavior is logical and 
> matches documentation. But having a separate command to upgrade now lets 
> us fix it separately without worry of breaking something more globally.

Except that, based on what we have (see below) ,we don't really have
an "upgrade" operation, we only have "install" and "delete"
(i.e. "uninstall").  So maybe we should preserve that, to minimize
problems and user surprise/confusion.

> > In interactive invocation, package-upgrade calls completing-read with
> > its 4th argument non-nil, so you cannot select a package which is not
> > in the collection returned by package--updateable-packages.  What I
> > meant above is to allow that collection to include built-in packages
> > as optional behavior.  I just tried invoking package-update for ElDoc,
> > and I get "No match" after typing "eldoc" to its prompt, although
> > eldoc version 1.14.0 is in the list presented by list-packages as
> > "available".
> 
> That's what I imagined: adding a new optional argument to 
> package--updateable-packages which would include builtins in the result.
> 
> And only pass it when called from package-upgrade.
> 
> Hopefully that's the kind of optional that you meant.

Yes, something like that.  Presumably activated by the same new option
introduced for package-installed.

> >>>> 'package-update' is an
> >>>> interactive function which itself it called from only one place:
> >>>> 'package-update-all', and since the plan is to improve both, we can make
> >>>> sure they only do what we ask of them: package-update will upgrade
> >>>> builtins when invoked directly, and package-update-all will upgrade them
> >>>> only when the builtin has been upgraded before (making it not a builtin
> >>>> anymore), or a new user option is set.
> >>>
> >>> This is one possibility, and it might make sense to some users.  But I
> >>> don't think we can be sure it will make sense to an overwhelming
> >>> majority of the users.
> >>
> >> Hence the user option?
> > 
> > Which one?  Are you suggesting to add a new one?  If so, why not use
> > the one we already added, package-install-upgrade-built-in?
> 
> The user option I was thinking of would probably be called a little 
> shorter: package-upgrade-built-in. And it would only affect the upgrader 
> commands.

We could rename the existing option, if the name is the problem.
Otherwise, I don't see why we would need two separate options: they do
the same job and have the same meaning from the user's POV.

> >> Because
> >> package-upgrade does not have a menu entry, or a button anywhere,
> >> whereas package-upgradable-builtins can be altered from the Customize UI.
> > 
> > Maybe marking a package in the list for update could be interpreted as
> > "upgrade that, no questions asked", and we will need no user options?
> 
> There is no handy "upgrade that" binding in the packages menu. The only 
> command that's available there related to upgrading is 
> package-menu-mark-upgrades, which does that to all packages (except 
> builtins).
> 
> To manually execute an upgrade of one package, one needs to both mark 
> the new version for installation (after first scrolling down the list to 
> find it), and mark the current version for deletion. This is what 
> currently an upgrade consists of.

Yes.  So users could manually mark an new version for installation,
and if needed mark the old version for deletion, thus indicating that
they want this upgrade.  No separate option to indicate the same is
needed.

> All this is to say, the first step (upgrading Eglot to the version from 
> ELPA) will be less user-friendly compared to the other UIs we have. But 
> it's probably manageable, especially if documented well.

I don't see why it would be less user-friendly.

> My point here, however, is that commit 580d8278c5f48 improved the 
> situation to a lesser degree that some people here might have expected.

One again: commit 580d8278c5f48 solved precisely the problem which
opened this bug report, nothing more, nothing less.  The rest are late
additions, which were not on the table when the original bug was
discussed.  So the above complaint is at least unfair.

> >>> If people agree with that, I don't think I'll object.  But this is in
> >>> a sense a breaking change: package-install will only install, and
> >>> thereafter users will need to use package-upgrade.  Some might dislike
> >>> such behavior changes.  And we will need to make sure that all the
> >>> available methods of "installing" do not "upgrade", for consistency.
> >>
> >> Yeah, apparently it won't be a breaking change, or a change at all.
> > 
> > I'm not sure, see above.
> > 
> > Also, when you mark packages for update from the list presented by
> > list-packages, the menu entry says
> > 
> >    i    Mark for Install
> > 
> > and its help-echo says "Mark a package for installation and move to
> > the next line", so we already confuse "install" and "upgrade".
> 
> But that's the thing: we don't have a separate action in the packages 
> list to "upgrade" a single package.

We agree.  I'm saying that your suggestion to make package-upgrade a
more important command is by itself a breaking change: there's no
"upgrade" operation, per se, in the current code.

> But we can conclude that we do have two different installation actions:
> 
>   - package-install which will install the latest version of a package, 
> but only if it's not installed;
>   - package-menu-mark-install, which will mark a specific version for 
> installation, disregarding whether some earlier version is already 
> installed; the previous version will remain installed still.

Which is again a breaking behavior change, AFAIU.  Is this a good idea
so late in the development of Emacs 29?

> >>>> Could you specify which problem it's currently solving? Some particular
> >>>> scenario.
> >>>
> >>> The scenario which started this bug report, see the message whose URL
> >>> I mentioned above.  IOW, we now allow users to explicitly request that
> >>> package-install includes built-in packages in the list of candidates,
> >>> and will therefore allow to upgrade them.
> >>
> >> After we fix 'package-upgrade', users will be able to 'M-x
> >> package-upgrade RET eglot RET'.
> > 
> > This goes back to the issue of having two confusingly-similar but
> > different commands, as mentioned above.  I guess we should first make
> > up our minds what, if anything, we want to do about this.
> 
> Sure.
> 
> But even if we decide that we want to eliminate that split, doing *that* 
> would really be a breaking change. I don't have a reasonable plan to 
> present for doing that in Emacs 29, so far.

There's no "split".  What I wanted to point out is that we don't seem
to have a clear vision of these two commands, since they are confusing
intertwined.  In fact, one could argue that package-upgrade in its
current form is simply a convenience shortcut for "delete old and
install new".

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 22/04/2023 11:26, Eli Zaretskii wrote:

>> The above scenarios (package-install or (use-package eglot :ensure t))
>> when ~/.emacs.d/elpa is empty, result in the very latest Eglot being
>> installed when using Emacs 28. And will result in something different
>> with Emacs 29 (not the latest version). That's not nothing: the CI
>> scripts will have to be updated, and the user instructions for reporting
>> problems will have to be made more complicated as well. Some
>> possibilities will simply be gone (the user won't be able to upgrade
>> Eglot to the very latest by deleting it from ~/.emacs.d/elpa and calling
>> package-install, for example). This *is* a non-backward compatible
>> change from the perspective of Eglot's maintainer.
> 
> We were talking about users installing and updating packages.  The CI
> scenario doesn't belong here.  It is also much less important one
> (test suites are always required to chase the changes in development).
> 
> Let's not complicate an already complicated set of issues by bringing
> up unimportant secondary use cases.

You asked. I relayed what's already been said on the matter by Joao.

>> I, personally, don't really buy this kind of argument, but I figured you
>> might. After all, it's rather in line with reasoning we've seen voiced
>> around these parts many times ("X has worked this way for Y years, let's
>> never change it from now on"). Classic https://xkcd.com/1172/.
> 
> If you allude to my reasoning, then it is never that simplistic, and
> always considers each case separately, not "by analogy".

Of course that's a simplification. Hence the "might" anyway.

>> Either way would be fine, IMO, as long as the behavior is logical and
>> matches documentation. But having a separate command to upgrade now lets
>> us fix it separately without worry of breaking something more globally.
> 
> Except that, based on what we have (see below) ,we don't really have
> an "upgrade" operation, we only have "install" and "delete"
> (i.e. "uninstall").  So maybe we should preserve that, to minimize
> problems and user surprise/confusion.

We do. We have commands for upgrading, both in "list-packages", and used 
interactively. Which do the thing of installing the new version and 
removing the old one.

Which is what upgrading means in various tools, e.g. 'apt'.

>>> In interactive invocation, package-upgrade calls completing-read with
>>> its 4th argument non-nil, so you cannot select a package which is not
>>> in the collection returned by package--updateable-packages.  What I
>>> meant above is to allow that collection to include built-in packages
>>> as optional behavior.  I just tried invoking package-update for ElDoc,
>>> and I get "No match" after typing "eldoc" to its prompt, although
>>> eldoc version 1.14.0 is in the list presented by list-packages as
>>> "available".
>>
>> That's what I imagined: adding a new optional argument to
>> package--updateable-packages which would include builtins in the result.
>>
>> And only pass it when called from package-upgrade.
>>
>> Hopefully that's the kind of optional that you meant.
> 
> Yes, something like that.  Presumably activated by the same new option
> introduced for package-installed.

Okay, then it's something different, and you didn't answer my question.

>>>>>> 'package-update' is an
>>>>>> interactive function which itself it called from only one place:
>>>>>> 'package-update-all', and since the plan is to improve both, we can make
>>>>>> sure they only do what we ask of them: package-update will upgrade
>>>>>> builtins when invoked directly, and package-update-all will upgrade them
>>>>>> only when the builtin has been upgraded before (making it not a builtin
>>>>>> anymore), or a new user option is set.
>>>>>
>>>>> This is one possibility, and it might make sense to some users.  But I
>>>>> don't think we can be sure it will make sense to an overwhelming
>>>>> majority of the users.
>>>>
>>>> Hence the user option?
>>>
>>> Which one?  Are you suggesting to add a new one?  If so, why not use
>>> the one we already added, package-install-upgrade-built-in?
>>
>> The user option I was thinking of would probably be called a little
>> shorter: package-upgrade-built-in. And it would only affect the upgrader
>> commands.
> 
> We could rename the existing option, if the name is the problem.
> Otherwise, I don't see why we would need two separate options: they do
> the same job and have the same meaning from the user's POV.

The name is a problem, yes. What could also be a problem is a user that 
customizes this option to have package-update update builtin packages (a 
reasonable behavior that should be on by default anyway), will also 
automatically have change the behavior of package-install to be more 
surprising (install an already installed package).

Further, if we have a user option affect package-update, we'll have to 
alter package-update-all and package-manu-mark-for-update in the same 
patch (otherwise we'll have more nonsense on our hands). Whereas the 
first version I sent is more minimal.

>> All this is to say, the first step (upgrading Eglot to the version from
>> ELPA) will be less user-friendly compared to the other UIs we have. But
>> it's probably manageable, especially if documented well.
> 
> I don't see why it would be less user-friendly.

The same reason we do have commands with "upgrade" in their names, 
rather than force everybody to use the "install" and "delete" ones.

>> My point here, however, is that commit 580d8278c5f48 improved the
>> situation to a lesser degree that some people here might have expected.
> 
> One again: commit 580d8278c5f48 solved precisely the problem which
> opened this bug report, nothing more, nothing less.

It doesn't seem like the originator of the report agrees with that.

>> But we can conclude that we do have two different installation actions:
>>
>>    - package-install which will install the latest version of a package,
>> but only if it's not installed;
>>    - package-menu-mark-install, which will mark a specific version for
>> installation, disregarding whether some earlier version is already
>> installed; the previous version will remain installed still.
> 
> Which is again a breaking behavior change, AFAIU.  Is this a good idea
> so late in the development of Emacs 29?

The above is not a breaking change, it's how things work already. And 
have been working for quite a while.

>> But even if we decide that we want to eliminate that split, doing *that*
>> would really be a breaking change. I don't have a reasonable plan to
>> present for doing that in Emacs 29, so far.
> 
> There's no "split".  What I wanted to point out is that we don't seem
> to have a clear vision of these two commands, since they are confusing
> intertwined.  In fact, one could argue that package-upgrade in its
> current form is simply a convenience shortcut for "delete old and
> install new".

What should an upgrade command do, in your opinion, if not "delete old 
and install new"?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Sat, 22 Apr 2023 13:48:41 +0300
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, 62720@debbugs.gnu.org,
>  philipk@posteo.net, monnier@iro.umontreal.ca, larsi@gnus.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> The user option I was thinking of would probably be called a little
> >> shorter: package-upgrade-built-in. And it would only affect the upgrader
> >> commands.
> > 
> > We could rename the existing option, if the name is the problem.
> > Otherwise, I don't see why we would need two separate options: they do
> > the same job and have the same meaning from the user's POV.
> 
> The name is a problem, yes. What could also be a problem is a user that 
> customizes this option to have package-update update builtin packages (a 
> reasonable behavior that should be on by default anyway), will also 
> automatically have change the behavior of package-install to be more 
> surprising (install an already installed package).

It's the same change in behavior, since for built-in packages
"install" and "upgrade" is the same.

> Further, if we have a user option affect package-update, we'll have to 
> alter package-update-all and package-manu-mark-for-update in the same 
> patch (otherwise we'll have more nonsense on our hands). Whereas the 
> first version I sent is more minimal.

How is this relevant to whether we need one or two separate user
options?

> >> All this is to say, the first step (upgrading Eglot to the version from
> >> ELPA) will be less user-friendly compared to the other UIs we have. But
> >> it's probably manageable, especially if documented well.
> > 
> > I don't see why it would be less user-friendly.
> 
> The same reason we do have commands with "upgrade" in their names, 
> rather than force everybody to use the "install" and "delete" ones.

I still don't think this is less user-friendly.

> > One again: commit 580d8278c5f48 solved precisely the problem which
> > opened this bug report, nothing more, nothing less.
> 
> It doesn't seem like the originator of the report agrees with that.

I'm aware of that.  But you are talking to me, not to him, and the
above is my opinion.  I also agree that the solution is not ideal,
just the best I could agree to.

> >>    - package-install which will install the latest version of a package,
> >> but only if it's not installed;
> >>    - package-menu-mark-install, which will mark a specific version for
> >> installation, disregarding whether some earlier version is already
> >> installed; the previous version will remain installed still.
> > 
> > Which is again a breaking behavior change, AFAIU.  Is this a good idea
> > so late in the development of Emacs 29?
> 
> The above is not a breaking change, it's how things work already. And 
> have been working for quite a while.

That's not what I understand.  E.g., package-install will install even
if the package is already installed.

But if this already works, then why are you bringing this up?

> >> But even if we decide that we want to eliminate that split, doing *that*
> >> would really be a breaking change. I don't have a reasonable plan to
> >> present for doing that in Emacs 29, so far.
> > 
> > There's no "split".  What I wanted to point out is that we don't seem
> > to have a clear vision of these two commands, since they are confusing
> > intertwined.  In fact, one could argue that package-upgrade in its
> > current form is simply a convenience shortcut for "delete old and
> > install new".
> 
> What should an upgrade command do, in your opinion, if not "delete old 
> and install new"?

Why is this question important, in the context of the current
discussion?  It's a tangent.  All I wanted to point out is that IMO
there's no "split" that we want to eliminate.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

Not sure about the reason emacs-devel was lost from Cc. That could be my 
fault: I removed it before sending the first version of the grandparent 
email, but then I thought I managed to abort that delivery and resend it 
properly.

Anyway, adding it back and replying with a full quote.

On 20/04/2023 01:49, João Távora wrote:
> On Wed, Apr 19, 2023 at 11:01 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> I think one of the conclusions to be made here is that even if
>> (package-install 'eglot) now installs the newest version of Eglot in
>> Emacs 29,
>>
>>     (use-package 'eglot :ensure t)
>>
>> still won't do that in Emacs 29 because (package-installed-p 'eglot)
>> returns t still. So the commit 580d8278c5f48 doesn't help with your
>> "most common upgrade method" cited below, if they rely on use-package
>> instead of calling 'package-install' directly.
> 
> Right, that's likely.
> 
>> The patch I +1'd here https://debbugs.gnu.org/62720#467 wouldn't help
>> with (use-package 'eglot :ensure t) either, IIUC.
> 
> That's also likely.  So we'd need this:
> 
> diff --git a/lisp/use-package/use-package-ensure.el
> b/lisp/use-package/use-package-ensure.el
> index e0ea982594e..95e6a9e95d5 100644
> --- a/lisp/use-package/use-package-ensure.el
> +++ b/lisp/use-package/use-package-ensure.el
> @@ -160,7 +160,9 @@ use-package-ensure-elpa
>           (when (consp package)
>             (use-package-pin-package (car package) (cdr package))
>             (setq package (car package)))
> -        (unless (package-installed-p package)
> +        (when (or (and (memq package package--safely-upgradeable-builtins)
> +                       (not (assoc 'eglot (package--alist))))
> +                  (not (package-installed-p package)))
>             (condition-case-unless-debug err
>                 (progn
>                   (when (assoc package (bound-and-true-p

All right, this part would "fix" (use-package eglot :ensure t).

>> Do we want to change the semantics of 'package-install' just so the
>> (minor) fraction of users who call this function directly will get Eglot
>> upgraded when starting over from an empty config with Emacs 29? And/or
>> change (package-installed-p 'eglot) to return nil when the appropriate
>> user option is set? I'm not sure that's wise, certainly not the latter.
> 
> We don't _have_ to change the semantics.  We can get exactly the same
> semantics if we want to.  Patch below:

What kind of semantics do we get with it?

1. (use-package eglot :ensure t) considers select builtin packages to be 
"not installed" for the purposes of ":ensure t".
2. 'M-x package-install' allows installing them. It doesn't allow 
installing any other package for which (package-installed-p 'xxx) 
returns t, but allows installing (essentially upgrading) these ones 
(either just eglot, or both eglot and use-package).
3. 'M-x package-update RET eglot RET' still doesn't work unless eglot 
has been "upgraded" at least once via other means.

Is that logical? Is even just 1+2 logical?

And what about capabilities that we lose that way? I guess one of the 
reasons to bundle ELPA packages is to make sure they can be used without 
additional installation. E.g. in some Internes-less network, or one 
that's firewalled off. Let's also imagine that (for example) clangd is 
already installed through other means, which is also within the realm of 
possibility.

And take use-package. Which some people position as the new way to write 
the Emacs configuration.

The user puts the snippet which they saw on the Internet

   (use-package eglot :ensure t)

either because they think it's a good idea, or because they intend to 
add some actual config in there, restart Emacs and... startup fails 
because the package can't be installed (no connection). Should they 
remove ":ensure t"? Perhaps. But the documentation says that that option 
checks that the package is "installed automatically if not already 
present on your system". Seems legit, right? Why would startup fail when 
Eglot is already present on the system?

Or, to put it another way, why did we decide to bundle Eglot with Emacs 
if the first thing we're going to do is to try to download it anyway?

So... I understand the problem, but I think we shouldn't change the 
functions in a way that makes them conflict with documentation or with 
each other.

> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index 685f983e285..850f4ad3a7a 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -652,6 +652,12 @@ package--builtins
>   name (a symbol) and DESC is a `package--bi-desc' structure.")
>   (put 'package--builtins 'risky-local-variable t)
> 
> +(defvar package--safely-upgradeable-builtins '(eglot use-package))
> +
> +(defun package--safely-upgradeable-builtin (p)
> +  (and (memq p package--safely-upgradeable-builtins) ; whitelisted
> +       (not (assoc p (package--alist))))) ; not installed already
> +
>   (defvar package-alist nil
>     "Alist of all packages available for activation.
>   Each element has the form (PKG . DESCS), where PKG is a package
> @@ -2201,14 +2207,18 @@ package-install
>        (package--archives-initialize)
>        (list (intern (completing-read
>                       "Install package: "
> +                    (append
>                        (delq nil
>                              (mapcar (lambda (elt)
>                                        (unless (package-installed-p (car elt))
>                                          (symbol-name (car elt))))
>                                      package-archive-contents))
> +                     package--safely-upgradeable-builtins)
>                       nil t))
>              nil)))
>     (package--archives-initialize)
> +  (when (package--safely-upgradeable-builtin pkg)
> +    (setq pkg (cadr (assoc pkg package-archive-contents))))
>     (add-hook 'post-command-hook #'package-menu--post-refresh)
>     (let ((name (if (package-desc-p pkg)
>                     (package-desc-name pkg)
> diff --git a/lisp/use-package/use-package-ensure.el
> b/lisp/use-package/use-package-ensure.el
> index e0ea982594e..cfa10f453d9 100644
> --- a/lisp/use-package/use-package-ensure.el
> +++ b/lisp/use-package/use-package-ensure.el
> @@ -160,7 +160,8 @@ use-package-ensure-elpa
>           (when (consp package)
>             (use-package-pin-package (car package) (cdr package))
>             (setq package (car package)))
> -        (unless (package-installed-p package)
> +        (when (or (package--safely-upgradeable-builtin package)
> +                  (not (package-installed-p package)))
>             (condition-case-unless-debug err
>                 (progn
>                   (when (assoc package (bound-and-true-p

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 12:25 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
>
> Anyway, adding it back and replying with a full quote.

Thanks, I also didn't notice emacs-devel was lost there.

> On 20/04/2023 01:49, João Távora wrote:
> What kind of semantics do we get with it?

The closest possible to Emacs 28 for _all_ users (Eglot users and
the Emacs users that are vulnerable to furtive upgrades and which
Eli is thinking of)

> 1. (use-package eglot :ensure t) considers select builtin packages to be
> "not installed" for the purposes of ":ensure t".
> 2. 'M-x package-install' allows installing them. It doesn't allow
> installing any other package for which (package-installed-p 'xxx)
> returns t, but allows installing (essentially upgrading) these ones
> (either just eglot, or both eglot and use-package).
> 3. 'M-x package-update RET eglot RET' still doesn't work unless eglot
> has been "upgraded" at least once via other means.
>
> Is that logical? Is even just 1+2 logical?

Neither is.  IMO we have to think in terms of expectations for each
package, not in terms of a heavy handed blunt class o packages like
"builtin".  Because Eglot wasn't a builtin and now is.  So this is
why my patch includes that predicate `package--safely-upgradeable-builtin`
The name is not good, I admit.  It should be
`package--hitherto-casually-upgradeable-builtin` maybe.

> because the package can't be installed (no connection). Should they
> remove ":ensure t"? Perhaps. But the documentation says that that option
> checks that the package is "installed automatically if not already
> present on your system". Seems legit, right? Why would startup fail when
> Eglot is already present on the system?

Because we want it to do the same it did in Emacs 28?  But I see what
you mean.  Maybe the :ensure for packages in that particular subclass
should mean "try a bit harder to upgrade these, but don't blow up if
you can't -- just try next time".

> Or, to put it another way, why did we decide to bundle Eglot with Emacs
> if the first thing we're going to do is to try to download it anyway?

Who is "we"?  I know for a fact some people are.  The enthusiasts are,
the feature seekers, are immediately.  Others will experiment, I
assume.  Very often one experimentation that happens is you just
save all your .emacs.d to the side safely and let er rip with
whatever init.el someone (trustworthy presumably) gave you.

( What, you're going to say you've never tried the spacemacs
  or doom emacs or nano emacs just to see what all the hype
  is about??? :-) )

But others, like me from time to time when working on VMs via ssh
machines,  will be very happy with sudo apt install emacs in
said machines (if not done so already) and use a vanilla
configurationless emacs.  And if you add a "clangd" to that
invocation and Emacs happens to be 29 you'll have nice LSP
for the first time in many years.  Personally, for _my_ quick
hacks in those situations, I don't bother with M-x package-install
latest-and-greatest, just as I don't bother bringing my heavy
config via ssh to that machine.  I can't really, I'm frequently
tmate'ing with other people there, we have to settle on Emacs -Q
(+ electric-pair-mode and a few obvious others).

But your question also has a completely different answer:
A very good reason I wanted Eglot to be bundled with Emacs is now
actually being put into question in this very thread.  I wanted
to simplify Eglot's development as it advances with other :core
packages.  For example, I wanted to split off its external
completion style easily into a separate core package, as I did
with Stefan earlier this year.  I want to be able to add a feature
to ElDoc and use it in Eglot in the same patch, etc.  That part
has worked fine.  It wasn't impossible with Eglot in GitHub, but
it was harder.  Hopefully Eli is not asking for that bliss to end
I understand hope he's asking to be circumspect in what
is added, and not do gratuitous things like bump dependencies
without reason.  I already do that, so no problem.

Finally, if you're taking all this in the direction of "I told
you so, don't you regret bringing Eglot into Emacs 29?", I don't
blame you.  But I have to say: no, not yet.  I think having the
beginnings of an LSP library in Emacs for major-modes to work
with is excellent.  I like the attention and the high-standards
other seasoned developers hold my code up to.  I like the Info
manual and I like that emacs -Q bla.c -f eglot works just fine.

This upgrade situation sucks, is true, and I should have foreseen
it.  Spent too much time in my emacs-master worktree I guess.

Who knows, maybe this will all be OK after all?? Half the
users are using "straight.el" anyway (and I really doubt
it is hampered by these minutiae about dependencies, seeing
as MELPA-land is teeming with way more "dependency hell").

If nothing more happens on Emacs 29, which is the most likely
outcome, I think I will just start recommending "straight.el"
more, maybe learn it myself.  I had problems with it in the
past but it's probably matured now.

> So... I understand the problem, but I think we shouldn't change the
> functions in a way that makes them conflict with documentation or with
> each other.

Does the latest patch introduce any such conflict besides the
"no internet" case you mentioned?  Not that it's really a
conflict IMO: it's just what Emacs 28 did with Eglot AFAICT.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 03:13, João Távora wrote:

>> On 20/04/2023 01:49, João Távora wrote:
>> What kind of semantics do we get with it?
> 
> The closest possible to Emacs 28 for _all_ users (Eglot users and
> the Emacs users that are vulnerable to furtive upgrades and which
> Eli is thinking of)

As if Eglot wasn't bundled in the first place, you mean.

>> 1. (use-package eglot :ensure t) considers select builtin packages to be
>> "not installed" for the purposes of ":ensure t".
>> 2. 'M-x package-install' allows installing them. It doesn't allow
>> installing any other package for which (package-installed-p 'xxx)
>> returns t, but allows installing (essentially upgrading) these ones
>> (either just eglot, or both eglot and use-package).
>> 3. 'M-x package-update RET eglot RET' still doesn't work unless eglot
>> has been "upgraded" at least once via other means.
>>
>> Is that logical? Is even just 1+2 logical?
> 
> Neither is.  IMO we have to think in terms of expectations for each
> package, not in terms of a heavy handed blunt class o packages like
> "builtin".  Because Eglot wasn't a builtin and now is.  So this is
> why my patch includes that predicate `package--safely-upgradeable-builtin`
> The name is not good, I admit.  It should be
> `package--hitherto-casually-upgradeable-builtin` maybe.

This kind of option still makes sense, but I'd put in a different place: 
it would affect which packages are upgraded by 'M-x package-update-all' 
or 'U' in the packages list.

>> because the package can't be installed (no connection). Should they
>> remove ":ensure t"? Perhaps. But the documentation says that that option
>> checks that the package is "installed automatically if not already
>> present on your system". Seems legit, right? Why would startup fail when
>> Eglot is already present on the system?
> 
> Because we want it to do the same it did in Emacs 28?  But I see what
> you mean.  Maybe the :ensure for packages in that particular subclass
> should mean "try a bit harder to upgrade these, but don't blow up if
> you can't -- just try next time".

... maybe.

Though a meticulous user (some of them, maybe not others) might still 
wonder why an upgrade if attempted at all, even though the package is 
already available.

>> Or, to put it another way, why did we decide to bundle Eglot with Emacs
>> if the first thing we're going to do is to try to download it anyway?
> 
> Who is "we"?

You/we/emacs-devel as a whole.

> I know for a fact some people are.  The enthusiasts are,
> the feature seekers, are immediately.  Others will experiment, I
> assume.  Very often one experimentation that happens is you just
> save all your .emacs.d to the side safely and let er rip with
> whatever init.el someone (trustworthy presumably) gave you.

I agree that upgrading it should be made as easy as possible. But it 
should require explicit actions.

> ( What, you're going to say you've never tried the spacemacs
>    or doom emacs or nano emacs just to see what all the hype
>    is about??? :-) )
> 
> But others, like me from time to time when working on VMs via ssh
> machines,  will be very happy with sudo apt install emacs in
> said machines (if not done so already) and use a vanilla
> configurationless emacs.  And if you add a "clangd" to that
> invocation and Emacs happens to be 29 you'll have nice LSP
> for the first time in many years.  Personally, for _my_ quick
> hacks in those situations, I don't bother with M-x package-install
> latest-and-greatest, just as I don't bother bringing my heavy
> config via ssh to that machine.  I can't really, I'm frequently
> tmate'ing with other people there, we have to settle on Emacs -Q
> (+ electric-pair-mode and a few obvious others).

All good reasons. But then you add 'use-package :ensure t' to the mix, 
and something unexpected starts to happen. Maybe expected to you, but 
not to somebody other like you.

Maybe you'll put the recommended use-package form somewhere in the Eglot 
manual, right? It will be published on the web as well. Will it contain 
":ensure t" or not? If not, the users of Emacs 28 might fail to get the 
package installed. If it will, then the offline users might have 
problems, as described previously.

> But your question also has a completely different answer:
> A very good reason I wanted Eglot to be bundled with Emacs is now
> actually being put into question in this very thread.  I wanted
> to simplify Eglot's development as it advances with other :core
> packages.  For example, I wanted to split off its external
> completion style easily into a separate core package, as I did
> with Stefan earlier this year.  I want to be able to add a feature
> to ElDoc and use it in Eglot in the same patch, etc.  That part
> has worked fine.  It wasn't impossible with Eglot in GitHub, but
> it was harder.  Hopefully Eli is not asking for that bliss to end
> I understand hope he's asking to be circumspect in what
> is added, and not do gratuitous things like bump dependencies
> without reason.  I already do that, so no problem.

It seems like most of this could have worked out if Eglot moved to the 
emacs.git repository, but was removed from the distro before packaging? 
Just a thought, not that I advocate for doing that.

> Finally, if you're taking all this in the direction of "I told
> you so, don't you regret bringing Eglot into Emacs 29?", I don't
> blame you.  But I have to say: no, not yet.  I think having the
> beginnings of an LSP library in Emacs for major-modes to work
> with is excellent.  I like the attention and the high-standards
> other seasoned developers hold my code up to.  I like the Info
> manual and I like that emacs -Q bla.c -f eglot works just fine.

I like my told-you-so's as much as the next guy (just kidding; 
definitely more than the next guy), but that's not the point I was 
making. Just that it's here now, and it seems odd to avoid using it and 
hurry up to install the next version right away.

> This upgrade situation sucks, is true, and I should have foreseen
> it.  Spent too much time in my emacs-master worktree I guess.
> 
> Who knows, maybe this will all be OK after all?? Half the
> users are using "straight.el" anyway (and I really doubt
> it is hampered by these minutiae about dependencies, seeing
> as MELPA-land is teeming with way more "dependency hell").

I'm pretty sure the users on straight.el are straight-up illusion. It's 
the latest fashion on reddit all right, but it has its own problems, and 
the silent majority are still using the older methods.

Not to say it's a bad project (I'll try to migrate my authored packages 
to it one day, since it seems to make that natural), but for an average 
user? I don't think so.

> If nothing more happens on Emacs 29, which is the most likely
> outcome, I think I will just start recommending "straight.el"
> more, maybe learn it myself.  I had problems with it in the
> past but it's probably matured now.

I would suggest to start recommending ways to perform an upgrade 
explicitly (somewhere in the README, in the manual, and so on). Like 
'M-x package-upgrade', if we manage to get it fixed in Emacs 29.

And, okay, having a separate section in README about Emacs 29 might suck 
at first, but then the majority of the users move to it, and support 
will get easier, more uniform.

>> So... I understand the problem, but I think we shouldn't change the
>> functions in a way that makes them conflict with documentation or with
>> each other.
> 
> Does the latest patch introduce any such conflict besides the
> "no internet" case you mentioned?  Not that it's really a
> conflict IMO: it's just what Emacs 28 did with Eglot AFAICT.

That's the only hard breakage I managed to come up with. The others are 
of logical kind (functions and commands doing things somewhat against 
their documented behavior), but that's also not so great because people 
like their tools to make sense.

To name an example: 'package-install' will upgrade Eglot even though 
(package-installed-p 'eglot) already returns t.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 2:13 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> >> because the package can't be installed (no connection). Should they
> >> remove ":ensure t"? Perhaps. But the documentation says that that option
> >> checks that the package is "installed automatically if not already
> >> present on your system". Seems legit, right? Why would startup fail when
> >> Eglot is already present on the system?
> >
> > Because we want it to do the same it did in Emacs 28?  But I see what
> > you mean.  Maybe the :ensure for packages in that particular subclass
> > should mean "try a bit harder to upgrade these, but don't blow up if
> > you can't -- just try next time".
>
> ... maybe.

I just looked in straight.el.  I haven't actually tried it, but I noticed
that the only place where it cares about if a package is built-in or not is
in straight--convert-recipe.  And to my (pleasant?) surprise, it is using
it to decide whether to report an error or not if the package is not found.
If it's builtin, the error it not thrown.

> Maybe you'll put the recommended use-package form somewhere in the Eglot
> manual, right? It will be published on the web as well. Will it contain
> ":ensure t" or not? If not, the users of Emacs 28 might fail to get the
> package installed. If it will, then the offline users might have
> problems, as described previously.

To be honest, I'm thinking of either not recommending a package manager
at all, or just recommending a third-party one that I think rhymes
with what Eglot users expect.  Might very well be straight.el or
elpaca.el.  One things is for sure, my previous recommendation
of "use package.el" is not good.

> > to ElDoc and use it in Eglot in the same patch, etc.  That part
> > has worked fine.  It wasn't impossible with Eglot in GitHub, but
> > it was harder.  Hopefully Eli is not asking for that bliss to end
> > I understand hope he's asking to be circumspect in what
> > is added, and not do gratuitous things like bump dependencies
> > without reason.  I already do that, so no problem.
>
> It seems like most of this could have worked out if Eglot moved to the
> emacs.git repository, but was removed from the distro before packaging?
> Just a thought, not that I advocate for doing that.

Then emacs -Q bla.py -f eglot would not work.

And major modes in Emacs could not use eglot.el as a library.

> > Finally, if you're taking all this in the direction of "I told
> > you so, don't you regret bringing Eglot into Emacs 29?", I don't
> > blame you.  But I have to say: no, not yet.  I think having the
> > beginnings of an LSP library in Emacs for major-modes to work
> > with is excellent.  I like the attention and the high-standards
> > other seasoned developers hold my code up to.  I like the Info
> > manual and I like that emacs -Q bla.c -f eglot works just fine.
>
> I like my told-you-so's as much as the next guy (just kidding;
> definitely more than the next guy), but that's not the point I was
> making. Just that it's here now, and it seems odd to avoid using it and
> hurry up to install the next version right away.

I agree.  I don't want users to hurry, but I don't want them to not
hurry either :-).  It depends on what they want.  Eglot 1.12.29 in
Emacs 29 is pretty good.

> > This upgrade situation sucks, is true, and I should have foreseen
> > it.  Spent too much time in my emacs-master worktree I guess.
> >
> > Who knows, maybe this will all be OK after all?? Half the
> > users are using "straight.el" anyway (and I really doubt
> > it is hampered by these minutiae about dependencies, seeing
> > as MELPA-land is teeming with way more "dependency hell").
>
> I'm pretty sure the users on straight.el are straight-up illusion. It's
> the latest fashion on reddit all right, but it has its own problems, and
> the silent majority are still using the older methods.
>
> Not to say it's a bad project (I'll try to migrate my authored packages
> to it one day, since it seems to make that natural), but for an average
> user? I don't think so.

I don't know.  I see a lot of people using it, and have seen that
for a long time.  The average user googles things and lands on reddits
hits.  ChatGPT teaches you "straight.el" too (if you ask it).

> > If nothing more happens on Emacs 29, which is the most likely
> > outcome, I think I will just start recommending "straight.el"
> > more, maybe learn it myself.  I had problems with it in the
> > past but it's probably matured now.
>
> I would suggest to start recommending ways to perform an upgrade
> explicitly (somewhere in the README, in the manual, and so on). Like
> 'M-x package-upgrade', if we manage to get it fixed in Emacs 29.

That's a big if.  My worry here is how to clearly control this
messaging, especially when dealing with cleanly deterministic
bug reports.  I have to know exactly the version that he user
is running.  Recently, I've become adept at doing:

HOME=`mktemp -d` emacs -l recipe.el

It's a very good way to establish sanity.  And until now recipe.el
could have just `(package-install 'eglot)` and I would know exactly
what packages the user has installed.  The answer will now
be different in Emacs 28 vs Emacs 29.  Mind you, I will still
know, of course, but the thing installed on Emacs 28 will be
wildly different than the one installed on Emacs 29.  And,
as I said before and everybody understands, it will get worse with
time.  So what form to give users??

I suggested adding a 'eglot-update' function to eglot.el only --
the only code I normally have control over.  A four-line long function
that I would put it Emacs 29.  It would have this property of
being absolutely consistent in whatever Emacs you're on.
But Eli called it "the worst of all worlds" and flatly refused it.
Maybe he could reconsider, who knows, if all this writing that
has happened since has provided some more perspective about
the problems.

> And, okay, having a separate section in README about Emacs 29 might suck
> at first, but then the majority of the users move to it, and support
> will get easier, more uniform.

Yeah, I guess.  BTW the tea leaves and some comments in the Philip-Eli
interaction tell me that the behaviour is going to again be different in
Emacs 30.  That will suck for a longer time.

> That's the only hard breakage I managed to come up with. The others are
> of logical kind (functions and commands doing things somewhat against
> their documented behavior), but that's also not so great because people
> like their tools to make sense.
>
> To name an example: 'package-install' will upgrade Eglot even though
> (package-installed-p 'eglot) already returns t.

Even with today's patch in Emacs 29 that already happens if you tweak
the new "heavy-handed" variable.  So I guess that isn't a problem.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 04:49, João Távora wrote:

>> Not to say it's a bad project (I'll try to migrate my authored packages
>> to it one day, since it seems to make that natural), but for an average
>> user? I don't think so.
> 
> I don't know.  I see a lot of people using it, and have seen that
> for a long time.  The average user googles things and lands on reddits
> hits.  ChatGPT teaches you "straight.el" too (if you ask it).

I've had problems reported because of outdated straight recipes on 
users' machines. It seems like a fine tool for a hacker, though.

>> I would suggest to start recommending ways to perform an upgrade
>> explicitly (somewhere in the README, in the manual, and so on). Like
>> 'M-x package-upgrade', if we manage to get it fixed in Emacs 29.
> 
> That's a big if.  My worry here is how to clearly control this
> messaging, especially when dealing with cleanly deterministic
> bug reports.  I have to know exactly the version that he user
> is running.  Recently, I've become adept at doing:
> 
> HOME=`mktemp -d` emacs -l recipe.el
> 
> It's a very good way to establish sanity.  And until now recipe.el
> could have just `(package-install 'eglot)` and I would know exactly
> what packages the user has installed.  The answer will now
> be different in Emacs 28 vs Emacs 29.  Mind you, I will still
> know, of course, but the thing installed on Emacs 28 will be
> wildly different than the one installed on Emacs 29.  And,
> as I said before and everybody understands, it will get worse with
> time.  So what form to give users??

If we go for the pinning approach, as Jim suggested, and the fix for 
that one makes it in in time, the form could like like

   (use-package eglot :ensure t :pin gnu)

and hopefully work across 28, 29 and later.

>> To name an example: 'package-install' will upgrade Eglot even though
>> (package-installed-p 'eglot) already returns t.
> 
> Even with today's patch in Emacs 29 that already happens if you tweak
> the new "heavy-handed" variable.  So I guess that isn't a problem.

I'm not sure I like that one either.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 7:07 PM Eli Zaretskii <eliz@gnu.org> wrote:

> It has similar problems: it will automatically update packages
> mentioned in package--safely-upgradeable-builtins, which might not be
> what users want for built-in packages.

It is what they want, by definition, this is why I named it
"safely-upgradeable-builtins".  These are the users:

  Specifically, users of Emacs 28 and older, who had Eglot installed,
  and expect Eglot to be automatically updated upon Emacs startup
  whenever a new Eglot version is available, will now have their
  expectations broken after they upgrade to Emacs 29, because Eglot is
  now a built-in package, and package.el won't by default upgrade a
  built-in package.

Recognize this writing? It is yours!

> You assume that everyone will
> want Eglot and use-package automatically updated, but this assumption
> has no real basis.

First, of course it has real statistical basis! Didn't I send you
links to tens and tens of issues were users reported their configurations
and one can actually see what users are doing to install Eglot?

Secondly, it has the theoretical basis of what you wrote yourself
barely 1 hour ago!  It shows you understand the problem that is
new in Emacs 29.

Using your language, we want to not "break those user's expectations".
if we can.  And we can, if you want to.  You want to, right? You want to
break as few user's expectations as possible, ideally 0.

And the code does exactly that! It avoids bothering that set of
users while also avoiding bothering the other set of users that
you mentioned.

And, for good measure, the set of users who had Eglot installed
and expect Eglot NOT to be updated when package-install is found
is the empty set.  Surely this is evident.

So there's no "dilemma".  There is rather some kind of spectacular
misunderstanding here.  There has to be, because I'm drawing these
conclusions from nothing more than elementary facts from set theory

> Plus, it adds to the maintenance burden of maintaining this
> (internal? not a defcustom??) variable for good.

We can make it a defcustom if you want.  That's exactly
Jim's idea.  I was just trying to make the simplest thing
possible.  This is just for Emacs 29.  You asked for little
code and this is much less code (less cyclomatic complexity,
etc) than what you eventually accepted.  So I was just trying to
cover that front too.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 20:15:20 +0100
> Cc: arne_bab@web.de, jporterbugs@gmail.com, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> On Wed, Apr 19, 2023 at 7:07 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > It has similar problems: it will automatically update packages
> > mentioned in package--safely-upgradeable-builtins, which might not be
> > what users want for built-in packages.
> 
> It is what they want, by definition, this is why I named it
> "safely-upgradeable-builtins".  These are the users:
> 
>   Specifically, users of Emacs 28 and older, who had Eglot installed,
>   and expect Eglot to be automatically updated upon Emacs startup
>   whenever a new Eglot version is available, will now have their
>   expectations broken after they upgrade to Emacs 29, because Eglot is
>   now a built-in package, and package.el won't by default upgrade a
>   built-in package.
> 
> Recognize this writing? It is yours!

Yes.  Of course, you conveniently omitted the next paragraph I wrote,
which described a different group of users, whose expectations would
be broken by the changes you proposed.  That was also "my writing".

> > You assume that everyone will
> > want Eglot and use-package automatically updated, but this assumption
> > has no real basis.
> 
> First, of course it has real statistical basis! Didn't I send you
> links to tens and tens of issues were users reported their configurations
> and one can actually see what users are doing to install Eglot?

Since when "tens" and "everyone" are the same thing?

> Secondly, it has the theoretical basis of what you wrote yourself
> barely 1 hour ago!  It shows you understand the problem that is
> new in Emacs 29.

Of course, I understand the problem.  But understanding the problem
doesn't mean agreement with the solutions you propose, or any other
solution, for that matter.  An acceptable solution should solve the
problem without causing other problems, and in this case it also must
solve the problem in a safe enough manner to be eligible for Emacs
29.1.

> Using your language, we want to not "break those user's expectations".
> if we can.  And we can, if you want to.  You want to, right? You want to
> break as few user's expectations as possible, ideally 0.
> 
> And the code does exactly that! It avoids bothering that set of
> users while also avoiding bothering the other set of users that
> you mentioned.
> 
> And, for good measure, the set of users who had Eglot installed
> and expect Eglot NOT to be updated when package-install is found
> is the empty set.  Surely this is evident.
> 
> So there's no "dilemma".  There is rather some kind of spectacular
> misunderstanding here.  There has to be, because I'm drawing these
> conclusions from nothing more than elementary facts from set theory

It is clear that you like the solution you proposed, and see no
problems with it.  But I disagree, and at this point I have explained
my disagreement enough times.  If you still think I'm wrong, please,
just let's leave this disagreement as it is.  Repeating the same
arguments over and over again just wastes bandwidth and our time.

> > Plus, it adds to the maintenance burden of maintaining this
> > (internal? not a defcustom??) variable for good.
> 
> We can make it a defcustom if you want.  That's exactly
> Jim's idea.  I was just trying to make the simplest thing
> possible.  This is just for Emacs 29.  You asked for little
> code and this is much less code (less cyclomatic complexity,
> etc) than what you eventually accepted.  So I was just trying to
> cover that front too.

I said I'm okay with a defcustom if its default value is nil.

Of course, we already added a defcustom to allow built-in packages to
be updated, so I'm not sure another one is needed.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 10:38 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > It is what they want, by definition, this is why I named it
> > "safely-upgradeable-builtins".  These are the users:
> >
> >   Specifically, users of Emacs 28 and older, who had Eglot installed,
> >   and expect Eglot to be automatically updated upon Emacs startup
> >   whenever a new Eglot version is available, will now have their
> >   expectations broken after they upgrade to Emacs 29, because Eglot is
> >   now a built-in package, and package.el won't by default upgrade a
> >   built-in package.
> >
> > Recognize this writing? It is yours!
>
> Yes.  Of course, you conveniently omitted the next paragraph I wrote,
> which described a different group of users, whose expectations would
> be broken by the changes you proposed.  That was also "my writing".

The code I'm trying to write should appease both.  But you should
help characterize these users.

> > > You assume that everyone will
> > > want Eglot and use-package automatically updated, but this assumption
> > > has no real basis.
> >
> > First, of course it has real statistical basis! Didn't I send you
> > links to tens and tens of issues were users reported their configurations
> > and one can actually see what users are doing to install Eglot?
>
> Since when "tens" and "everyone" are the same thing?

Come on, you know that "everyone" is impossible to prove.  Aren't tens
(actually hundreds, I think) a good data point.  Can you show even
one issue where someone was surprised/harmed by furtive unintented
updates of dependencies?

> > Secondly, it has the theoretical basis of what you wrote yourself
> > barely 1 hour ago!  It shows you understand the problem that is
> > new in Emacs 29.
>
> Of course, I understand the problem.  But understanding the problem
> doesn't mean agreement with the solutions you propose, or any other
> solution, for that matter.  An acceptable solution should solve the
> problem without causing other problems, and in this case it also must
> solve the problem in a safe enough manner to be eligible for Emacs
> 29.1.

And that's what I'm looking for.

> > Using your language, we want to not "break those user's expectations".
> > if we can.  And we can, if you want to.  You want to, right? You want to
> > break as few user's expectations as possible, ideally 0.
> >
> > And the code does exactly that! It avoids bothering that set of
> > users while also avoiding bothering the other set of users that
> > you mentioned.
> >
> > And, for good measure, the set of users who had Eglot installed
> > and expect Eglot NOT to be updated when package-install is found
> > is the empty set.  Surely this is evident.
> >
> > So there's no "dilemma".  There is rather some kind of spectacular
> > misunderstanding here.  There has to be, because I'm drawing these
> > conclusions from nothing more than elementary facts from set theory
>
> It is clear that you like the solution you proposed, and see no
> problems with it.  But I disagree, and at this point I have explained
> my disagreement enough times.

No you haven't, sorry.  You've not said "João, if your patch goes in,
at least user X will have this specific problem Y when she does Z, and
I think Y is bad because reasons".  Dmitry is doing that, critiquing
actual code.  I don't think that's time wasted: it's the only way the
discussion can advance.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 10:48:03 +0100
> Cc: arne_bab@web.de, jporterbugs@gmail.com, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> > > Recognize this writing? It is yours!
> >
> > Yes.  Of course, you conveniently omitted the next paragraph I wrote,
> > which described a different group of users, whose expectations would
> > be broken by the changes you proposed.  That was also "my writing".
> 
> The code I'm trying to write should appease both.  But you should
> help characterize these users.

I have.

> > > > You assume that everyone will
> > > > want Eglot and use-package automatically updated, but this assumption
> > > > has no real basis.
> > >
> > > First, of course it has real statistical basis! Didn't I send you
> > > links to tens and tens of issues were users reported their configurations
> > > and one can actually see what users are doing to install Eglot?
> >
> > Since when "tens" and "everyone" are the same thing?
> 
> Come on, you know that "everyone" is impossible to prove.

I didn't expect you to prove it.  My point is that your proposed
solution is only correct if "everyone" indeed wants to update Eglot,
and if not, your solution helps one group and breaks another.  I
cannot agree to such lopsided solutions.

> Aren't tens (actually hundreds, I think) a good data point.

Not enough to justify breaking the rest.

> Can you show even one issue where someone was surprised/harmed by
> furtive unintented updates of dependencies?

You just heard one of them in this thread.

> > It is clear that you like the solution you proposed, and see no
> > problems with it.  But I disagree, and at this point I have explained
> > my disagreement enough times.
> 
> No you haven't, sorry.

Surer, I did.  You just don't like the explanation, and so you pretend
it doesn't exist.

> I don't think that's time wasted: it's the only way the discussion
> can advance.

I don't think this discussion with you can make any progress.  That
hope was lost long ago, when I first said we should agree to disagree
and leave it at that.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 12:47 PM Eli Zaretskii <eliz@gnu.org> wrote:
> Not enough to justify breaking the rest

But there is no characterization of this rest.

> > Can you show even one issue where someone was surprised/harmed by
> > furtive unintented updates of dependencies?
>
> You just heard one of them in this thread.

I must have missed this. Maybe my spam filter?  Who is this user
and can I interview them to understand how my patch would hurt
them?  Dmitry? Arne? Jim?  The only person who actively looked
at my patch and read it and explained real-world implications
was Dmitry.

> I don't think this discussion with you can make any progress.
> That hope was lost long ago

I'm beginning to agree.  I've also kind of lost hope.  I'll
take the damage and cut the losses with eglot-update.
Thank you for you efforts anyway.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 13:00:23 +0100
> Cc: arne_bab@web.de, jporterbugs@gmail.com, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> Thank you for you efforts anyway.

I cannot thank you, unfortunately.  Not in this discussion.  I'm
deeply hurt by your conduct.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 1:16 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 20 Apr 2023 13:00:23 +0100
> > Cc: arne_bab@web.de, jporterbugs@gmail.com, dmitry@gutov.dev,
> >       emacs-devel@gnu.org
> >
> > Thank you for you efforts anyway.
>
> I cannot thank you, unfortunately.  Not in this discussion.  I'm
> deeply hurt by your conduct.

I spent a lot of hours in this discussion, conducted experiments,
proposed almost countless patches, interviewed people, etc.

So I don't understand but that's OK, can't argue feelings.
I'm very sorry I hurt you somehow, is all I can say.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[John Yates]

On Wed, Apr 19, 2023 at 1:03 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Specifically, users of Emacs 28 and older, who had Eglot installed,
> and expect Eglot to be automatically updated upon Emacs startup
> whenever a new Eglot version is available, will now have their
> expectations broken after they upgrade to Emacs 29, because Eglot is
> now a built-in package, and package.el won't by default upgrade a
> built-in package.
>
> OTOH, the proposal to change package.el so it automatically updates
> built-in packages as well would break the workflows of people who
> don't expect built-in packages to be updated: they would suddenly see
> packages like ElDoc or Xref or Project (and others) being updated
> automatically at startup instead of staying at the version shipped
> with Emacs.

Full disclosure: jumping in without having read much of the prior thread.

Perhaps package.el should allow a user to declare whether (s)he wants
a stable experience or a "rolling release" experience.  In that way the
issue is no longer a function of whether a package is :core or not.

My initial thought was that this would be a single, global toggle.  But I can
also imagine more granular choices (stable, rolling or unspecified) per
package archive and per individual package.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 6:35 PM John Yates <john@yates-sheets.org> wrote:

> My initial thought was that this would be a single, global toggle.
> But I can also imagine more granular choices (stable, rolling or
> unspecified) per package archive and per individual package.

Logical! That global toggle exists now, added very recently.

But, exactly as you say, it should be more granular.  And by
default the value of the switch for Eglot should be "yes, install and
upgrade its dependencies like it happened in Emacs 28".  Whether that is
because Eglot is listed under a special category or just listed
specifically, I don't think is very important.

I already proposed this (but without the toggle) before the
heavy-handed global toggle that forcibly breaks a part of
the users was introduced.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: John Yates <john@yates-sheets.org>
> Date: Wed, 19 Apr 2023 13:35:36 -0400
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>, joaotavora@gmail.com, jporterbugs@gmail.com, 
> 	dmitry@gutov.dev, emacs-devel@gnu.org
> 
> Perhaps package.el should allow a user to declare whether (s)he wants
> a stable experience or a "rolling release" experience.  In that way the
> issue is no longer a function of whether a package is :core or not.

I agree.  But such a change in package.el was impossible when the
pretest already started.  It should, however, be a necessary feature
of package.el.

> My initial thought was that this would be a single, global toggle.  But I can
> also imagine more granular choices (stable, rolling or unspecified) per
> package archive and per individual package.

There's a lot of new turf here to be explored.  I hope we get it done
in time for Emacs 30.  But it wasn't a reasonable alternative for
Emacs 29, because this issue was raised too late, and I don't think it
would be TRT to delay Emacs 29.1 by another 6 months or so.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 10:03 AM, Eli Zaretskii wrote:
> Specifically, users of Emacs 28 and older, who had Eglot installed,
> and expect Eglot to be automatically updated upon Emacs startup
> whenever a new Eglot version is available, will now have their
> expectations broken after they upgrade to Emacs 29, because Eglot is
> now a built-in package, and package.el won't by default upgrade a
> built-in package.

I have some thoughts on this but before I go further, I want to be sure 
I understand the problem. How is the user upgrading their packages in 
this scenario? I installed Eglot 1.14 on Emacs 28 and then tried loading 
up Emacs 29. Then I did the following:

1. Add gnu-devel to 'package-archives' (this way, I can be sure there's 
a newer Eglot to upgrade to in one of the archives)
2. M-x list-packages
3. U  ;; package-menu-mark-upgrades
4. x  ;; package-menu-execute

When Emacs tells me what packages it will upgrade, Eglot is in the list. 
However, ERC (which is in ELPA, but I didn't install via package.el) is 
*not* in the list. Isn't this the behavior we want?[1]

[1] For now, ignore the part where this upgrade changes where we get 
Eglot from GNU ELPA to GNU-devel ELPA. (I think this should be fixed, 
but I don't think it poses an immediate issue for Eglot's problem, since 
Emacs doesn't enable GNU-devel by default.)

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 11:04:50 -0700
> Cc: joaotavora@gmail.com, dmitry@gutov.dev, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 4/19/2023 10:03 AM, Eli Zaretskii wrote:
> > Specifically, users of Emacs 28 and older, who had Eglot installed,
> > and expect Eglot to be automatically updated upon Emacs startup
> > whenever a new Eglot version is available, will now have their
> > expectations broken after they upgrade to Emacs 29, because Eglot is
> > now a built-in package, and package.el won't by default upgrade a
> > built-in package.
> 
> I have some thoughts on this but before I go further, I want to be sure 
> I understand the problem. How is the user upgrading their packages in 
> this scenario? I installed Eglot 1.14 on Emacs 28 and then tried loading 
> up Emacs 29. Then I did the following:
> 
> 1. Add gnu-devel to 'package-archives' (this way, I can be sure there's 
> a newer Eglot to upgrade to in one of the archives)
> 2. M-x list-packages
> 3. U  ;; package-menu-mark-upgrades
> 4. x  ;; package-menu-execute
> 
> When Emacs tells me what packages it will upgrade, Eglot is in the list. 
> However, ERC (which is in ELPA, but I didn't install via package.el) is 
> *not* in the list. Isn't this the behavior we want?[1]

AFAIU, this is not the scenario that João was bothered about.  But I
let him respond.

But if this is the scenario, then there's no problem, AFAIU what you
are saying.  So what exactly would you like to add to this discussion?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 11:34 AM, Eli Zaretskii wrote:
>> Date: Wed, 19 Apr 2023 11:04:50 -0700
>> Cc: joaotavora@gmail.com, dmitry@gutov.dev, emacs-devel@gnu.org
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> 1. Add gnu-devel to 'package-archives' (this way, I can be sure there's
>> a newer Eglot to upgrade to in one of the archives)
>> 2. M-x list-packages
>> 3. U  ;; package-menu-mark-upgrades
>> 4. x  ;; package-menu-execute
>>
>> When Emacs tells me what packages it will upgrade, Eglot is in the list.
>> However, ERC (which is in ELPA, but I didn't install via package.el) is
>> *not* in the list. Isn't this the behavior we want?[1]
> 
> AFAIU, this is not the scenario that João was bothered about.  But I
> let him respond.
> 
> But if this is the scenario, then there's no problem, AFAIU what you
> are saying.  So what exactly would you like to add to this discussion?

Two main things (once I hear back from João to confirm):

1) If there are any package-upgrade actions that *don't* work in the way 
I described, we should fix them, using the behavior of 
'package-menu-mark-upgrades' for guidance. As far as I can tell, that's 
the behavior everyone wants, but there could be other scenarios where it 
does something else.

2) More-generally, there's the question of "stability gradations". 
Elsewhere, you suggested listing these in the *Packages* buffer with 
values like "alpha", "current", "stable", etc. We can already do 
something similar to this with additional package archives (e.g. GNU 
ELPA vs GNU-devel ELPA). However, package.el doesn't automatically keep 
track of which channel you used to install a package, so you have to go 
through a fair amount of extra effort to pin your packages to particular 
release channels.

I think (1) is the immediate concern though, and it might be best to 
have a resolution for that before going too far into general solutions 
like (2).

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 12:35:15 -0700
> Cc: arne_bab@web.de, joaotavora@gmail.com, dmitry@gutov.dev,
>  emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> > But if this is the scenario, then there's no problem, AFAIU what you
> > are saying.  So what exactly would you like to add to this discussion?
> 
> Two main things (once I hear back from João to confirm):
> 
> 1) If there are any package-upgrade actions that *don't* work in the way 
> I described, we should fix them, using the behavior of 
> 'package-menu-mark-upgrades' for guidance. As far as I can tell, that's 
> the behavior everyone wants, but there could be other scenarios where it 
> does something else.
> 
> 2) More-generally, there's the question of "stability gradations". 
> Elsewhere, you suggested listing these in the *Packages* buffer with 
> values like "alpha", "current", "stable", etc. We can already do 
> something similar to this with additional package archives (e.g. GNU 
> ELPA vs GNU-devel ELPA). However, package.el doesn't automatically keep 
> track of which channel you used to install a package, so you have to go 
> through a fair amount of extra effort to pin your packages to particular 
> release channels.
> 
> I think (1) is the immediate concern though, and it might be best to 
> have a resolution for that before going too far into general solutions 
> like (2).

These are serious issues we need to discuss and solve.  But they will
probably require significant changes in package.el, and so are stuff
for Emacs 30, not for the emacs-29 branch.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:

>> But there are too many different tasks to actually check them all every
>> time I update. And I usually don’t choose to do an Emacs update, but
>> just update when my distro ships the new version.
>
> I don't think anyone will argue that breaking changes are good.

I’ve seen that more than once, but I’m glad that that’s not on the table :-)

> Specifically, users of Emacs 28 and older, who had Eglot installed,
> and expect Eglot to be automatically updated upon Emacs startup
> whenever a new Eglot version is available, will now have their
> expectations broken after they upgrade to Emacs 29, because Eglot is
> now a built-in package, and package.el won't by default upgrade a
> built-in package.
…
> So there's a dilemma here: which of the two groups of users to break?

Not updating eglot until the next Emacs release shouldn’t cause breakage
in any other packages, right?

Except if a more modern eglot is a dependency of a non-built-in package.

I think that’s what I would prefer: I would treat being pulled into
Emacs as a stabilization step that switches the package from being on
the latest version to being at the version in Emacs or the minimum
version required by dependencies — if that version is higher than the
version in Emacs. Basically minimize the distance from the Emacs
release.

But that’s just my position — thank you for working to find a solution
that works best for most!

> And that is the real issue here, at least from my POV.

Thank you for the clarification! I had thought that I had read enough of
the thread to understand the context … thank you for pulling me in
nonetheless!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, dmitry@gutov.dev,
>  emacs-devel@gnu.org
> Date: Wed, 19 Apr 2023 21:40:35 +0200
> 
> > Specifically, users of Emacs 28 and older, who had Eglot installed,
> > and expect Eglot to be automatically updated upon Emacs startup
> > whenever a new Eglot version is available, will now have their
> > expectations broken after they upgrade to Emacs 29, because Eglot is
> > now a built-in package, and package.el won't by default upgrade a
> > built-in package.
> …
> > So there's a dilemma here: which of the two groups of users to break?
> 
> Not updating eglot until the next Emacs release shouldn’t cause breakage
> in any other packages, right?

No, it shouldn't.  With the obvious exception of the breakage that is
already part of the current Emacs release, which we somehow failed to
detect and fix before releasing it.

> Except if a more modern eglot is a dependency of a non-built-in package.

Right.

> I think that’s what I would prefer: I would treat being pulled into
> Emacs as a stabilization step that switches the package from being on
> the latest version to being at the version in Emacs or the minimum
> version required by dependencies — if that version is higher than the
> version in Emacs. Basically minimize the distance from the Emacs
> release.

AFAIU, that is what will happen with Emacs 29 when it is released.
But things might change in Emacs 30, as we are currently discussing
what needs to be done to better support updating core packages.

Re: Stability of core packages

[emacs]

I have been following this thread from a meta perspective.

The essence of the difficulty is that we are
following an established pattern that has various
weaknesses and we are now trying to adjust that pattern.

What I am suggesting breaks the existing
model. Let me start with some context.

Because we use git for everything and because it
is possible to take complete git snap shots of all
relevant packages, it is possible to easily
provide stability for the user based on snapshots.

Consistent upgrades are then possible for new
consistent git snapshots. So, based on the snapshots
the packages are not stale and remain consistent
while evolving.

This is what doomemacs does. For every package,
there is a git tag which doomemacs keeps with the
adoption of the package. The tags are guaranteed
to be consistent. With Blee (ByStar Libre Emacs
Environment) I do the same but instead of keeping
that tag with the packages, I keep central
manifests for emacs releases. So, emacs-28.1 has
its own packages manifest which can be upgraded
from time to time. All upgrades go to the
appropriate repo and get the appropriate tag based
on the manifest. I do this across all emacs
package archives -- which I see as collections of
git repositories.

If you want to go with something like this, you
have to revisit the package retrieval model.

For every release of emacs, you maintain a
separate manifest. And you keep updating that at
new releases and even in between releases. Since
all emacs archives are git based, it is possible
to apply this strategy to all of them -- where
each take care of its own consistency.

In practice, this has already happened for layers
that sit on top of emacs (doomemacs, blee). It is
a matter of adopting the higher layer model in the
core. There are already many variations on the
model of packages.el.

If you want to go with this model, much needs to
be revisited. But much will be gained.

Again, these comments only deal with the meta
topic not the at hand  Eglot topic. But, it shows
how eglot upgrades could have rolled back to
emacs-28.

...Mohsen


Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
>> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, dmitry@gutov.dev,
>>  emacs-devel@gnu.org
>> Date: Wed, 19 Apr 2023 21:40:35 +0200
>>
>> > Specifically, users of Emacs 28 and older, who had Eglot installed,
>> > and expect Eglot to be automatically updated upon Emacs startup
>> > whenever a new Eglot version is available, will now have their
>> > expectations broken after they upgrade to Emacs 29, because Eglot is
>> > now a built-in package, and package.el won't by default upgrade a
>> > built-in package.
>> …
>> > So there's a dilemma here: which of the two groups of users to break?
>>
>> Not updating eglot until the next Emacs release shouldn’t cause breakage
>> in any other packages, right?
>
> No, it shouldn't.  With the obvious exception of the breakage that is
> already part of the current Emacs release, which we somehow failed to
> detect and fix before releasing it.
>
>> Except if a more modern eglot is a dependency of a non-built-in package.
>
> Right.
>
>> I think that’s what I would prefer: I would treat being pulled into
>> Emacs as a stabilization step that switches the package from being on
>> the latest version to being at the version in Emacs or the minimum
>> version required by dependencies — if that version is higher than the
>> version in Emacs. Basically minimize the distance from the Emacs
>> release.
>
> AFAIU, that is what will happen with Emacs 29 when it is released.
> But things might change in Emacs 30, as we are currently discussing
> what needs to be done to better support updating core packages.

Re: Stability of core packages

[Eli Zaretskii]

> From: emacs@mohsen.1.banan.byname.net
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>,  joaotavora@gmail.com,
>   jporterbugs@gmail.com,  dmitry@gutov.dev,  emacs-devel@gnu.org
> Date: Fri, 28 Apr 2023 22:21:46 -0700
> 
> 
> I have been following this thread from a meta perspective.

Thanks for chiming in.

> Because we use git for everything and because it
> is possible to take complete git snap shots of all
> relevant packages, it is possible to easily
> provide stability for the user based on snapshots.
> 
> Consistent upgrades are then possible for new
> consistent git snapshots. So, based on the snapshots
> the packages are not stale and remain consistent
> while evolving.
> 
> This is what doomemacs does. For every package,
> there is a git tag which doomemacs keeps with the
> adoption of the package. The tags are guaranteed
> to be consistent. With Blee (ByStar Libre Emacs
> Environment) I do the same but instead of keeping
> that tag with the packages, I keep central
> manifests for emacs releases. So, emacs-28.1 has
> its own packages manifest which can be upgraded
> from time to time. All upgrades go to the
> appropriate repo and get the appropriate tag based
> on the manifest. I do this across all emacs
> package archives -- which I see as collections of
> git repositories.
> 
> If you want to go with something like this, you
> have to revisit the package retrieval model.
> 
> For every release of emacs, you maintain a
> separate manifest. And you keep updating that at
> new releases and even in between releases. Since
> all emacs archives are git based, it is possible
> to apply this strategy to all of them -- where
> each take care of its own consistency.

I've read this several times, and I admit I still don't really
understand the suggestion.  Maybe it's because you are using some
terminology without explaining it in enough details, perhaps because
you assume prior knowledge of what it means.

Here are some questions for which I didn't find answers, and which
precluded me from following your suggestions:

  . what is a "git snapshot"? what does it include in the context of
    this discussion?
  . what is meant by "upgrades for new consistent git snapshots"? how
    is such an upgrade done, and how does it work in practice?
  . what are those git tags which doomemacs keeps with the packages?
    what does each such tag indicate, and how are these tags
    maintained and updated?
  . what does it mean for a git tag to be "consistent"?
  . what are those "central manifests" you describe for Blee? what
    does each manifest contain and how is it used?
  . wrt your proposal of keeping separate manifests for each Emacs
    release, how will a manifest for a specific release be updated,
    and where will it be kept and made available for users?

Re: Stability of core packages

[Mohsen BANAN]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: emacs@mohsen.1.banan.byname.net
>> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>,  joaotavora@gmail.com,
>>   jporterbugs@gmail.com,  dmitry@gutov.dev,  emacs-devel@gnu.org
>> Date: Fri, 28 Apr 2023 22:21:46 -0700
>>
>>
>> I have been following this thread from a meta perspective.
>
> Thanks for chiming in.
>
>> Because we use git for everything and because it
>> is possible to take complete git snap shots of all
>> relevant packages, it is possible to easily
>> provide stability for the user based on snapshots.
>>
>> Consistent upgrades are then possible for new
>> consistent git snapshots. So, based on the snapshots
>> the packages are not stale and remain consistent
>> while evolving.
>>
>> This is what doomemacs does. For every package,
>> there is a git tag which doomemacs keeps with the
>> adoption of the package. The tags are guaranteed
>> to be consistent. With Blee (ByStar Libre Emacs
>> Environment) I do the same but instead of keeping
>> that tag with the packages, I keep central
>> manifests for emacs releases. So, emacs-28.1 has
>> its own packages manifest which can be upgraded
>> from time to time. All upgrades go to the
>> appropriate repo and get the appropriate tag based
>> on the manifest. I do this across all emacs
>> package archives -- which I see as collections of
>> git repositories.
>>
>> If you want to go with something like this, you
>> have to revisit the package retrieval model.
>>
>> For every release of emacs, you maintain a
>> separate manifest. And you keep updating that at
>> new releases and even in between releases. Since
>> all emacs archives are git based, it is possible
>> to apply this strategy to all of them -- where
>> each take care of its own consistency.
>
> I've read this several times, and I admit I still don't really
> understand the suggestion.  Maybe it's because you are using some
> terminology without explaining it in enough details, perhaps because
> you assume prior knowledge of what it means.

Sorry about that.

In fact, that is the case. I was assuming that you
and others would be familiar with the lessons
learned from the "straight.el" and doom emacs
efforts over the past many years. My apologies.

To better set the context, let me start with a
quote from:
https://github.com/doomemacs/doomemacs/blob/master/docs/faq.org

-----------
Why does Doom use straight.el and not package.el?

package.el simply doesn’t cut it. Its flaws become
apparent the more packages you manage, the more
complex your config becomes, and how often those
packages see updates:

- No Disaster recovery
...

- Rolling release or bust

package.el installs the latest version of every
package, or none at all. There’s no rollback, no
pinning to a stable ref, and no git history to
peek into. It offers little reproducibility;
wiping/losing/copying your config becomes a gamble
if you aren’t constantly backing up your packages
(then recompiling them if you’re up/downgrading
Emacs). melpa-stable was well intentioned, but it
offers no guarantees or standards on how/when
maintainers tag their projects. I’d rather the
user decide for themselves, because they’re the
ones in the trenches.

...

Package management needs to be easier, because
Emacs is hard enough already. Doom fills these
gaps with straight.el’s help, and beyond.
------------

I am not saying that everything that Henrik
Lissner is saying is correct. But I also think
that the model of package.el is inadequate and
that it should be replaced by something like
straight.el.

The readme of
https://github.com/radian-software/straight.el
has a Guiding principles section.

As João Távora observes, straight.el can be a
solution for the eglot use case:

João Távora> Who knows, maybe this will all be OK after all?? Half the
João Távora> users are using "straight.el" anyway (and I really doubt
João Távora> it is hampered by these minutiae about dependencies, seeing
João Távora> as MELPA-land is teeming with way more "dependency hell").

>
> Here are some questions for which I didn't find answers, and which
> precluded me from following your suggestions:
>
>   . what is a "git snapshot"? what does it include in the context of
>     this discussion?

By a "git snapshot" I mean a collection of Git
commit IDs which can be used as pins.

In the doom model, with streight.el and with the
eglot use case, consider:

(package! eglot :pin "e501275e06952889056268dabe08ccd0dbaf23e5")

https://github.com/doomemacs/doomemacs/blob/master/modules/tools/lsp/packages.el

That is an example of a git snapshot.


>   . what is meant by "upgrades for new consistent git snapshots"? how
>     is such an upgrade done, and how does it work in practice?

New collection of Git commit IDs can then become
basis for upgrades.

>   . what are those git tags which doomemacs keeps with the packages?
>     what does each such tag indicate, and how are these tags
>     maintained and updated?

In the context of evolution of say ELPA, I am
suggesting use of the model of streight.el.

And its use based on the manifest model -- not
doom's package adoption model.

For each release of emacs, emacs{28,29,30}
the archive will maintain a manifest-{28,29,30}.

Each of those files enumerates a list of Git
commit IDs for each of the packages in the
archive. An update to the Git commit ID becomes a
basis for a potential upgrade.

>   . what does it mean for a git tag to be "consistent"?

It means that it is known to work well with all
other Git commit IDs in that manifest.

>   . what are those "central manifests" you describe for Blee? what
>     does each manifest contain and how is it used?

See manifest-{28,29,30} above.

>   . wrt your proposal of keeping separate manifests for each Emacs
>     release, how will a manifest for a specific release be updated,
>     and where will it be kept and made available for users?

The manifests are an internal part of the archive.
Users can see them, but they need not.

Maintenance of a manifests is the work of the ELPA
maintainers. They need to be subjected to
regression tests/verification's that guarantee
consistent sets.

I think what I am saying above reflects current
ad-doc practices of many straight.el users and
reflects a natural evolution direction for
package.el.

Eli, if you have not used straight.el, I suggest
experimenting with it. And again sorry about
having been overly cryptic.

...Mohsen

Re: Stability of core packages

[Eli Zaretskii]

> From: Mohsen BANAN <emacs@mohsen.1.banan.byname.net>
> Cc: emacs-devel@gnu.org, João Távora
>  <joaotavora@gmail.com>
> Date: Sat, 29 Apr 2023 14:47:51 -0700
> 
> >
> > I've read this several times, and I admit I still don't really
> > understand the suggestion.  Maybe it's because you are using some
> > terminology without explaining it in enough details, perhaps because
> > you assume prior knowledge of what it means.
> 
> Sorry about that.
> 
> In fact, that is the case. I was assuming that you
> and others would be familiar with the lessons
> learned from the "straight.el" and doom emacs
> efforts over the past many years. My apologies.
> [...]

Thanks.  I think I understand now what you are proposing.

However, this is not what the current discussion thread was about.
You are describing the technical means of tracking the collections of
versions/snapshots/commits of packages that are known to work well
with a given release of Emacs.  By contrast, the issue I raised, which
I think is a prerequisite for selecting any such technical means, was:
what are the levels of "package stability" we want to support, and
what are the criteria for each supported level?  Once we have that
figured out and agreed-upon, we can proceed to discussing how to
support those stability levels from the technical POV.

> Eli, if you have not used straight.el, I suggest
> experimenting with it.

Sorry, I cannot afford that: not enough time left, after all my other
duties here are done.

But I don't think I need to do that in order to understand what you
are proposing.  For that matter, I don't really see a significant
difference between explicit versioning of package releases and using
commits/tags for the same purpose.  After all, a version is just a tag
in the repository, i.e. a label of a commit.  So they are all
equivalent.  And since in practice the only source of stability data
of a package is the maintainer(s) of that package, the reason for
using commits instead of release version tags seems weak to me;
moreover, it assumes some significant team of people unrelated to any
package that keeps track of stability and dependencies of a large
number of packages, something that IMO is impractical for Emacs.

But as I said: these are secondary issues, from my POV.  We are not
yet at a point where these issues are of any practical importance for
our package system.  And one more thing to keep in mind: these issues
are from my POV most important for the so-called "core" packages,
which are those that need to be shipped as part of the Emacs release
tarball, but are developed and maintained outside of emacs.git.  None
of the package systems you mention have ever faced and handled this
situation (with the possible exception of XEmacs, years ago, but we
all know where that went...).

Re: Stability of core packages

[Philip Kaludercic]

Mohsen BANAN <emacs@mohsen.1.banan.byname.net> writes:

[...]

> Eli, if you have not used straight.el, I suggest
> experimenting with it. And again sorry about
> having been overly cryptic.

I have never used straight.el either (mainly because of the need
aggressive incompatibility with package.el and my dislike of having a
big bootstrap-blob at the beginning of my init), but I have been working
on package-vc that a lot of people have apparently been using as a
replacement for package.el.  Recently there has been some work on
supporting arbitrary build-commands that could be executed alongside the
package compilation.  I have been told that this is a major feature
difference between the current state of package-vc and the
straight/elpaca family of package managers.  As you seem to have some
experience with the topic, do you believe this could be helpful to
address your issue?  I am uncertain if this is something worth investing
an effort into, as the issues that Emacs lisp packaging struggles with
are not only those of finding a set of compatible revisions.

Re: Stability of core packages

[Corwin Brust]

On Sat, Apr 29, 2023 at 4:47 PM Mohsen BANAN
<emacs@mohsen.1.banan.byname.net> wrote:
>
> >> This is what doomemacs does. For every package,
> >> there is a git tag which doomemacs keeps with the
> >> adoption of the package. The tags are guaranteed
> >> to be consistent. With Blee (ByStar Libre Emacs
> >> Environment) I do the same but instead of keeping
> >> that tag with the packages, I keep central
> >> manifests for emacs releases. So, emacs-28.1 has
> >> its own packages manifest which can be upgraded
> >> from time to time. All upgrades go to the
> >> appropriate repo and get the appropriate tag based
> >> on the manifest. I do this across all emacs
> >> package archives -- which I see as collections of
> >> git repositories.
> >>
> >> If you want to go with something like this, you
> >> have to revisit the package retrieval model.

[SNIP]

> -----------
> Why does Doom use straight.el and not package.el?
>
> package.el simply doesn’t cut it. Its flaws become
> apparent the more packages you manage, the more
> complex your config becomes, and how often those
> packages see updates:

This is the same doomemacs project[1] that does all development
against HEAD of master, correct?    I track three Emacs branches, all
of which see frequent pushes. Work is regularly merged back to trunk.
I don't see anything like this level of practice maturity from doom
(or straight), so I think the lesson here isn't technical, but
something more like:

1. For powerusers, package.el isn't better than manually tracking
package versions.
2. Tracking package versions is so difficult that Emacs
re-distributers find it worthwhile to hard-code working combinations.

I think the issue is the packaging landscape has matured significantly
thus it is time to reconsider the uses-cases Emacs should support -
and that's the point of this thread.

Meanwhile, I think the practical upshot of the approach taken by doom
(and straight.el, in general) is to push tracking what versions of
which packages work together to individual uses.  I feel it would be a
step backward for Emacs to adopt this approach as the default,
expecting all users to learn to do it.  User-managed explicit package
version pinning does seem like a cool thing to support -- I hope
package-vc helps pave the way for core support for that use-case.  But
it should not be the default: I would be sad if we decide to expect
new users to master package pinning to try out new features.

[1] https://github.com/doomemacs/doomemacs/branches

Re: Stability of core packages

[Mohsen BANAN]

Corwin Brust <corwin@bru.st> writes:

> On Sat, Apr 29, 2023 at 4:47 PM Mohsen BANAN
> <emacs@mohsen.1.banan.byname.net> wrote:
>> ...
>> >> If you want to go with something like this, you
>> >> have to revisit the package retrieval model.
>
> [SNIP]
>
>> -----------
>> Why does Doom use straight.el and not package.el?
>>
>> package.el simply doesn’t cut it. Its flaws become
>> apparent the more packages you manage, the more
>> complex your config becomes, and how often those
>> packages see updates:

> This is the same doomemacs project[1] that does all development
> against HEAD of master, correct?    I track three Emacs branches, all
> of which see frequent pushes. Work is regularly merged back to trunk.

My mentioning doomemacs was not an endorsement of
doomemacs ...

> so I think the lesson here isn't technical, but
> something more like:
>
> 1. For powerusers, package.el isn't better than manually tracking
> package versions.
> 2. Tracking package versions is so difficult that Emacs
> re-distributers find it worthwhile to hard-code working combinations.
>
> I think the issue is the packaging landscape has matured significantly
> thus it is time to reconsider the uses-cases Emacs should support -
> and that's the point of this thread.

Exactly.

> Meanwhile, I think the practical upshot of the approach taken by doom
> (and straight.el, in general) is to push tracking what versions of
> which packages work together to individual uses.  I feel it would be a
> step backward for Emacs to adopt this approach as the default,
> expecting all users to learn to do it.  User-managed explicit package
> version pinning does seem like a cool thing to support -- I hope
> package-vc helps pave the way for core support for that use-case.  But
> it should not be the default: I would be sad if we decide to expect
> new users to master package pinning to try out new features.

That is not what I am saying or suggesting.

Over the past several years, we have learned
various things from:

 1) the packaging landscape
 2) package archive providers
 3) Emacs re-distributers

Right now, in practice, Emacs re-distributers
maintain the stable manifest of the matching sets
for each release of emacs.

I am suggesting two things:

1) package.el should evolve or be replaced with
   something that is the equivalent of
   straight.el.

2) The responsibility for maintaining a stable
matching set of packages for each release of emacs
should come down to package archive providers (not
the Emacs re-distributers). Package archive
providers should maintain stable and consistent
manifests. Emacs re-distributers and emacs
end-users can selectively overwrite these, but
that won't be the default.

The challenge here is that in the existing emacs
ecosystem, basic emacs (emacs+elpa), package
archive providers and emacs re-distributers have
no structure to facilitate convergence.

If basic emacs (emacs+elpa) could solve this
properly, the rest could have a pattern to
emulate. For that to happen, basic emacs needs to
learn the lessons of emacs re-distributers. That
was why I mentioned that experimenting with
straight.el is a good idea. Not as a capability,
but as a basis for existing policies and
practices.

...Mohsen

Re: Stability of core packages

[David Masterson]

emacs@mohsen.1.banan.byname.net writes:

> What I am suggesting breaks the existing
> model. Let me start with some context.
>
> Because we use git for everything and because it
> is possible to take complete git snap shots of all
> relevant packages, it is possible to easily
> provide stability for the user based on snapshots.
>
> Consistent upgrades are then possible for new
> consistent git snapshots. So, based on the snapshots
> the packages are not stale and remain consistent
> while evolving.
>
> This is what doomemacs does. For every package,
> there is a git tag which doomemacs keeps with the
> adoption of the package. The tags are guaranteed
> to be consistent. With Blee (ByStar Libre Emacs
> Environment) I do the same but instead of keeping
> that tag with the packages, I keep central
> manifests for emacs releases. So, emacs-28.1 has
> its own packages manifest which can be upgraded
> from time to time. All upgrades go to the
> appropriate repo and get the appropriate tag based
> on the manifest. I do this across all emacs
> package archives -- which I see as collections of
> git repositories.
>
> If you want to go with something like this, you
> have to revisit the package retrieval model.
>
> For every release of emacs, you maintain a
> separate manifest. And you keep updating that at
> new releases and even in between releases. Since
> all emacs archives are git based, it is possible
> to apply this strategy to all of them -- where
> each take care of its own consistency.
>
> In practice, this has already happened for layers
> that sit on top of emacs (doomemacs, blee). It is
> a matter of adopting the higher layer model in the
> core. There are already many variations on the
> model of packages.el.
>
> If you want to go with this model, much needs to
> be revisited. But much will be gained.
>
> Again, these comments only deal with the meta
> topic not the at hand  Eglot topic. But, it shows
> how eglot upgrades could have rolled back to
> emacs-28.
>
> ...Mohsen
>
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
>>> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, dmitry@gutov.dev,
>>>  emacs-devel@gnu.org
>>> Date: Wed, 19 Apr 2023 21:40:35 +0200
>>>
>>> > Specifically, users of Emacs 28 and older, who had Eglot installed,
>>> > and expect Eglot to be automatically updated upon Emacs startup
>>> > whenever a new Eglot version is available, will now have their
>>> > expectations broken after they upgrade to Emacs 29, because Eglot is
>>> > now a built-in package, and package.el won't by default upgrade a
>>> > built-in package.
>>> …
>>> > So there's a dilemma here: which of the two groups of users to break?
>>>
>>> Not updating eglot until the next Emacs release shouldn’t cause breakage
>>> in any other packages, right?
>>
>> No, it shouldn't.  With the obvious exception of the breakage that is
>> already part of the current Emacs release, which we somehow failed to
>> detect and fix before releasing it.
>>
>>> Except if a more modern eglot is a dependency of a non-built-in package.
>>
>> Right.
>>
>>> I think that’s what I would prefer: I would treat being pulled into
>>> Emacs as a stabilization step that switches the package from being on
>>> the latest version to being at the version in Emacs or the minimum
>>> version required by dependencies — if that version is higher than the
>>> version in Emacs. Basically minimize the distance from the Emacs
>>> release.
>>
>> AFAIU, that is what will happen with Emacs 29 when it is released.
>> But things might change in Emacs 30, as we are currently discussing
>> what needs to be done to better support updating core packages.
>

-- 
David Masterson

Re: Stability of core packages

[David Masterson]

emacs@mohsen.1.banan.byname.net writes:

> What I am suggesting breaks the existing
> model. Let me start with some context.
>
> Because we use git for everything and because it
> is possible to take complete git snap shots of all
> relevant packages, it is possible to easily
> provide stability for the user based on snapshots.
>
> Consistent upgrades are then possible for new
> consistent git snapshots. So, based on the snapshots
> the packages are not stale and remain consistent
> while evolving.

[...]

> If you want to go with something like this, you
> have to revisit the package retrieval model.
>
> For every release of emacs, you maintain a
> separate manifest. And you keep updating that at
> new releases and even in between releases. Since
> all emacs archives are git based, it is possible
> to apply this strategy to all of them -- where
> each take care of its own consistency.
>
> In practice, this has already happened for layers
> that sit on top of emacs (doomemacs, blee). It is
> a matter of adopting the higher layer model in the
> core. There are already many variations on the
> model of packages.el.

How would Emacs deal with those system layers.  In theory, Emacs
packages could depend on packages outside of the Emacs layer (system
libraries or other language tools).  My thought is that you'd either
have to be explicit all the way up the tree (making it a logistical
nightmare) or you'd have to prune the tree (making the definition of a
release intentionally fuzzy).

Currently, the model is that each package (supposedly) calls out it's
package version dependencies that it has been tested with and assumes
that later versions are compatible.  Isn't the above approach going to
devolve into this?

How does doom handle it?

-- 
David Masterson

Re: Stability of core packages

[David Masterson]

David Masterson <dsmasterson@gmail.com> writes:

oops ignore this 

> emacs@mohsen.1.banan.byname.net writes:
>
>> What I am suggesting breaks the existing
>> model. Let me start with some context.
>>
>> Because we use git for everything and because it
>> is possible to take complete git snap shots of all
>> relevant packages, it is possible to easily
>> provide stability for the user based on snapshots.
>>
>> Consistent upgrades are then possible for new
>> consistent git snapshots. So, based on the snapshots
>> the packages are not stale and remain consistent
>> while evolving.
>>
>> This is what doomemacs does. For every package,
>> there is a git tag which doomemacs keeps with the
>> adoption of the package. The tags are guaranteed
>> to be consistent. With Blee (ByStar Libre Emacs
>> Environment) I do the same but instead of keeping
>> that tag with the packages, I keep central
>> manifests for emacs releases. So, emacs-28.1 has
>> its own packages manifest which can be upgraded
>> from time to time. All upgrades go to the
>> appropriate repo and get the appropriate tag based
>> on the manifest. I do this across all emacs
>> package archives -- which I see as collections of
>> git repositories.
>>
>> If you want to go with something like this, you
>> have to revisit the package retrieval model.
>>
>> For every release of emacs, you maintain a
>> separate manifest. And you keep updating that at
>> new releases and even in between releases. Since
>> all emacs archives are git based, it is possible
>> to apply this strategy to all of them -- where
>> each take care of its own consistency.
>>
>> In practice, this has already happened for layers
>> that sit on top of emacs (doomemacs, blee). It is
>> a matter of adopting the higher layer model in the
>> core. There are already many variations on the
>> model of packages.el.
>>
>> If you want to go with this model, much needs to
>> be revisited. But much will be gained.
>>
>> Again, these comments only deal with the meta
>> topic not the at hand  Eglot topic. But, it shows
>> how eglot upgrades could have rolled back to
>> emacs-28.
>>
>> ...Mohsen
>>
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
>>>> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, dmitry@gutov.dev,
>>>>  emacs-devel@gnu.org
>>>> Date: Wed, 19 Apr 2023 21:40:35 +0200
>>>>
>>>> > Specifically, users of Emacs 28 and older, who had Eglot installed,
>>>> > and expect Eglot to be automatically updated upon Emacs startup
>>>> > whenever a new Eglot version is available, will now have their
>>>> > expectations broken after they upgrade to Emacs 29, because Eglot is
>>>> > now a built-in package, and package.el won't by default upgrade a
>>>> > built-in package.
>>>> …
>>>> > So there's a dilemma here: which of the two groups of users to break?
>>>>
>>>> Not updating eglot until the next Emacs release shouldn’t cause breakage
>>>> in any other packages, right?
>>>
>>> No, it shouldn't.  With the obvious exception of the breakage that is
>>> already part of the current Emacs release, which we somehow failed to
>>> detect and fix before releasing it.
>>>
>>>> Except if a more modern eglot is a dependency of a non-built-in package.
>>>
>>> Right.
>>>
>>>> I think that’s what I would prefer: I would treat being pulled into
>>>> Emacs as a stabilization step that switches the package from being on
>>>> the latest version to being at the version in Emacs or the minimum
>>>> version required by dependencies — if that version is higher than the
>>>> version in Emacs. Basically minimize the distance from the Emacs
>>>> release.
>>>
>>> AFAIU, that is what will happen with Emacs 29 when it is released.
>>> But things might change in Emacs 30, as we are currently discussing
>>> what needs to be done to better support updating core packages.
>>

-- 
David Masterson

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 09:50:12 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, Dmitry Gutov <dmitry@gutov.dev>,
>  emacs-devel@gnu.org
> 
> On Tue, Apr 18, 2023 at 7:57 PM Jim Porter <jporterbugs@gmail.com> wrote:
> >
> > * Stable: the version of a package included in the latest Emacs tarball
> > * Latest: the latest version on GNU ELPA (etc)
> > * Devel: the latest version on GNU-devel ELPA (etc)
> >
> > You could possibly add:
> >
> > * Core(?): the version of a package included in the tarball of the
> > user's *current* Emacs installation
> 
> These are interesting levels, but I was under the impression
> that the goal is to partition the set of :core packages according
> to some kind of gradation, what Eli called "stability gradation".
> The set can be found in the variable package--builtins.

No you've misunderstood what I mean by "stability gradation".  It has
nothing to do with the value of package--builtins, even when that is
non-nil.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 1:55 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 19 Apr 2023 09:50:12 +0100
> > Cc: Eli Zaretskii <eliz@gnu.org>, Dmitry Gutov <dmitry@gutov.dev>,
> >  emacs-devel@gnu.org
> >
> > On Tue, Apr 18, 2023 at 7:57 PM Jim Porter <jporterbugs@gmail.com> wrote:
> > >
> > > * Stable: the version of a package included in the latest Emacs tarball
> > > * Latest: the latest version on GNU ELPA (etc)
> > > * Devel: the latest version on GNU-devel ELPA (etc)
> > >
> > > You could possibly add:
> > >
> > > * Core(?): the version of a package included in the tarball of the
> > > user's *current* Emacs installation
> >
> > These are interesting levels, but I was under the impression
> > that the goal is to partition the set of :core packages according
> > to some kind of gradation, what Eli called "stability gradation".
> > The set can be found in the variable package--builtins.
>
> No you've misunderstood what I mean by "stability gradation".  It has
> nothing to do with the value of package--builtins, even when that is
> non-nil.

OK, but you wrote:

  IOW, shouldn't packages have some "stability gradation" that is
  visible when users look at the list of packages via package.el?

By "have some" I interpreted that you wanted to map a given
characteristic to each package.   And by "packages" I interpreted
that you meant the packages that the Emacs project has control
over.  And those packages are, by definition, the ":core" or
"builtin" packages (although if the Emacs project is taken to
include ELPA .git, it will also include all the packages there).

If I've misunderstood, I think you could give examples of
real or made up packages and assign a "stability gradation" to
them and define what that gradation means.

Then explain what you want to do with that stability gradation
in Emacs (if anything, perhaps you just want to show it).

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 14:18:21 +0100
> Cc: jporterbugs@gmail.com, dmitry@gutov.dev, emacs-devel@gnu.org
> 
> > > These are interesting levels, but I was under the impression
> > > that the goal is to partition the set of :core packages according
> > > to some kind of gradation, what Eli called "stability gradation".
> > > The set can be found in the variable package--builtins.
> >
> > No you've misunderstood what I mean by "stability gradation".  It has
> > nothing to do with the value of package--builtins, even when that is
> > non-nil.
> 
> OK, but you wrote:
> 
>   IOW, shouldn't packages have some "stability gradation" that is
>   visible when users look at the list of packages via package.el?

Yes.

> By "have some" I interpreted that you wanted to map a given
> characteristic to each package.   And by "packages" I interpreted
> that you meant the packages that the Emacs project has control
> over.  And those packages are, by definition, the ":core" or
> "builtin" packages (although if the Emacs project is taken to
> include ELPA .git, it will also include all the packages there).
> 
> If I've misunderstood, I think you could give examples of
> real or made up packages and assign a "stability gradation" to
> them and define what that gradation means.

In the "list-packages" display, instead of

  eglot       1.14    available  gnu   The Emacs Client for LSP servers

I envisioned seeing something like

  eglot       2.01    alpha      gnu   The Emacs Client for LSP servers
  eglot       1.20    current    gnu   The Emacs Client for LSP servers
  eglot       1.15    stable     gnu   The Emacs Client for LSP servers
  eglot       1.14    previous   gnu   The Emacs Client for LSP servers
  eglot       1.12    built-in   gnu   The Emacs Client for LSP servers

where the 3rd column is the "stability gradation" I had in mind.

I also envisioned some user option, say, package-preferred-stabilty,
which users could set to a value such as 'stable', and then package.el
will only automatically update a package to a newer version when the
new version satisfies the stability criteria per the value of that
option.

> Then explain what you want to do with that stability gradation
> in Emacs (if anything, perhaps you just want to show it).

If the above doesn't explain this, I don't think I understand the
question.  What do you mean by "do with that in Emacs"?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 2:44 PM Eli Zaretskii <eliz@gnu.org> wrote:

> I envisioned seeing something like
>
>   eglot       2.01    alpha      gnu   The Emacs Client for LSP servers
>   eglot       1.20    current    gnu   The Emacs Client for LSP servers
>   eglot       1.15    stable     gnu   The Emacs Client for LSP servers
>   eglot       1.14    previous   gnu   The Emacs Client for LSP servers
>   eglot       1.12    built-in   gnu   The Emacs Client for LSP servers
>
> where the 3rd column is the "stability gradation" I had in mind.
>
> I also envisioned some user option, say, package-preferred-stabilty,
> which users could set to a value such as 'stable', and then package.el
> will only automatically update a package to a newer version when the
> new version satisfies the stability criteria per the value of that
> option.
>
> > Then explain what you want to do with that stability gradation
> > in Emacs (if anything, perhaps you just want to show it).
>
> If the above doesn't explain this, I don't think I understand the
> question.

Thank you, it explains what you mean.  Seems like you want distribution
channels like Debian has.  No objection here, but IMO seems to be
solving problems that I've never personally seen happening in
Emacs packages.  And it also seems like a lot of work.

I really can't remember seeing a bug report where someone was
discontent with an upgrade of some package to a higher version
that was accidently buggy, let alone a "furtive" upgrade related to
dependencies.  Doesn't mean they don't exist, I've just never seen
them.

I remember quite well that upgrading versions of dependencies _fixes_
bugs.  One of the first code packages was cl-lib and its first versions
were off, the later versions were progressively better.

> What do you mean by "do with that in Emacs"?

I meant how is Emacs's code to react to this gradation.  You've more
or less suggested that "package-preferred-stability" would be
consulted by package-install to know which one to pick, so that also
answers my question.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 18/04/2023 15:57, Eli Zaretskii wrote:

>>>> On 14/04/2023 22:28, Eli Zaretskii wrote:
>>>>> If, OTOH,
>>>>> you think that it's imperative to allow_all_  users of Eglot with
>>>>> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
>>>>> become available), then we should release Emacs 29 with 1.14.
>>>>
>>>> Was this question about stability only?
>>>
>>> It was about the criteria for which versions of core packages to ship
>>> with a release.
>>
>> I don't think we can get a single set of criteria across core packages.
> 
> We don't need to have just one set.  Packages are different in both
> their complexity, their dependence on other packages, and dependence
> of other packages on them.  So one set is unlikely to fit the bill,
> indeed.
> 
> But that doesn't mean we shouldn't have criteria at all.  We should
> strive to have a small number of them, and we should know which set is
> applicable to which class of packages.

It's a good question, but not a deciding one WRT what we should do with 
Eglot for Emacs 29, I think. I hope I'll be able to explain that below.

> This will be one of the serious issues if we ever move to having some
> packages only in elpa.git, and will then bundle them when preparing an
> Emacs release tarball.  It will be imperative to know at that time
> which version/branch of each such package to take as part of preparing
> a release.  We must have a solution by then, so this is as good time
> as any to start discussing the issue.

Sure. Please keep in mind, however, that very few of external packages 
have separate branches for releases and development. I guess Org does, 
but I'm not sure if others exist. One of the reasons for that is that 
the ELPA repositories only package the very latest released version 
anyway and quickly delete the older ones. And package.el is the foremost 
distribution method for Elisp code.

For :core package, we also have Emacs's own development and release 
branches. But only master branch has versioned releases cut from it. 
It's hard to say whether the time the code has spent in a given 
'emacs-xy' branch brings maturity to it, and how much.

So every time a new version if tagged, it's a value judgment on the part 
of the maintainer: whether enough time has passed since the most recent 
big feature was added, whether there were bug reports or not.

The MELPA project has helped a lot with this over the years because it 
popularized snapshot versions. So when a package is feature on MELPA, 
the author could be fairly sure that a lot of users have downloaded the 
latest snapshot and gave it a try, and the lack of new reports means 
it's probably okay. MELPA has the problem that the "Stable" repository 
is not very popular, though. We here have it in the reverse: I'm not 
sure how many people make use of elpa-devel (GNU-devel ELPA). Probably 
not many.

>> E.g. Org is developed externally, has its own community of significant
>> size, and does split off release branches (with additional testing, I',m
>> guessing).
>>
>> Eglot, OTOH, is developed only here, with no additional release workflow
>> other than what MELPA/GNU ELPA historically provided: collect up some
>> features/fixes, bump the Version header, and push a new release out to
>> the users. The lack of extended testing period is made up for with the
>> capability to push out a new fixed version overnight. That's why the
>> difficulty in upgrading to the latest version (for Emacs 29 users) is
>> going to hurt.
> 
> If some core package is not tested enough before it gets a new
> version, then why are we okay with telling users of a stable Emacs to
> update the package willy-nilly as soon as another version is on ELPA?
> Shouldn't we at least warn them, or, better, somehow indicate that
> this version is not yet considered stable?

I have a different answer from all that had been presented here: because 
the user can uninstall it.

Uninstall any newer installed version and stay with that one that had 
been bundled with the Emacs release. As long as *that* one has been 
picked well enough to be stable, and can be reverted to, we can afford 
not to worry too much about the user installing another version.

It's like reverting to a more stable distribution channel, to use the 
analogy from the bug discussion.

> IOW, shouldn't packages have some "stability gradation" that is
> visible when users look at the list of packages via package.el?
> Shouldn't we allow users to tell package.el which stability they want
> to download, so that unstable packages don't inadvertently get
> installed and mess up their production environment?

We have elpa and elpa-devel. The latter is "explicitly unstable" and the 
former is "checkpoint releases".

Neither keeps the previous versions around, so it's not like the user at 
any point could make a choice to install a minor version update instead 
of going up a full major version. The only choice is to update to the 
latest, or not (by using elpa-devel, though, the user might opt into 
having "the latest" mean "the latest commit in master").

>> BTW, if you recall the threads before Eglot was added, I was against
>> that, and one of the things I cited is an LSP client has inherently high
>> development velocity. Maybe the LSP community will settle/mature/stop
>> adding features one day, but it's not there yet.
> 
> I don't see what development pace has to do with this issue.  If a
> package is being developed at a high pace, it might mean that the
> stable version will lag more.  But what does this change in principle?

It matters when we're talking not about simple additional, optional 
features, but about changes that keep pace with the evolution of the LSP 
standard, with new LSP servers that arrive, new changes in existing 
protocols. Things that users come to expect to be able to use now, and 
not in 3 years.

Taking a conservative stance can work when the ecosystem supports it 
too. E.g. if every LSP server that we support now had an implicit 
promise to be properly maintained for 3 years since, at least. I don't 
think that's the case. Almost every one could be deprecated in that time 
frame, with community being recommended to switch to a newer one.

>>>> Because since we've decided in favor of stability of package.el, and
>>>> against eglot's easy upgradability, I would suggest to backport Eglot
>>>> 1.14 to emacs-29.
>>>
>>> I won't object.  In fact, I asked up front why not.
>>
>> Note that that suggestion comes with a fix to eldoc which you so far
>> have rejected for emacs-29.
> 
> You mean, the change in ElDoc that avoids the "jumping mode line"
> issue?  If so, it is unfortunate for Eglot to depend on that, because
> it means users of Emacs 29 will be unable to upgrade to Eglot 1.15
> without by side effect installing a newer and potentially less stable
> ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)

Sorry if that was unclear: yes, the fix for "jumping mode-line and 
blinking text", but not as a bump in dependency. Just to backport the 
fix to eldoc.el inside emacs-29.

But Joao has explained that that idea is impractical because Eglot 1.14 
depends on Eldoc 1.14.0. And I just wanted a particular fix from it. So 
backporting the thing I wanted from Eglot 1.14 (taking up much less 
space in the echo area) is not really feasible because it depends on a 
fairly recent addition to eldoc.

To clarify: I made that suggestion from the premise that we do expect 
and encourage a fair fraction of the users to use just the versions of 
Eglot and Eldoc that come with Emacs 29, and never upgrade to the latest 
ones.

> So this again goes back to the main issue: how should the stability
> considerations affect development of core packages and their
> "stability gradation"?  Developers of core packages should keep these
> aspects in mind at all times, and, for example, avoid dependencies on
> other packages that aren't absolutely necessary.

I agree with that stance, in general.

And as per above explanation, Eglot 0.14 depends on Eldoc 0.14.0 not 
because it's the very latest and spiffiest version, but because it needs 
a particular feature from it.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 11:11 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> And as per above explanation, Eglot 0.14 depends on Eldoc 0.14.0 not
> because it's the very latest and spiffiest version, but because it needs
> a particular feature from it.

Of course.  And if there are doubts as to when and why Eglot's
"Package-Require"'s are updated, one can always M-x vc-region-history
on that single line of lisp/progmodes/eglot.el.  For example, recently
Xref requirement was bumped because:

  commit f72a394716f4373dbbdc79ad0816da90bdb032a1
  Author: Basil L. Contovounesios <contovob@tcd.ie>
  Date:   Fri Jan 27 00:27:26 2023 +0000

      Work around package.el transitive dependency bug

      Eglot already depends transitively on Xref 1.4.0 via Project,
      but package.el doesn't pick up on this in Emacs 28
      (which has Xref 1.3.0).

      * lisp/progmodes/eglot.el (Version): Bump to 1.11.
      (Package-Requires): Explicitly require Xref 1.4.0, which is
      the version already required by Project, for the benefit of
      Emacs 28 (bug#61048).

If you find some dependency is set unnecessarily too high, I
don't see any problem in downgrading it.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 01:10:56 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > This will be one of the serious issues if we ever move to having some
> > packages only in elpa.git, and will then bundle them when preparing an
> > Emacs release tarball.  It will be imperative to know at that time
> > which version/branch of each such package to take as part of preparing
> > a release.  We must have a solution by then, so this is as good time
> > as any to start discussing the issue.
> 
> Sure. Please keep in mind, however, that very few of external packages 
> have separate branches for releases and development. I guess Org does, 
> but I'm not sure if others exist. One of the reasons for that is that 
> the ELPA repositories only package the very latest released version 
> anyway and quickly delete the older ones. And package.el is the foremost 
> distribution method for Elisp code.

Then maybe ELPA will have to change as well.

> So every time a new version if tagged, it's a value judgment on the part 
> of the maintainer: whether enough time has passed since the most recent 
> big feature was added, whether there were bug reports or not.

Similar judgment calls are necessary for Emacs.  So I see nothing here
that brings some fundamentally new issues.

Moreover, I don't see how this affects the need to have good criteria
for stability, and the importance of applying those criteria when
deciding which versions to bundle with Emacs.

> > If some core package is not tested enough before it gets a new
> > version, then why are we okay with telling users of a stable Emacs to
> > update the package willy-nilly as soon as another version is on ELPA?
> > Shouldn't we at least warn them, or, better, somehow indicate that
> > this version is not yet considered stable?
> 
> I have a different answer from all that had been presented here: because 
> the user can uninstall it.

User can also downgrade to a previous version of the package, don't
they?  Or if they cannot, they should be able to do that.  Once that
is possible, what exactly is the difference between these two kinds of
packages?  Downgrading to an older version when a newer one is buggy
or otherwise unsatisfactory should be supported for all packages.

Also, does package.el support "downgrading" to the bundled version?
Did anyone actually try that?  In particular, what happens with the
dependencies the user upgraded together with the package being
"uninstalled", due to the minimum requirements of that package?

> > IOW, shouldn't packages have some "stability gradation" that is
> > visible when users look at the list of packages via package.el?
> > Shouldn't we allow users to tell package.el which stability they want
> > to download, so that unstable packages don't inadvertently get
> > installed and mess up their production environment?
> 
> We have elpa and elpa-devel. The latter is "explicitly unstable" and the 
> former is "checkpoint releases".

So what is the stability measure of "elpa", again?  Is it on par with
the Emacs's release branch?  Could it be?  If not, why not?

> Neither keeps the previous versions around, so it's not like the user at 
> any point could make a choice to install a minor version update instead 
> of going up a full major version.

That in itself is a serious deficiency, IMO, and we should fix it by
providing the "downgrade" option.

> > I don't see what development pace has to do with this issue.  If a
> > package is being developed at a high pace, it might mean that the
> > stable version will lag more.  But what does this change in principle?
> 
> It matters when we're talking not about simple additional, optional 
> features, but about changes that keep pace with the evolution of the LSP 
> standard, with new LSP servers that arrive, new changes in existing 
> protocols. Things that users come to expect to be able to use now, and 
> not in 3 years.

Users who must have these features (presumably because they must use
servers which require that) will have to give up some stability
expectations.  Exactly like users who must have some new enough
feature of Emacs which is only available on the master branch.

So once again: what is fundamentally different or new about packages
which develop at fast pace, in the context of this discussion?  Why do
we need to bring up those details here?

> Taking a conservative stance can work when the ecosystem supports it 
> too. E.g. if every LSP server that we support now had an implicit 
> promise to be properly maintained for 3 years since, at least. I don't 
> think that's the case. Almost every one could be deprecated in that time 
> frame, with community being recommended to switch to a newer one.

How does this help us move forward with the discussion and with
handling this issue?  Conservative doesn't mean stupid; if some
external factor changes outside of our control and forces us to make
potentially destabilizing changes, what else can we do that requires a
discussion?  And how does it impact what we should do about
categorizing package releases according to their stability and about
the decisions which version to bundle with Emacs?

> To clarify: I made that suggestion from the premise that we do expect 
> and encourage a fair fraction of the users to use just the versions of 
> Eglot and Eldoc that come with Emacs 29, and never upgrade to the latest 
> ones.

AFAIU, João is of the opposite opinion: he thinks that most Eglot
users will want the latest version on ELPA, or at least a version that
is newer than the one bundled with Emacs.

> And as per above explanation, Eglot 0.14 depends on Eldoc 0.14.0 not 
> because it's the very latest and spiffiest version, but because it needs 
> a particular feature from it.

And there's no reasonable way of adding to Eglot 1.14 some fallback or
compatibility shim to remove that dependency?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 5:47 AM, Eli Zaretskii wrote:
> Also, does package.el support "downgrading" to the bundled version?
> Did anyone actually try that?  In particular, what happens with the
> dependencies the user upgraded together with the package being
> "uninstalled", due to the minimum requirements of that package?

I've done this in the past and everything works pretty much as expected 
from my recollection: you can uninstall a package that you got from 
ELPA, so afterwards, you'd just get the bundled version (you might need 
to restart; I always do).

In addition, any automatically-installed dependent packages are marked 
with the status "dependency". You can remove no-longer-needed deps via 
'package-autoremove'. When uninstalling a package interactively in the 
*Packages* buffer, it will even suggest that you remove unneeded deps 
when appropriate (see 'package-menu-execute').

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 11:22:10 -0700
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 4/19/2023 5:47 AM, Eli Zaretskii wrote:
> > Also, does package.el support "downgrading" to the bundled version?
> > Did anyone actually try that?  In particular, what happens with the
> > dependencies the user upgraded together with the package being
> > "uninstalled", due to the minimum requirements of that package?
> 
> I've done this in the past and everything works pretty much as expected 
> from my recollection: you can uninstall a package that you got from 
> ELPA, so afterwards, you'd just get the bundled version (you might need 
> to restart; I always do).
> 
> In addition, any automatically-installed dependent packages are marked 
> with the status "dependency". You can remove no-longer-needed deps via 
> 'package-autoremove'. When uninstalling a package interactively in the 
> *Packages* buffer, it will even suggest that you remove unneeded deps 
> when appropriate (see 'package-menu-execute').

IMO, downgrading to the bundled version should be much simpler and by
default should remove all the dependencies without asking and without
any need for manual user actions.

In any case, I don't think this use case was considered or tried
enough for us to consider it a solved issue.  I'm quite sure there's
more here than meets the eye, simply because this is rarely if ever
done.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 11:37 AM, Eli Zaretskii wrote:
> IMO, downgrading to the bundled version should be much simpler and by
> default should remove all the dependencies without asking and without
> any need for manual user actions.

For what it's worth, Apt (the Debian package manager) works the same 
way: if you remove a package, it will just inform you that there are 
unneeded deps and that you can use 'apt autoremove' to clean them up. I 
think that makes sense, since you might really want the new version of 
some package, and you just don't realize that it's marked as a dep 
instead of an explicitly-installed package.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Lynn Winebarger]

On Wed, Apr 19, 2023 at 2:37 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Wed, 19 Apr 2023 11:22:10 -0700
> > Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> > From: Jim Porter <jporterbugs@gmail.com>
> >
> > On 4/19/2023 5:47 AM, Eli Zaretskii wrote:
> > > Also, does package.el support "downgrading" to the bundled version?
> > > Did anyone actually try that?  In particular, what happens with the
> > > dependencies the user upgraded together with the package being
> > > "uninstalled", due to the minimum requirements of that package?
> >
> > I've done this in the past and everything works pretty much as expected
> > from my recollection: you can uninstall a package that you got from
> > ELPA, so afterwards, you'd just get the bundled version (you might need
> > to restart; I always do).
> >
> > In addition, any automatically-installed dependent packages are marked
> > with the status "dependency". You can remove no-longer-needed deps via
> > 'package-autoremove'. When uninstalling a package interactively in the
> > *Packages* buffer, it will even suggest that you remove unneeded deps
> > when appropriate (see 'package-menu-execute').
>
> IMO, downgrading to the bundled version should be much simpler and by
> default should remove all the dependencies without asking and without
> any need for manual user actions.
>
> In any case, I don't think this use case was considered or tried
> enough for us to consider it a solved issue.  I'm quite sure there's
> more here than meets the eye, simply because this is rarely if ever
> done.

One of the design points of the unboxing package-management-wrapper
I'm writing is maintaining distinct package areas with a notion of
scoping.  My purpose was to enable separate management of
site-packages and user packages.  A site administrator might have one
set of packages, and a user might want different versions of those
packages.  So, the dependency calculation for user packages should
allow for dependencies to be satisfied by site packages, but not vice
versa (that's the scoping).  The "delta" of the package set installed
in the user area has to be enough to make the set of libraries "seen"
by the user coherent.  That is, say package A and B both depend on
package C and all are in the site-level packaging area.  Then the user
wants a different version of package A, which requires a different
version of package C.  Coherence means that if the version of package
B installed at the site level is incompatible with the desired version
of package C, then a compatible version of package B will also have to
be installed in the user packaging area.  That set has to be
transitively closed.

But the separation into those two is just a default configuration.
Arbitrary package areas can be defined, each with their own set of
"in-scope" areas.  One easily reversible way to add a new version of
eglot would be to set up a new package area for it, with whatever
other package areas are loaded being put in its scope, then install
the minimum coherent set of packages into the new area.  Then rolling
back the update would be trivial, and it could be "committed" to the
usual package area once the user was satisfied.

My only other thought on this discussion, as a package (ab)user
running multiple versions of emacs, is that package releases should be
dependent on the emacs version installing it.  What I see as available
packages/upgrades when I list packages while running emacs 28 should
probably be different (for some packages) than what I see while
running emacs 25 or emacs 29.  It seems like the release philosophy
after version 24 has been conservative about changes to the run-time
semantics per major release, so the versioning probably doesn't need
to be conditioned on the minor version of the Emacs binary.

Lynn

Re: history of ELPA packages and dependencies (was: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot))

[Lynn Winebarger]

On Wed, Apr 19, 2023 at 6:51 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> The "delta" of the package set installed
> in the user area has to be enough to make the set of libraries "seen"
> by the user coherent.  That is, say package A and B both depend on
> package C and all are in the site-level packaging area.  Then the user
> wants a different version of package A, which requires a different
> version of package C.  Coherence means that if the version of package
> B installed at the site level is incompatible with the desired version
> of package C, then a compatible version of package B will also have to
> be installed in the user packaging area.  That set has to be
> transitively closed.

Calculating the set of sets of package/version pairs satsifying the
"coherence" constraint requires a lot of data from each published
version of each package in an ELPA.

Looking at the Savannah repos for GNU/nonGNU ELPAs, some package
branches appear to have multiple elisp files with package header
fields.  For example, externals/corfu in the GNU ELPA repo has a
subdirectory "extensions" in which some elisp files appear to contain
package definitions.  However, the GNU ELPA only has one "corfu"
package.  Unfortunately, I can't tell from that example if elpa-admin
completely ignores the package-requires in the other elisp files, or
determines the set of requirements based on fields from all included
elisp files.

So, can I determine package-requires of each package commit by looking
only at the main elisp file for the package after checking out that
commit?

I'm sure MELPA will be more challenging, but building a history for
GNU/nonGNU ELPA packages will be a good starting point.

Thanks,
Lynn

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Lynn Winebarger]

On Wed, Apr 19, 2023 at 6:51 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> On Wed, Apr 19, 2023 at 2:37 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > In any case, I don't think this use case was considered or tried
> > enough for us to consider it a solved issue.  I'm quite sure there's
> > more here than meets the eye, simply because this is rarely if ever
> > done.

> My only other thought on this discussion, as a package (ab)user
> running multiple versions of emacs, is that package releases should be
> dependent on the emacs version installing it.  What I see as available
> packages/upgrades when I list packages while running emacs 28 should
> probably be different (for some packages) than what I see while
> running emacs 25 or emacs 29.  It seems like the release philosophy
> after version 24 has been conservative about changes to the run-time
> semantics per major release, so the versioning probably doesn't need
> to be conditioned on the minor version of the Emacs binary.

Another point in favor of maintaining separate dependencies based on
emacs major version (and possibly more) are addition of features like
the builtin sqlite support.  Some packages might require the sqlite
module for versions prior to 29, or the sqlite-builtin module for 29+,
but the latter dependency is incompatible with version 28 and below.

Lynn

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 15:47, Eli Zaretskii wrote:

>>> This will be one of the serious issues if we ever move to having some
>>> packages only in elpa.git, and will then bundle them when preparing an
>>> Emacs release tarball.  It will be imperative to know at that time
>>> which version/branch of each such package to take as part of preparing
>>> a release.  We must have a solution by then, so this is as good time
>>> as any to start discussing the issue.
>>
>> Sure. Please keep in mind, however, that very few of external packages
>> have separate branches for releases and development. I guess Org does,
>> but I'm not sure if others exist. One of the reasons for that is that
>> the ELPA repositories only package the very latest released version
>> anyway and quickly delete the older ones. And package.el is the foremost
>> distribution method for Elisp code.
> 
> Then maybe ELPA will have to change as well.

I don't think that should be, or can be reasonably required. But that 
can be put it to a separate discussion. With Stefan and Philip involved, 
probably.

>> So every time a new version if tagged, it's a value judgment on the part
>> of the maintainer: whether enough time has passed since the most recent
>> big feature was added, whether there were bug reports or not.
> 
> Similar judgment calls are necessary for Emacs.  So I see nothing here
> that brings some fundamentally new issues.
> 
> Moreover, I don't see how this affects the need to have good criteria
> for stability, and the importance of applying those criteria when
> deciding which versions to bundle with Emacs.

Ok, if the question is just "how to choose the version of :core package 
to bundle with an Emacs release", then just see the criterion "N weeks 
have passed without issue" that I just described in the bug thread.

This one doesn't need any changes to ELPA, and I've been using it, more 
or less, for some years now.

>>> If some core package is not tested enough before it gets a new
>>> version, then why are we okay with telling users of a stable Emacs to
>>> update the package willy-nilly as soon as another version is on ELPA?
>>> Shouldn't we at least warn them, or, better, somehow indicate that
>>> this version is not yet considered stable?
>>
>> I have a different answer from all that had been presented here: because
>> the user can uninstall it.
> 
> User can also downgrade to a previous version of the package, don't
> they?  Or if they cannot, they should be able to do that.

Even if there was such possibility, they wouldn't have a single stable 
version to go back to (they'd have to make a choice). When we bundle a 
version, that's one fewer question to answer.

> Once that
> is possible, what exactly is the difference between these two kinds of
> packages?  Downgrading to an older version when a newer one is buggy
> or otherwise unsatisfactory should be supported for all packages.

Then the difference would be that we hand-picked a set of particular 
versions of packages, whereas for third-party packages that was done (or 
failed to be done) by other people we have no responsibility over.

> Also, does package.el support "downgrading" to the bundled version?
> Did anyone actually try that?  In particular, what happens with the
> dependencies the user upgraded together with the package being
> "uninstalled", due to the minimum requirements of that package?

It should work by "uninstalling" the package. An when you uninstall a 
package, package.el warns you about any dependencies that would be 
broken this way. Someone should test it just in case, of course.

But the important part is that the bundled package stays installed. You 
asked why we can encourage people to upgrade said packages freely. One 
answer is that they have a safety net.

>>> IOW, shouldn't packages have some "stability gradation" that is
>>> visible when users look at the list of packages via package.el?
>>> Shouldn't we allow users to tell package.el which stability they want
>>> to download, so that unstable packages don't inadvertently get
>>> installed and mess up their production environment?
>>
>> We have elpa and elpa-devel. The latter is "explicitly unstable" and the
>> former is "checkpoint releases".
> 
> So what is the stability measure of "elpa", again?  Is it on par with
> the Emacs's release branch?  Could it be?

It's more stable than 'git clone' but less stable than using a release.

And this will stay that way while we're using it to help stabilize 
package versions, too.

 > If not, why not?

I don't think we want ELPA to have the same frequency of package 
releases as Emacs itself.

We could add a new archive, though. Like elpa-very-stable. Which would 
be pushed, for example, the same versions of bundled packages that go 
into Emacs releases. Or something like that.

>> Neither keeps the previous versions around, so it's not like the user at
>> any point could make a choice to install a minor version update instead
>> of going up a full major version.
> 
> That in itself is a serious deficiency, IMO, and we should fix it by
> providing the "downgrade" option.

Doing that for every package will mean more work for everybody around, 
and it'll also increase the costs of hosting packages considerably.

>>> I don't see what development pace has to do with this issue.  If a
>>> package is being developed at a high pace, it might mean that the
>>> stable version will lag more.  But what does this change in principle?
>>
>> It matters when we're talking not about simple additional, optional
>> features, but about changes that keep pace with the evolution of the LSP
>> standard, with new LSP servers that arrive, new changes in existing
>> protocols. Things that users come to expect to be able to use now, and
>> not in 3 years.
> 
> Users who must have these features (presumably because they must use
> servers which require that) will have to give up some stability
> expectations.  Exactly like users who must have some new enough
> feature of Emacs which is only available on the master branch.
> 
> So once again: what is fundamentally different or new about packages
> which develop at fast pace, in the context of this discussion?  Why do
> we need to bring up those details here?

It's an issue of whether a "stable" Eglot could actually be useful 2 
years later for most people. If not (I admit I don't know for certain), 
we should choose to focus on other scenarios and needs, and less on 
stability in this particular case.

>> Taking a conservative stance can work when the ecosystem supports it
>> too. E.g. if every LSP server that we support now had an implicit
>> promise to be properly maintained for 3 years since, at least. I don't
>> think that's the case. Almost every one could be deprecated in that time
>> frame, with community being recommended to switch to a newer one.
> 
> How does this help us move forward with the discussion and with
> handling this issue?  Conservative doesn't mean stupid; if some
> external factor changes outside of our control and forces us to make
> potentially destabilizing changes, what else can we do that requires a
> discussion?

No, I don't think that conservative means stupid. I am fairly 
conservative in some things, just not in others.

Or take Joao: he's at the moment being conservative about how the users 
install Eglot and have had it upgraded over time when using all previous 
releases of Emacs.

> And how does it impact what we should do about
> categorizing package releases according to their stability and about
> the decisions which version to bundle with Emacs?

No, that's a different question. One I had hopefully addressed above.

>> To clarify: I made that suggestion from the premise that we do expect
>> and encourage a fair fraction of the users to use just the versions of
>> Eglot and Eldoc that come with Emacs 29, and never upgrade to the latest
>> ones.
> 
> AFAIU, João is of the opposite opinion: he thinks that most Eglot
> users will want the latest version on ELPA, or at least a version that
> is newer than the one bundled with Emacs.

I am of the above opposite opinion as well, but I can appreciate your 
POV as well, I think. And, well, if the ultimate decision is made using 
the latter principle, we still could make the most of it.

>> And as per above explanation, Eglot 0.14 depends on Eldoc 0.14.0 not
>> because it's the very latest and spiffiest version, but because it needs
>> a particular feature from it.
> 
> And there's no reasonable way of adding to Eglot 1.14 some fallback or
> compatibility shim to remove that dependency?

Everything's possible. It might not even be too hard.

But if we're talking about the improvement I was eyeing (better 
eglot<->eldoc behavior all around), then a simple shim won't cut it.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 8:25 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > AFAIU, João is of the opposite opinion: he thinks that most Eglot
> > users will want the latest version on ELPA, or at least a version that
> > is newer than the one bundled with Emacs.
> I am of the above opposite opinion as well, but I can appreciate your

From my experience, yes.  Overwhelmingly so.  People get excited.
Reddit posts I didn't order appear out of some anonymous user
enthusiasm annoucing the feature :-) But I can also, like Dmitry,
appreciate your opinion, Eli.  Users that don't  want the latest version
on ELPA don't have to get it, by any means.  Of course not.

Here is, taken from the sibling discussion, myrecommendation to
Eglot users:

 "Dear user,

  Emacs 29 will come with the most stable version I can offer, and I
  can tell you, prospective user, that it has been through as much
  of the Emacs 29 pretest period as possible. If you, prospective user,
  are interested in the features listed in etc/EGLOT-NEWS, you may try
  upgrading to the latest version (I hope it will be easy, good luck!)
  In the unlikely event that things go sour for your particular
  LSP server and use case, you have my deepest apologies, and you
  should revert back to the version bundled with Emacs 29.

  Thank you,
  Your maintainer"


João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 22:25:08 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> I have a different answer from all that had been presented here: because
> >> the user can uninstall it.
> > 
> > User can also downgrade to a previous version of the package, don't
> > they?  Or if they cannot, they should be able to do that.
> 
> Even if there was such possibility, they wouldn't have a single stable 
> version to go back to (they'd have to make a choice). When we bundle a 
> version, that's one fewer question to answer.
> 
> > Once that
> > is possible, what exactly is the difference between these two kinds of
> > packages?  Downgrading to an older version when a newer one is buggy
> > or otherwise unsatisfactory should be supported for all packages.
> 
> Then the difference would be that we hand-picked a set of particular 
> versions of packages, whereas for third-party packages that was done (or 
> failed to be done) by other people we have no responsibility over.
> 
> > Also, does package.el support "downgrading" to the bundled version?
> > Did anyone actually try that?  In particular, what happens with the
> > dependencies the user upgraded together with the package being
> > "uninstalled", due to the minimum requirements of that package?
> 
> It should work by "uninstalling" the package. An when you uninstall a 
> package, package.el warns you about any dependencies that would be 
> broken this way. Someone should test it just in case, of course.
> 
> But the important part is that the bundled package stays installed. You 
> asked why we can encourage people to upgrade said packages freely. One 
> answer is that they have a safety net.

All those differences disappear when we are discussing core packages.
Which is my only focus in this discussion.

> >> We have elpa and elpa-devel. The latter is "explicitly unstable" and the
> >> former is "checkpoint releases".
> > 
> > So what is the stability measure of "elpa", again?  Is it on par with
> > the Emacs's release branch?  Could it be?
> 
> It's more stable than 'git clone' but less stable than using a release.
> 
> And this will stay that way while we're using it to help stabilize 
> package versions, too.

I very much hope it doesn't stay that way for core packages.

>  > If not, why not?
> 
> I don't think we want ELPA to have the same frequency of package 
> releases as Emacs itself.

This is a red herring.  Emacs releases are less frequent because Emacs
is a large collection of packages, and thus the period required for
its stabilization is much longer.  A single core package can apply
stability criteria and still not go anywhere near the Emacs release
frequency.  It does need more effort on the part of the package
developers, but the effect on the release frequency will be minor.

> > Users who must have these features (presumably because they must use
> > servers which require that) will have to give up some stability
> > expectations.  Exactly like users who must have some new enough
> > feature of Emacs which is only available on the master branch.
> > 
> > So once again: what is fundamentally different or new about packages
> > which develop at fast pace, in the context of this discussion?  Why do
> > we need to bring up those details here?
> 
> It's an issue of whether a "stable" Eglot could actually be useful 2 
> years later for most people.

"Two years" because you think the release frequency of Eglot will be
slowed down to such abysmally low value?  As explained above, I think
this is not going to happen, so there's no need to assume this will be
the outcome.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 12:47, Eli Zaretskii wrote:

>>> Also, does package.el support "downgrading" to the bundled version?
>>> Did anyone actually try that?  In particular, what happens with the
>>> dependencies the user upgraded together with the package being
>>> "uninstalled", due to the minimum requirements of that package?
>>
>> It should work by "uninstalling" the package. An when you uninstall a
>> package, package.el warns you about any dependencies that would be
>> broken this way. Someone should test it just in case, of course.
>>
>> But the important part is that the bundled package stays installed. You
>> asked why we can encourage people to upgrade said packages freely. One
>> answer is that they have a safety net.
> 
> All those differences disappear when we are discussing core packages.
> Which is my only focus in this discussion.

Disappear, why? The core packages are the only kind of "bundled" 
packages we have (for now), and so they are the only ones that can be 
safely reverted to the bundled version via uninstallation.

>>>> We have elpa and elpa-devel. The latter is "explicitly unstable" and the
>>>> former is "checkpoint releases".
>>>
>>> So what is the stability measure of "elpa", again?  Is it on par with
>>> the Emacs's release branch?  Could it be?
>>
>> It's more stable than 'git clone' but less stable than using a release.
>>
>> And this will stay that way while we're using it to help stabilize
>> package versions, too.
> 
> I very much hope it doesn't stay that way for core packages.

Let me try again: one of the uses of ELPA is to push out the package 
release "into the real world" so that more people can test it.

So that "N weeks later" we can consider is "stable" and be content to 
have it in an Emacs release.

By the very definition, the same release wasn't "stable" N weeks ago, 
just as it was published. So ELPA cannot be considered to have the same 
level of stability because we also use it for testing this way (among 
other things).

>>   > If not, why not?
>>
>> I don't think we want ELPA to have the same frequency of package
>> releases as Emacs itself.
> 
> This is a red herring.  Emacs releases are less frequent because Emacs
> is a large collection of packages, and thus the period required for
> its stabilization is much longer.  A single core package can apply
> stability criteria and still not go anywhere near the Emacs release
> frequency.  It does need more effort on the part of the package
> developers, but the effect on the release frequency will be minor.

I guess you're right.

But note that as per above, if we wanted ELPA to be the "stable" source, 
we'd need some other testing ground before pushing releases to it. E.g. 
we could just wait for people using the master branch to report 
problems. But since there are fewer of them than there are ELPA users, 
it stands to reason that the stabilization periods must become longer. 
Maybe not two years, but longer than with ELPA.

>>> Users who must have these features (presumably because they must use
>>> servers which require that) will have to give up some stability
>>> expectations.  Exactly like users who must have some new enough
>>> feature of Emacs which is only available on the master branch.
>>>
>>> So once again: what is fundamentally different or new about packages
>>> which develop at fast pace, in the context of this discussion?  Why do
>>> we need to bring up those details here?
>>
>> It's an issue of whether a "stable" Eglot could actually be useful 2
>> years later for most people.
> 
> "Two years" because you think the release frequency of Eglot will be
> slowed down to such abysmally low value?  As explained above, I think
> this is not going to happen, so there's no need to assume this will be
> the outcome.

No, no. I'm talking the scenario with users who are going to stay with 
the version of Eglot that comes with Emacs 29, for two years without 
upgrading. That might not be wise in a lot of cases (admittedly, this is 
just a guess at this point, looking at the speed of changes around the 
community), so we should make it easy to upgrade.

I don't believe the upgrade must be made automatic, however.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 16:03:16 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/04/2023 12:47, Eli Zaretskii wrote:
> 
> >>> Also, does package.el support "downgrading" to the bundled version?
> >>> Did anyone actually try that?  In particular, what happens with the
> >>> dependencies the user upgraded together with the package being
> >>> "uninstalled", due to the minimum requirements of that package?
> >>
> >> It should work by "uninstalling" the package. An when you uninstall a
> >> package, package.el warns you about any dependencies that would be
> >> broken this way. Someone should test it just in case, of course.
> >>
> >> But the important part is that the bundled package stays installed. You
> >> asked why we can encourage people to upgrade said packages freely. One
> >> answer is that they have a safety net.
> > 
> > All those differences disappear when we are discussing core packages.
> > Which is my only focus in this discussion.
> 
> Disappear, why? The core packages are the only kind of "bundled" 
> packages we have (for now), and so they are the only ones that can be 
> safely reverted to the bundled version via uninstallation.

Exactly.  That answers your 'why?" question, AFAICT.

> Let me try again: one of the uses of ELPA is to push out the package 
> release "into the real world" so that more people can test it.
> 
> So that "N weeks later" we can consider is "stable" and be content to 
> have it in an Emacs release.
> 
> By the very definition, the same release wasn't "stable" N weeks ago, 
> just as it was published. So ELPA cannot be considered to have the same 
> level of stability because we also use it for testing this way (among 
> other things).

Perhaps there's some kind of misunderstanding here.  To me, declaring
a version of a package "stable" just means some label in its metadata,
which is exposed to package.el and to the users.  So when the package
developer decides that "N weeks have passed", he or she will add that
label, thus making the corresponding version be considered "stable".
Next time users check for updates they will see that, and could then
upgrade to the "next stable" version if they so desire.

IOW, the version on ELPA that wasn't "stable" before, and was used for
testing, now becomes "stable".

Am I missing something here?

> But note that as per above, if we wanted ELPA to be the "stable" source, 
> we'd need some other testing ground before pushing releases to it.

For core packages, that testing ground is the master branch.  Exactly
as for code in Emacs itself.  Once a version is declared "stable", its
changes will be cherry-picked to the Emacs release branch, thus making
it part of the future bug-fix release.

> E.g. 
> we could just wait for people using the master branch to report 
> problems. But since there are fewer of them than there are ELPA users, 
> it stands to reason that the stabilization periods must become longer. 
> Maybe not two years, but longer than with ELPA.

Why do you assume that people who use the Emacs master branch don't
use core packages? or that there are fewer of them than those who use
the release branch?  And, since the code of the core package that is
on master is also on ELPA, I don't see how we can lose any testing,
because reports from users who use non-"stable" versions from ELPA are
as good as from those who use the package via the builds of the Emacs
master branch.  So nothing is lost here, not that I could see.

> >>> Users who must have these features (presumably because they must use
> >>> servers which require that) will have to give up some stability
> >>> expectations.  Exactly like users who must have some new enough
> >>> feature of Emacs which is only available on the master branch.
> >>>
> >>> So once again: what is fundamentally different or new about packages
> >>> which develop at fast pace, in the context of this discussion?  Why do
> >>> we need to bring up those details here?
> >>
> >> It's an issue of whether a "stable" Eglot could actually be useful 2
> >> years later for most people.
> > 
> > "Two years" because you think the release frequency of Eglot will be
> > slowed down to such abysmally low value?  As explained above, I think
> > this is not going to happen, so there's no need to assume this will be
> > the outcome.
> 
> No, no. I'm talking the scenario with users who are going to stay with 
> the version of Eglot that comes with Emacs 29, for two years without 
> upgrading. That might not be wise in a lot of cases (admittedly, this is 
> just a guess at this point, looking at the speed of changes around the 
> community), so we should make it easy to upgrade.
> 
> I don't believe the upgrade must be made automatic, however.

Users of Emacs 29 that need a newer Eglot don't need to stay with the
version we shipped.  First, if the machinery to promote Eglot versions
to "stable" status is in place, bug-fix 29.x releases could have newer
versions of Eglot bundled with them; our rate of putting out bug-fix
releases is much higher than once every 2 years.  Users who cannot
wait until the next bug-fix release, but still want stability, will be
able to upgrade their Eglot to a newer version, provided that we mark
some newer version on ELPA "stable" after "N weeks" or so.  Finally,
users who want newer Eglot badly and are willing to sacrifice some
stability will update to the latest-and-greatest version on ELPA, even
if it is not "stable" yet.

So I don't see how this can lose, if we indeed have the above system,
or something like it, in place.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 17:03, Eli Zaretskii wrote:
> Perhaps there's some kind of misunderstanding here.  To me, declaring
> a version of a package "stable" just means some label in its metadata,
> which is exposed to package.el and to the users.  So when the package
> developer decides that "N weeks have passed", he or she will add that
> label, thus making the corresponding version be considered "stable".
> Next time users check for updates they will see that, and could then
> upgrade to the "next stable" version if they so desire.
> 
> IOW, the version on ELPA that wasn't "stable" before, and was used for
> testing, now becomes "stable".

Okay. That can work if:

- We're able to force at least some older versions of the package to 
stay around in the repository. And we'll probably need some new UI in 
package.el to be able to choose among the versions.

- We're able to retroactively add some metadata to already published 
versions, like calling some older one "stable" after some time has passed.

> Users of Emacs 29 that need a newer Eglot don't need to stay with the
> version we shipped.  First, if the machinery to promote Eglot versions
> to "stable" status is in place, bug-fix 29.x releases could have newer
> versions of Eglot bundled with them; our rate of putting out bug-fix
> releases is much higher than once every 2 years.

That's fair.

> Users who cannot
> wait until the next bug-fix release, but still want stabil
> 
> ity, will be
> able to upgrade their Eglot to a newer version, provided that we mark
> some newer version on ELPA "stable" after "N weeks" or so.  Finally,
> users who want newer Eglot badly and are willing to sacrifice some
> stability will update to the latest-and-greatest version on ELPA, even
> if it is not "stable" yet.

So we seem to agree that the ability to upgrade is important as well.

Whether we're able to transition to a new system with stability tags, 
etc, is yet to be seen.

> So I don't see how this can lose, if we indeed have the above system,
> or something like it, in place.

The main downsides of this are probably obvious:

- The development work required.
- The additional ongoing package maintenance work that is implied by 
this design.

Whether it's really worth it in the end, I don't know. E.g. Joao seems 
to think that additional stable releases for Eglot won't have much of an 
audience (which seems sensible to me), but Arne's messages seem to 
indicate a preference for stability. And the larger community seems to 
be using MELPA quite happily, which is as unstable as it gets.

Maybe it won't be used much by Eglot, but Org devs take a liking to it. 
On my part, I would maybe tag a "stable" release once a year or so.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 17:22:03 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > Users who cannot
> > wait until the next bug-fix release, but still want stabil
> > 
> > ity, will be
> > able to upgrade their Eglot to a newer version, provided that we mark
> > some newer version on ELPA "stable" after "N weeks" or so.  Finally,
> > users who want newer Eglot badly and are willing to sacrifice some
> > stability will update to the latest-and-greatest version on ELPA, even
> > if it is not "stable" yet.
> 
> So we seem to agree that the ability to upgrade is important as well.

Yes.

> Whether we're able to transition to a new system with stability tags, 
> etc, is yet to be seen.

Indeed.  In fact, it remains to be seen even if we have a broad-enough
agreement to consider transitioning to such a system a Good Thing.

> The main downsides of this are probably obvious:
> 
> - The development work required.
> - The additional ongoing package maintenance work that is implied by 
> this design.

Agreed.  I just don't see how will we be ever able to move to keeping
core packages only on ELPA if we don't transition to such a system, or
something like it.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 17:42, Eli Zaretskii wrote:
>> The main downsides of this are probably obvious:
>>
>> - The development work required.
>> - The additional ongoing package maintenance work that is implied by
>> this design.
> Agreed.  I just don't see how will we be ever able to move to keeping
> core packages only on ELPA if we don't transition to such a system, or
> something like it.

When you say "keeping core packages only on ELPA", what do you mean exactly?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 18:30:47 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> When you say "keeping core packages only on ELPA", what do you mean exactly?

That core packages will be only in elpa.git, not in emacs.git.  Then
we'd "bundle" them with Emacs when preparing the release tarballs.

I believe this is a long-term plan wrt core packages.  It was even
discussed a couple of times here.

Re: Stability of core packages

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 20 Apr 2023 18:30:47 +0300
>> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> When you say "keeping core packages only on ELPA", what do you mean exactly?
>
> That core packages will be only in elpa.git, not in emacs.git.  Then
> we'd "bundle" them with Emacs when preparing the release tarballs.
>
> I believe this is a long-term plan wrt core packages.  It was even
> discussed a couple of times here.

For people building Emacs from source, would this mean that they
would have to manually install core packages?  Also, this sounds like it
would make it impossible for package.el to be distributed on ELPA, as
proposed in another thread.

Re: Stability of core packages

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Dmitry Gutov <dmitry@gutov.dev>,  joaotavora@gmail.com,
>   emacs-devel@gnu.org
> Date: Thu, 20 Apr 2023 17:26:57 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That core packages will be only in elpa.git, not in emacs.git.  Then
> > we'd "bundle" them with Emacs when preparing the release tarballs.
> >
> > I believe this is a long-term plan wrt core packages.  It was even
> > discussed a couple of times here.
> 
> For people building Emacs from source, would this mean that they
> would have to manually install core packages?

No.  We'd need to change either the build process or the structure of
emacs.git (and consequently what happens when you say "git pull") so
that manual installation will not be required.

This was all discussed in the past, and this aspect was mentioned as
one of the issues that would have to be solved.

> Also, this sounds like it would make it impossible for package.el to
> be distributed on ELPA, as proposed in another thread.

It's possible, yes.  But no one has yet got this far into designing
such a system as to figure all that out.  So other alternatives could
be workable.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 18:49, Eli Zaretskii wrote:
>> Date: Thu, 20 Apr 2023 18:30:47 +0300
>> Cc:joaotavora@gmail.com,emacs-devel@gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> When you say "keeping core packages only on ELPA", what do you mean exactly?
> That core packages will be only in elpa.git, not in emacs.git.  Then
> we'd "bundle" them with Emacs when preparing the release tarballs.

Okay then, though I'd call them just "bundled" packages from that point. 
Since "core" is an ELPA term for packages developed inside emacs.git.

That separation could also be considered a win, saving time for some of 
the developers. Which could make up for the extra overhead with managing 
stable tags.

> I believe this is a long-term plan wrt core packages.  It was even
> discussed a couple of times here.

Note that to do the above, we don't really have to implement this stuff 
as a feature in Emacs (as a UI in package.el), or in elpa-admin. At 
least not right away.

As long as we've agreed on a principle to choose releases to tag as 
stable, we could just write those git revision down somewhere (using git 
tags, or some text file in the Emacs repo, or a text file on a server 
somewhere, ...). And then use that info when bundling the packages.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]

On Thu, Apr 20, 2023, 11:50 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Thu, 20 Apr 2023 18:30:47 +0300
> > Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> > From: Dmitry Gutov <dmitry@gutov.dev>
> >
> > When you say "keeping core packages only on ELPA", what do you mean
> exactly?
>
> That core packages will be only in elpa.git, not in emacs.git.  Then
> we'd "bundle" them with Emacs when preparing the release tarballs.
>
> I believe this is a long-term plan wrt core packages.  It was even
> discussed a couple of times here.
>

The search function for the emacs-devel archive is still broken.  For
example, the first result (in chronological ordering) for "concurrency" is
from 2020.

Lynn

[-- Attachment #2: Type: text/html, Size: 1588 bytes --]

Re: What is :core? (was: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot))

[Lynn Winebarger]

On Tue, Apr 18, 2023 at 8:58 AM Eli Zaretskii <eliz@gnu.org> wrote:
> This discussion no longer belongs to the bug tracker, so I'm moving it
> to emacs-devel and changing its Subject.  Please reply here, not
> there.

Is there any documentation for what ":core" refers to?  Executing
"grep -rn ':core' *" in the emacs source yields the following, which
doesn't include any explanation:


ChangeLog.3:81561: Turn Eldoc, Xref and Project into GNU ELPA :core packages
ChangeLog.3:236132: * lisp/emacs-lisp/let-alist.el: Now an Elpa :core package
ChangeLog.4:12044: project.el is a GNU ELPA :core package, so this
kind of trick isn't
lisp/emacs-lisp/eldoc.el:11:;; This is a GNU ELPA :core package.
Avoid functionality that is not
lisp/emacs-lisp/let-alist.el:12:;; This is an Elpa :core package.
Don't use functionality that is not
lisp/jsonrpc.el:10:;; This is a GNU ELPA :core package.  Avoid
functionality that is not
lisp/progmodes/project.el:7:;; This is a GNU ELPA :core package.
Avoid using functionality that
lisp/progmodes/flymake.el:11:;; This is a GNU ELPA :core package.
Avoid functionality that is not
lisp/progmodes/xref.el:7:;; This is a GNU ELPA :core package.  Avoid
functionality that is not
lisp/progmodes/eglot.el:12:;; This is a GNU ELPA :core package.  Avoid
adding functionality
lisp/progmodes/eglot.el:57:;;   available as GNU ELPA :core packages.
Historically, a number of
lisp/progmodes/eglot.el:58:;;   :core packages were added or reworked
in Emacs to make this
test/lisp/progmodes/eglot-tests.el:39:;; IMPORTANT: Since Eglot is a
:core ELPA package, these tests are

Lynn

Re: What is :core? (was: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot))

[João Távora]

No one knows, apparently, :-D

But what it means in practice, is that some files (or sets of files)
in the Emacs.git repository on the master branch are watched daily by the
scripts that govern the distribution of packages  in https://elpa.gnu.org.
If those scripts notice a change in the ";; Version:" header of said files,
a packaged-up version of that code is tar'ed into a file named
package-<version>.tar.lz and archived somewhere in that site.

Then the tar.lz file can be installed in Emacs via 'package-install',
'use-package', or various other third-party means.  Crucially (!) the
package can be installed even in older (sometimes quite a bit older)
versions of Emacs, depending on the value of "Emacs" in the
";; Package-Requires" header.  When installed in these Emacs versions,
the package will dutifully  perform the job _as if_ you were using it
as part of Emacs master version at the commit where the ";; Version:"
header was updated.

The ";; Version:" header of the Emacs.git repository files is
updated when the maintainers of this package deem the code and its
dependencies sufficiently stable and ready to enter this distribution
channel.

All in all, it is, IMHO, the best recent development we have had
to escape the glacially slow frequency of rolling out new features
in Emacs to the general public.  As a user, you don't have to
compile a master Emacs, and -- frequently -- you don't even need to
upgrade your Emacs.

Entire "Emacs distributions" are built around this.
Doom Emacs, for example, requires just Emacs 27.2.  But it gets the
user all the new stuff via ELPA (and MELPA, a cousin of ELPA).

There is an issue that no one foresaw, and that is bug#62720 where
this all started: When a package that wasn't :core becomes :core (it
had never happened but it did shortly before Emacs 29 was cut), then
some of the installation methods some people were using (notably Emacs's
built-in package manager) will start doing unexpected things when given
the same package to install.

João

On Wed, Apr 19, 2023 at 1:31 PM Lynn Winebarger <owinebar@gmail.com> wrote:
>
> On Tue, Apr 18, 2023 at 8:58 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > This discussion no longer belongs to the bug tracker, so I'm moving it
> > to emacs-devel and changing its Subject.  Please reply here, not
> > there.
>
> Is there any documentation for what ":core" refers to?  Executing
> "grep -rn ':core' *" in the emacs source yields the following, which
> doesn't include any explanation:
>
>
> ChangeLog.3:81561: Turn Eldoc, Xref and Project into GNU ELPA :core packages
> ChangeLog.3:236132: * lisp/emacs-lisp/let-alist.el: Now an Elpa :core package
> ChangeLog.4:12044: project.el is a GNU ELPA :core package, so this
> kind of trick isn't
> lisp/emacs-lisp/eldoc.el:11:;; This is a GNU ELPA :core package.
> Avoid functionality that is not
> lisp/emacs-lisp/let-alist.el:12:;; This is an Elpa :core package.
> Don't use functionality that is not
> lisp/jsonrpc.el:10:;; This is a GNU ELPA :core package.  Avoid
> functionality that is not
> lisp/progmodes/project.el:7:;; This is a GNU ELPA :core package.
> Avoid using functionality that
> lisp/progmodes/flymake.el:11:;; This is a GNU ELPA :core package.
> Avoid functionality that is not
> lisp/progmodes/xref.el:7:;; This is a GNU ELPA :core package.  Avoid
> functionality that is not
> lisp/progmodes/eglot.el:12:;; This is a GNU ELPA :core package.  Avoid
> adding functionality
> lisp/progmodes/eglot.el:57:;;   available as GNU ELPA :core packages.
> Historically, a number of
> lisp/progmodes/eglot.el:58:;;   :core packages were added or reworked
> in Emacs to make this
> test/lisp/progmodes/eglot-tests.el:39:;; IMPORTANT: Since Eglot is a
> :core ELPA package, these tests are
>
> Lynn



-- 
João Távora

Re: What is :core? (was: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot))

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Wed, 19 Apr 2023 08:31:16 -0400
> Cc: Dmitry Gutov <dmitry@gutov.dev>,
>  João Távora <joaotavora@gmail.com>, 
>  emacs-devel <emacs-devel@gnu.org>
> 
> Is there any documentation for what ":core" refers to?  Executing
> "grep -rn ':core' *" in the emacs source yields the following, which
> doesn't include any explanation:

It is ELPA terminology, so the answer is in ELPA's file elpa-packages:

  ;; List of packages that are maintained externally.
  ;; The list is made of elements of the form (NAME KIND URL OPTS...).
  ;; See `admin/README' for further documentation about the format.
  ;;
  ;; Where NAME is the name of the package;
  ;;
  ;; KIND can be one of:
  ;;  :url      = kept in a separate `externals/<name>' branch.
  ;;  :core     = part of GNU Emacs repository.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Philip Kaludercic <philipk@posteo.net>,  larsi@gnus.org,
>   62720@debbugs.gnu.org,  joaotavora@gmail.com,  monnier@iro.umontreal.ca
> Date: Fri, 14 Apr 2023 14:34:16 +0200
> 
> >>>>> On Fri, 14 Apr 2023 13:54:11 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> Yes, we can change the default to t on master.  But that would also
>     Eli> require to adjust a few doc strings, see below.
> 
> So on master if I upgrade all packages, ':core' packages would be
> automatically upgraded as well?

is that what the default of that option means?

> I strongly object to that as a default; just because thereʼs a newer
> version on elpa of a :core package doesnʼt mean emacs should upgrade
> to it unless *explicitly* told to do so.

I said we _can_ change the default; I didn't say we _must_.  If enough
people object to making that the default, it won't be changed.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 16:40, Eli Zaretskii wrote:
>> So on master if I upgrade all packages, ':core' packages would be
>> automatically upgraded as well?
> is that what the default of that option means?
> 
>> I strongly object to that as a default; just because thereʼs a newer
>> version on elpa of a :core package doesnʼt mean emacs should upgrade
>> to it unless*explicitly*  told to do so.
> I said we_can_  change the default; I didn't say we_must_.  If enough
> people object to making that the default, it won't be changed.

We need to have a change in behavior that allows 'M-x package-install' 
to upgrade built-in packages. But that shouldn't automatically mean that 
package-menu-mark-upgrades marks all built-in packages for upgrade, or 
package-upgrade-all (nee package-update-all) does that either.

We could have another option that enables the latter to upgrade all 
built-ins too, of course.

Regarding the currently proposed user option, does it make sense to you 
to have such option that decides whether package-install upgrades 
built-ins? Whereas one can always upgrade a built-in package using 'i' 
(package-menu-mark-install) in the list-packages menu, no matter the 
value of that option. I get the backward-compatibility intent, but user 
options should also do something logical from a user's point of view.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 14 Apr 2023 19:04:20 +0300
> Cc: 62720@debbugs.gnu.org, philipk@posteo.net, larsi@gnus.org,
>  monnier@iro.umontreal.ca, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 14/04/2023 16:40, Eli Zaretskii wrote:
> >> So on master if I upgrade all packages, ':core' packages would be
> >> automatically upgraded as well?
> > is that what the default of that option means?
> > 
> >> I strongly object to that as a default; just because thereʼs a newer
> >> version on elpa of a :core package doesnʼt mean emacs should upgrade
> >> to it unless*explicitly*  told to do so.
> > I said we_can_  change the default; I didn't say we_must_.  If enough
> > people object to making that the default, it won't be changed.
> 
> We need to have a change in behavior that allows 'M-x package-install' 
> to upgrade built-in packages. But that shouldn't automatically mean that 
> package-menu-mark-upgrades marks all built-in packages for upgrade, or 
> package-upgrade-all (nee package-update-all) does that either.
> 
> We could have another option that enables the latter to upgrade all 
> built-ins too, of course.
> 
> Regarding the currently proposed user option, does it make sense to you 
> to have such option that decides whether package-install upgrades 
> built-ins? Whereas one can always upgrade a built-in package using 'i' 
> (package-menu-mark-install) in the list-packages menu, no matter the 
> value of that option. I get the backward-compatibility intent, but user 
> options should also do something logical from a user's point of view.

All these questions should have been raised and discussed months ago.
Then we could have done the best for the users (what that is, is still
unclear even at this point).  Now it's too late, and the main factor
that will decide how Emacs 29.1 will behave in that regard is whether
the changes to implement that are safe enough to go into Emacs 29.1.

To answer your question more specifically: yes, it does make sense to
me.  The logic, so it seems, is in the eyes of the beholder, and my
eyes do see a certain logic there.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 14/04/2023 20:43, Eli Zaretskii wrote:
>> Regarding the currently proposed user option, does it make sense to you
>> to have such option that decides whether package-install upgrades
>> built-ins? Whereas one can always upgrade a built-in package using 'i'
>> (package-menu-mark-install) in the list-packages menu, no matter the
>> value of that option. I get the backward-compatibility intent, but user
>> options should also do something logical from a user's point of view.
> All these questions should have been raised and discussed months ago.
> Then we could have done the best for the users (what that is, is still
> unclear even at this point).  Now it's too late, and the main factor
> that will decide how Emacs 29.1 will behave in that regard is whether
> the changes to implement that are safe enough to go into Emacs 29.1.

We certainly dropped the ball on this.

> To answer your question more specifically: yes, it does make sense to
> me.  The logic, so it seems, is in the eyes of the beholder, and my
> eyes do see a certain logic there.

Perhaps it does, and my concern is in vain.

But if we ultimately decide that it has weird semantics and adds 
undesirable cost to improving the package further, the 
obsoletion-deprecation cycle has always been a pain, so I really prefer 
to solve these issues without adding options, when possible.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Fri, 14 Apr 2023 20:47:01 +0300
> Cc: rpluim@gmail.com, 62720@debbugs.gnu.org, philipk@posteo.net,
>  larsi@gnus.org, monnier@iro.umontreal.ca, joaotavora@gmail.com
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> I really prefer to solve these issues without adding options, when
> possible.

As a general principle, I cannot agree more, of course.  But we are
not talking in general here, and have some unique constraints as well.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

Philip,

On 13/04/2023 21:49, Philip Kaludercic wrote:
> +    (when (and (or current-prefix-arg package-install-upgrade-built-in)
> +               (package--active-built-in-p pkg))
> +      (setq pkg (cadr (assq name package-archive-contents))))

How sure are you that the first element in (cdr (assq name 
package-archive-contents)) will contain the latest version?

When reading the code I stumbled on these two items:

- Comment in package-compute-transaction:

                 ;; pkg-descs is sorted by priority, not version, so
                 ;; don't error just yet.

- package-menu--find-upgrades iterates through installed packages and 
checks for new versions with this condition:

              (version-list-< (package-desc-priority-version pkg-desc)
                              (package-desc-priority-version avail-pkg))

That's why I came with the more complex way to choose the appropriate 
package-desc in my latest attempt:

  (car
   (last (seq-sort-by #'package-desc-priority-version
                      #'version-list-<
                      (cdr (assq package package-archive-contents)))))

Would be happy to hear it's an overkill and we can use the more simple 
version.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Philip,
>
> On 13/04/2023 21:49, Philip Kaludercic wrote:
>> +    (when (and (or current-prefix-arg package-install-upgrade-built-in)
>> +               (package--active-built-in-p pkg))
>> +      (setq pkg (cadr (assq name package-archive-contents))))
>
> How sure are you that the first element in (cdr (assq name
> package-archive-contents)) will contain the latest version?

This is not certain, but the same approach is used in other places in
package.el, so I just stocked to it.

> When reading the code I stumbled on these two items:
>
> - Comment in package-compute-transaction:
>
>                 ;; pkg-descs is sorted by priority, not version, so
>                 ;; don't error just yet.
>
> - package-menu--find-upgrades iterates through installed packages and
>   checks for new versions with this condition:
>
>              (version-list-< (package-desc-priority-version pkg-desc)
>                              (package-desc-priority-version avail-pkg))
>
> That's why I came with the more complex way to choose the appropriate
> package-desc in my latest attempt:
>
>  (car
>   (last (seq-sort-by #'package-desc-priority-version
>                      #'version-list-<
>                      (cdr (assq package package-archive-contents)))))

If we insist on package.el installing the newest version, then this
would make sense.  AFAIK there is no guarantees on the order of packages
in `package-archive-contents'.  This shouldn't be an issue for Eglot,
but might be one for use-package.  I would actually pull it out into a
separate utility function that we would re-use in other places.

> Would be happy to hear it's an overkill and we can use the more simple
> version.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 23/04/2023 16:02, Philip Kaludercic wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> Philip,
>>
>> On 13/04/2023 21:49, Philip Kaludercic wrote:
>>> +    (when (and (or current-prefix-arg package-install-upgrade-built-in)
>>> +               (package--active-built-in-p pkg))
>>> +      (setq pkg (cadr (assq name package-archive-contents))))
>>
>> How sure are you that the first element in (cdr (assq name
>> package-archive-contents)) will contain the latest version?
> 
> This is not certain, but the same approach is used in other places in
> package.el, so I just stocked to it.

Perhaps they conceal latent bugs as well. I don't know the codebase too 
well, but the places I did examine used the comparison together with the 
priority.

>> That's why I came with the more complex way to choose the appropriate
>> package-desc in my latest attempt:
>>
>>   (car
>>    (last (seq-sort-by #'package-desc-priority-version
>>                       #'version-list-<
>>                       (cdr (assq package package-archive-contents)))))
> 
> If we insist on package.el installing the newest version, then this
> would make sense.

What's the alternative? Upgrading to "some version"?

> AFAIK there is no guarantees on the order of packages
> in `package-archive-contents'.  This shouldn't be an issue for Eglot,
> but might be one for use-package.

Ah, it seems they removed it from Melpa, that shouldn't surprise me.

Eglot is in the GNU-devel archive as well, though. So there is some 
potential for conflict.

 > I would actually pull it out into a
 > separate utility function that we would re-use in other places.

Since I had to drop the respective code from the latest version of the 
patch, I guess you could put it in package-install instead.

A reusable helper is fine, of course, but what are the other places we 
would use it at?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 23/04/2023 16:02, Philip Kaludercic wrote:
>> Dmitry Gutov <dmitry@gutov.dev> writes:
>> 
>>> Philip,
>>>
>>> On 13/04/2023 21:49, Philip Kaludercic wrote:
>>>> +    (when (and (or current-prefix-arg package-install-upgrade-built-in)
>>>> +               (package--active-built-in-p pkg))
>>>> +      (setq pkg (cadr (assq name package-archive-contents))))
>>>
>>> How sure are you that the first element in (cdr (assq name
>>> package-archive-contents)) will contain the latest version?
>> This is not certain, but the same approach is used in other places
>> in
>> package.el, so I just stocked to it.
>
> Perhaps they conceal latent bugs as well. I don't know the codebase
> too well, but the places I did examine used the comparison together
> with the priority.

I will look into this.

>>> That's why I came with the more complex way to choose the appropriate
>>> package-desc in my latest attempt:
>>>
>>>   (car
>>>    (last (seq-sort-by #'package-desc-priority-version
>>>                       #'version-list-<
>>>                       (cdr (assq package package-archive-contents)))))
>> If we insist on package.el installing the newest version, then this
>> would make sense.
>
> What's the alternative? Upgrading to "some version"?

Respect `package-archive-priorities'.

>> AFAIK there is no guarantees on the order of packages
>> in `package-archive-contents'.  This shouldn't be an issue for Eglot,
>> but might be one for use-package.
>
> Ah, it seems they removed it from Melpa, that shouldn't surprise me.

Oh, I did not know it was on MELPA in the first place.  I haven't been
using the archive for a while.

> Eglot is in the GNU-devel archive as well, though. So there is some
> potential for conflict.

Hmm, I guess that is true, but I have never been a fan of advertising
the -devel archives for day-to-day usage.

>> I would actually pull it out into a
>> separate utility function that we would re-use in other places.
>
> Since I had to drop the respective code from the latest version of the
> patch, I guess you could put it in package-install instead.
>
> A reusable helper is fine, of course, but what are the other places we
> would use it at?

At any place where (cadr ...) is used, I would assume.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> If we insist on package.el installing the newest version, then this
> would make sense.

I don't see any reason we should ignore priorities here, so we shouldn't
blindly aim for the "newest version".


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Why did the original code use symbol-name, but the new one doesn't?
> To my knowledge, completing-read given a collection of symbols will use
> the symbol names as candidates, or is this more complicated?

It mostly works.  Strictly speaking a completion table can be either:
- a list of strings
- an alist whose keys are either strings or symbols
and the code handles those two cases together so a list of symbols tend
to work just as well, but if your first symbol happens to be `lambda` or
`closure`, you're out of luck.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Philip Kaludercic <philipk@posteo.net>
>>> Cc: João Távora <joaotavora@gmail.com>,
>>>   monnier@iro.umontreal.ca,
>>>   62720@debbugs.gnu.org,  larsi@gnus.org
>>> Date: Wed, 12 Apr 2023 19:39:20 +0000
>>> 
>>> >> Please, in normal non-shouting case, explain to me how you think
>>> >> that the behavior of an existing
>>> >> command can be changed with "completely separate code". 
>>> >
>>> > I already did: either (1) add a prefix argument to an existing
>>> > command, which will then trigger the new behavior, or (2) add a
>>> > separate command.
>>> 
>>> Here you have (1):
>>
>> Thanks.  This is almost on-target, but it modifies
>> package-compute-transaction.  Is that necessary?
>
> I have found an alternative that doesn't change the way
> `package-compute-transaction' works, but requires a small change in
> `package-install':

I have found a smaller but equivalent change that would also solve the
issue:


[-- Attachment #2: Type: text/plain, Size: 2782 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..842a475290d 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2187,7 +2198,9 @@ package-install
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
 available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+called interactively, prompt for the package name.  When invoked
+with a prefix argument, the prompt will include built-in packages
+that can be upgraded via an archive.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2205,11 +2218,13 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and current-prefix-arg
+                                     (package--upgradable-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2235,8 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (package--upgradable-built-in-p pkg)
+      (setq pkg (cadr (assq name package-archive-contents))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)

[-- Attachment #3: Type: text/plain, Size: 381 bytes --]


This relies on the fact that package-install handles package descriptor
objects in a different way than when you just call the function with a
package name.

This would also mean that an alternative solution to this issue would be
to tell users to evaluate

  (package-install (cadr (assq 'eglot package-archive-contents)))

but I don't know how user-friendly of an idea this is.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 15:14:51 +0000
> 
> > I have found an alternative that doesn't change the way
> > `package-compute-transaction' works, but requires a small change in
> > `package-install':
> 
> I have found a smaller but equivalent change that would also solve the
> issue:

Are we still sure we want to change package-install, not
package-upgrade?  AFAIU, there were several voices that preferred the
latter, with the rationale that a built-in package is always
"installed", so "installing" it makes little or no sense.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 13/04/2023 18:59, Eli Zaretskii wrote:
>> From: Philip Kaludercic<philipk@posteo.net>
>> Cc:joaotavora@gmail.com,monnier@iro.umontreal.ca,62720@debbugs.gnu.org,
>>    larsi@gnus.org
>> Date: Thu, 13 Apr 2023 15:14:51 +0000
>>
>>> I have found an alternative that doesn't change the way
>>> `package-compute-transaction' works, but requires a small change in
>>> `package-install':
>> I have found a smaller but equivalent change that would also solve the
>> issue:
> Are we still sure we want to change package-install, not
> package-upgrade?  AFAIU, there were several voices that preferred the
> latter, with the rationale that a built-in package is always
> "installed", so "installing" it makes little or no sense.

Note that this notion would probably go better together with changing 
package-install's behavior to never update, for any packages that are 
already installed. That would make it consistent, but it's also a 
breaking change.

So I really just mentioned it as a weak justification for not fixing 
package-install just now (so we'd make the decision about it later). 
Where the main cause for (not) doing that was your reticence for 
changing it this late in the cycle.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 15:14:51 +0000
>> 
>> > I have found an alternative that doesn't change the way
>> > `package-compute-transaction' works, but requires a small change in
>> > `package-install':
>> 
>> I have found a smaller but equivalent change that would also solve the
>> issue:
>
> Are we still sure we want to change package-install, not
> package-upgrade?  AFAIU, there were several voices that preferred the
> latter, with the rationale that a built-in package is always
> "installed", so "installing" it makes little or no sense.

If you want that, here is one proposal.  Should be the smallest so far:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages-with-package-insta.patch --]
[-- Type: text/x-diff, Size: 4855 bytes --]

From 99e92d78560ef1aec7217d90ed3a8108bdf2924c Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 13 Apr 2023 20:13:59 +0200
Subject: [PATCH] Allow upgrading built-in packages with 'package-install'

* etc/NEWS: Mention the change
* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
predicate.
(package-install-upgrade-built-in): Add new user option to enable
feature.
(package-install): Respect new user option.
---
 etc/NEWS                   |  5 +++++
 lisp/emacs-lisp/package.el | 44 +++++++++++++++++++++++++++++++-------
 2 files changed, 41 insertions(+), 8 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 84dbb94a71a..a7834cd0d2b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1876,6 +1876,11 @@ package maintainers.
 By customizing this user option you can specify specific packages to
 install.
 
+---
+*** New user option 'package-install-upgrade-built-in'.
+When enabled, 'package-install' can be used to install potentially
+newer versions of built-in packages.
+
 ** Emacs Sessions (Desktop)
 
 +++
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..882db8db719 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--active-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2182,12 +2193,19 @@ package--archives-initialize
   (unless package-archive-contents
     (package-refresh-contents)))
 
+(defcustom package-install-upgrade-built-in nil
+  "Non-nil means that built-in packages can be upgraded via a package archive.
+If disabled, then `package-install' will not allow installing
+potentially newer versions of built-in packages from package
+archives."
+  :type 'boolean
+  :version "29.1")
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
-available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+available packages in an archive in `package-archives'.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2197,7 +2215,11 @@ package-install
 `package-selected-packages'.
 
 If PKG is a `package-desc' and it is already installed, don't try
-to install it but still mark it as selected."
+to install it but still mark it as selected.
+
+If the command is invoked with a prefix argument, the upgrading
+of built-in packages will be possible, as if
+`package-install-upgrade-built-in' had been enabled."
   (interactive
    (progn
      ;; Initialize the package system to get the list of package
@@ -2205,11 +2227,14 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and (or current-prefix-arg
+                                         package-install-upgrade-built-in)
+                                     (package--active-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (symbol-name (car elt)))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2245,9 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (and (or current-prefix-arg package-install-upgrade-built-in)
+               (package--active-built-in-p pkg))
+      (setq pkg (cadr (assq name package-archive-contents))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Thu, 13 Apr 2023 19:14:37 +0000
> 
> > Are we still sure we want to change package-install, not
> > package-upgrade?  AFAIU, there were several voices that preferred the
> > latter, with the rationale that a built-in package is always
> > "installed", so "installing" it makes little or no sense.
> 
> If you want that, here is one proposal.  Should be the smallest so far:

Hmm... looks identical to the previous patch you sent, which changes
package-install?  Or what am I missing?

As for the question I asked: I'm okay with changing package-install if
there are no objections from those who proposed to change
package-upgrade instead.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Thu, 13 Apr 2023 19:14:37 +0000
>> 
>> > Are we still sure we want to change package-install, not
>> > package-upgrade?  AFAIU, there were several voices that preferred the
>> > latter, with the rationale that a built-in package is always
>> > "installed", so "installing" it makes little or no sense.
>> 
>> If you want that, here is one proposal.  Should be the smallest so far:
>
> Hmm... looks identical to the previous patch you sent, which changes
> package-install?  Or what am I missing?
>
> As for the question I asked: I'm okay with changing package-install if
> there are no objections from those who proposed to change
> package-upgrade instead.

Did I send the wrong patch.  I double-checked now and this one should be
using package-update:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-package-update-to-upgrade-built-in-packages.patch --]
[-- Type: text/x-diff, Size: 2283 bytes --]

From 7b3ca355785b7539dd417317442054647e66c045 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 13 Apr 2023 21:12:31 +0200
Subject: [PATCH] Allow 'package-update' to upgrade built-in packages

* lisp/emacs-lisp/package.el (package-update): Add support for
upgrading a built-in package to a version available from
ELPA.  (Bug#62720)
---
 lisp/emacs-lisp/package.el | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..8e7bc115a73 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2236,18 +2236,32 @@ package-install
 
 ;;;###autoload
 (defun package-update (name)
-  "Update package NAME if a newer version exists."
+  "Update package NAME if a newer version exists.
+When invoked with a prefix argument, the prompt will also include
+built-in packages that can be upgraded via a package archive."
   (interactive
    (list (completing-read
-          "Update package: " (package--updateable-packages) nil t)))
+          "Update package: "
+          (append
+           (and current-prefix-arg
+                (mapcan
+                 (lambda (elt)
+                   (and (not (assq (car elt) (package--alist)))
+                        (package-built-in-p (car elt))
+                        (list (symbol-name (car elt)))))
+                 package-archive-contents))
+           (package--updateable-packages))
+          nil t)))
   (let* ((package (if (symbolp name)
                       name
                     (intern name)))
-         (pkg-desc (cadr (assq package package-alist))))
-    (if (package-vc-p pkg-desc)
-        (package-vc-update pkg-desc)
-      (package-delete pkg-desc 'force)
-      (package-install package 'dont-select))))
+         (old-desc (cadr (assq package package-alist)))
+         (new-desc (cadr (assq package package-archive-contents))))
+    (if (and old-desc (package-vc-p old-desc))
+        (package-vc-update old-desc)
+      (when old-desc
+        (package-delete old-desc 'force))
+      (package-install new-desc 'dont-select))))
 
 (defun package--updateable-packages ()
   ;; Initialize the package system to get the list of package
-- 
2.39.2

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Fri, 14 Apr 2023 16:40:01 +0000
> 
> > Hmm... looks identical to the previous patch you sent, which changes
> > package-install?  Or what am I missing?
> >
> > As for the question I asked: I'm okay with changing package-install if
> > there are no objections from those who proposed to change
> > package-upgrade instead.
> 
> Did I send the wrong patch.  I double-checked now and this one should be
> using package-update:

Thanks.  It doesn't seem to be simpler than the change in
package-install, perhaps even more complex than that.

What is your preference? change package-install or package-update?  I
tend to the former.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Fri, 14 Apr 2023 16:40:01 +0000
>> 
>> > Hmm... looks identical to the previous patch you sent, which changes
>> > package-install?  Or what am I missing?
>> >
>> > As for the question I asked: I'm okay with changing package-install if
>> > there are no objections from those who proposed to change
>> > package-upgrade instead.
>> 
>> Did I send the wrong patch.  I double-checked now and this one should be
>> using package-update:
>
> Thanks.  It doesn't seem to be simpler than the change in
> package-install, perhaps even more complex than that.
>
> What is your preference? change package-install or package-update?  I
> tend to the former.

I have no preference either way.  If you think that package-install is
fine, and João has expressed interest in that route as well, we might as
well go that way.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 10:41:52 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What is your preference? change package-install or package-update?  I
> > tend to the former.
> 
> I have no preference either way.  If you think that package-install is
> fine, and João has expressed interest in that route as well, we might as
> well go that way.

OK, then let's do that, and thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 10:41:52 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > What is your preference? change package-install or package-update?  I
>> > tend to the former.
>> 
>> I have no preference either way.  If you think that package-install is
>> fine, and João has expressed interest in that route as well, we might as
>> well go that way.
>
> OK, then let's do that, and thanks.

Great, then just to that we are on the same page, what approach do we
finally want to decide on?

- User option to enable upgrading built-in packages
- Prefix argument to enable upgrading built-in packages
- Always upgrade built-in packages

I argue the last option should be safe.  Semantically it would also make
sense, since invoking the command can be taken to be take to be an
explicit request, and if it is not what a user wants (I assume that João
think this is not probable), then it is easy to revert.

If we decide that this is not acceptable, then we can fall back onto the
patch that uses a prefix argument or a user option.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 11:37:40 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I have no preference either way.  If you think that package-install is
> >> fine, and João has expressed interest in that route as well, we might as
> >> well go that way.
> >
> > OK, then let's do that, and thanks.
> 
> Great, then just to that we are on the same page, what approach do we
> finally want to decide on?
> 
> - User option to enable upgrading built-in packages
> - Prefix argument to enable upgrading built-in packages
> - Always upgrade built-in packages

The first two on emacs-29, the last one on master (if enough people
think it's a good idea; me, I think we should wait for a while before
deciding).

> I argue the last option should be safe.  Semantically it would also make
> sense, since invoking the command can be taken to be take to be an
> explicit request, and if it is not what a user wants (I assume that João
> think this is not probable), then it is easy to revert.
> 
> If we decide that this is not acceptable, then we can fall back onto the
> patch that uses a prefix argument or a user option.

Yes, that is what I think we should install on emacs-29.

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 961 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 11:37:40 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> I have no preference either way.  If you think that package-install is
>> >> fine, and João has expressed interest in that route as well, we might as
>> >> well go that way.
>> >
>> > OK, then let's do that, and thanks.
>> 
>> Great, then just to that we are on the same page, what approach do we
>> finally want to decide on?
>> 
>> - User option to enable upgrading built-in packages
>> - Prefix argument to enable upgrading built-in packages
>> - Always upgrade built-in packages
>
> The first two on emacs-29, the last one on master (if enough people
> think it's a good idea; me, I think we should wait for a while before
> deciding).

OK, so let us use this change:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages-with-package-insta.patch --]
[-- Type: text/x-diff, Size: 2782 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..842a475290d 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,17 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--upgradable-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2187,7 +2198,9 @@ package-install
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
 available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+called interactively, prompt for the package name.  When invoked
+with a prefix argument, the prompt will include built-in packages
+that can be upgraded via an archive.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2205,11 +2218,13 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and current-prefix-arg
+                                     (package--upgradable-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (car elt))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2235,8 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (package--upgradable-built-in-p pkg)
+      (setq pkg (cadr (assq name package-archive-contents))))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)

[-- Attachment #3: Type: text/plain, Size: 749 bytes --]


(I intentionally picked this one without the user option since that
will probably become unnecessary with Emacs 30+).  I believe there was
an issue with the name `package--upgradable-built-in-p' and the
docstring?

>> I argue the last option should be safe.  Semantically it would also make
>> sense, since invoking the command can be taken to be take to be an
>> explicit request, and if it is not what a user wants (I assume that João
>> think this is not probable), then it is easy to revert.
>> 
>> If we decide that this is not acceptable, then we can fall back onto the
>> patch that uses a prefix argument or a user option.
>
> Yes, that is what I think we should install on emacs-29.
>
> Thanks.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 13:21:53 +0000
> 
> >> - User option to enable upgrading built-in packages
> >> - Prefix argument to enable upgrading built-in packages
> >> - Always upgrade built-in packages
> >
> > The first two on emacs-29, the last one on master (if enough people
> > think it's a good idea; me, I think we should wait for a while before
> > deciding).
> 
> OK, so let us use this change:
> 
> (I intentionally picked this one without the user option since that
> will probably become unnecessary with Emacs 30+).

The user option allows those users who always want package-install to
upgrade core package to have what they want, easily.  So I think we
should keep it.  On master, the option could be t by default, or
become unnecessary if that's what happens (but I wouldn't bet on
that).

> I believe there was an issue with the name
> `package--upgradable-built-in-p' and the docstring?

Yes.  The doc string has a typo:

  "Return non-nil if PACKAGE if the built-in version is used."

See those two "if"s?  And even if I replace the second "if" with "is",
the sentence doesn't make sense.

As for the name, I think we can leave it at that.

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 13:21:53 +0000
>> 
>> >> - User option to enable upgrading built-in packages
>> >> - Prefix argument to enable upgrading built-in packages
>> >> - Always upgrade built-in packages
>> >
>> > The first two on emacs-29, the last one on master (if enough people
>> > think it's a good idea; me, I think we should wait for a while before
>> > deciding).
>> 
>> OK, so let us use this change:
>> 
>> (I intentionally picked this one without the user option since that
>> will probably become unnecessary with Emacs 30+).
>
> The user option allows those users who always want package-install to
> upgrade core package to have what they want, easily.  So I think we
> should keep it.  On master, the option could be t by default, or
> become unnecessary if that's what happens (but I wouldn't bet on
> that).

My argument against a user option is just that the whole deal is
something that will in practice at most affect two packages (if we
change the behaviour in Emacs 29).  Is it really worth adding a general
option for this very specific situation?

>> I believe there was an issue with the name
>> `package--upgradable-built-in-p' and the docstring?
>
> Yes.  The doc string has a typo:
>
>   "Return non-nil if PACKAGE if the built-in version is used."
>
> See those two "if"s?  And even if I replace the second "if" with "is",
> the sentence doesn't make sense.

Right, how does

  "Return non-nil if the built-in version of PACKAGE is used."

sound?

> As for the name, I think we can leave it at that.

Ok.

> Thanks.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 17:14:41 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The user option allows those users who always want package-install to
> > upgrade core package to have what they want, easily.  So I think we
> > should keep it.  On master, the option could be t by default, or
> > become unnecessary if that's what happens (but I wouldn't bet on
> > that).
> 
> My argument against a user option is just that the whole deal is
> something that will in practice at most affect two packages (if we
> change the behaviour in Emacs 29).  Is it really worth adding a general
> option for this very specific situation?

I think we should count users of those packages, not just the packages
themselves.  Yes, I think it's worth it, because we don't know how
many of the users will want the built-in packages to be included in an
update.

> >   "Return non-nil if PACKAGE if the built-in version is used."
> >
> > See those two "if"s?  And even if I replace the second "if" with "is",
> > the sentence doesn't make sense.
> 
> Right, how does
> 
>   "Return non-nil if the built-in version of PACKAGE is used."
> 
> sound?

I think we should explain what does "the built-in version of PACKAGE
is used" mean, in the context in which this predicate is used.  Maybe
say something like

    Return non-nil if the built-in version of PACKAGE is used.
  If the built-in version of PACKAGE is used and PACKAGE is
  also available for installation from an archive, it is an
  indication that PACKAGE was never upgraded to any newer
  version from the archive.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 17:14:41 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > The user option allows those users who always want package-install to
>> > upgrade core package to have what they want, easily.  So I think we
>> > should keep it.  On master, the option could be t by default, or
>> > become unnecessary if that's what happens (but I wouldn't bet on
>> > that).
>> 
>> My argument against a user option is just that the whole deal is
>> something that will in practice at most affect two packages (if we
>> change the behaviour in Emacs 29).  Is it really worth adding a general
>> option for this very specific situation?
>
> I think we should count users of those packages, not just the packages
> themselves.  Yes, I think it's worth it, because we don't know how
> many of the users will want the built-in packages to be included in an
> update.

OK, see below.

>> >   "Return non-nil if PACKAGE if the built-in version is used."
>> >
>> > See those two "if"s?  And even if I replace the second "if" with "is",
>> > the sentence doesn't make sense.
>> 
>> Right, how does
>> 
>>   "Return non-nil if the built-in version of PACKAGE is used."
>> 
>> sound?
>
> I think we should explain what does "the built-in version of PACKAGE
> is used" mean, in the context in which this predicate is used.  Maybe
> say something like
>
>     Return non-nil if the built-in version of PACKAGE is used.
>   If the built-in version of PACKAGE is used and PACKAGE is
>   also available for installation from an archive, it is an
>   indication that PACKAGE was never upgraded to any newer
>   version from the archive.

Sounds good to me.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-upgrading-built-in-packages-with-package-insta.patch --]
[-- Type: text/x-diff, Size: 5042 bytes --]

From dae52770fa50bb68a4fdd0983de7811e427733a5 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 13 Apr 2023 20:13:59 +0200
Subject: [PATCH] Allow upgrading built-in packages with 'package-install'

* etc/NEWS: Mention the change
* lisp/emacs-lisp/package.el (package--upgradable-built-in-p): Add new
predicate.
(package-install-upgrade-built-in): Add new user option to enable
feature.
(package-install): Respect new user option.
---
 etc/NEWS                   |  5 ++++
 lisp/emacs-lisp/package.el | 47 +++++++++++++++++++++++++++++++-------
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 84dbb94a71a..a7834cd0d2b 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1876,6 +1876,11 @@ package maintainers.
 By customizing this user option you can specify specific packages to
 install.
 
+---
+*** New user option 'package-install-upgrade-built-in'.
+When enabled, 'package-install' can be used to install 
+newer versions of built-in packages.
+
 ** Emacs Sessions (Desktop)
 
 +++
diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index f92afe56b76..c0cc7bebeb2 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -797,6 +797,21 @@ package-built-in-p
         (require 'finder-inf nil t) ; For `package--builtins'.
         (assq package package--builtins))))))
 
+(defun package--active-built-in-p (package)
+  "Return non-nil if PACKAGE if the built-in version is used.
+If the built-in version of PACKAGE is used and PACKAGE is
+also available for installation from an archive, it is an
+indication that PACKAGE was never upgraded to any newer
+version from the archive."
+  (and (not (assq (cond
+                   ((package-desc-p package)
+                    (package-desc-name package))
+                   ((stringp package) (intern package))
+                   ((symbolp package) package)
+                   ((error "Unknown package format: %S" package)))
+                  (package--alist)))
+       (package-built-in-p package)))
+
 (defun package--autoloads-file-name (pkg-desc)
   "Return the absolute name of the autoloads file, sans extension.
 PKG-DESC is a `package-desc' object."
@@ -2182,12 +2197,18 @@ package--archives-initialize
   (unless package-archive-contents
     (package-refresh-contents)))
 
+(defcustom package-install-upgrade-built-in nil
+  "Non-nil means that built-in packages can be upgraded via a package archive.
+If disabled, then `package-install' will not suggest to replace a
+built-in package with a version from a package archive."
+  :type 'boolean
+  :version "29.1")
+
 ;;;###autoload
 (defun package-install (pkg &optional dont-select)
   "Install the package PKG.
 PKG can be a `package-desc' or a symbol naming one of the
-available packages in an archive in `package-archives'.  When
-called interactively, prompt for the package name.
+available packages in an archive in `package-archives'.
 
 Mark the installed package as selected by adding it to
 `package-selected-packages'.
@@ -2197,7 +2218,11 @@ package-install
 `package-selected-packages'.
 
 If PKG is a `package-desc' and it is already installed, don't try
-to install it but still mark it as selected."
+to install it but still mark it as selected.
+
+If the command is invoked with a prefix argument, the upgrading
+of built-in packages will be possible, as if
+`package-install-upgrade-built-in' had been enabled."
   (interactive
    (progn
      ;; Initialize the package system to get the list of package
@@ -2205,11 +2230,14 @@ package-install
      (package--archives-initialize)
      (list (intern (completing-read
                     "Install package: "
-                    (delq nil
-                          (mapcar (lambda (elt)
-                                    (unless (package-installed-p (car elt))
-                                      (symbol-name (car elt))))
-                                  package-archive-contents))
+                    (mapcan
+                     (lambda (elt)
+                       (and (or (and (or current-prefix-arg
+                                         package-install-upgrade-built-in)
+                                     (package--active-built-in-p (car elt)))
+                                (not (package-installed-p (car elt))))
+                            (list (symbol-name (car elt)))))
+                     package-archive-contents)
                     nil t))
            nil)))
   (package--archives-initialize)
@@ -2220,6 +2248,9 @@ package-install
     (unless (or dont-select (package--user-selected-p name))
       (package--save-selected-packages
        (cons name package-selected-packages)))
+    (when (and (or current-prefix-arg package-install-upgrade-built-in)
+               (package--active-built-in-p pkg))
+      (setq pkg (or (cadr (assq name package-archive-contents)) pkg)))
     (if-let* ((transaction
                (if (package-desc-p pkg)
                    (unless (package-installed-p pkg)
-- 
2.30.2


[-- Attachment #3: Type: text/plain, Size: 23 bytes --]


-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sat, 15 Apr 2023 18:19:31 +0000
> 
> Sounds good to me.

The patch LGTM, thanks.  Please install it.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
>> Date: Sat, 15 Apr 2023 18:19:31 +0000
>> 
>> Sounds good to me.
>
> The patch LGTM, thanks.  Please install it.

Done.

-- 
Philip Kaludercic

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> Date: Sun, 16 Apr 2023 13:45:24 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Philip Kaludercic <philipk@posteo.net>
> >> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org
> >> Date: Sat, 15 Apr 2023 18:19:31 +0000
> >> 
> >> Sounds good to me.
> >
> > The patch LGTM, thanks.  Please install it.
> 
> Done.

Thanks, I'm therefore closing this bug.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Philip Kaludercic <philipk@posteo.net> writes:

> +---
> +*** New user option 'package-install-upgrade-built-in'.
> +When enabled, 'package-install' can be used to install 
> +newer versions of built-in packages.
> +

If this is the final version, at least please add to NEWS:

"Note that you must set this variable to t early in your configuration
if you expect to keep the existing behaviour of the Eglot and
'use-package' libraries after the upgrade to Emacs 29."

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Kévin Le Gouguec]

João Távora <joaotavora@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> +---
>> +*** New user option 'package-install-upgrade-built-in'.
>> +When enabled, 'package-install' can be used to install 
>> +newer versions of built-in packages.
>> +
>
> If this is the final version, at least please add to NEWS:
>
> "Note that you must set this variable to t early in your configuration
> if you expect to keep the existing behaviour of the Eglot and
> 'use-package' libraries after the upgrade to Emacs 29."

And/or something to the same effect in those packages's entries, which
might have more chances of getting noticed?


diff --git a/etc/NEWS b/etc/NEWS
index 0789fa49d75..a2160fa2a92 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -3249,6 +3249,12 @@ based on data provided by language servers using the Language Server
 Protocol (LSP).  See the new Info manual "(eglot) Top" for more.  Also
 see "etc/EGLOT-NEWS".
 
+Note that by default, Emacs's package manager does not fetch updates
+to built-in packages.  If you would like to upgrade to new versions of
+Eglot as they are released on GNU ELPA, enable the new user option
+'package-install-upgrade-built-in' and re-install Eglot with
+'package-install'.
+
 +++
 ** use-package: Declarative package configuration.
 use-package is now shipped with Emacs.  It provides the 'use-package'
@@ -3256,6 +3262,12 @@ macro, which allows you to isolate package configuration in your init
 file in a way that is declarative, tidy, and performance-oriented.
 See the new Info manual "(use-package) Top" for more.
 
+Note that by default, Emacs's package manager does not fetch updates
+to built-in packages.  If you would like to upgrade to new versions of
+use-package as they are released on GNU ELPA, enable the new user
+option 'package-install-upgrade-built-in' and re-install use-package
+with 'package-install'.
+
 ---
 ** New package 'wallpaper'.
 This package provides the command 'wallpaper-set', which sets the

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>>   larsi@gnus.org
>> Date: Wed, 12 Apr 2023 13:42:56 +0000
>> 
>> >> After thinking about this for a bit, I think that the right approach is
>> >> to use package-install instead of writing a separate command.  After
>> >> all, this will make the behaviour of package-install consistent with
>> >> that of the package menu.
>> >
>> > Is this for master or for the release branch?
>> 
>> Personally I am indifferent, it should be compatible with both
>
> If we want to install this on the release branch, the changes must be
> very safe, which in this case basically means "do not touch any code
> used when updating non-core packages".
>
> If the patch you presented is all there is to it, then I'm afraid it
> doesn't satisfy the above condition, because it does affect the use
> case of updating an ELPA package that is not in core.  So I cannot
> agree to installing this on emacs-29 in the form you posted, sorry.

I understand your concern, 

>> > And I thought we all agreed built-in packages need special treatment
>> > anyway, didn't we?  Then why having a separate command is not a
>> > natural next step?
>> 
>> I don't necessarily agree that "special treatment" requires a separate
>> command.
>
> Even if you don't agree with that in general, having a separate
> command would allow us to install that command on emacs-29 without any
> fears.  So that alone is a significant advantage, even if the rest are
> not yet agreed upon.

Would this be a permanent thing, or would the command be deprecated by
emacs-30?  I get the need under the circumstances, but it doesn't seem
like the best solution if we were to set aside release-concerns.

>> I think it is wrong the assume that an built-in package should
>> automatically be updated to a ELPA package whenever possible.
>
> This seems to be an argument in favor of a separate command?  Or what
> did you mean by that, and how is it related to the issue at hand?

I don't think package-update should switch from a built-in package to a
package that was installed from ELPA.  The user should at least once
commit to the switch (be it with a separate command or package-install).

>> >> It might work but it should be tested somewhat thoroughly before the
>> >> patch is applied.  In the meantime, I just finished a similar approach
>> >> that does not modify `package-installed-p', but just adds another
>> >> utility function:
>> >
>> > A new utility function is fine by me, even if this is e branch.  But I
>> > don't quite understand how this is supposed to work in package-install
>> > to allow updating built-in packages, and do that in a way that will
>> > not touch the existing code for non-built-in packages in significant
>> > ways (assuming you propose this from the release branch).  Can you
>> > elaborate on that?
>> 
>> The only reason we couldn't install built-in packages is that when
>> planning to install packages `package-compute-transaction' believes that
>> if a built-in package is provided, then everything is fine and we don't
>> need to proceed with installing any packages.  All I propose is to lift
>> this assumption, then this works fine.
>
> My problem is _how_ to lift this assumption.  The way you propose
> doing that affects updating non-core packages in ways that we will not
> have enough time to test well enough, not compared to the year that
> these commands exist in package.el and were used by many people in
> many cases.  So we have the following alternatives for the way
> forward:
>
>  . install your changes on master only, and leave the problem of
>    updating a core package unsolved in Emacs 29 (with the workaround
>    mentioned in the beginning of this bug's discussion available to
>    alleviate the problem to some extent)
>  . come up with safer changes for package-install that could be
>    installed on emacs-29
>  . add a new command for updating a core package, which can then be
>    safely installed on emacs-29
>
> The last 2 alternatives can be for emacs-29 only, whereas on master we
> install your proposed change (or something else).

I would like to investigate option 2, but if nothing is found we can
fall back to 3.  But even if there are issues in this case, I don't
consider the matter in general to be that drastic.  If all Joao wants is
to avoid confusion, we can also improve the error message that
`package-install' generates when it says that a package ins already
installed.

> For the 2nd alternative above to be acceptable, the added/modified
> code must be completely separate from the code we have now, so that
> any possibility of its destabilizing the current code could be
> eliminated.  It could be a separate code, triggered by the prefix
> argument, for example.

OK.

>> One point that might be deliberated is that this means all built-in
>> dependencies are also installed, even if these are not strictly
>> necessary.  It shouldn't matter that much, since most users would
>> upgrade them eventually, but worth mentioning I guess? 
>
> That just confirms my fears that we are opening a Pandora's box.  We
> have no idea what this will do, and no time to test the results.
> Unintended consequences are abundant.  We must draw any such
> consequences to the absolute minimum, at least the way the commands
> work by default.  Even if the result is less than elegant.

Intuitively I would want to argue that this change has an upper-bound to
how much harm it could do, but as I cannot prove it in any way I'll
rather not assert that the point.

>> >> +(defun package-core-p (package)
>> >> +  "Return non-nil the built-in version of PACKAGE is loaded."
>> >
>> > Didn't you say the "core" terminology was confusing people?
>> 
>> TBH I am not really satisfied with the name (so any other suggestion is
>> just as fine for me), and as Joao said it would be better to make the
>> predicate as internal so that users are not expected to deal with it.
>
> The name should still be self-explanatory enough, because we the Emacs
> maintainers will read this code and should be able to understand what
> the function does without reading is source every time.

OK.

>> 
>> >> +  (let ((package (if (package-desc-p package)
>> >> +                     (package-desc-name package)
>> >> +                   package)))
>> >> +    (and (assq package (package--alist))
>> >> +         (package-built-in-p package))))
>> >
>> > It sounds like this doesn't check whether a package is "core", it
>> > checks whether it's built-in and can be updated?  So maybe the name
>> > should be changed to reflect that?  And the doc string as well (what
>> > it means by "is loaded")?
>> 
>> Right the "loaded" doesn't make sense.  How about this:
>> 
>> +(defun package--upgradable-built-in-p (package)
>> +  "Check if a built-in PACKAGE can be upgraded.
>> +This command differs from `package-built-in-p' in that it only
>> +returns a non-nil value if the user has not installed a more
>> +recent version of the package from a package archive."
>
> Note that what the doc string says is not what the name tells us.
> "Upgradeable" and "user has not installed a more recent version" are
> not the same.  What the doc string says calls for a name like
> package--built-in-and-up-to-date or something.
>
>> +  (and (not (assq (cond
>> +                   ((package-desc-p package)
>> +                    (package-desc-name package))
>> +                   ((stringp package) (intern package))
>> +                   ((symbolp package) package)
>> +                   ((error "Unknown package format: %S" package)))
>> +                  (package--alist)))
>> +       (package-built-in-p package)))
>
> Why do we need all these conditions, where we didn't need them in the
> current code?

Practically speaking these conditions are not necessary, I just added it
in case there was the need to use the function elsewhere at some point.

> Also, package-alist is documented as "alist of all packages available
> for activation", so it is not clear how the fact that assq returns nil
> is evidence that "the user has not installed a more recent version".
> Looking at what package--alist and package-load-all-descriptors do, it
> looks like they just collect packages that were downloaded into the
> relevant directories?  Is that enough to consider any package not in
> the list to be "not installed"?  

The point is that package.el will add all packages it manages (ie. are
"available to activation") to package-alist.  Built-in/core packages are
not managed by package.el, but just "acknowledged" via
`package--builtins'.  So this looks like a safe assumption to me.

>                                  And what about the "more recent
> version" condition -- this doesn't seem to be tested anywhere in
> package--alist?

You are right, but this is a misphrasing.

> The above questions and undocumented subtleties is what scares me in
> installing such changes at this late stage.  I'm not sure everyone
> involved, yourself included, have a clear understanding of what the
> modified code will do in each possible use case.  That is why I very
> much prefer separate code, which will then free us from the need of
> considering all these subtleties, as the last year of user's
> experience with this code can vouch that it does its job correctly, by
> and large.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: joaotavora@gmail.com,  monnier@iro.umontreal.ca,  62720@debbugs.gnu.org,
>   larsi@gnus.org
> Date: Wed, 12 Apr 2023 20:10:50 +0000
> 
> >> > And I thought we all agreed built-in packages need special treatment
> >> > anyway, didn't we?  Then why having a separate command is not a
> >> > natural next step?
> >> 
> >> I don't necessarily agree that "special treatment" requires a separate
> >> command.
> >
> > Even if you don't agree with that in general, having a separate
> > command would allow us to install that command on emacs-29 without any
> > fears.  So that alone is a significant advantage, even if the rest are
> > not yet agreed upon.
> 
> Would this be a permanent thing, or would the command be deprecated by
> emacs-30?  I get the need under the circumstances, but it doesn't seem
> like the best solution if we were to set aside release-concerns.

We could keep the separate command on master as well, at least until
it is obvious that a special command is not needed in any use case.
The time to revisit this is when we take some practical steps towards
removing from emacs.git packages that are on ELPA and figuring out how
to bundle them with a release tarball and how to let users upgrade
them after they install a released Emacs.  It could be that when we
implement all that, there will be some aspects of upgrading core
packages that will benefit from having a separate command.  If we find
this is not needed, we can deprecate the command at that time.

Mind you, the command I have in mind is one that will allow upgrading
_only_ built-in packages, i.e. it will not deal with the packages that
don't come with an Emacs tarball.

> >> I think it is wrong the assume that an built-in package should
> >> automatically be updated to a ELPA package whenever possible.
> >
> > This seems to be an argument in favor of a separate command?  Or what
> > did you mean by that, and how is it related to the issue at hand?
> 
> I don't think package-update should switch from a built-in package to a
> package that was installed from ELPA.  The user should at least once
> commit to the switch (be it with a separate command or package-install).

I tend to agree.  Although, once again, we should revisit this when
core packages are no longer in emacs.git.  Taking a perhaps somewhat
different analogy of updating my smartphone, there's no particular
difference between updating a "built-in" app, one that is part of the
smartphone's system components and came with it OOTB, and updating any
other app, including those I installed myself.  I think the same is
true for updating system components and applications from any modern
distro, isn't it?  So this is yet to be decided, but for now we can
use this more conservative policy.

> >  . install your changes on master only, and leave the problem of
> >    updating a core package unsolved in Emacs 29 (with the workaround
> >    mentioned in the beginning of this bug's discussion available to
> >    alleviate the problem to some extent)
> >  . come up with safer changes for package-install that could be
> >    installed on emacs-29
> >  . add a new command for updating a core package, which can then be
> >    safely installed on emacs-29
> >
> > The last 2 alternatives can be for emacs-29 only, whereas on master we
> > install your proposed change (or something else).
> 
> I would like to investigate option 2, but if nothing is found we can
> fall back to 3.  But even if there are issues in this case, I don't
> consider the matter in general to be that drastic.  If all Joao wants is
> to avoid confusion, we can also improve the error message that
> `package-install' generates when it says that a package ins already
> installed.

Yes, I think if we go with the modified package-install or
package-update, then invoking them in a way that doesn't update
built-in packages should reflect that in the error message, and also
tell how to invoke them to be able to upgrade a built-in package.

> >> One point that might be deliberated is that this means all built-in
> >> dependencies are also installed, even if these are not strictly
> >> necessary.  It shouldn't matter that much, since most users would
> >> upgrade them eventually, but worth mentioning I guess? 
> >
> > That just confirms my fears that we are opening a Pandora's box.  We
> > have no idea what this will do, and no time to test the results.
> > Unintended consequences are abundant.  We must draw any such
> > consequences to the absolute minimum, at least the way the commands
> > work by default.  Even if the result is less than elegant.
> 
> Intuitively I would want to argue that this change has an upper-bound to
> how much harm it could do, but as I cannot prove it in any way I'll
> rather not assert that the point.

There is, of course, an upper bound.  But I don't think we know how
high that is at this time.  Bitter experience has taught me that we
are not very good in predicting that.

> >> +  (and (not (assq (cond
> >> +                   ((package-desc-p package)
> >> +                    (package-desc-name package))
> >> +                   ((stringp package) (intern package))
> >> +                   ((symbolp package) package)
> >> +                   ((error "Unknown package format: %S" package)))
> >> +                  (package--alist)))
> >> +       (package-built-in-p package)))
> >
> > Why do we need all these conditions, where we didn't need them in the
> > current code?
> 
> Practically speaking these conditions are not necessary, I just added it
> in case there was the need to use the function elsewhere at some point.

Then maybe these conditions should be only on master.

> > Also, package-alist is documented as "alist of all packages available
> > for activation", so it is not clear how the fact that assq returns nil
> > is evidence that "the user has not installed a more recent version".
> > Looking at what package--alist and package-load-all-descriptors do, it
> > looks like they just collect packages that were downloaded into the
> > relevant directories?  Is that enough to consider any package not in
> > the list to be "not installed"?  
> 
> The point is that package.el will add all packages it manages (ie. are
> "available to activation") to package-alist.  Built-in/core packages are
> not managed by package.el, but just "acknowledged" via
> `package--builtins'.  So this looks like a safe assumption to me.

These subtleties should ideally be documented somewhere in package.el.
Without being aware of all that, it is hard to read the code of
package.el and understand what it does and why, and the names and doc
strings of the relevant symbols in most cases don't help, as they all
use vague and too-broad words to describe their purpose and effects.

> >                                  And what about the "more recent
> > version" condition -- this doesn't seem to be tested anywhere in
> > package--alist?
> 
> You are right, but this is a misphrasing.

How so? what should be said instead?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 12/04/2023 14:08, Eli Zaretskii wrote:
>> From: João Távora<joaotavora@gmail.com>
>> Cc:philipk@posteo.net,monnier@iro.umontreal.ca,62720@debbugs.gnu.org,
>>    larsi@gnus.org
>> Date: Wed, 12 Apr 2023 12:00:04 +0100
>>
>> Had another idea: what about this very tiny patch, then?  It makes `M-x
>> package-install` work for installing a :core package.  This also rhymes
>> exactly with Stefan's intution/feeling that :core packages need to be
>> "installed" to promote them to installable.  The current M-x
>> package-install recommendation could remain flawlessly and then you can
>> do whatever you think is best for M-x package-update & friends.
> This has the same problem: it modifies a function that is called in
> too many places.  package-installed-p has half a dozen callers in
> package.el alone.  The change is tiny, but what about its
> implications on every use case where it is involved?

What if we only fix 'package-upgrade' (nee package-update) in emacs-29?

It's a new function, after all. And semantically, the result would be 
somewhat sensible: since 'eglot' is already installed ("bundled" or 
whatever), having 'package-install' fail is not that big a deal, as long 
as 'M-x package-upgrade' works.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Date: Wed, 12 Apr 2023 18:49:09 +0300
> Cc: larsi@gnus.org, 62720@debbugs.gnu.org, philipk@posteo.net,
>  monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 12/04/2023 14:08, Eli Zaretskii wrote:
> >> From: João Távora<joaotavora@gmail.com>
> >> Cc:philipk@posteo.net,monnier@iro.umontreal.ca,62720@debbugs.gnu.org,
> >>    larsi@gnus.org
> >> Date: Wed, 12 Apr 2023 12:00:04 +0100
> >>
> >> Had another idea: what about this very tiny patch, then?  It makes `M-x
> >> package-install` work for installing a :core package.  This also rhymes
> >> exactly with Stefan's intution/feeling that :core packages need to be
> >> "installed" to promote them to installable.  The current M-x
> >> package-install recommendation could remain flawlessly and then you can
> >> do whatever you think is best for M-x package-update & friends.
> > This has the same problem: it modifies a function that is called in
> > too many places.  package-installed-p has half a dozen callers in
> > package.el alone.  The change is tiny, but what about its
> > implications on every use case where it is involved?
> 
> What if we only fix 'package-upgrade' (nee package-update) in emacs-29?

I believe that's what João was proposing.

I don't mind, but changes in package-update will also need to be safe
enough, since this command, while new in Emacs 29, is with us for the
last year, so is relatively well tested, and I don't want to risk
breaking it by last-minute changes of a non-trivial nature.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 4:58 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > >> Had another idea: what about this very tiny patch, then?  It makes `M-x
> > >> package-install` work for installing a :core package.  This also rhymes
> > >> exactly with Stefan's intution/feeling that :core packages need to be
> > >> "installed" to promote them to installable.  The current M-x
> > >> package-install recommendation could remain flawlessly and then you can
> > >> do whatever you think is best for M-x package-update & friends.
> > > This has the same problem: it modifies a function that is called in
> > > too many places.  package-installed-p has half a dozen callers in
> > > package.el alone.  The change is tiny, but what about its
> > > implications on every use case where it is involved?
> >
> > What if we only fix 'package-upgrade' (nee package-update) in emacs-29?
>
> I believe that's what João was proposing.

AFAICT, Dmitry was asking only for package-update, not
package-update-all.  Stefan was also inclined for that.

In my changes, I changed both.  But it is not hard for me to touch
only package-update and to do it with the utmost care for
separation and stability.

For the moment, I'm focusing on M-x package-install, like Philip is.
There seems to be more consensus there that it should offer to update
builtin packages that have never been updated.

I do believe there is high demand for a "upgrade/update" mechanism
that just "updates whatever there is to update, don't care if core
or whatnot" and people looking at package-update-all and
package-menu--mark-upgrades (the "U" command Dmitry brought up)
will eventually be disappointed.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 12/04/2023 19:29, João Távora wrote:
> On Wed, Apr 12, 2023 at 4:58 PM Eli Zaretskii<eliz@gnu.org>  wrote:
> 
>>>>> Had another idea: what about this very tiny patch, then?  It makes `M-x
>>>>> package-install` work for installing a :core package.  This also rhymes
>>>>> exactly with Stefan's intution/feeling that :core packages need to be
>>>>> "installed" to promote them to installable.  The current M-x
>>>>> package-install recommendation could remain flawlessly and then you can
>>>>> do whatever you think is best for M-x package-update & friends.
>>>> This has the same problem: it modifies a function that is called in
>>>> too many places.  package-installed-p has half a dozen callers in
>>>> package.el alone.  The change is tiny, but what about its
>>>> implications on every use case where it is involved?
>>> What if we only fix 'package-upgrade' (nee package-update) in emacs-29?
>> I believe that's what João was proposing.
> AFAICT, Dmitry was asking only for package-update, not
> package-update-all.  Stefan was also inclined for that.
> 
> In my changes, I changed both.  But it is not hard for me to touch
> only package-update and to do it with the utmost care for
> separation and stability.

I think that would make sense. Like Philip phrased it, updating from a 
bundled version is like switching to a different package repository, 
with different stability expectations.

I also seem to recall some logic somewhere that made sure that the 
package is only updated from the source that it was installed from 
(unless explicitly instructed otherwise by the user). We could treat 
"builtins" as a separate source for that purpose, too.

> For the moment, I'm focusing on M-x package-install, like Philip is.
> There seems to be more consensus there that it should offer to update
> builtin packages that have never been updated.
> 
> I do believe there is high demand for a "upgrade/update" mechanism
> that just "updates whatever there is to update, don't care if core
> or whatnot" and people looking at package-update-all and
> package-menu--mark-upgrades (the "U" command Dmitry brought up)
> will eventually be disappointed.

I think "U" should only update the packages that are either not 
built-in, or the built-ins that have been at least updated once 
(meaning, some "external" version is already installed). For reasons of 
stability, mostly. But using 'M-x package-upgrade' would opt-in 
individual packages into that upgrading mechanism too.

That might not be everyone's cup of tea, so adding a user option like 
'package-upgrade-all-builtins' would work too, allowing the user to opt 
into the more risky behavior.

The semantics of 'package-install' are less clear to me. E.g. it 
wouldn't be out of the question to always error out when the package 
(some version) is already installed. So I could see it being "fixed" 
either way sometime in the future.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 12/04/2023 14:00, João Távora wrote:
> Eli Zaretskii<eliz@gnu.org>  writes:
> 
>> the restrictions.  What else can I say to express clearly that I would
>> like such a command to not be installed?
> Had another idea: what about this very tiny patch, then?  It makes `M-x
> package-install` work for installing a :core package.  This also rhymes
> exactly with Stefan's intution/feeling that :core packages need to be
> "installed" to promote them to installable.  The current M-x
> package-install recommendation could remain flawlessly and then you can
> do whatever you think is best for M-x package-update & friends.
> 
> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index f92afe56b76..134505a96af 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -2164,8 +2164,10 @@ package-installed-p
>          (and pkg-descs
>               (version-list-<= min-version
>                                (package-desc-version (car pkg-descs)))))
> -     ;; Also check built-in packages.
> -     (package-built-in-p package min-version)))))
> +     ;; Also check built-in packages.  But if min-version is nil, just
> +     ;; act as if a package-desc had been passed (bug#62720)
> +     (and min-version (package-built-in-p package min-version))))))
>   
>   (defun package-download-transaction (packages)
>     "Download and install all the packages in PACKAGES.

If the package becomes "mass-upgradable" afterwards, this seems like the 
best solution, behavior-wise (imminent 29 release aside, of course).

Does 'M-x package-update/upgrade' get fixed automatically with this, or 
does it need separate changes?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
Status "incompat"?  If I press RET on its name, I see

  Package eglot is incompatible.

       Status: Incompatible because it depends on uninstallable packages.

Why is that?

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Tue, Apr 11, 2023, 19:53 Eli Zaretskii <eliz@gnu.org> wrote:

> Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
> Status "incompat"?  If I press RET on its name, I see
>
>   Package eglot is incompatible.
>
>        Status: Incompatible because it depends on uninstallable packages.
>
> Why is that?
>

No idea. Is this with our without my patch?

João

>

[-- Attachment #2: Type: text/html, Size: 952 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 11 Apr 2023 21:28:26 +0100
> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 
> 	Lars Ingebrigtsen <larsi@gnus.org>, 62720@debbugs.gnu.org
> 
>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
>  Status "incompat"?  If I press RET on its name, I see
> 
>    Package eglot is incompatible.
> 
>         Status: Incompatible because it depends on uninstallable packages.
> 
>  Why is that?
> 
> No idea. Is this with our without my patch?

Without.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Tue, 11 Apr 2023 21:28:26 +0100
>> Cc: "Philip K." <philipk@posteo.net>, Stefan Monnier <monnier@iro.umontreal.ca>, 
>> 	Lars Ingebrigtsen <larsi@gnus.org>, 62720@debbugs.gnu.org
>> 
>>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
>>  Status "incompat"?  If I press RET on its name, I see
>> 
>>    Package eglot is incompatible.
>> 
>>         Status: Incompatible because it depends on uninstallable packages.
>> 
>>  Why is that?
>> 
>> No idea. Is this with our without my patch?
>
> Without.

I can't reproduce this in a recent emacs-29.  Here's what I start Emacs
with

HOME=`mktemp -d` && src/emacs -Q -f list-packages

I get:

     Status: Available from gnu -- [Install]

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  larsi@gnus.org,
>   62720@debbugs.gnu.org
> Date: Wed, 12 Apr 2023 10:18:08 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
> >>  Status "incompat"?  If I press RET on its name, I see
> >> 
> >>    Package eglot is incompatible.
> >> 
> >>         Status: Incompatible because it depends on uninstallable packages.
> >> 
> >>  Why is that?
> >> 
> >> No idea. Is this with our without my patch?
> >
> > Without.
> 
> I can't reproduce this in a recent emacs-29.  Here's what I start Emacs
> with
> 
> HOME=`mktemp -d` && src/emacs -Q -f list-packages
> 
> I get:
> 
>      Status: Available from gnu -- [Install]

I did just

  emacs -Q
  M-x list-packages RET

and I still see what I described.

I get the same if I point HOME to a scratch empty directory.

Maybe it's a Windows thing? why are some dependencies marked as "not
available"?

  Package eglot is incompatible.

       Status: Incompatible because it depends on uninstallable packages.
      Archive: gnu
      Version: 1.14
       Commit: 8125d4cfc5605ead9102b7d823c4241029eb76cc
      Summary: The Emacs Client for LSP servers
     Requires: emacs-26.3, jsonrpc-1.0.16, flymake-1.2.1,
		 project-0.9.8 (not available), xref-1.6.2 (not available),
		 eldoc-1.14.0, seq-2.23 (not available),
		 external-completion-0.1 (not available)

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Wed, Apr 12, 2023 at 10:53 AM Eli Zaretskii <eliz@gnu.org> wrote:

> I did just
>
>   emacs -Q
>   M-x list-packages RET
>
> and I still see what I described.
>
> I get the same if I point HOME to a scratch empty directory.
>
> Maybe it's a Windows thing? why are some dependencies marked as "not
> available"?

Can't reproduce on Windows, either.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  larsi@gnus.org,
>>   62720@debbugs.gnu.org
>> Date: Wed, 12 Apr 2023 10:18:08 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
>> >>  Status "incompat"?  If I press RET on its name, I see
>> >> 
>> >>    Package eglot is incompatible.
>> >> 
>> >>         Status: Incompatible because it depends on uninstallable packages.
>> >> 
>> >>  Why is that?
>> >> 
>> >> No idea. Is this with our without my patch?
>> >
>> > Without.
>> 
>> I can't reproduce this in a recent emacs-29.  Here's what I start Emacs
>> with
>> 
>> HOME=`mktemp -d` && src/emacs -Q -f list-packages
>> 
>> I get:
>> 
>>      Status: Available from gnu -- [Install]
>
> I did just
>
>   emacs -Q
>   M-x list-packages RET
>
> and I still see what I described.
>
> I get the same if I point HOME to a scratch empty directory.
>
> Maybe it's a Windows thing? why are some dependencies marked as "not
> available"?
>
>   Package eglot is incompatible.
>
>        Status: Incompatible because it depends on uninstallable packages.
>       Archive: gnu
>       Version: 1.14
>        Commit: 8125d4cfc5605ead9102b7d823c4241029eb76cc
>       Summary: The Emacs Client for LSP servers
>      Requires: emacs-26.3, jsonrpc-1.0.16, flymake-1.2.1,
> 		 project-0.9.8 (not available), xref-1.6.2 (not available),
> 		 eldoc-1.14.0, seq-2.23 (not available),
> 		 external-completion-0.1 (not available)

On GNU/Linux: I have experienced a similar issue to this one in the
past, but found that it resolved itself without any action on my end
after a while.  Sadly I did not take the time to investigate what is
going on, but I believe that this is a subtle bug in package.el.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: João Távora <joaotavora@gmail.com>,
>   monnier@iro.umontreal.ca,
>   larsi@gnus.org,  62720@debbugs.gnu.org
> Date: Wed, 12 Apr 2023 13:20:27 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: João Távora <joaotavora@gmail.com>
> >> Cc: philipk@posteo.net,  monnier@iro.umontreal.ca,  larsi@gnus.org,
> >>   62720@debbugs.gnu.org
> >> Date: Wed, 12 Apr 2023 10:18:08 +0100
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >>  Btw, why, when I do "M-x list-packages RET", I see Eglot in red with
> >> >>  Status "incompat"?  If I press RET on its name, I see
> >> >> 
> >> >>    Package eglot is incompatible.
> >> >> 
> >> >>         Status: Incompatible because it depends on uninstallable packages.
> >> >> 
> >> >>  Why is that?
> >> >> 
> >> >> No idea. Is this with our without my patch?
> >> >
> >> > Without.
> >> 
> >> I can't reproduce this in a recent emacs-29.  Here's what I start Emacs
> >> with
> >> 
> >> HOME=`mktemp -d` && src/emacs -Q -f list-packages
> >> 
> >> I get:
> >> 
> >>      Status: Available from gnu -- [Install]
> >
> > I did just
> >
> >   emacs -Q
> >   M-x list-packages RET
> >
> > and I still see what I described.
> >
> > I get the same if I point HOME to a scratch empty directory.
> >
> > Maybe it's a Windows thing? why are some dependencies marked as "not
> > available"?
> >
> >   Package eglot is incompatible.
> >
> >        Status: Incompatible because it depends on uninstallable packages.
> >       Archive: gnu
> >       Version: 1.14
> >        Commit: 8125d4cfc5605ead9102b7d823c4241029eb76cc
> >       Summary: The Emacs Client for LSP servers
> >      Requires: emacs-26.3, jsonrpc-1.0.16, flymake-1.2.1,
> > 		 project-0.9.8 (not available), xref-1.6.2 (not available),
> > 		 eldoc-1.14.0, seq-2.23 (not available),
> > 		 external-completion-0.1 (not available)
> 
> On GNU/Linux: I have experienced a similar issue to this one in the
> past, but found that it resolved itself without any action on my end
> after a while.  Sadly I did not take the time to investigate what is
> going on, but I believe that this is a subtle bug in package.el.

I've now tried on GNU/Linux (one of gnu.org machines), and I see the
same there: eglot, project, and use-package are marked as "incompat".

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
> updates the built-in and the manually installed packages that can be
> updated.  It _doesn't_ install any packages that weren't installed yet,
> of course.

Hmm... it might make sense to treat builtins specially in this respect.
Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
with the idea that `M-x package-update-all` would update all the
`:core` packages.

I'm not sure why, admittedly, but I think it comes down to the fact that
the first upgrade of a `:core` package from GNU ELPA feels to me more
like an "install" than an "upgrade".


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

On Tue, Apr 11, 2023, 20:44 Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
> > updates the built-in and the manually installed packages that can be
> > updated.  It _doesn't_ install any packages that weren't installed yet,
> > of course.
>
> Hmm... it might make sense to treat builtins specially in this respect.
> Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
> with the idea that `M-x package-update-all` would update all the
> `:core` packages.
>
> I'm not sure why, admittedly, but I think it comes down to the fact that
> the first upgrade of a `:core` package from GNU ELPA feels to me more
> like an "install" than an "upgrade".


Can't argue against feelings :)

João

>
>

[-- Attachment #2: Type: text/html, Size: 1425 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I'm not sure why, admittedly, but I think it comes down to the fact that
>> the first upgrade of a `:core` package from GNU ELPA feels to me more
>> like an "install" than an "upgrade".
> Can't argue against feelings :)

What can I say: I just know I'm right!

More seriously, I agree it's not a very strong/compelling argument.
I just wanted to mention it, in case I'm not the only weirdo.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I'm not sure why, admittedly, but I think it comes down to the fact that
>>> the first upgrade of a `:core` package from GNU ELPA feels to me more
>>> like an "install" than an "upgrade".
>> Can't argue against feelings :)
>
> What can I say: I just know I'm right!
>
> More seriously, I agree it's not a very strong/compelling argument.
> I just wanted to mention it, in case I'm not the only weirdo.

I would agree as well, and my argument would be that by installing a
core package from ELPA, I am indicating that I want to follow the newest
version of a package as it is developed upstream, which might be
perceived as less stable/predictable than tracking the development
through Emacs updates.

>         Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Philip Kaludercic <philipk@posteo.net>,  eliz@gnu.org,
>   62720@debbugs.gnu.org,  Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 11 Apr 2023 15:44:22 -0400
> 
> > Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
> > updates the built-in and the manually installed packages that can be
> > updated.  It _doesn't_ install any packages that weren't installed yet,
> > of course.
> 
> Hmm... it might make sense to treat builtins specially in this respect.
> Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
> with the idea that `M-x package-update-all` would update all the
> `:core` packages.
> 
> I'm not sure why, admittedly, but I think it comes down to the fact that
> the first upgrade of a `:core` package from GNU ELPA feels to me more
> like an "install" than an "upgrade".

Which means my proposal of adding a new command
package-update-core-package makes more and more sense: we will
probably need to handle such packages specially for any number of
reasons, more so as we go with our plan to have them only on ELPA and
"bundle" them when the release is tarred.  So having such a command
now will be a good investment for the future.

Philip, if this makes sense, would you please add such a command on
the emacs-29 branch?  If the exact purpose and effects of the command
are not clear yet, let's talk about it and finalize that.

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Philip Kaludercic <philipk@posteo.net>,  eliz@gnu.org,
>>   62720@debbugs.gnu.org,  Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Tue, 11 Apr 2023 15:44:22 -0400
>> 
>> > Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
>> > updates the built-in and the manually installed packages that can be
>> > updated.  It _doesn't_ install any packages that weren't installed yet,
>> > of course.
>> 
>> Hmm... it might make sense to treat builtins specially in this respect.
>> Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
>> with the idea that `M-x package-update-all` would update all the
>> `:core` packages.
>> 
>> I'm not sure why, admittedly, but I think it comes down to the fact that
>> the first upgrade of a `:core` package from GNU ELPA feels to me more
>> like an "install" than an "upgrade".
>
> Which means my proposal of adding a new command
> package-update-core-package makes more and more sense: 

I am not sure that "core package" is necessarily a term or concept that
users would be familiar with.  Or at least I have seen users confused
about the concept online, not realising that a core package can be
updated via ELPA.

How about something along the lines of `package-update-from-built-in'?

>                                                        we will
> probably need to handle such packages specially for any number of
> reasons, more so as we go with our plan to have them only on ELPA and
> "bundle" them when the release is tarred.  

Could you elaborate on this plan.  Or perhaps I just lack the background
to see how these issues are related?

>                                            So having such a command
> now will be a good investment for the future.
>
> Philip, if this makes sense, would you please add such a command on
> the emacs-29 branch?  If the exact purpose and effects of the command
> are not clear yet, let's talk about it and finalize that.

Can do.  This would prompt the user for a core package that hasn't been
installed from ELPA yet, and would make sure the package instead of the
core code is loaded?  If there is no difference in version between the
ELPA and the core package, should we say anything?

> Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  joaotavora@gmail.com,
>   62720@debbugs.gnu.org,  larsi@gnus.org
> Date: Wed, 12 Apr 2023 07:44:20 +0000
> 
> >> I'm not sure why, admittedly, but I think it comes down to the fact that
> >> the first upgrade of a `:core` package from GNU ELPA feels to me more
> >> like an "install" than an "upgrade".
> >
> > Which means my proposal of adding a new command
> > package-update-core-package makes more and more sense: 
> 
> I am not sure that "core package" is necessarily a term or concept that
> users would be familiar with.  Or at least I have seen users confused
> about the concept online, not realising that a core package can be
> updated via ELPA.

I'm okay with an alternative terminology, like "built-in".

> How about something along the lines of `package-update-from-built-in'?

Not sure about the "from" part.  How about package-update-built-in
instead? or maybe package-upgrade-builtin (since built-in packages
will only ever be upgraded, at least normally)?

> >                                                        we will
> > probably need to handle such packages specially for any number of
> > reasons, more so as we go with our plan to have them only on ELPA and
> > "bundle" them when the release is tarred.  
> 
> Could you elaborate on this plan.  Or perhaps I just lack the background
> to see how these issues are related?

The plan is to provide a command similar to package-update, but which
will be used only for built-in packages which are also on ELPA
(a.k.a. "core packages").  The internals will probably be different,
since package-update offers only non-built-in packages as completion
candidates, the data structures package.el uses for built-in and
non-built-in packages are different, and installing an updated package
should NOT delete the bundled one, just install the newer version so
that it is used in preference to the bundled one.  Other than that,
the UI is supposed to be the same as in package-update.

If something else is unclear, please ask.

> >                                            So having such a command
> > now will be a good investment for the future.
> >
> > Philip, if this makes sense, would you please add such a command on
> > the emacs-29 branch?  If the exact purpose and effects of the command
> > are not clear yet, let's talk about it and finalize that.
> 
> Can do.  This would prompt the user for a core package that hasn't been
> installed from ELPA yet, and would make sure the package instead of the
> core code is loaded?

Yes.  But when you say "hasn't been installed from ELPA yet", do you
mean that once it has been installed, users will have to use a
different command for updating it further?  That might be confusing;
I'd prefer that the same command is used for all the updates of
built-in packages.  For example, if we'd later want to support
downgrading back to the bundled version, the implementation will be
different for built-in packages, so having a separate command will
save us some maintenance headaches.

> If there is no difference in version between the ELPA and the core
> package, should we say anything?

Probably just user-error, I guess?  Or maybe optionally provide a way
of forcing the update, like if the user wants to reinstall the package
anyway for whatever reasons?

Thanks.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Hmm... it might make sense to treat builtins specially in this respect.
>> Of course, maybe it's OK, but for some reason I feel a bit uncomfortable
>> with the idea that `M-x package-update-all` would update all the
>> `:core` packages.
>> 
>> I'm not sure why, admittedly, but I think it comes down to the fact that
>> the first upgrade of a `:core` package from GNU ELPA feels to me more
>> like an "install" than an "upgrade".
>
> Which means my proposal of adding a new command
> package-update-core-package makes more and more sense: we will

FWIW, I'm not sure it's needed: when the user asks to `package-update`
the Eglot package, then we should upgrade `eglot`, regardless if
it's builtin.

My comment was specifically for `M-x package-update-all`.


        Stefan

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 11/04/2023 14:02, João Távora wrote:
> Philip Kaludercic<philipk@posteo.net>  writes:
> 
>> Will this not affect `package-update-all'?  I don't if we want that the
>> command installs all packages from ELPA that it can find.
> Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
> updates the built-in and the manually installed packages that can be
> updated.  It_doesn't_  install any packages that weren't installed yet,
> of course.

On a related note, do you know whether we upgrade the built-in packages 
when the user presses 'U' in the list-packages buffer?

Using the command package-menu-mark-upgrades, that is.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 11/04/2023 14:02, João Távora wrote:
>> Philip Kaludercic<philipk@posteo.net>  writes:
>> 
>>> Will this not affect `package-update-all'?  I don't if we want that the
>>> command installs all packages from ELPA that it can find.
>> Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
>> updates the built-in and the manually installed packages that can be
>> updated.  It_doesn't_  install any packages that weren't installed yet,
>> of course.
>
> On a related note, do you know whether we upgrade the built-in
> packages when the user presses 'U' in the list-packages buffer?
>
> Using the command package-menu-mark-upgrades, that is.

Nope, doesn't work, doesn't do anything to those packages.  I wish it
did, of course.

I also don't understand why this is using separate, but repeated logic
from package-update.  What is the difference between "upgrade" and
"update", if any?  Is "upgrade" more powerful?

BTW, I also noticed that Eglot's version on Emacs 29 is garbled.  I had
wanted 1.12-emacs29 to somehow show that it is Emacs 29 specific.  But
version-to-list doesn't like it and the package shows up as version
"nil" in package--builtins.  Will just change it to 1.12.29, which is
less perceptible but works fine (none of this makes any difference to
this bug, of course).

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Dmitry Gutov]

On 12/04/2023 12:34, João Távora wrote:
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
>> On 11/04/2023 14:02, João Távora wrote:
>>> Philip Kaludercic<philipk@posteo.net>  writes:
>>>
>>>> Will this not affect `package-update-all'?  I don't if we want that the
>>>> command installs all packages from ELPA that it can find.
>>> Thanks.  I've just tested 'M-x package-update-all' with my patch.  It
>>> updates the built-in and the manually installed packages that can be
>>> updated.  It_doesn't_  install any packages that weren't installed yet,
>>> of course.
>>
>> On a related note, do you know whether we upgrade the built-in
>> packages when the user presses 'U' in the list-packages buffer?
>>
>> Using the command package-menu-mark-upgrades, that is.
> 
> Nope, doesn't work, doesn't do anything to those packages.  I wish it
> did, of course.

All right, then it's not just about package-install.

> I also don't understand why this is using separate, but repeated logic
> from package-update.  What is the difference between "upgrade" and
> "update", if any?  Is "upgrade" more powerful?

They're supposed to be the same. But that's bug#62750 which you've 
already read by now.

> BTW, I also noticed that Eglot's version on Emacs 29 is garbled.  I had
> wanted 1.12-emacs29 to somehow show that it is Emacs 29 specific.  But
> version-to-list doesn't like it and the package shows up as version
> "nil" in package--builtins.  Will just change it to 1.12.29, which is
> less perceptible but works fine (none of this makes any difference to
> this bug, of course).

Cool.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca
> From: João Távora <joaotavora@gmail.com>
> Date: Fri, 07 Apr 2023 23:12:43 +0100
> 
> I've been meaning to report this for some time.  I really think this
> should be fixed in Emacs 29 before it ships.
> 
> src/emacs -Q # Emacs 29
> 
> M-x package-install RET eglot RET
> 
> Echoes "No match", even though eglot 1.14 is available from GNU ELPA.
> That's because Eglot is a :core ELPA package and it's already in Emacs
> 29, so package.el thinks there is nothing to install.

Is :core new in Emacs 29?  If not, then this problem is not new: it
existed earlier for every :core package in ELPA, it just didn't exist
for Eglot.

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2132 bytes --]

On Sat, Apr 8, 2023, 08:09 Eli Zaretskii <eliz@gnu.org> wrote:

> > Cc: monnier@iro.umontreal.ca
> > From: João Távora <joaotavora@gmail.com>
> > Date: Fri, 07 Apr 2023 23:12:43 +0100
> >
> > I've been meaning to report this for some time.  I really think this
> > should be fixed in Emacs 29 before it ships.
> >
> > src/emacs -Q # Emacs 29
> >
> > M-x package-install RET eglot RET
> >
> > Echoes "No match", even though eglot 1.14 is available from GNU ELPA.
> > That's because Eglot is a :core ELPA package and it's already in Emacs
> > 29, so package.el thinks there is nothing to install.
>
> Is :core new in Emacs 29?  If not, then this problem is not new: it
> existed earlier for every :core package in ELPA, it just didn't exist
> for Eglot.
>

Yes, it existed before. It's a bug but not a regression. But many of these
:core packages were primarily used as libraries supporting other packages,
not do much as user-facing, command-providing, UI-enhancing packages.

From what I've been seeing in the Eglot bug tracker there is a very
significant number of people using third-party package managers to install
and upgrade Eglot (straight, elpaca, ...)

I always tell them to report problems sweet package-install instead of
these managers, as I've seen more than my fair share of problems created by
these other managers.

My experience and the internal traffic reports tem me Eglot is a popular
Emacs package and people are not satisfied with just any old release. If
this isn't fixed, it'll be heaps of text to tell people how to upgrade. Not
the end of the world, but really awkward.

There is certainly code in Emacs 29 that allows upgrading Eglot or other
core packages but it's buried deep in that odd workflow. If the code for a
M-x package-upgrade, as Stefan suggests, is simple, then i think it should
be added to emacs-29.

Alternatively, there could be some kind of 'package-bugfixes' non-:core
package in elpa.git that adds this new command. Asking people to install
that package first is still easier than guiding then through that odd
workflow.

João

>

[-- Attachment #2: Type: text/html, Size: 3151 bytes --]

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

> Alternatively, there could be some kind of 'package-bugfixes' non-:core
> package in elpa.git that adds this new command. Asking people to install
> that package first is still easier than guiding then through that odd
> workflow.

I think that adding this into compat.el could be also justified.
With compat.el being ELPA dependency.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

On Sat, Apr 8, 2023 at 3:49 PM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Alternatively, there could be some kind of 'package-bugfixes' non-:core
> > package in elpa.git that adds this new command. Asking people to install
> > that package first is still easier than guiding then through that odd
> > workflow.
>
> I think that adding this into compat.el could be also justified.
> With compat.el being ELPA dependency.

But compat.el is not :core right?  So Eglot cannot depend on it.

Also I think that's a bit beyond the purpose of compat, though
I don't oppose it. I really think package.el, like other package
managers, should learn to update itself.  It should be a :core
ELPA package itself.  But then that also requires this bug to
be fixed.

João

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[Ihor Radchenko]

João Távora <joaotavora@gmail.com> writes:

>> I think that adding this into compat.el could be also justified.
>> With compat.el being ELPA dependency.
>
> But compat.el is not :core right?  So Eglot cannot depend on it.

Several packages in :core do depend on compat. The idea is to use
(require 'compat nil 'noerror), which does nothing in core, but enables
forward-compatibility features on ELPA.

See https://elpa.gnu.org/packages/doc/compat.html#Usage

> Also I think that's a bit beyond the purpose of compat, though
> I don't oppose it. I really think package.el, like other package
> managers, should learn to update itself.  It should be a :core
> ELPA package itself.  But then that also requires this bug to
> be fixed.

I agree that it is slightly out of scope of compat. But this particular
problem appears to be important enough as exception.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

bug#62720: 29.0.60; Not easy at all to upgrade :core packages like Eglot

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

On Sat, Apr 8, 2023, 16:28 Ihor Radchenko <yantar92@posteo.net> wrote:

> João Távora <joaotavora@gmail.com> writes:
>
> >> I think that adding this into compat.el could be also justified.
> >> With compat.el being ELPA dependency.
> >
> > But compat.el is not :core right?  So Eglot cannot depend on it.
>
> Several packages in :core do depend on compat. The idea is to use
> (require 'compat nil 'noerror), which does nothing in core, but enables
> forward-compatibility features on ELPA.
>

Sure, that works, your right. And it's a good idea.

But it doesn't solve _this_ problem even if compat.el patches package.el,
because once you get Emacs 29, you're locked out of ELPA Eglot (unless you
take that very circuitous route).

> Also I think that's a bit beyond the purpose of compat, though
> > I don't oppose it. I really think package.el, like other package
> > managers, should learn to update itself.  It should be a :core
> > ELPA package itself.  But then that also requires this bug to
> > be fixed.
>
> I agree that it is slightly out of scope of compat. But this particular
> problem appears to be important enough as exception.
>

Yes, but more importantly, i don't think there's a fix to this bug that can
be done fully ELPA-side.

João

>

[-- Attachment #2: Type: text/html, Size: 2279 bytes --]