Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 06/05/2023 09:44, Eli Zaretskii wrote:
> +using the more general command 'package-install', which by default
> +will not upgrade "built-in" packages, those that come with Emacs.

Friendly reminder: while 'M-x package-install' does not upgrade 
"built-in" packages "by default", it doesn't upgrade any other packages 
at all, ever.

I'd hate to create an impression in some users that 'package-install' is 
a suitable instrument to upgrade packages.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sat, 6 May 2023 13:14:32 +0300
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 06/05/2023 09:44, Eli Zaretskii wrote:
> > +using the more general command 'package-install', which by default
> > +will not upgrade "built-in" packages, those that come with Emacs.
> 
> Friendly reminder: while 'M-x package-install' does not upgrade 
> "built-in" packages "by default", it doesn't upgrade any other packages 
> at all, ever.

Really? so what does package-install do?

> I'd hate to create an impression in some users that 'package-install' is 
> a suitable instrument to upgrade packages.

It isn't? then what is it for?

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 06/05/2023 13:35, Eli Zaretskii wrote:
>> Date: Sat, 6 May 2023 13:14:32 +0300
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 06/05/2023 09:44, Eli Zaretskii wrote:
>>> +using the more general command 'package-install', which by default
>>> +will not upgrade "built-in" packages, those that come with Emacs.
>> Friendly reminder: while 'M-x package-install' does not upgrade
>> "built-in" packages "by default", it doesn't upgrade any other packages
>> at all, ever.
> Really? so what does package-install do?

'M-x package-install' installs a package if it's not already installed.

>> I'd hate to create an impression in some users that 'package-install' is
>> a suitable instrument to upgrade packages.
> It isn't? then what is it for?

See above.

It also _can_ install a newer version of the package if the first 
argument is of type 'package-desc', but that's basically an internal 
overload, not something that can be user interactively, or easily in the 
init script. BTW, the value of package-install-upgrade-built-in has no 
effect on that "overload".

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sat, 6 May 2023 13:46:26 +0300
> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 06/05/2023 13:35, Eli Zaretskii wrote:
> >> Date: Sat, 6 May 2023 13:14:32 +0300
> >> From: Dmitry Gutov<dmitry@gutov.dev>
> >>
> >> On 06/05/2023 09:44, Eli Zaretskii wrote:
> >>> +using the more general command 'package-install', which by default
> >>> +will not upgrade "built-in" packages, those that come with Emacs.
> >> Friendly reminder: while 'M-x package-install' does not upgrade
> >> "built-in" packages "by default", it doesn't upgrade any other packages
> >> at all, ever.
> > Really? so what does package-install do?
> 
> 'M-x package-install' installs a package if it's not already installed.
> 
> >> I'd hate to create an impression in some users that 'package-install' is
> >> a suitable instrument to upgrade packages.
> > It isn't? then what is it for?
> 
> See above.
> 
> It also _can_ install a newer version of the package if the first 
> argument is of type 'package-desc', but that's basically an internal 
> overload, not something that can be user interactively, or easily in the 
> init script. BTW, the value of package-install-upgrade-built-in has no 
> effect on that "overload".

I'm surprised, to say the least.  To recap:

  . the entire discussion of bug#62720, of which this is fallout, was
    about allowing package-install to upgrade Eglot, and according to
    João, that's what package-install did before Emacs 29.  Was this
    all based on a mistake? if so, why no one has said so in all that
    long argument??
  . the text I changed also talks about package-install, and yet you
    didn't object to it, why?

So please forgive me if I don't quite believe my eyes when I read your
messages in this matter.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[João Távora]

On Sat, May 6, 2023 at 11:52 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > It also _can_ install a newer version of the package if the first
> > argument is of type 'package-desc', but that's basically an internal
> > overload, not something that can be user interactively, or easily in the
> > init script. BTW, the value of package-install-upgrade-built-in has no
> > effect on that "overload".
>
> I'm surprised, to say the least.  To recap:
>
>   . the entire discussion of bug#62720, of which this is fallout, was
>     about allowing package-install to upgrade Eglot, and according to
>     João, that's what package-install did before Emacs 29.  Was this
>     all based on a mistake? if so, why no one has said so in all that
>     long argument??


One more time: in Emacs 28 package-install doesn't upgrade,
but it installs the latest, which is incompatible behaviour
if you move to Emacs 29, where that won't happen.

So if you're used to setting up a brand new Emacs 28 and
package-install Eglot to get versions with nice features and
bugfixes, you may be dismayed to find that doing the very
same thing in Emacs 29 results in what will probably be a
old version.  And this problem will persist _and_ worsen
during the lifetime of Emacs 29, which I expect to be a number
of years.

Anyway, experimenting and experiencing this yourself will
take at most a minute if you have Emacs 28 and Emacs 29
handy, so there's no shroud of mystery whatsoever.  just

   HOME=`mktemp -d` emacs-{28,29}/src/emacs

But we already went through all this, and all these decisions
have been made already, and they are ultimately all legitimate
even if we all disagree.

I don't think the NEWS entry needed rewording: it didn't
say anything inaccurate or misleading.  You made it more
complete, and perfectly accurate, but also more inescrutable,
which is not really what NEWS should be about IMO.  But it's
not that  tragic either, so please Dmitry and Eli let's just
leave this.

João

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 6 May 2023 14:03:02 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> 
> On Sat, May 6, 2023 at 11:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I'm surprised, to say the least.  To recap:
> >
> >   . the entire discussion of bug#62720, of which this is fallout, was
> >     about allowing package-install to upgrade Eglot, and according to
> >     João, that's what package-install did before Emacs 29.  Was this
> >     all based on a mistake? if so, why no one has said so in all that
> >     long argument??
> 
> 
> One more time: in Emacs 28 package-install doesn't upgrade,
> but it installs the latest, which is incompatible behaviour
> if you move to Emacs 29, where that won't happen.

Is this for Eglot (and use-package), or is this for any package
installed from ELPA?  IOW, does Emacs 29's package-install still
install the latest version of a package from ELPA?  And does that
happen even if some older version of a package is already installed?

> So if you're used to setting up a brand new Emacs 28 and
> package-install Eglot to get versions with nice features and
> bugfixes, you may be dismayed to find that doing the very
> same thing in Emacs 29 results in what will probably be a
> old version.

Are you talking about users who didn't update their Eglot, except when
a new Emacs version was released?  Or are you talking about users who
updated Eglot from ELPA (using package-install) even between Emacs
releases?  Or are you talking about something else entirely?

> And this problem will persist _and_ worsen during the lifetime of
> Emacs 29

Why do you insist on reiterating past disagreements, especially when
they are tangents, is beyond me.

> Anyway, experimenting and experiencing this yourself will
> take at most a minute if you have Emacs 28 and Emacs 29
> handy, so there's no shroud of mystery whatsoever.

Why do you think I even have a minute to spare?

I'm asking the experts on package.el to describe what happens
precisely _because_ I don't have that time.  For example, today, I've
been at the keyboard, working on various Emacs issues for the last 7
hours without any breaks, and this is my weekend!

> But we already went through all this, and all these decisions
> have been made already, and they are ultimately all legitimate
> even if we all disagree.

It sounds like what we went through was based on a misunderstanding,
or at least Dmitry says it was.

> I don't think the NEWS entry needed rewording: it didn't
> say anything inaccurate or misleading.  You made it more
> complete, and perfectly accurate, but also more inescrutable,
> which is not really what NEWS should be about IMO.

Clearly, if I thought the original wording was acceptable, I wouldn't
have touched it (with the single exception of a spelling mistake).

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[João Távora]

On Sat, May 6, 2023 at 2:21 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Sat, 6 May 2023 14:03:02 +0100
> > Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> >
> > On Sat, May 6, 2023 at 11:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > I'm surprised, to say the least.  To recap:
> > >
> > >   . the entire discussion of bug#62720, of which this is fallout, was
> > >     about allowing package-install to upgrade Eglot, and according to
> > >     João, that's what package-install did before Emacs 29.  Was this
> > >     all based on a mistake? if so, why no one has said so in all that
> > >     long argument??
> >
> >
> > One more time: in Emacs 28 package-install doesn't upgrade,
> > but it installs the latest, which is incompatible behaviour
> > if you move to Emacs 29, where that won't happen.
>
> Is this for Eglot (and use-package), or is this for any package
> installed from ELPA?  IOW, does Emacs 29's package-install still
> install the latest version of a package from ELPA?  And does that
> happen even if some older version of a package is already installed?

Yes to the first, no to the second.

> > So if you're used to setting up a brand new Emacs 28 and
> > package-install Eglot to get versions with nice features and
> > bugfixes, you may be dismayed to find that doing the very
> > same thing in Emacs 29 results in what will probably be a
> > old version.
>
> Are you talking about users who didn't update their Eglot, except when
> a new Emacs version was released?  Or are you talking about users who
> updated Eglot from ELPA (using package-install) even between Emacs
> releases?  Or are you talking about something else entirely?

Just spend that minute, Eli.   Call it an investment.  And then
re-read what I wrote in that paragraph.  It will make sense.

> > And this problem will persist _and_ worsen during the lifetime of
> > Emacs 29
>
> Why do you insist on reiterating past disagreements, especially when
> they are tangents, is beyond me.

You brought it up!!! You asked "was this all based on a mistake?"
I just clarified and summarized what happens!

> > Anyway, experimenting and experiencing this yourself will
> > take at most a minute if you have Emacs 28 and Emacs 29
> > handy, so there's no shroud of mystery whatsoever.
>
> Why do you think I even have a minute to spare?

Clearly, unless you're some kind of superhuman being, you're
wasting much more time writing these messages. So I figured
a minute of experimentation would probably save a lot of
your time and our collective time.

You're still asking me questions, and then when I answer
them you say I'm bringing up old past disagreements.

> I'm asking the experts on package.el to describe what happens
> precisely _because_ I don't have that time.  For example, today, I've
> been at the keyboard, working on various Emacs issues for the last 7
> hours without any breaks, and this is my weekend!

For your health, take a break Eli.

> > But we already went through all this, and all these decisions
> > have been made already, and they are ultimately all legitimate
> > even if we all disagree.
>
> It sounds like what we went through was based on a misunderstanding,
> or at least Dmitry says it was.

Again, please invest that minute (after taking a break, of course).

> > I don't think the NEWS entry needed rewording: it didn't
> > say anything inaccurate or misleading.  You made it more
> > complete, and perfectly accurate, but also more inescrutable,
> > which is not really what NEWS should be about IMO.
>
> Clearly, if I thought the original wording was acceptable, I wouldn't
> have touched it

There was nothing wrong or misleading about it.  What possibly
can be wrong or unacceptable in announcing "Eglot can upgrade
itself to the latest version." to Eglot users?  But let's just let it go, your
reworking isn't that horrible either.

João

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 6 May 2023 14:48:15 +0100
> Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> 
> On Sat, May 6, 2023 at 2:21 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > From: João Távora <joaotavora@gmail.com>
> > > Date: Sat, 6 May 2023 14:03:02 +0100
> > > Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> > >
> > > On Sat, May 6, 2023 at 11:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > > I'm surprised, to say the least.  To recap:
> > > >
> > > >   . the entire discussion of bug#62720, of which this is fallout, was
> > > >     about allowing package-install to upgrade Eglot, and according to
> > > >     João, that's what package-install did before Emacs 29.  Was this
> > > >     all based on a mistake? if so, why no one has said so in all that
> > > >     long argument??
> > >
> > >
> > > One more time: in Emacs 28 package-install doesn't upgrade,
> > > but it installs the latest, which is incompatible behaviour
> > > if you move to Emacs 29, where that won't happen.
> >
> > Is this for Eglot (and use-package), or is this for any package
> > installed from ELPA?  IOW, does Emacs 29's package-install still
> > install the latest version of a package from ELPA?  And does that
> > happen even if some older version of a package is already installed?
> 
> Yes to the first, no to the second.
> 
> > > So if you're used to setting up a brand new Emacs 28 and
> > > package-install Eglot to get versions with nice features and
> > > bugfixes, you may be dismayed to find that doing the very
> > > same thing in Emacs 29 results in what will probably be a
> > > old version.
> >
> > Are you talking about users who didn't update their Eglot, except when
> > a new Emacs version was released?  Or are you talking about users who
> > updated Eglot from ELPA (using package-install) even between Emacs
> > releases?  Or are you talking about something else entirely?
> 
> Just spend that minute, Eli.   Call it an investment.  And then
> re-read what I wrote in that paragraph.  It will make sense.
> 
> > > And this problem will persist _and_ worsen during the lifetime of
> > > Emacs 29
> >
> > Why do you insist on reiterating past disagreements, especially when
> > they are tangents, is beyond me.
> 
> You brought it up!!! You asked "was this all based on a mistake?"
> I just clarified and summarized what happens!
> 
> > > Anyway, experimenting and experiencing this yourself will
> > > take at most a minute if you have Emacs 28 and Emacs 29
> > > handy, so there's no shroud of mystery whatsoever.
> >
> > Why do you think I even have a minute to spare?
> 
> Clearly, unless you're some kind of superhuman being, you're
> wasting much more time writing these messages. So I figured
> a minute of experimentation would probably save a lot of
> your time and our collective time.
> 
> You're still asking me questions, and then when I answer
> them you say I'm bringing up old past disagreements.
> 
> > I'm asking the experts on package.el to describe what happens
> > precisely _because_ I don't have that time.  For example, today, I've
> > been at the keyboard, working on various Emacs issues for the last 7
> > hours without any breaks, and this is my weekend!
> 
> For your health, take a break Eli.
> 
> > > But we already went through all this, and all these decisions
> > > have been made already, and they are ultimately all legitimate
> > > even if we all disagree.
> >
> > It sounds like what we went through was based on a misunderstanding,
> > or at least Dmitry says it was.
> 
> Again, please invest that minute (after taking a break, of course).
> 
> > > I don't think the NEWS entry needed rewording: it didn't
> > > say anything inaccurate or misleading.  You made it more
> > > complete, and perfectly accurate, but also more inescrutable,
> > > which is not really what NEWS should be about IMO.
> >
> > Clearly, if I thought the original wording was acceptable, I wouldn't
> > have touched it
> 
> There was nothing wrong or misleading about it.  What possibly
> can be wrong or unacceptable in announcing "Eglot can upgrade
> itself to the latest version." to Eglot users?  But let's just let it go, your
> reworking isn't that horrible either.

You are not helping.  None of this helps.  I asked specific questions,
but got no answers I can use.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sat, 06 May 2023 17:11:28 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> 
> You are not helping.  None of this helps.  I asked specific questions,
> but got no answers I can use.

Philip and Dmitry, would you please help me to figure out the answers
to my questions?  TIA.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 06/05/2023 16:48, João Távora wrote:
> There was nothing wrong or misleading about it.  What possibly
> can be wrong or unacceptable in announcing "Eglot can upgrade
> itself to the latest version." to Eglot users?  But let's just let it go, your
> reworking isn't that horrible either.

Is it really fine? Saying

   command 'package-install', which by default
   will not upgrade "built-in" packages, those that come with Emacs

implies that 'package-install' *will* upgrade some other kinds of packages.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 06/05/2023 16:22, Eli Zaretskii wrote:
>> From: João Távora <joaotavora@gmail.com>
>> Date: Sat, 6 May 2023 14:03:02 +0100
>> Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
>>
>> On Sat, May 6, 2023 at 11:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
>>
>>> I'm surprised, to say the least.  To recap:
>>>
>>>    . the entire discussion of bug#62720, of which this is fallout, was
>>>      about allowing package-install to upgrade Eglot, and according to
>>>      João, that's what package-install did before Emacs 29.  Was this
>>>      all based on a mistake? if so, why no one has said so in all that
>>>      long argument??
>>
>>
>> One more time: in Emacs 28 package-install doesn't upgrade,
>> but it installs the latest, which is incompatible behaviour
>> if you move to Emacs 29, where that won't happen.
> 
> Is this for Eglot (and use-package), or is this for any package
> installed from ELPA?  IOW, does Emacs 29's package-install still
> install the latest version of a package from ELPA?  And does that
> happen even if some older version of a package is already installed?

Like Joao said:

   Yes to the first, no to the second.

Meaning, 'M-x package-install' will install the latest version (or some 
available version) from ELPA, if the package is not installed.

If some version of it is installed from ELPA (!) already, 'M-x 
package-install' won't upgrade.

>> So if you're used to setting up a brand new Emacs 28 and
>> package-install Eglot to get versions with nice features and
>> bugfixes, you may be dismayed to find that doing the very
>> same thing in Emacs 29 results in what will probably be a
>> old version.
> 
> Are you talking about users who didn't update their Eglot, except when
> a new Emacs version was released?  Or are you talking about users who
> updated Eglot from ELPA (using package-install) even between Emacs
> releases?  Or are you talking about something else entirely?

He is mostly talking about users who e.g. wiped their config directory 
and ~/.emacs.d/elpa and are starting anew. Or just deleted 
~/.emacs.d/elpa/eglot-xxxxxx. In that situation, indeed, 'M-x 
package-install' will install the latest version from ELPA.

In Emacs 29, however, it won't. Because one version of Eglot is 
available already (built-in).

I'm pretty sure I have outlined this twice already, not too long ago. 
Prefacing the first message with an apology, saying I had been 
previously confused myself.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sat, 6 May 2023 18:26:11 +0300
> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> One more time: in Emacs 28 package-install doesn't upgrade,
> >> but it installs the latest, which is incompatible behaviour
> >> if you move to Emacs 29, where that won't happen.
> > 
> > Is this for Eglot (and use-package), or is this for any package
> > installed from ELPA?  IOW, does Emacs 29's package-install still
> > install the latest version of a package from ELPA?  And does that
> > happen even if some older version of a package is already installed?
> 
> Like Joao said:
> 
>    Yes to the first, no to the second.

Yes, except that I asked 3 questions, not 2, and the first was "either
or", so saying "yes" doesn't help.  But never mind.

> Meaning, 'M-x package-install' will install the latest version (or some 
> available version) from ELPA, if the package is not installed.
> 
> If some version of it is installed from ELPA (!) already, 'M-x 
> package-install' won't upgrade.

Then I don't understand why you decided to drop the similar change to
package-upgrade.  At the time I thought package-install can be used as
an alternative, but if it cannot, I think we should add to
package-upgrade the same optional behavior of upgrading a built-in
package as we have in package-install.

What other methods currently exist to upgrade an already installed
package (or a non-built-in package that is already installed)?  I know
about one -- via lisp-packages (a.k.a. package menu); are there
others?

Will any of these methods upgrade a built-in package, at least as an
optional behavior?

> >> So if you're used to setting up a brand new Emacs 28 and
> >> package-install Eglot to get versions with nice features and
> >> bugfixes, you may be dismayed to find that doing the very
> >> same thing in Emacs 29 results in what will probably be a
> >> old version.
> > 
> > Are you talking about users who didn't update their Eglot, except when
> > a new Emacs version was released?  Or are you talking about users who
> > updated Eglot from ELPA (using package-install) even between Emacs
> > releases?  Or are you talking about something else entirely?
> 
> He is mostly talking about users who e.g. wiped their config directory 
> and ~/.emacs.d/elpa and are starting anew. Or just deleted 
> ~/.emacs.d/elpa/eglot-xxxxxx.

Is this what many users do?  I'd be surprised, but maybe I'm missing
something.

> In that situation, indeed, 'M-x 
> package-install' will install the latest version from ELPA.
> 
> In Emacs 29, however, it won't. Because one version of Eglot is 
> available already (built-in).

But if emptying ~/.emacs.d/elpa is not a frequent use case, why should
we care about it so much?  It sounds like bug#62720 and the entire
long dispute that followed were focused on this strange use pattern,
instead of talking about more reasonable upgrade scenarios?

> I'm pretty sure I have outlined this twice already, not too long ago. 
> Prefacing the first message with an apology, saying I had been 
> previously confused myself.

I apologize for being confused and for asking almost the same
questions repeatedly, but given that fact that even people who are
familiar with package.el are confused, I think I'm in good company.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 06/05/2023 18:44, Eli Zaretskii wrote:
>> Date: Sat, 6 May 2023 18:26:11 +0300
>> Cc: emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> One more time: in Emacs 28 package-install doesn't upgrade,
>>>> but it installs the latest, which is incompatible behaviour
>>>> if you move to Emacs 29, where that won't happen.
>>>
>>> Is this for Eglot (and use-package), or is this for any package
>>> installed from ELPA?  IOW, does Emacs 29's package-install still
>>> install the latest version of a package from ELPA?  And does that
>>> happen even if some older version of a package is already installed?
>>
>> Like Joao said:
>>
>>     Yes to the first, no to the second.
> 
> Yes, except that I asked 3 questions, not 2, and the first was "either
> or", so saying "yes" doesn't help.  But never mind.

The answers were for the two questions after "IOW".

>> Meaning, 'M-x package-install' will install the latest version (or some
>> available version) from ELPA, if the package is not installed.
>>
>> If some version of it is installed from ELPA (!) already, 'M-x
>> package-install' won't upgrade.
> 
> Then I don't understand why you decided to drop the similar change to
> package-upgrade.  At the time I thought package-install can be used as
> an alternative, but if it cannot, I think we should add to
> package-upgrade the same optional behavior of upgrading a built-in
> package as we have in package-install.

We now have a better solution on master: 'M-x package-upgrade' simply 
upgrades the built-ins, no questions asked.

If we added the behavior similar to the addition in package-install 
(with prefix arguments and guarded by an option, possibly even a new 
optional argument), we'd have to carry over that awkward convention to 
Emacs 30 in some form. And as you recall, Joao wasn't happy with either 
solution anyway (of those that you liked enough).

> What other methods currently exist to upgrade an already installed
> package (or a non-built-in package that is already installed)?  I know
> about one -- via lisp-packages (a.k.a. package menu); are there
> others?

Also:
M-x package-upgrade
M-x package-upgrade-all

> Will any of these methods upgrade a built-in package, at least as an
> optional behavior?

Not in Emacs 29.

>>>> So if you're used to setting up a brand new Emacs 28 and
>>>> package-install Eglot to get versions with nice features and
>>>> bugfixes, you may be dismayed to find that doing the very
>>>> same thing in Emacs 29 results in what will probably be a
>>>> old version.
>>>
>>> Are you talking about users who didn't update their Eglot, except when
>>> a new Emacs version was released?  Or are you talking about users who
>>> updated Eglot from ELPA (using package-install) even between Emacs
>>> releases?  Or are you talking about something else entirely?
>>
>> He is mostly talking about users who e.g. wiped their config directory
>> and ~/.emacs.d/elpa and are starting anew. Or just deleted
>> ~/.emacs.d/elpa/eglot-xxxxxx.
> 
> Is this what many users do?  I'd be surprised, but maybe I'm missing
> something.

I have no idea. Joao says this is common enough. That has not been my 
experience, but I don't deal with support for Eglot users either.

>> In that situation, indeed, 'M-x
>> package-install' will install the latest version from ELPA.
>>
>> In Emacs 29, however, it won't. Because one version of Eglot is
>> available already (built-in).
> 
> But if emptying ~/.emacs.d/elpa is not a frequent use case, why should
> we care about it so much?  It sounds like bug#62720 and the entire
> long dispute that followed were focused on this strange use pattern,
> instead of talking about more reasonable upgrade scenarios?

We focused on it because, apparently, using 'M-x package-install' worked 
in more cases in Emacs 28 than in Emacs 29. And some think it's 
important. And because 'package-upgrade' is not in Emacs 28 at all.

Personally, I think it's better to focus on fixing 'package-upgrade' 
(which I did). But I don't think it's constructive to hide that fix 
behind a pref.

>> I'm pretty sure I have outlined this twice already, not too long ago.
>> Prefacing the first message with an apology, saying I had been
>> previously confused myself.
> 
> I apologize for being confused and for asking almost the same
> questions repeatedly, but given that fact that even people who are
> familiar with package.el are confused, I think I'm in good company.

It's okay. Just answering the question from your second response in this 
thread.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sat, 6 May 2023 18:54:47 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> If some version of it is installed from ELPA (!) already, 'M-x
> >> package-install' won't upgrade.
> > 
> > Then I don't understand why you decided to drop the similar change to
> > package-upgrade.  At the time I thought package-install can be used as
> > an alternative, but if it cannot, I think we should add to
> > package-upgrade the same optional behavior of upgrading a built-in
> > package as we have in package-install.
> 
> We now have a better solution on master: 'M-x package-upgrade' simply 
> upgrades the built-ins, no questions asked.

What we have on master is not relevant to what we discuss here, which
is Emacs 29.

> If we added the behavior similar to the addition in package-install 
> (with prefix arguments and guarded by an option, possibly even a new 
> optional argument), we'd have to carry over that awkward convention to 
> Emacs 30 in some form. And as you recall, Joao wasn't happy with either 
> solution anyway (of those that you liked enough).

The question is: is it reasonable not to allow package-upgrade in
Emacs 29 to upgrade a built-in package?  Not even as an option?

> > What other methods currently exist to upgrade an already installed
> > package (or a non-built-in package that is already installed)?  I know
> > about one -- via lisp-packages (a.k.a. package menu); are there
> > others?
> 
> Also:
> M-x package-upgrade
> M-x package-upgrade-all
> 
> > Will any of these methods upgrade a built-in package, at least as an
> > optional behavior?
> 
> Not in Emacs 29.

So I think we have a problem, and I think we need to solve it.

Philip, Stefan: WDYT about this?

What about installation from the list-packages menu: will it upgrade a
built-in package if package-install-upgrade-built-in is non-nil?

> > But if emptying ~/.emacs.d/elpa is not a frequent use case, why should
> > we care about it so much?  It sounds like bug#62720 and the entire
> > long dispute that followed were focused on this strange use pattern,
> > instead of talking about more reasonable upgrade scenarios?
> 
> We focused on it because, apparently, using 'M-x package-install' worked 
> in more cases in Emacs 28 than in Emacs 29. And some think it's 
> important. And because 'package-upgrade' is not in Emacs 28 at all.

If package-upgrade was not in Emacs 28, how did users upgrade
installed packages in Emacs 28 and before?

> Personally, I think it's better to focus on fixing 'package-upgrade' 
> (which I did). But I don't think it's constructive to hide that fix 
> behind a pref.

I don't see a zero-sum game here.  We could focus on both.  But I
don't use package.el and never will, so if those who use it and
maintain it think otherwise, I won't insist.  Although I find this
stance very strange indeed, to say the least.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 6 May 2023 18:54:47 +0300
>> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> >> If some version of it is installed from ELPA (!) already, 'M-x
>> >> package-install' won't upgrade.
>> > 
>> > Then I don't understand why you decided to drop the similar change to
>> > package-upgrade.  At the time I thought package-install can be used as
>> > an alternative, but if it cannot, I think we should add to
>> > package-upgrade the same optional behavior of upgrading a built-in
>> > package as we have in package-install.
>> 
>> We now have a better solution on master: 'M-x package-upgrade' simply 
>> upgrades the built-ins, no questions asked.
>
> What we have on master is not relevant to what we discuss here, which
> is Emacs 29.
>
>> If we added the behavior similar to the addition in package-install 
>> (with prefix arguments and guarded by an option, possibly even a new 
>> optional argument), we'd have to carry over that awkward convention to 
>> Emacs 30 in some form. And as you recall, Joao wasn't happy with either 
>> solution anyway (of those that you liked enough).
>
> The question is: is it reasonable not to allow package-upgrade in
> Emacs 29 to upgrade a built-in package?  Not even as an option?

I think that would be fine.  The code looks fine, and considering the
commotion about this functionality and the effort that has been put into
preparing a minimal functional change, I think that it would be warranted
to allow for this change to be applies to emacs-29.

>> > What other methods currently exist to upgrade an already installed
>> > package (or a non-built-in package that is already installed)?  I know
>> > about one -- via lisp-packages (a.k.a. package menu); are there
>> > others?
>> 
>> Also:
>> M-x package-upgrade
>> M-x package-upgrade-all
>> 
>> > Will any of these methods upgrade a built-in package, at least as an
>> > optional behavior?
>> 
>> Not in Emacs 29.
>
> So I think we have a problem, and I think we need to solve it.
>
> Philip, Stefan: WDYT about this?
>
> What about installation from the list-packages menu: will it upgrade a
> built-in package if package-install-upgrade-built-in is non-nil?

Yes it will, which is why I find this discussion silly and have not been
following it in detail (also other non-emacs responsibilities have been
intervening).

>> > But if emptying ~/.emacs.d/elpa is not a frequent use case, why should
>> > we care about it so much?  It sounds like bug#62720 and the entire
>> > long dispute that followed were focused on this strange use pattern,
>> > instead of talking about more reasonable upgrade scenarios?
>> 
>> We focused on it because, apparently, using 'M-x package-install' worked 
>> in more cases in Emacs 28 than in Emacs 29. And some think it's 
>> important. And because 'package-upgrade' is not in Emacs 28 at all.
>
> If package-upgrade was not in Emacs 28, how did users upgrade
> installed packages in Emacs 28 and before?

They invoked M-x list-packages, waited for the upgrade to appear,
selected them with U and then executed the update with x.  This is what
used to work, and what will continue to work.  There was some mention
about problems with M-x list-packages, but I haven't heard of any
specific issues that could be addressed (I personally suspect it might
be an issue related to the tracking of the master or close-to-master
branches).

>> Personally, I think it's better to focus on fixing 'package-upgrade' 
>> (which I did). But I don't think it's constructive to hide that fix 
>> behind a pref.
>
> I don't see a zero-sum game here.  We could focus on both.  But I
> don't use package.el and never will, so if those who use it and
> maintain it think otherwise, I won't insist.  Although I find this
> stance very strange indeed, to say the least.

(This explains some of the confusion, I was under the assumption that
some of your questions were from a position of Socratic ignorance,
but if you don't use it at all, then some of the past confusion makes
more sense to me.)

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Dmitry Gutov <dmitry@gutov.dev>,  Stefan Monnier
>  <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
> Date: Sat, 06 May 2023 18:44:35 +0000
> 
> > If package-upgrade was not in Emacs 28, how did users upgrade
> > installed packages in Emacs 28 and before?
> 
> They invoked M-x list-packages, waited for the upgrade to appear,
> selected them with U and then executed the update with x.  This is what
> used to work, and what will continue to work.

But only if package-install-upgrade-built-in is non-nil, right?

> > I don't see a zero-sum game here.  We could focus on both.  But I
> > don't use package.el and never will, so if those who use it and
> > maintain it think otherwise, I won't insist.  Although I find this
> > stance very strange indeed, to say the least.
> 
> (This explains some of the confusion, I was under the assumption that
> some of your questions were from a position of Socratic ignorance,

You should know me better.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Dmitry Gutov <dmitry@gutov.dev>,  Stefan Monnier
>>  <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
>> Date: Sat, 06 May 2023 18:44:35 +0000
>> 
>> > If package-upgrade was not in Emacs 28, how did users upgrade
>> > installed packages in Emacs 28 and before?
>> 
>> They invoked M-x list-packages, waited for the upgrade to appear,
>> selected them with U and then executed the update with x.  This is what
>> used to work, and what will continue to work.
>
> But only if package-install-upgrade-built-in is non-nil, right?

No, that is unrelated since that user option has no effect on the
package menu.

Upgrading via U only works if the user has already installed a package
from ELPA.  So in this case, a user would have to run M-x list-packages,
select and install Eglot, thereby instructing package.el to load the
local version instead of the version bundled with Emacs.  This procedure
is how users would have upgraded from a older version of project.el, for
example.
                                                          
This works regardless of what package-install-upgrade-built-in has been
set to.  That is only related to how the `package-install' command
works.

>> > I don't see a zero-sum game here.  We could focus on both.  But I
>> > don't use package.el and never will, so if those who use it and
>> > maintain it think otherwise, I won't insist.  Although I find this
>> > stance very strange indeed, to say the least.
>> 
>> (This explains some of the confusion, I was under the assumption that
>> some of your questions were from a position of Socratic ignorance,
>
> You should know me better.

In that case I admit to plain ignorance. :)

-- 
Philip Kaludercic

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 06/05/2023 21:44, Philip Kaludercic wrote:
>>>> But if emptying ~/.emacs.d/elpa is not a frequent use case, why should
>>>> we care about it so much?  It sounds like bug#62720 and the entire
>>>> long dispute that followed were focused on this strange use pattern,
>>>> instead of talking about more reasonable upgrade scenarios?
>>> We focused on it because, apparently, using 'M-x package-install' worked
>>> in more cases in Emacs 28 than in Emacs 29. And some think it's
>>> important. And because 'package-upgrade' is not in Emacs 28 at all.
>> If package-upgrade was not in Emacs 28, how did users upgrade
>> installed packages in Emacs 28 and before?
> They invoked M-x list-packages, waited for the upgrade to appear,
> selected them with U and then executed the update with x.  This is what
> used to work, and what will continue to work.

Not for "active built-ins", though.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 06/05/2023 19:40, Eli Zaretskii wrote:
>> Date: Sat, 6 May 2023 18:54:47 +0300
>> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> If some version of it is installed from ELPA (!) already, 'M-x
>>>> package-install' won't upgrade.
>>>
>>> Then I don't understand why you decided to drop the similar change to
>>> package-upgrade.  At the time I thought package-install can be used as
>>> an alternative, but if it cannot, I think we should add to
>>> package-upgrade the same optional behavior of upgrading a built-in
>>> package as we have in package-install.
>>
>> We now have a better solution on master: 'M-x package-upgrade' simply
>> upgrades the built-ins, no questions asked.
> 
> What we have on master is not relevant to what we discuss here, which
> is Emacs 29.

What's on master is how I think this function should look and work. Any 
potential change in Emacs 29 should be compatible with it.

>> If we added the behavior similar to the addition in package-install
>> (with prefix arguments and guarded by an option, possibly even a new
>> optional argument), we'd have to carry over that awkward convention to
>> Emacs 30 in some form. And as you recall, Joao wasn't happy with either
>> solution anyway (of those that you liked enough).
> 
> The question is: is it reasonable not to allow package-upgrade in
> Emacs 29 to upgrade a built-in package?  Not even as an option?

We've documented that it does not upgrade built-ins. That's what we 
settled on.

The discussion about options brings us back to complications:

- That package-upgrade could (probably should) work differently from 
package-upgrade-all WRT built-ins
- That it would probably make sense to add a new different option rather 
than reuse package-install-upgrade-built-in
- package-upgrade-all would require a yet another option (or a separate 
value, at least)

So we might as well skip all this, retouch the docs and call it a day.

Even fixing 'M-x package-upgrade' properly won't please everyone either: 
e.g. the latest question on Reddit 
(https://www.reddit.com/r/emacs/comments/139f8r9/comment/jj3kmty/?context=3) 
is about package-menu-mark-upgrades including built-in Eglot as well.

>>> What other methods currently exist to upgrade an already installed
>>> package (or a non-built-in package that is already installed)?  I know
>>> about one -- via lisp-packages (a.k.a. package menu); are there
>>> others?
>>
>> Also:
>> M-x package-upgrade
>> M-x package-upgrade-all
>>
>>> Will any of these methods upgrade a built-in package, at least as an
>>> optional behavior?
>>
>> Not in Emacs 29.
> 
> So I think we have a problem, and I think we need to solve it.
> 
> Philip, Stefan: WDYT about this?
> 
> What about installation from the list-packages menu: will it upgrade a
> built-in package if package-install-upgrade-built-in is non-nil?

Installation or upgrade?

package-menu-mark-upgrades ('U') is not affected by 
package-install-upgrade-built-in. It won't.

But a version from ELPA can still be installed by using 'i' in the 
list-packages menu. Just as we've written in the docstrings.

>>> But if emptying ~/.emacs.d/elpa is not a frequent use case, why should
>>> we care about it so much?  It sounds like bug#62720 and the entire
>>> long dispute that followed were focused on this strange use pattern,
>>> instead of talking about more reasonable upgrade scenarios?
>>
>> We focused on it because, apparently, using 'M-x package-install' worked
>> in more cases in Emacs 28 than in Emacs 29. And some think it's
>> important. And because 'package-upgrade' is not in Emacs 28 at all.
> 
> If package-upgrade was not in Emacs 28, how did users upgrade
> installed packages in Emacs 28 and before?

Using package-menu-mark-upgrades ('U').

The built-in packages like project or xref often didn't get upgraded at 
all, or if they did, that happened usually because of some other package 
(like Eglot) declaring dependency on some newer version of either. Then 
that happened together with Eglot's upgrade.

>> Personally, I think it's better to focus on fixing 'package-upgrade'
>> (which I did). But I don't think it's constructive to hide that fix
>> behind a pref.
> 
> I don't see a zero-sum game here.  We could focus on both.  But I
> don't use package.el and never will, so if those who use it and
> maintain it think otherwise, I won't insist.  Although I find this
> stance very strange indeed, to say the least.

We haven't been able to settle on a reasonable change that would satisfy 
your safety requirements.

We should probably focus on getting Emacs 29 out soon and then look into 
backporting 53cc61d60db to Emacs 29.2.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sat, 6 May 2023 22:14:13 +0300
> Cc: emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > What about installation from the list-packages menu: will it upgrade a
> > built-in package if package-install-upgrade-built-in is non-nil?
> 
> Installation or upgrade?
> 
> package-menu-mark-upgrades ('U') is not affected by 
> package-install-upgrade-built-in. It won't.

Shouldn't it?

> But a version from ELPA can still be installed by using 'i' in the 
> list-packages menu. Just as we've written in the docstrings.

That was stated at the very beginning of bug#62720.

> > If package-upgrade was not in Emacs 28, how did users upgrade
> > installed packages in Emacs 28 and before?
> 
> Using package-menu-mark-upgrades ('U').

So we should allow that, at least as an optional behavior in Emacs 29,
right?

> We should probably focus on getting Emacs 29 out soon

Why do you think this is not what happens?

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 06/05/2023 22:38, Eli Zaretskii wrote:
>> Date: Sat, 6 May 2023 22:14:13 +0300
>> Cc: emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> What about installation from the list-packages menu: will it upgrade a
>>> built-in package if package-install-upgrade-built-in is non-nil?
>>
>> Installation or upgrade?
>>
>> package-menu-mark-upgrades ('U') is not affected by
>> package-install-upgrade-built-in. It won't.
> 
> Shouldn't it?

Maybe, maybe not.

A user that customized that option to have (package-install 'eglot) 
ensure that a version from ELPA is installed might not want or expect 
for it to affect package-menu-mark-upgrades and/or package-upgrade-all. 
Or anticipate the full consequences anyway.

After all, in the former case you choose a specific package to install, 
and the last two act on all built-ins together.

>> But a version from ELPA can still be installed by using 'i' in the
>> list-packages menu. Just as we've written in the docstrings.
> 
> That was stated at the very beginning of bug#62720.

Just making sure we're on the same page.

>>> If package-upgrade was not in Emacs 28, how did users upgrade
>>> installed packages in Emacs 28 and before?
>>
>> Using package-menu-mark-upgrades ('U').
> 
> So we should allow that, at least as an optional behavior in Emacs 29,
> right?

I don't believe in "optionality" here.

If the user has to hunt for the option to toggle, they might as well 
find the "one little trick" that does the thing they want.

Most people will only have to do that once (per config), if at all: 
after Eglot is upgraded this way, it's smooth sailing from then on.

>> We should probably focus on getting Emacs 29 out soon
> 
> Why do you think this is not what happens?

I'm just saying we've spent enough time on this particular issue. We can 
improve the docs the best we can and move on.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[João Távora]

On Sat, May 6, 2023 at 9:32 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > So we should allow that, at least as an optional behavior in Emacs 29,
> > right?
>
> I don't believe in "optionality" here.
>
> If the user has to hunt for the option to toggle, they might as well
> find the "one little trick" that does the thing they want.
>
> Most people will only have to do that once (per config), if at all:
> after Eglot is upgraded this way, it's smooth sailing from then on.

Very well put, and exactly my thoughts.

Also, let's assume eglot-x.el makes it to GNU ELPA as is
being proposed in a side thread here. Then, that new
package will depend on Eglot.  And thus simply M-x
package-install RET eglot-x will also rev up Eglot, the :core
package to the latest version -- on any Emacs version.  That
will be another "trick".  In fact if eglot-x gains popularity
and is minimally useful, we might as well start recommending
that people ask for eglot-x as a means to upgrade Eglot itself.

João

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sat, 6 May 2023 23:31:31 +0300
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 06/05/2023 22:38, Eli Zaretskii wrote:
> >> package-menu-mark-upgrades ('U') is not affected by
> >> package-install-upgrade-built-in. It won't.
> > 
> > Shouldn't it?
> 
> Maybe, maybe not.

I think it better did, because using "U" would upgrade Eglot and
use-package in Emacs 28 and before.  So we should give users who want
that the capability of keeping that workflow in Emacs 29, if only as
opt-in behavior.

Also, "/ u" should ideally show built-in packages as well, when
package-install-upgrade-built-in is non-nil.

Philip, can these two changes be implemented safely for Emacs 29?

> A user that customized that option to have (package-install 'eglot) 
> ensure that a version from ELPA is installed might not want or expect 
> for it to affect package-menu-mark-upgrades and/or package-upgrade-all. 

That is true, but denying them the possibility of upgrading would be
worse, I think.  And since this is opt-in behavior, the user is less
likely to be tripped by that without realizing it.

> Or anticipate the full consequences anyway.

Documentation should solve this aspect.

> >>> If package-upgrade was not in Emacs 28, how did users upgrade
> >>> installed packages in Emacs 28 and before?
> >>
> >> Using package-menu-mark-upgrades ('U').
> > 
> > So we should allow that, at least as an optional behavior in Emacs 29,
> > right?
> 
> I don't believe in "optionality" here.
> 
> If the user has to hunt for the option to toggle, they might as well 
> find the "one little trick" that does the thing they want.
> 
> Most people will only have to do that once (per config), if at all: 
> after Eglot is upgraded this way, it's smooth sailing from then on.

I think allowing such an optional behavior and documenting it in NEWS
and in the manual should go a long way towards eliminating at least
some of your fears.

> >> We should probably focus on getting Emacs 29 out soon
> > 
> > Why do you think this is not what happens?
> 
> I'm just saying we've spent enough time on this particular issue. We can 
> improve the docs the best we can and move on.

This is not the only issue that holds the next pretest.  So nothing is
lost by spending some more time on this, especially since it's now
clear the original longish discussion was at least partially based on
some kind of Rashomon effect.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 6 May 2023 23:31:31 +0300
>> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> On 06/05/2023 22:38, Eli Zaretskii wrote:
>> >> package-menu-mark-upgrades ('U') is not affected by
>> >> package-install-upgrade-built-in. It won't.
>> > 
>> > Shouldn't it?
>> 
>> Maybe, maybe not.
>
> I think it better did, because using "U" would upgrade Eglot and
> use-package in Emacs 28 and before.  So we should give users who want
> that the capability of keeping that workflow in Emacs 29, if only as
> opt-in behavior.

"Workflow" is really to high of a term for the problem being discussed
here.  Installing packages is not a periodical thing people to on a
regular basis.

> Also, "/ u" should ideally show built-in packages as well, when
> package-install-upgrade-built-in is non-nil.

So the point here would be that a user who enables this option, regards
any newer version of a core-package on ELPA as something that should be
installed?

Perhaps this is a tangent, but what would happen if a third-party
repository like MELPA adds a package with the same name as a core
package?

> Philip, can these two changes be implemented safely for Emacs 29?

It certainly can be done.

>> A user that customized that option to have (package-install 'eglot) 
>> ensure that a version from ELPA is installed might not want or expect 
>> for it to affect package-menu-mark-upgrades and/or package-upgrade-all. 
>
> That is true, but denying them the possibility of upgrading would be
> worse, I think.  And since this is opt-in behavior, the user is less
> likely to be tripped by that without realizing it.
>
>> Or anticipate the full consequences anyway.
>
> Documentation should solve this aspect.
>
>> >>> If package-upgrade was not in Emacs 28, how did users upgrade
>> >>> installed packages in Emacs 28 and before?
>> >>
>> >> Using package-menu-mark-upgrades ('U').
>> > 
>> > So we should allow that, at least as an optional behavior in Emacs 29,
>> > right?
>> 
>> I don't believe in "optionality" here.
>> 
>> If the user has to hunt for the option to toggle, they might as well 
>> find the "one little trick" that does the thing they want.
>> 
>> Most people will only have to do that once (per config), if at all: 
>> after Eglot is upgraded this way, it's smooth sailing from then on.
>
> I think allowing such an optional behavior and documenting it in NEWS
> and in the manual should go a long way towards eliminating at least
> some of your fears.
>
>> >> We should probably focus on getting Emacs 29 out soon
>> > 
>> > Why do you think this is not what happens?
>> 
>> I'm just saying we've spent enough time on this particular issue. We can 
>> improve the docs the best we can and move on.
>
> This is not the only issue that holds the next pretest.  So nothing is
> lost by spending some more time on this, especially since it's now
> clear the original longish discussion was at least partially based on
> some kind of Rashomon effect.

-- 
Philip Kaludercic

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Dmitry Gutov <dmitry@gutov.dev>,  monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> Date: Sun, 07 May 2023 08:46:53 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think it better did, because using "U" would upgrade Eglot and
> > use-package in Emacs 28 and before.  So we should give users who want
> > that the capability of keeping that workflow in Emacs 29, if only as
> > opt-in behavior.
> 
> "Workflow" is really to high of a term for the problem being discussed
> here.  Installing packages is not a periodical thing people to on a
> regular basis.

Since we are talking about an optional feature, it will suit this
well, I think.  The target audience for this are those users who were
used to upgrade Eglot regularly when it was not a core package.
Likewise for other packages that would be brought into core in the
future.

> > Also, "/ u" should ideally show built-in packages as well, when
> > package-install-upgrade-built-in is non-nil.
> 
> So the point here would be that a user who enables this option, regards
> any newer version of a core-package on ELPA as something that should be
> installed?

Yes.

> Perhaps this is a tangent, but what would happen if a third-party
> repository like MELPA adds a package with the same name as a core
> package?

This problem exists today already, with non-core packages.  Right?

> > Philip, can these two changes be implemented safely for Emacs 29?
> 
> It certainly can be done.

I'd appreciate if you could show a patch that we could then consider.
TIA.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 1706 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Dmitry Gutov <dmitry@gutov.dev>,  monnier@iro.umontreal.ca,
>>   emacs-devel@gnu.org
>> Date: Sun, 07 May 2023 08:46:53 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > I think it better did, because using "U" would upgrade Eglot and
>> > use-package in Emacs 28 and before.  So we should give users who want
>> > that the capability of keeping that workflow in Emacs 29, if only as
>> > opt-in behavior.
>> 
>> "Workflow" is really to high of a term for the problem being discussed
>> here.  Installing packages is not a periodical thing people to on a
>> regular basis.
>
> Since we are talking about an optional feature, it will suit this
> well, I think.  The target audience for this are those users who were
> used to upgrade Eglot regularly when it was not a core package.
> Likewise for other packages that would be brought into core in the
> future.
>
>> > Also, "/ u" should ideally show built-in packages as well, when
>> > package-install-upgrade-built-in is non-nil.
>> 
>> So the point here would be that a user who enables this option, regards
>> any newer version of a core-package on ELPA as something that should be
>> installed?
>
> Yes.
>
>> Perhaps this is a tangent, but what would happen if a third-party
>> repository like MELPA adds a package with the same name as a core
>> package?
>
> This problem exists today already, with non-core packages.  Right?
>
>> > Philip, can these two changes be implemented safely for Emacs 29?
>> 
>> It certainly can be done.
>
> I'd appreciate if you could show a patch that we could then consider.
> TIA.

This is an initial, quick sketch:


[-- Attachment #2: Type: text/plain, Size: 1498 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 2892728ebd9..4c85db1aedb 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -3740,7 +3740,9 @@ package-menu--find-upgrades
       ;; ENTRY is (PKG-DESC [NAME VERSION STATUS DOC])
       (let ((pkg-desc (car entry))
             (status (aref (cadr entry) 2)))
-        (cond ((member status '("installed" "dependency" "unsigned" "external"))
+        (cond ((member status (append
+                               '("installed" "dependency" "unsigned" "external")
+                               (and package-install-upgrade-built-in '("built-in"))))
                (push pkg-desc installed))
               ((member status '("available" "new"))
                (setq available (package--append-to-alist pkg-desc available))))))
@@ -3749,8 +3751,10 @@ package-menu--find-upgrades
       (let* ((name (package-desc-name pkg-desc))
              (avail-pkg (cadr (assq name available))))
         (and avail-pkg
-             (version-list-< (package-desc-priority-version pkg-desc)
-                             (package-desc-priority-version avail-pkg))
+             (or (version-list-< (package-desc-priority-version pkg-desc)
+                                 (package-desc-priority-version avail-pkg))
+                 (and package-install-upgrade-built-in
+                      (package--active-built-in-p pkg-desc)))
              (push (cons name avail-pkg) upgrades))))
     upgrades))
 

[-- Attachment #3: Type: text/plain, Size: 105 bytes --]


but on my system, it immediately suggests upgrading 24 packages, which
is a lot.

-- 
Philip Kaludercic

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 07 May 2023 17:16:31 +0000
> 
> > I'd appreciate if you could show a patch that we could then consider.
> > TIA.
> 
> This is an initial, quick sketch:

LGTM.

> but on my system, it immediately suggests upgrading 24 packages, which
> is a lot.

Can you show the list of those 24 packages?

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sun, 07 May 2023 17:16:31 +0000
>> 
>> > I'd appreciate if you could show a patch that we could then consider.
>> > TIA.
>> 
>> This is an initial, quick sketch:
>
> LGTM.
>
>> but on my system, it immediately suggests upgrading 24 packages, which
>> is a lot.
>
> Can you show the list of those 24 packages?

I am on a different system now, so the prompt differs slightly and have
other updates, but this is what it looks like:

Packages to install: 23 (xref-1.6.3 verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4 svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8 org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6 jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5 eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).  Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).  Proceed? (y or n) 

M-x emacs-version
GNU Emacs 29.0.90 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0) of 2023-05-04

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 07 May 2023 19:24:26 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> but on my system, it immediately suggests upgrading 24 packages, which
> >> is a lot.
> >
> > Can you show the list of those 24 packages?
> 
> I am on a different system now, so the prompt differs slightly and have
> other updates, but this is what it looks like:
> 
> Packages to install: 23 (xref-1.6.3 verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4 svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8 org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6 jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5 eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).  Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).  Proceed? (y or n) 

I see nothing unexpected in this list.  And since the feature is
opt-in, what are the dangers of having it?

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sun, 07 May 2023 19:24:26 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> but on my system, it immediately suggests upgrading 24 packages, which
>> >> is a lot.
>> >
>> > Can you show the list of those 24 packages?
>> 
>> I am on a different system now, so the prompt differs slightly and have
>> other updates, but this is what it looks like:
>> 
>> Packages to install: 23 (xref-1.6.3
>> verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4
>> svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8
>> org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6
>> jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5
>> eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).
>> Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).
>> Proceed? (y or n)
>
> I see nothing unexpected in this list.  And since the feature is
> opt-in, what are the dangers of having it?

There is no danger, it might just be overwhelming?  I was a bit
surprised at first.  But if you think it is fine, then I have no
objections.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 07 May 2023 19:44:01 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Packages to install: 23 (xref-1.6.3
> >> verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4
> >> svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8
> >> org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6
> >> jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5
> >> eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).
> >> Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).
> >> Proceed? (y or n)
> >
> > I see nothing unexpected in this list.  And since the feature is
> > opt-in, what are the dangers of having it?
> 
> There is no danger, it might just be overwhelming?  I was a bit
> surprised at first.  But if you think it is fine, then I have no
> objections.

Assuming users who turn this on understand the meaning, I see no
problem.  Once this is on the branch, I will take care of documenting
this to make the consequences clear.

Thanks.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Mon, 08 May 2023 14:19:33 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: dmitry@gutov.dev, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > From: Philip Kaludercic <philipk@posteo.net>
> > Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> > Date: Sun, 07 May 2023 19:44:01 +0000
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > >> Packages to install: 23 (xref-1.6.3
> > >> verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4
> > >> svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8
> > >> org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6
> > >> jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5
> > >> eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).
> > >> Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).
> > >> Proceed? (y or n)
> > >
> > > I see nothing unexpected in this list.  And since the feature is
> > > opt-in, what are the dangers of having it?
> > 
> > There is no danger, it might just be overwhelming?  I was a bit
> > surprised at first.  But if you think it is fine, then I have no
> > objections.
> 
> Assuming users who turn this on understand the meaning, I see no
> problem.  Once this is on the branch, I will take care of documenting
> this to make the consequences clear.

Done; comments welcome.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sun, 07 May 2023 19:44:01 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Philip Kaludercic <philipk@posteo.net>
> >> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> >> Date: Sun, 07 May 2023 19:24:26 +0000
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> but on my system, it immediately suggests upgrading 24 packages, which
> >> >> is a lot.
> >> >
> >> > Can you show the list of those 24 packages?
> >> 
> >> I am on a different system now, so the prompt differs slightly and have
> >> other updates, but this is what it looks like:
> >> 
> >> Packages to install: 23 (xref-1.6.3
> >> verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4
> >> svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8
> >> org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6
> >> jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5
> >> eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).
> >> Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).
> >> Proceed? (y or n)
> >
> > I see nothing unexpected in this list.  And since the feature is
> > opt-in, what are the dangers of having it?
> 
> There is no danger, it might just be overwhelming?  I was a bit
> surprised at first.  But if you think it is fine, then I have no
> objections.

One more question: the patch you proposed affects both "U" and "/ u",
right?

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sun, 07 May 2023 19:44:01 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Philip Kaludercic <philipk@posteo.net>
>> >> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> >> Date: Sun, 07 May 2023 19:24:26 +0000
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> >> but on my system, it immediately suggests upgrading 24 packages, which
>> >> >> is a lot.
>> >> >
>> >> > Can you show the list of those 24 packages?
>> >> 
>> >> I am on a different system now, so the prompt differs slightly and have
>> >> other updates, but this is what it looks like:
>> >> 
>> >> Packages to install: 23 (xref-1.6.3
>> >> verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4
>> >> svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8
>> >> org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6
>> >> jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5
>> >> eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).
>> >> Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).
>> >> Proceed? (y or n)
>> >
>> > I see nothing unexpected in this list.  And since the feature is
>> > opt-in, what are the dangers of having it?
>> 
>> There is no danger, it might just be overwhelming?  I was a bit
>> surprised at first.  But if you think it is fine, then I have no
>> objections.
>
> One more question: the patch you proposed affects both "U" and "/ u",
> right?

Yes it does, since both of these functions use
`package-menu--find-upgrades'.  In the future, we should probably
centralise the upgrade logic in a single function, like
`package--upgradeable-packages'.

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 7 May 2023 23:36:24 +0300
>> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> On 07/05/2023 22:32, Eli Zaretskii wrote:
>> > Packages to install: 23 (xref-1.6.3
>> > verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4
>> > svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8
>> > org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6
>> > jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5
>> > eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3
>> > bind-key-2.4.1).  Packages to upgrade: 2 (editorconfig-0.9.1
>> > inspector-0.29).  Proceed? (y or n)
>> 
>> At least nadvice, cl-lib and cl-generic seem to be the odd ones (the 
>> built-in versions are higher, and the ELPA packages are supposed to be 
>> used as shims or backward compatibility wrappers). That looks like a bug.

I think you are right, I can extend my previous patch by a version check.

> Yes, I think it's an unrelated bug.  We had already reports about
> strange status values, and there's a new bug#63064 today about similar
> problems.  We should try to fix this for Emacs 29, I think.
>
>> Regarding potential downsides in general:
>> 
>> - We simply increase the odds of breakage when a built-in package is 
>> upgraded because it will reach more people, across different Emacs 
>> versions. There is good and bad in that (features will reach them faster 
>> too), but it's something to consider.
>> 
>> - I very rarely use Tramp or Org. I don't use ERC or bind-key. Or 
>> so-long, python, ntlm, use-package, verilog-mode. Unlike other installed 
>> packages (which I have hand-picked), these packages are here just by the 
>> virtue of being included in Emacs, but flipping the pref will also cause 
>> them to be upgraded (downloaded, take time unarchiving, take up space) 
>> every time there is a new version out. It's nothing dangerous 
>> (probably), but seems unfortunate anyway.
>
> AFAIU, the "U" command is not for everyone.  There are alternatives
> which allow selective upgrades, and users who don't want "surprises",
> or want the upgrade to take as little time and disk space as possible,
> should use those alternatives instead of "U".  I'll try to make that
> clear in the documentation.

I think that U is pretty standard, but yes, those who cannot use it are
free to select the packages they wish to upgrade manually (that being
said, Emacs was never known for being popular among people who are
struggling for every kilobyte of disk space).

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 08 May 2023 13:34:26 +0000
> 
> >> At least nadvice, cl-lib and cl-generic seem to be the odd ones (the 
> >> built-in versions are higher, and the ELPA packages are supposed to be 
> >> used as shims or backward compatibility wrappers). That looks like a bug.
> 
> I think you are right, I can extend my previous patch by a version check.

Good idea, IMO.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 571 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Mon, 08 May 2023 13:34:26 +0000
>> 
>> >> At least nadvice, cl-lib and cl-generic seem to be the odd ones (the 
>> >> built-in versions are higher, and the ELPA packages are supposed to be 
>> >> used as shims or backward compatibility wrappers). That looks like a bug.
>> 
>> I think you are right, I can extend my previous patch by a version check.
>
> Good idea, IMO.

OK, then this is my proposal:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Ensure-that-package-menu-respects-package-install-up.patch --]
[-- Type: text/x-diff, Size: 2040 bytes --]

From 8d78dc4ab9cb188f62c656d6d55326240816f8c6 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 10 May 2023 08:58:34 +0200
Subject: [PATCH] Ensure that package menu respects
 'package-install-upgrade-built-in'

* lisp/emacs-lisp/package.el (package-menu--find-upgrades): Check if
built-in packages can be upgraded if
'package-install-upgrade-built-in' is non-nil.
---
 lisp/emacs-lisp/package.el | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 9476354ebe0..7f94ed2e57b 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -3731,7 +3731,9 @@ package-menu--find-upgrades
       ;; ENTRY is (PKG-DESC [NAME VERSION STATUS DOC])
       (let ((pkg-desc (car entry))
             (status (aref (cadr entry) 2)))
-        (cond ((member status '("installed" "dependency" "unsigned" "external"))
+        (cond ((member status (append
+                               '("installed" "dependency" "unsigned" "external" "built-in")
+                               (and package-install-upgrade-built-in '("built-in"))))
                (push pkg-desc installed))
               ((member status '("available" "new"))
                (setq available (package--append-to-alist pkg-desc available))))))
@@ -3740,8 +3742,11 @@ package-menu--find-upgrades
       (let* ((name (package-desc-name pkg-desc))
              (avail-pkg (cadr (assq name available))))
         (and avail-pkg
-             (version-list-< (package-desc-priority-version pkg-desc)
-                             (package-desc-priority-version avail-pkg))
+             (and (version-list-< (package-desc-priority-version pkg-desc)
+                                  (package-desc-priority-version avail-pkg))
+                  (if package-install-upgrade-built-in
+                      (package--active-built-in-p pkg-desc)
+                    t))
              (push (cons name avail-pkg) upgrades))))
     upgrades))
 
-- 
2.39.2


[-- Attachment #3: Type: text/plain, Size: 101 bytes --]


This also reduced the number of packages that are suggested for an
upgrade to only 5 (on Emacs 29).

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Philip Kaludercic <philipk@posteo.net>
>>> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>>> Date: Mon, 08 May 2023 13:34:26 +0000
>>> 
>>> >> At least nadvice, cl-lib and cl-generic seem to be the odd ones (the 
>>> >> built-in versions are higher, and the ELPA packages are supposed to be 
>>> >> used as shims or backward compatibility wrappers). That looks like a bug.
>>> 
>>> I think you are right, I can extend my previous patch by a version check.
>>
>> Good idea, IMO.
>
> OK, then this is my proposal:

I noticed a bug, here is a revised version:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Ensure-that-package-menu-respects-package-install-up.patch --]
[-- Type: text/x-diff, Size: 1746 bytes --]

From 8f53ac64620db17a7b163889bb319b621ab97a25 Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Wed, 10 May 2023 08:58:34 +0200
Subject: [PATCH] Ensure that package menu respects
 'package-install-upgrade-built-in'

* lisp/emacs-lisp/package.el (package-menu--find-upgrades): Check if
built-in packages can be upgraded if
'package-install-upgrade-built-in' is non-nil.
---
 lisp/emacs-lisp/package.el | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 9476354ebe0..24ba67cef2f 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -3731,7 +3731,9 @@ package-menu--find-upgrades
       ;; ENTRY is (PKG-DESC [NAME VERSION STATUS DOC])
       (let ((pkg-desc (car entry))
             (status (aref (cadr entry) 2)))
-        (cond ((member status '("installed" "dependency" "unsigned" "external"))
+        (cond ((member status (append
+                               '("installed" "dependency" "unsigned" "external" "built-in")
+                               (and package-install-upgrade-built-in '("built-in"))))
                (push pkg-desc installed))
               ((member status '("available" "new"))
                (setq available (package--append-to-alist pkg-desc available))))))
@@ -3742,6 +3744,8 @@ package-menu--find-upgrades
         (and avail-pkg
              (version-list-< (package-desc-priority-version pkg-desc)
                              (package-desc-priority-version avail-pkg))
+             (xor (not package-install-upgrade-built-in)
+                  (package--active-built-in-p pkg-desc))
              (push (cons name avail-pkg) upgrades))))
     upgrades))
 
-- 
2.39.2

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 10 May 2023 11:03:58 +0000
> 
> Philip Kaludercic <philipk@posteo.net> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >>> From: Philip Kaludercic <philipk@posteo.net>
> >>> Cc: dmitry@gutov.dev,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> >>> Date: Mon, 08 May 2023 13:34:26 +0000
> >>> 
> >>> >> At least nadvice, cl-lib and cl-generic seem to be the odd ones (the 
> >>> >> built-in versions are higher, and the ELPA packages are supposed to be 
> >>> >> used as shims or backward compatibility wrappers). That looks like a bug.
> >>> 
> >>> I think you are right, I can extend my previous patch by a version check.
> >>
> >> Good idea, IMO.
> >
> > OK, then this is my proposal:
> 
> I noticed a bug, here is a revised version:

Thanks, LGTM.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Ruijie Yu via Emacs development discussions.]

Philip Kaludercic <philipk@posteo.net> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> OK, then this is my proposal:
>
> I noticed a bug, here is a revised version:
>
> -        (cond ((member status '("installed" "dependency" "unsigned" "external"))
> +        (cond ((member status (append
> +                               '("installed" "dependency" "unsigned" "external" "built-in")
> +                               (and package-install-upgrade-built-in '("built-in"))))

I'm having a hard time understanding the significance of this portion.
Why using (and ... '("built-in")) ?  AFACT, it adds matching status with
"built-in", but unconditionally, because of the added element in the
literal list?

And since the result of `member' is unused (except for checking its
boolean value), whether it returns one or two copies of "built-in" would
make no difference either.

-- 
Best,


RY

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Ruijie Yu <ruijie@netyu.xyz> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> OK, then this is my proposal:
>>
>> I noticed a bug, here is a revised version:
>>
>> - (cond ((member status '("installed" "dependency" "unsigned"
>> "external"))
>> +        (cond ((member status (append
>> + '("installed" "dependency" "unsigned" "external" "built-in")
>> + (and package-install-upgrade-built-in '("built-in"))))
>
> I'm having a hard time understanding the significance of this portion.
> Why using (and ... '("built-in")) ?  AFACT, it adds matching status with
> "built-in", but unconditionally, because of the added element in the
> literal list?

You are right, the `and' was not necessary, I've removed it locally.
Will push the changes soon.

> And since the result of `member' is unused (except for checking its
> boolean value), whether it returns one or two copies of "built-in" would
> make no difference either.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 10/05/2023 14:03, Philip Kaludercic wrote:
> I noticed a bug, here is a revised version:

Should package-upgrade-all have a similar adjustment?

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 10/05/2023 14:03, Philip Kaludercic wrote:
>> I noticed a bug, here is a revised version:
>
> Should package-upgrade-all have a similar adjustment?

Why not?  Didn't you have a patch that extended
`package--upgradeable-packages' by an optional argument, we could use
that here an just pass `package-install-upgrade-built-in'.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 11/05/2023 10:26, Philip Kaludercic wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> On 10/05/2023 14:03, Philip Kaludercic wrote:
>>> I noticed a bug, here is a revised version:
>> Should package-upgrade-all have a similar adjustment?
> Why not?  Didn't you have a patch that extended
> `package--upgradeable-packages' by an optional argument, we could use
> that here an just pass `package-install-upgrade-built-in'.

I'd be happy to see commit 53cc61d60db backported from master.

Not crazy about the idea of adding a prefix argument to package-upgrade, 
though.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Thu, 11 May 2023 12:43:36 +0300
> Cc: Eli Zaretskii <eliz@gnu.org>, monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 11/05/2023 10:26, Philip Kaludercic wrote:
> > Dmitry Gutov<dmitry@gutov.dev>  writes:
> > 
> >> On 10/05/2023 14:03, Philip Kaludercic wrote:
> >>> I noticed a bug, here is a revised version:
> >> Should package-upgrade-all have a similar adjustment?
> > Why not?  Didn't you have a patch that extended
> > `package--upgradeable-packages' by an optional argument, we could use
> > that here an just pass `package-install-upgrade-built-in'.
> 
> I'd be happy to see commit 53cc61d60db backported from master.

Sorry, no, not in Emacs 29.1.

> Not crazy about the idea of adding a prefix argument to package-upgrade, 
> though.

Yes, I know.  I'm not crazy either, but this issue should have been
raised many moons earlier that it has been, and then we'd have a
potentially better solution.  As things are, we have no better choice
but go forward a half-step.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Philip Kaludercic]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 11/05/2023 10:26, Philip Kaludercic wrote:
>> Dmitry Gutov<dmitry@gutov.dev>  writes:
>> 
>>> On 10/05/2023 14:03, Philip Kaludercic wrote:
>>>> I noticed a bug, here is a revised version:
>>> Should package-upgrade-all have a similar adjustment?
>> Why not?  Didn't you have a patch that extended
>> `package--upgradeable-packages' by an optional argument, we could use
>> that here an just pass `package-install-upgrade-built-in'.
>
> I'd be happy to see commit 53cc61d60db backported from master.

In that case I guess it is better to leave it alone.

> Not crazy about the idea of adding a prefix argument to
> package-upgrade, though.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 07/05/2023 22:32, Eli Zaretskii wrote:
> Packages to install: 23 (xref-1.6.3 verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4 svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8 org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6 jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5 eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).  Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).  Proceed? (y or n)

At least nadvice, cl-lib and cl-generic seem to be the odd ones (the 
built-in versions are higher, and the ELPA packages are supposed to be 
used as shims or backward compatibility wrappers). That looks like a bug.

Regarding potential downsides in general:

- We simply increase the odds of breakage when a built-in package is 
upgraded because it will reach more people, across different Emacs 
versions. There is good and bad in that (features will reach them faster 
too), but it's something to consider.

- I very rarely use Tramp or Org. I don't use ERC or bind-key. Or 
so-long, python, ntlm, use-package, verilog-mode. Unlike other installed 
packages (which I have hand-picked), these packages are here just by the 
virtue of being included in Emacs, but flipping the pref will also cause 
them to be upgraded (downloaded, take time unarchiving, take up space) 
every time there is a new version out. It's nothing dangerous 
(probably), but seems unfortunate anyway.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sun, 7 May 2023 23:36:24 +0300
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 07/05/2023 22:32, Eli Zaretskii wrote:
> > Packages to install: 23 (xref-1.6.3 verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4 svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8 org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6 jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5 eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).  Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).  Proceed? (y or n)
> 
> At least nadvice, cl-lib and cl-generic seem to be the odd ones (the 
> built-in versions are higher, and the ELPA packages are supposed to be 
> used as shims or backward compatibility wrappers). That looks like a bug.

Yes, I think it's an unrelated bug.  We had already reports about
strange status values, and there's a new bug#63064 today about similar
problems.  We should try to fix this for Emacs 29, I think.

> Regarding potential downsides in general:
> 
> - We simply increase the odds of breakage when a built-in package is 
> upgraded because it will reach more people, across different Emacs 
> versions. There is good and bad in that (features will reach them faster 
> too), but it's something to consider.
> 
> - I very rarely use Tramp or Org. I don't use ERC or bind-key. Or 
> so-long, python, ntlm, use-package, verilog-mode. Unlike other installed 
> packages (which I have hand-picked), these packages are here just by the 
> virtue of being included in Emacs, but flipping the pref will also cause 
> them to be upgraded (downloaded, take time unarchiving, take up space) 
> every time there is a new version out. It's nothing dangerous 
> (probably), but seems unfortunate anyway.

AFAIU, the "U" command is not for everyone.  There are alternatives
which allow selective upgrades, and users who don't want "surprises",
or want the upgrade to take as little time and disk space as possible,
should use those alternatives instead of "U".  I'll try to make that
clear in the documentation.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 08/05/2023 14:24, Eli Zaretskii wrote:
>> Date: Sun, 7 May 2023 23:36:24 +0300
>> Cc:monnier@iro.umontreal.ca,emacs-devel@gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> On 07/05/2023 22:32, Eli Zaretskii wrote:
>>> Packages to install: 23 (xref-1.6.3 verilog-mode-2022.12.18.181110314 use-package-2.4.5 tramp-2.6.0.4 svg-1.1 soap-client-3.2.1 so-long-1.1.2 python-0.28 project-0.9.8 org-9.6.5 ntlm-2.1.0 nadvice-0.4 map-3.3.1 let-alist-1.0.6 jsonrpc-1.0.17 flymake-1.3.4 external-completion-0.1 erc-5.5 eldoc-1.14.0 eglot-1.15 cl-lib-0.7.1 cl-generic-0.3 bind-key-2.4.1).  Packages to upgrade: 2 (editorconfig-0.9.1 inspector-0.29).  Proceed? (y or n)
>> At least nadvice, cl-lib and cl-generic seem to be the odd ones (the
>> built-in versions are higher, and the ELPA packages are supposed to be
>> used as shims or backward compatibility wrappers). That looks like a bug.
> Yes, I think it's an unrelated bug.  We had already reports about
> strange status values, and there's a new bug#63064 today about similar
> problems.  We should try to fix this for Emacs 29, I think.
> 

We might have somewhat similar reports, but this problem must be 
specific to the proposed patch.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Mon, 08 May 2023 14:24:25 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > Regarding potential downsides in general:
> > 
> > - We simply increase the odds of breakage when a built-in package is 
> > upgraded because it will reach more people, across different Emacs 
> > versions. There is good and bad in that (features will reach them faster 
> > too), but it's something to consider.
> > 
> > - I very rarely use Tramp or Org. I don't use ERC or bind-key. Or 
> > so-long, python, ntlm, use-package, verilog-mode. Unlike other installed 
> > packages (which I have hand-picked), these packages are here just by the 
> > virtue of being included in Emacs, but flipping the pref will also cause 
> > them to be upgraded (downloaded, take time unarchiving, take up space) 
> > every time there is a new version out. It's nothing dangerous 
> > (probably), but seems unfortunate anyway.
> 
> AFAIU, the "U" command is not for everyone.  There are alternatives
> which allow selective upgrades, and users who don't want "surprises",
> or want the upgrade to take as little time and disk space as possible,
> should use those alternatives instead of "U".  I'll try to make that
> clear in the documentation.

Now done, please review and comment.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Dmitry Gutov]

On 07/05/2023 08:51, Eli Zaretskii wrote:
>> Date: Sat, 6 May 2023 23:31:31 +0300
>> Cc: philipk@posteo.net, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 06/05/2023 22:38, Eli Zaretskii wrote:
>>>> package-menu-mark-upgrades ('U') is not affected by
>>>> package-install-upgrade-built-in. It won't.
>>>
>>> Shouldn't it?
>>
>> Maybe, maybe not.
> 
> I think it better did, because using "U" would upgrade Eglot and
> use-package in Emacs 28 and before.  So we should give users who want
> that the capability of keeping that workflow in Emacs 29, if only as
> opt-in behavior.
> 
> Also, "/ u" should ideally show built-in packages as well, when
> package-install-upgrade-built-in is non-nil.

So we think upgrading built-in packages could be dangerous, but if the 
user wants to upgrade Eglot, then upgrading all built-ins is suddenly fine?

> Philip, can these two changes be implemented safely for Emacs 29?
> 
>> A user that customized that option to have (package-install 'eglot)
>> ensure that a version from ELPA is installed might not want or expect
>> for it to affect package-menu-mark-upgrades and/or package-upgrade-all.
> 
> That is true, but denying them the possibility of upgrading would be
> worse, I think.  And since this is opt-in behavior, the user is less
> likely to be tripped by that without realizing it.
> 
>> Or anticipate the full consequences anyway.
> 
> Documentation should solve this aspect.

If the only way to get a basic scenario to work is to toggle a pref, we 
won't be able to go back and say "you shouldn't have toggled it".

If we don't actually consider upgrading of all built-ins to be a 
problem, and are just hesitant to enable it in Emacs 29, then I guess it 
is fine.

Otherwise, we've had a pretty good idea posted previously: patch where a 
variable holds the list of builtins that should be upgraded automatically.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[Eli Zaretskii]

> Date: Sun, 7 May 2023 12:50:31 +0300
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > Also, "/ u" should ideally show built-in packages as well, when
> > package-install-upgrade-built-in is non-nil.
> 
> So we think upgrading built-in packages could be dangerous, but if the 
> user wants to upgrade Eglot, then upgrading all built-ins is suddenly fine?

It is not dangerous, it just isn't what Emacs 28 did.

> If we don't actually consider upgrading of all built-ins to be a 
> problem, and are just hesitant to enable it in Emacs 29, then I guess it 
> is fine.

There are arguments both ways, and situations where Emacs 28 behaved
this way or the other.  So optional behavior seems to strike the right
balance between the opposites.

> Otherwise, we've had a pretty good idea posted previously: patch where a 
> variable holds the list of builtins that should be upgraded automatically.

I don't think this is TRT.  There's nothing special about those two
packages in this regard.

Re: emacs-29 9b775ddc057 1/2: ; * etc/EGLOT-NEWS: Fix wording of last change.

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]

On Sat, May 6, 2023, 16:26 Dmitry Gutov <dmitry@gutov.dev> wrote:

>
>
> He is mostly talking about users who e.g. wiped their config directory
> and ~/.emacs.d/elpa and are starting anew. Or just deleted
> ~/.emacs.d/elpa/eglot-xxxxxx. In that situation,


Or, more simply, just launching Emacs on a new host you've never logged
into.

João

[-- Attachment #2: Type: text/html, Size: 742 bytes --]