Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

This discussion no longer belongs to the bug tracker, so I'm moving it
to emacs-devel and changing its Subject.  Please reply here, not
there.

For those who see this for the first time, and want more background,
here are some relevant messages which discussed this aspect as part of
the otherwise huge thread, which is only loosely related to this
particular issue:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#383
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#398
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#401
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#410
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#413

> Date: Tue, 18 Apr 2023 04:25:01 +0300
> Cc: joaotavora@gmail.com, rpluim@gmail.com, philipk@posteo.net,
>  62720@debbugs.gnu.org, larsi@gnus.org, monnier@iro.umontreal.ca
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 17/04/2023 05:24, Eli Zaretskii wrote:
> >> Date: Sun, 16 Apr 2023 23:46:37 +0300
> >> Cc: rpluim@gmail.com, philipk@posteo.net, 62720@debbugs.gnu.org,
> >>   larsi@gnus.org, monnier@iro.umontreal.ca
> >> From: Dmitry Gutov <dmitry@gutov.dev>
> >>
> >> On 14/04/2023 22:28, Eli Zaretskii wrote:
> >>> If, OTOH,
> >>> you think that it's imperative to allow_all_  users of Eglot with
> >>> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
> >>> become available), then we should release Emacs 29 with 1.14.
> >>
> >> Was this question about stability only?
> > 
> > It was about the criteria for which versions of core packages to ship
> > with a release.
> 
> I don't think we can get a single set of criteria across core packages.

We don't need to have just one set.  Packages are different in both
their complexity, their dependence on other packages, and dependence
of other packages on them.  So one set is unlikely to fit the bill,
indeed.

But that doesn't mean we shouldn't have criteria at all.  We should
strive to have a small number of them, and we should know which set is
applicable to which class of packages.

This will be one of the serious issues if we ever move to having some
packages only in elpa.git, and will then bundle them when preparing an
Emacs release tarball.  It will be imperative to know at that time
which version/branch of each such package to take as part of preparing
a release.  We must have a solution by then, so this is as good time
as any to start discussing the issue.

> E.g. Org is developed externally, has its own community of significant 
> size, and does split off release branches (with additional testing, I',m 
> guessing).
> 
> Eglot, OTOH, is developed only here, with no additional release workflow 
> other than what MELPA/GNU ELPA historically provided: collect up some 
> features/fixes, bump the Version header, and push a new release out to 
> the users. The lack of extended testing period is made up for with the 
> capability to push out a new fixed version overnight. That's why the 
> difficulty in upgrading to the latest version (for Emacs 29 users) is 
> going to hurt.

If some core package is not tested enough before it gets a new
version, then why are we okay with telling users of a stable Emacs to
update the package willy-nilly as soon as another version is on ELPA?
Shouldn't we at least warn them, or, better, somehow indicate that
this version is not yet considered stable?

IOW, shouldn't packages have some "stability gradation" that is
visible when users look at the list of packages via package.el?
Shouldn't we allow users to tell package.el which stability they want
to download, so that unstable packages don't inadvertently get
installed and mess up their production environment?

> BTW, if you recall the threads before Eglot was added, I was against 
> that, and one of the things I cited is an LSP client has inherently high 
> development velocity. Maybe the LSP community will settle/mature/stop 
> adding features one day, but it's not there yet.

I don't see what development pace has to do with this issue.  If a
package is being developed at a high pace, it might mean that the
stable version will lag more.  But what does this change in principle?

> >> Because since we've decided in favor of stability of package.el, and
> >> against eglot's easy upgradability, I would suggest to backport Eglot
> >> 1.14 to emacs-29.
> > 
> > I won't object.  In fact, I asked up front why not.
> 
> Note that that suggestion comes with a fix to eldoc which you so far 
> have rejected for emacs-29.

You mean, the change in ElDoc that avoids the "jumping mode line"
issue?  If so, it is unfortunate for Eglot to depend on that, because
it means users of Emacs 29 will be unable to upgrade to Eglot 1.15
without by side effect installing a newer and potentially less stable
ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)

So this again goes back to the main issue: how should the stability
considerations affect development of core packages and their
"stability gradation"?  Developers of core packages should keep these
aspects in mind at all times, and, for example, avoid dependencies on
other packages that aren't absolutely necessary.  Otherwise, we will
have to advise users who value stability not to upgrade their core
packages for fear of destabilizing their environments, and the whole
advantage of having core packages on ELPA will be effectively lost, at
least for those users who want stability.

IOW, having a core package on ELPA comes with some serious strings
attached, and just ignoring them, or leaving that up to the users
(without duly informing them about their new responsibilities) is not
a good solution, IMO.

We should seriously ponder these aspects, and the sooner the better.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 1:56 PM Eli Zaretskii <eliz@gnu.org> wrote:

> without by side effect installing a newer and potentially less stable
> ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)

It's not a "must".  Eglot can work without it.  The problem happens in
ElDoc and doesn't affect its interface.

But Eglot relies on ElDoc as a whole.  In general you can't expect
to have new features in Eglot without some advancing of the
infrastructure that Eglot depends on.  As highlighted in the manual
and elsewhere, Eglot is relatively thin glue between LSP abstractions
and Emacs abstractions.  It relies on this client-agnostic
infrastructure of ElDoc, Flymake, Xref, Project, Jsonrpc (and
soon others) and drives advances to that infrastructure too.
For example, I completely reworked Flymake in 2017 (in
backward-compatible fashion of course) precisely to support
Eglot's use case.  It supports many others of course, and there is
a healthy collection of non-Eglot Flymake clients.  Exactly the
same happened for ElDoc, and similar processes happen with Xref,
for example. These processes have been happening for a while now,
before Eglot was in core, and the model has proven its value IMO.

Anyway, this was just an introduction to what is Eglot's main
characteristic (and why it's sometimes called "minimal" and
well integrated with Emacs's existing features).

So there _isn't_ a way to partially upgrade a package and not its
dependencies.  And there _shouldn't_ ever be this way, for our
collective sanity.  And there's no good reason this should ever
exist.

> We should seriously ponder these aspects, and the sooner the better.

I don't actually think there is any actual problem present if
we all agree (as you seem to) with Dmitry's preposition that
"there shouldn't be a single set of criteria governing core
packages".  Then we can teach Emacs's upgrade mechanisms to
deal with each set differently, carefully examining the
requirements for each set.

I'd like you, if possible, to also respond (here, if you prefer) to the
points I raised in my own reply to Dmitry's message you're replying
to.  This is the message;:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#529

Thank you,
João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 18 Apr 2023 15:02:01 +0100
> Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> 
> On Tue, Apr 18, 2023 at 1:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > without by side effect installing a newer and potentially less stable
> > ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)
> 
> It's not a "must".  Eglot can work without it.  The problem happens in
> ElDoc and doesn't affect its interface.

Then what Dmitry said about Eglot 1.15 being dependent on that change
in ElDoc is not relevant to the issue at hand, which is whether Eglot
1.15 could be bundled with Emacs 29.1.

> But Eglot relies on ElDoc as a whole.  In general you can't expect
> to have new features in Eglot without some advancing of the
> infrastructure that Eglot depends on.

Understood.  My point is that if you want Eglot users to be able to
upgrade to a newer versions, you need to have compatibility layers, to
avoid the need to upgrade too many other packages, which might hamper
stability.  Otherwise, we cannot in good faith recommend that users of
stable Emacs update their core packages without a second thought.

> So there _isn't_ a way to partially upgrade a package and not its
> dependencies.

Updating a package P1 should require update of as few packages P2...Pn
as possible.  Ideally, none at all.  Users should be able to decide
whether they want or don't want to update any single package without
also needing to decide whether they are okay with updating half a
dozen of others.  This should be our goal, because otherwise updating
a package will be unsafe if you use any Emacs except master (and thus
don't care much about stability anyway).

> > We should seriously ponder these aspects, and the sooner the better.
> 
> I don't actually think there is any actual problem present if
> we all agree (as you seem to) with Dmitry's preposition that
> "there shouldn't be a single set of criteria governing core
> packages".  Then we can teach Emacs's upgrade mechanisms to
> deal with each set differently, carefully examining the
> requirements for each set.

Sure, but we need to have these sets, actually.  Right now, we don't,
not for every core package out there anyway.

> I'd like you, if possible, to also respond (here, if you prefer) to the
> points I raised in my own reply to Dmitry's message you're replying
> to.  This is the message;:
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#529

I didn't respond because I had nothing to say to that which I didn't
already say in response to Dmitry's message.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 3:47 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Tue, 18 Apr 2023 15:02:01 +0100
> > Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> >
> > On Tue, Apr 18, 2023 at 1:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > without by side effect installing a newer and potentially less stable
> > > ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)
> >
> > It's not a "must".  Eglot can work without it.  The problem happens in
> > ElDoc and doesn't affect its interface.
>
> Then what Dmitry said about Eglot 1.15 being dependent on that change
> in ElDoc is not relevant to the issue at hand, which is whether Eglot
> 1.15 could be bundled with Emacs 29.1.

It is quite relevant.

Eglot 1.15 depends on many other things in ElDoc.  That particular
bugfix might not -- or might indeed -- included, depending on what
other non-bugfix things  Eglot will require of ElDoc at the time.
ElDoc is now 1.14 but it could be 1.15 at the time motivated by
Eglot 1.15/16/17.

And even in ElDoc 1.14 there are already things (the :echo display
option) that are not in Emacs 29. And Eglot 1.14 directly depends
on those things.  It relies on them to do a good job.

I would think that if you oppose a bugfix backport you would also
oppose a _feature_ backport, which usually is (and is indeed in
this case) a much more complicated, "scary", bug-prone development.

> > But Eglot relies on ElDoc as a whole.  In general you can't expect
> > to have new features in Eglot without some advancing of the
> > infrastructure that Eglot depends on.
>
> Understood.  My point is that if you want Eglot users to be able to
> upgrade to a newer versions, you need to have compatibility layers, to
> avoid the need to upgrade too many other packages, which might hamper
> stability.  Otherwise, we cannot in good faith recommend that users of
> stable Emacs update their core packages without a second thought.

Yes, and this is why each released version of Eglot specifies exactly
the _released_ versions of its dependencies that it depends on. The
dependency language isn't very elaborated (there is no way to say
Eglot 1.1234  depends on ElDoc between 1.23 and 1.56), but it's been
enough.  It's not much different from what is found in numerous
other package systems.

Another matter is what package.el does with this info and the
implied graph. Transitive dependencies are known not to matter.
And if a package requires ElDoc 1.14 but 1.99 is already available
package.el will just go ahead and install the latest.  Always has,
I'm afraid (anyone can correct me on this if I'm mistaken, which
I'd love to be).

I think the experience of the straight.el and elpaca.el package
manager authors could be useful here to us.  If someone knows them
by heart, please do tell me.  Who knows, maybe what we want
is to bring one of these into Emacs and kill package.el. straight.el
is what a lot of people seem to be using these days anyway (and the cooler
kids elpaca.el).

> > So there _isn't_ a way to partially upgrade a package and not its
> > dependencies.
>
> Updating a package P1 should require update of as few packages P2...Pn
> as possible.  Ideally, none at all.

And very often that does happen, I suppose.  Not every Eglot release
_requires_ installation of new versions of its dependencies.  But some
do.

> Users should be able to decide
> whether they want or don't want to update any single package without
> also needing to decide whether they are okay with updating half a
> dozen of others.  This should be our goal, because otherwise updating
> a package will be unsafe if you use any Emacs except master (and thus
> don't care much about stability anyway).

I don't know how you can meet that goal in general.  We should
indeed work to minimize dependencies and do things that don't affect
interfaces and don't require changes other's interfaces.  As much as
possible, I agree.  But in general programs rely on other programs.
Dependencies exist.  Like in many other package managers, users
should be presented with the consequences of their wishes, when
that is feasible and when we can do so without breaking their
configs.

That's my idea of stability anyways.

> > "there shouldn't be a single set of criteria governing core
> > packages".  Then we can teach Emacs's upgrade mechanisms to
> > deal with each set differently, carefully examining the
> > requirements for each set.
>
> Sure, but we need to have these sets, actually.  Right now, we don't,
> not for every core package out there anyway.

[ Just a note that :core packages are not "out there", they're
"in here" -- by definition.  That's their main selling point
and precisely what we should take advantage of :-) ]

I propose two main sets of :core packages to start with.

Set 1 - :core packages that have always been core, i.e. they started
their life in the code

Set 2 - :core packages that started their life somewhere else
(GNU ELPA, Github, etc), were installable through ELPA interfaces
and now are :core (which means Git-versioned in Emacs.git but
still installable via ELPA).  Two known such packages are Eglot
and Use-Package.  Both depend on _other_ :core packages.  Eglot
on many of these, Use-Package only on "bind-key", which is
also new, but didn't seem to be installable independently
before Use-Package appeared.

This isn't the end of the analysis, of course.  I'm just
providing these two sets to see if it rings a bell with participants,
because from the opinions I collected in that very long bug, they
seem to make and map neatly onto the requirements that each
party put forth:

- That members of set 1 shouldn't be upgradable "willy nilly" to
maintain exact backward-compatibility.

- That members of set 2 should be upgraded in much easier fashion
because that's what guarantees that people's configs already
doing so won't break.

> > I'd like you, if possible, to also respond (here, if you prefer) to the
> > points I raised in my own reply to Dmitry's message you're replying
> > to.  This is the message;:
> >
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#529
>
> I didn't respond because I had nothing to say to that which I didn't
> already say in response to Dmitry's message.

I proposed a simple bugfix to bug#62720, based precisely on
the above idea of separation of sets.  A dead-simple 7-line
fix.

That patch was +1'ed by Philip and Dmitry and hasn't been
addressed by you.  Philip, by the way, tried until the end to
give you a patch that also didn't have the non-interactive
package-install/use-package lockout, but you insisted.  Why? Why
keep it for those members of Set 1?  What is there to be gained?
Do you acknowledge what there is to be lost?

But moving on from that minor tragedy, it's nevertheless easy
to note that bug#62720 conflates the two sets.  We should work
to improve that, hopefully in time to avert damage.  The code
required to express these sets in Elisp is, of course, quite trivial.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 18 Apr 2023 16:45:33 +0100
> Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> 
> > > It's not a "must".  Eglot can work without it.  The problem happens in
> > > ElDoc and doesn't affect its interface.
> >
> > Then what Dmitry said about Eglot 1.15 being dependent on that change
> > in ElDoc is not relevant to the issue at hand, which is whether Eglot
> > 1.15 could be bundled with Emacs 29.1.
> 
> It is quite relevant.
> 
> Eglot 1.15 depends on many other things in ElDoc.

It isn't like ElDoc is not available in Emacs 29.

> That particular bugfix might not -- or might indeed -- included,
> depending on what other non-bugfix things Eglot will require of
> ElDoc at the time.  ElDoc is now 1.14 but it could be 1.15 at the
> time motivated by Eglot 1.15/16/17.

My point is that Eglot and any other core package will do its users a
favor if it deliberately and consistently makes a point to depend on
as few _new_ features of other core packages as possible.  If you
disagree with that, then we will have to agree to disagree, because
for me this is a very basic issue with core packages and the future of
Emacs.

> And even in ElDoc 1.14 there are already things (the :echo display
> option) that are not in Emacs 29. And Eglot 1.14 directly depends
> on those things.  It relies on them to do a good job.

Which IMO is not a good thing, not unless Eglot 1.14 has fallbacks for
when these features are not available.

But if that is impossible or impractical for some reason in this
particular case, then it simply means that users of Emacs 29 will be
unable to upgrade Eglot without also risking less stability due to
upgrading the dependencies.  Again, if you don't think this
"dependencies hell" is a Bad Thing in general, we will have to agree
to disagree.

> I would think that if you oppose a bugfix backport you would also
> oppose a _feature_ backport, which usually is (and is indeed in
> this case) a much more complicated, "scary", bug-prone development.

I consider each case of a request to backport on its own merit.  So
conclusions such as the above, which don't consider the specifics of
each change, but instead go by some abstract principles, are not what
I usually make.

> > Understood.  My point is that if you want Eglot users to be able to
> > upgrade to a newer versions, you need to have compatibility layers, to
> > avoid the need to upgrade too many other packages, which might hamper
> > stability.  Otherwise, we cannot in good faith recommend that users of
> > stable Emacs update their core packages without a second thought.
> 
> Yes, and this is why each released version of Eglot specifies exactly
> the _released_ versions of its dependencies that it depends on.

You are missing my point.  I'm not talking about dependencies with
other packages.  My point is not about other core packages, it's about
Emacs itself and its stability as a whole.  Users of a stable release
of Emacs can, and usually do, expect their entire Emacs configurations
to be stable, and telling them to upgrade packages without any clear
indication of their stability goes against that.  Yes, requiring
specific versions of the other N packages will minimize breakage due
to incompatibilities between those packages, but what about unintended
consequences, regressions, etc.?  Suppose Eglot 1.14 brings with it
some package whose updated version has a bug -- why should a user who
wants to have a stable Emacs environment to risk this?

> > Updating a package P1 should require update of as few packages P2...Pn
> > as possible.  Ideally, none at all.
> 
> And very often that does happen, I suppose.  Not every Eglot release
> _requires_ installation of new versions of its dependencies.  But some
> do.

I'm asking whether you make an extra effort to avoid such requirements
whenever possible, even if that would call for extra work on your
part?  I hope you and other package developers do, because otherwise
proliferation of core packages would be a very bad deal for the future
of Emacs, which traditionally is perceived as an extremely stable
platform.

> > Users should be able to decide
> > whether they want or don't want to update any single package without
> > also needing to decide whether they are okay with updating half a
> > dozen of others.  This should be our goal, because otherwise updating
> > a package will be unsafe if you use any Emacs except master (and thus
> > don't care much about stability anyway).
> 
> I don't know how you can meet that goal in general.

If you at least think we should try, then we have something to work
with.  Even if the ideal of having to upgrade no package is
unattainable, bringing the number close to zero is a worthy goal.

> We should indeed work to minimize dependencies and do things that
> don't affect interfaces and don't require changes other's
> interfaces.  As much as possible, I agree.  But in general programs
> rely on other programs.  Dependencies exist.  Like in many other
> package managers, users should be presented with the consequences of
> their wishes, when that is feasible and when we can do so without
> breaking their configs.

I'm saying that we should make an extra effort to avoid that, not
accept dependencies without a "fight".

> I propose two main sets of :core packages to start with.
> 
> Set 1 - :core packages that have always been core, i.e. they started
> their life in the code
> 
> Set 2 - :core packages that started their life somewhere else
> (GNU ELPA, Github, etc), were installable through ELPA interfaces
> and now are :core (which means Git-versioned in Emacs.git but
> still installable via ELPA).  Two known such packages are Eglot
> and Use-Package.  Both depend on _other_ :core packages.  Eglot
> on many of these, Use-Package only on "bind-key", which is
> also new, but didn't seem to be installable independently
> before Use-Package appeared.
> 
> This isn't the end of the analysis, of course.

I'm not sure history is the aspect that distinguishes them.  An
important aspect is how "low' is the functionality provided by the
package.  Some packages provide infrastructure -- those affect many
other places by definition.  Others are applications on which nothing
else depends.  Perhaps these are the important factors, I don't know.

> - That members of set 1 shouldn't be upgradable "willy nilly" to
> maintain exact backward-compatibility.
> 
> - That members of set 2 should be upgraded in much easier fashion
> because that's what guarantees that people's configs already
> doing so won't break.

That is completely irrelevant for this discussion, IMO.  It is up to
the users whether to upgrade and how aggressively.  We don't know
enough about the configurations, the environment, and the usage
patterns of the users, we can only give them information -- in this
case regarding stability of each available version -- and let them
make the conclusions as they see fit.  We will never be able to second
guess them well enough.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 5:19 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Tue, 18 Apr 2023 16:45:33 +0100
> > Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> >
> > > > It's not a "must".  Eglot can work without it.  The problem happens in
> > > > ElDoc and doesn't affect its interface.
> > >
> > > Then what Dmitry said about Eglot 1.15 being dependent on that change
> > > in ElDoc is not relevant to the issue at hand, which is whether Eglot
> > > 1.15 could be bundled with Emacs 29.1.
> >
> > It is quite relevant.
> >
> > Eglot 1.15 depends on many other things in ElDoc.
>
> It isn't like ElDoc is not available in Emacs 29.

It is.  But in Emacs 29, it's not the bleeding edge anymore.  It's
1.13.0 which doesn't have an important feature of 1.14.0

  commit e19994fe8c000b0ed2dbc667cdec26cf54356907
  Author: João Távora Date: Thu Mar 23 09:02:18 2023 +0000

  ElDoc: rework rendering of echo area (bug#62029)

And it doesn't have the fix for bug#62816, which is in master
and will appear in ElDoc 1.14.1.  Unless you want it to, of
course.  In the case of that single backport, ElDoc bundled
with Emacs 29 (and with Emacs 29 _only_) should become known
as 1.13.29 (imperfect versioning scheme, but not terrible).

> > That particular bugfix might not -- or might indeed -- included,
> > depending on what other non-bugfix things Eglot will require of
> > ElDoc at the time.  ElDoc is now 1.14 but it could be 1.15 at the
> > time motivated by Eglot 1.15/16/17.
>
> My point is that Eglot and any other core package will do its users a
> favor if it deliberately and consistently makes a point to depend on
> as few _new_ features of other core packages as possible.  If you
> disagree with that, then we will have to agree to disagree, because
> for me this is a very basic issue with core packages and the future of
> Emacs.

Yes, as few _new_ features as possible, but not fewer than that.
Yes, I agree good design is when there is nothing more to take out.
But sometimes you have to add to make things appear.

> > And even in ElDoc 1.14 there are already things (the :echo display
> > option) that are not in Emacs 29. And Eglot 1.14 directly depends
> > on those things.  It relies on them to do a good job.
>
> Which IMO is not a good thing, not unless Eglot 1.14 has fallbacks for
> when these features are not available.

It has.  It doesn't fail.  I try to make everything forward and backward
compatible.  But certainly your at-point documentation experience in Eglot
if you merge if you backport it to Emacs 29 without also backporting ElDoc
will _not_ be the same.  Users will notice this and if we tell them
"why yes, Eglot 1.15 is in Emacs 29!"  they will be triply befuddled,
and rightfully so, because Eglot 1.15 running on Emacs 28 will have much
better behaviour and fewer bugs and run smoother.

> But if that is impossible or impractical for some reason in this
> particular case, then it simply means that users of Emacs 29 will be
> unable to upgrade Eglot without also risking less stability due to
> upgrading the dependencies.  Again, if you don't think this
> "dependencies hell" is a Bad Thing in general, we will have to agree
> to disagree.

No, it doesn't mean that, at all.  First, dependencies exist.  And here
it's not hell at all, not IME.  Secondly, stability is a matter of
expectations.  Eglot users _expect_ :core dependencies to be upgraded
when they request Eglot to be  upgraded.  That's the way it has always
worked for ~5 years.  And non-Eglot users _also_ expect that, because
installing any non-:core ELPA package that depends on :core packages
has _also_ always produced that behaviour for dependencies.

Thirdly, in practice, I have been following this for some time, I've not
seen many (any?) bugs related to these kinds of "furtive" dependency
hell upgrade by package-install.  Maybe Dmitry has?   Or you have?
If anything, I've seen bugs related to upgrades of dependencies
_not_ happening (because of package.el's inability to understand
transitive dependencies -- Basil spotted that, I think).

> > I would think that if you oppose a bugfix backport you would also
> > oppose a _feature_ backport, which usually is (and is indeed in
> > this case) a much more complicated, "scary", bug-prone development.
>
> I consider each case of a request to backport on its own merit.  So
> conclusions such as the above, which don't consider the specifics of
> each change, but instead go by some abstract principles, are not what
> I usually make.

That makes full sense, btw.

> > > Understood.  My point is that if you want Eglot users to be able to
> > > upgrade to a newer versions, you need to have compatibility layers, to
> > > avoid the need to upgrade too many other packages, which might hamper
> > > stability.  Otherwise, we cannot in good faith recommend that users of
> > > stable Emacs update their core packages without a second thought.
> >
> > Yes, and this is why each released version of Eglot specifies exactly
> > the _released_ versions of its dependencies that it depends on.
>
> You are missing my point.  I'm not talking about dependencies with
> other packages.  My point is not about other core packages, it's about
> Emacs itself and its stability as a whole.  Users of a stable release
> of Emacs can, and usually do, expect their entire Emacs configurations
> to be stable, and telling them to upgrade packages without any clear
> indication of their stability goes against that.  Yes, requiring
> specific versions of the other N packages will minimize breakage due
> to incompatibilities between those packages, but what about unintended
> consequences, regressions, etc.?  Suppose Eglot 1.14 brings with it
> some package whose updated version has a bug -- why should a user who
> wants to have a stable Emacs environment to risk this?

She shouldn't.  She should never M-x package-install RET eglot or
have it in her configuration.  This is not a new problem.  Things that
download code bear risks, at least when compared to the stock Emacs 29
that went through a lenghty pretest period.  This is _precisely_ why
I think backporting Eglot 1.15/16/whatever to Emacs 29 is not a good
idea.  This is why that user should use Eglot bundled with Emacs 29.
It's a good, stable, well-tested Eglot that you can do lots of cool
stuff in.

> > > Updating a package P1 should require update of as few packages P2...Pn
> > > as possible.  Ideally, none at all.
> >
> > And very often that does happen, I suppose.  Not every Eglot release
> > _requires_ installation of new versions of its dependencies.  But some
> > do.
>
> I'm asking whether you make an extra effort to avoid such requirements
> whenever possible, even if that would call for extra work on your
> part?  I hope you and other package developers do, because otherwise
> proliferation of core packages would be a very bad deal for the future
> of Emacs, which traditionally is perceived as an extremely stable
> platform.

Yep, I do those efforts, but I wouldn't go so far as to call them "extra".
I think things should go where they belong.  That's, again, the main
design philosophy of Eglot.  Don't invent (too much) UI in Eglot.  Put
UI and other functionality in client-agnostic libraries and use Eglot
to link up LSP interfaces to those libraries.  Consequently, that
requires new libraries and updates to the libraries.

For example, for the "breadcrumb headerline" feature of bug#58431 I'll
be proposing the introduction of a new lisp/progmodes/breadcrumb.el
:core library that depends solely on Emacs's imenu.el.  It works
with Eglot and without Eglot (and it's turning out quite nice, should
work with the imenu info produced by the tree-sitter modes, too).

So is that "proliferation of :core libraries"?  Yes maybe, but it's
also what we want.  We don't want a from-scratch LSP-specific breadcrumb
implementation in Eglot, I think.  So a library is the way to go.

If you don't want it in core, I can very well put it in GNU ELPA, or
even MELPA.  Then Eglot will "optionally" depend on it, much in the
same way it already depends "optionally" on Markdown.el, Company and
Yasnippet.

> > > Users should be able to decide
> > > whether they want or don't want to update any single package without
> > > also needing to decide whether they are okay with updating half a
> > > dozen of others.  This should be our goal, because otherwise updating
> > > a package will be unsafe if you use any Emacs except master (and thus
> > > don't care much about stability anyway).
> >
> > I don't know how you can meet that goal in general.
>
> If you at least think we should try, then we have something to work
> with.  Even if the ideal of having to upgrade no package is
> unattainable, bringing the number close to zero is a worthy goal.
...
> I'm saying that we should make an extra effort to avoid that, not
> accept dependencies without a "fight".

OK, you have a foot soldier to "fight" against dependency hell :-).
But sometimes I will enter talks with these dependencies who
may not be from hell, rather from some kind of more well behaved
purgatory.

> > This isn't the end of the analysis, of course.
>
> I'm not sure history is the aspect that distinguishes them.  An
> important aspect is how "low' is the functionality provided by the
> package.  Some packages provide infrastructure -- those affect many
> other places by definition.  Others are applications on which nothing
> else depends.  Perhaps these are the important factors, I don't know.

Yes, these are other important factors.  But history is important
as a factor because the _past_ dictates user expectations, which
are by definition based on it.  And stability is about expectations
(as someone wisely wrote here recently, wisely, apropos a security
bug).

> > - That members of set 1 shouldn't be upgradable "willy nilly" to
> > maintain exact backward-compatibility.
> >
> > - That members of set 2 should be upgraded in much easier fashion
> > because that's what guarantees that people's configs already
> > doing so won't break.
>
> That is completely irrelevant for this discussion, IMO.  It is up to
> the users whether to upgrade and how aggressively.  We don't know
> enough about the configurations, the environment, and the usage
> patterns of the users, we can only give them information -- in this
> case regarding stability of each available version -- and let them
> make the conclusions as they see fit.  We will never be able to second
> guess them well enough.

It is not irrelevant.  We can know _exactly_ how certain things worked
in Emacs 26, 27 and 28 and how much of a backward-incompatible change
we're making.  Thus we can act accordingly to minimize that change.
Or even eliminate it for the specific cases where we want to eliminate
it.  Minimizing change is the bread-and-butter of stability, which
seems to be something you're also interested in, but also at the same
time, contradictingly, aren't.  That part I still don't understand.

In this case, the "certain thing" is the behaviour of the very
popular and extremely simple (use-package eglot :ensure t) or the
presence or a (package-install eglot) in a configuration. It does one
thing in Emacs 26/7/8 it will do another different thing in Emacs 29.

How "popular" is this (use-package eglot :ensure t), you ask?  Just check
it out for yourself:
https://github.com/joaotavora/eglot/search?q=%22use-package%22&type=issues

it pops up every other issue.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 18/04/2023 20:49, João Távora wrote:
>> But if that is impossible or impractical for some reason in this
>> particular case, then it simply means that users of Emacs 29 will be
>> unable to upgrade Eglot without also risking less stability due to
>> upgrading the dependencies.  Again, if you don't think this
>> "dependencies hell" is a Bad Thing in general, we will have to agree
>> to disagree.
> No, it doesn't mean that, at all.  First, dependencies exist.  And here
> it's not hell at all, not IME.  Secondly, stability is a matter of
> expectations.  Eglot users_expect_  :core dependencies to be upgraded
> when they request Eglot to be  upgraded.  That's the way it has always
> worked for ~5 years.  And non-Eglot users_also_  expect that, because
> installing any non-:core ELPA package that depends on :core packages
> has_also_  always produced that behaviour for dependencies.

To my recollection, Eglot historically only raised the dependency 
versions when it needed to -- to bring in some required feature. Which 
is quite reasonable from my POV.

Although I see some very recent more bumps for xref and project there, 
those I didn't expect, but they probably follow the same logic.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/18/2023 5:57 AM, Eli Zaretskii wrote:
> So this again goes back to the main issue: how should the stability
> considerations affect development of core packages and their
> "stability gradation"?

It sounds to me like there are 3 or 4 levels (depending on how you count):

* Stable: the version of a package included in the latest Emacs tarball
* Latest: the latest version on GNU ELPA (etc)
* Devel: the latest version on GNU-devel ELPA (etc)

You could possibly add:

* Core(?): the version of a package included in the tarball of the 
user's *current* Emacs installation

I think these 3 or 4 levels should be plenty for 99% of scenarios. From 
this, we can also propose a hard rule: packages at a certain level 
should only depend on other packages at the same level or lower (so a 
"latest" package can't depend on a "devel" package). We could also add 
as soft guidance: a package should do its best to require only stable 
versions of dependencies where feasible.

However, this runs into the problem João saw: if your package (e.g. 
Eglot) would strongly benefit from requiring a newer version of a 
dependency (ElDoc), what should you do? Currently, the only options are 
a) do nothing and let users have a worse experience or b) make the user 
upgrade the dependency too, even if they don't particularly want it. 
Neither solution seems ideal to me.

One alternative would be for packages to be able to *recommend* 
dependencies. Then, Eglot could recommend newer versions of ElDoc, but 
they wouldn't actually be required. We could do this via some extra 
package metadata, or maybe the follow simple solution would be enough: 
when installing a package from a particular archive[1], offer the user 
the option of also installing any of the package's requirements from 
that archive too. So, if I installed Eglot from GNU ELPA, Emacs would 
suggest that I also install ElDoc, etc from GNU ELPA too. (Of course, if 
a package truly *requires* a newer version of a package that's only 
available in ELPA, it wouldn't need to prompt the user.)

[1] You could also think of this as "installing a package of a 
particular stability level".

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Tue, 18 Apr 2023 11:56:58 -0700
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 4/18/2023 5:57 AM, Eli Zaretskii wrote:
> > So this again goes back to the main issue: how should the stability
> > considerations affect development of core packages and their
> > "stability gradation"?
> 
> It sounds to me like there are 3 or 4 levels (depending on how you count):
> 
> * Stable: the version of a package included in the latest Emacs tarball
> * Latest: the latest version on GNU ELPA (etc)
> * Devel: the latest version on GNU-devel ELPA (etc)

I think we need only two.  Stable can move to the next version, since
packages are released more frequently than Emacs.

> However, this runs into the problem João saw: if your package (e.g. 
> Eglot) would strongly benefit from requiring a newer version of a 
> dependency (ElDoc), what should you do? Currently, the only options are 
> a) do nothing and let users have a worse experience or b) make the user 
> upgrade the dependency too, even if they don't particularly want it. 
> Neither solution seems ideal to me.

How is this different from what we have in Emacs?  An exciting new
feature is sometimes deferred to the next major release if the release
branch is close enough to a release.  There's nothing new here, just
the fact that sometimes useful new features could destabilize Emacs,
so one needs to choose which one it wants more.

> One alternative would be for packages to be able to *recommend* 
> dependencies. Then, Eglot could recommend newer versions of ElDoc, but 
> they wouldn't actually be required.

This is probably needed, but it requires non-trivial support from
package.el, to let informed users select the updates that fit their
stability requirements and feature needs.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/18/2023 12:21 PM, Eli Zaretskii wrote:
>> Date: Tue, 18 Apr 2023 11:56:58 -0700
>> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> It sounds to me like there are 3 or 4 levels (depending on how you count):
>>
>> * Stable: the version of a package included in the latest Emacs tarball
>> * Latest: the latest version on GNU ELPA (etc)
>> * Devel: the latest version on GNU-devel ELPA (etc)
> 
> I think we need only two.  Stable can move to the next version, since
> packages are released more frequently than Emacs.

At least from a user POV, I think it makes sense to distinguish among 
all three of these. For example, I might use the version of Eglot in 
Emacs 29, or I might install it from GNU ELPA, or I could even install 
it from GNU-devel ELPA. I might even switch back and forth depending on 
what my needs are (and in fact, that's exactly what I do with Eglot; 
while I usually prefer to stick on the GNU ELPA version, sometimes I 
switch to GNU-devel ELPA if there's a fix for a bug I find very bothersome).

I think this set of three levels also makes it easier - at least for me 
- to reason about what to do with something like ElDoc. If a user 
installs Eglot from GNU ELPA (i.e. the user gets "Eglot latest"), should 
it automatically install ElDoc from GNU ELPA ("ElDoc latest") or should 
it use the ElDoc from the Emacs tarball ("ElDoc stable")?

If there were only two levels - latest and devel - then I think the 
answer to the Eglot/ElDoc problem would simply be: installing Eglot 
latest should pull in ElDoc latest. Since there's no higher "stability 
gradation" than latest, we wouldn't really be able to say that 
ElDoc-from-Emacs is better/stabler than ElDoc-from-ELPA. (Well, we can 
still *say* that, but I think it helps to embed our reasoning for it 
into these stability gradations.)

> How is this different from what we have in Emacs?  An exciting new
> feature is sometimes deferred to the next major release if the release
> branch is close enough to a release.  There's nothing new here, just
> the fact that sometimes useful new features could destabilize Emacs,
> so one needs to choose which one it wants more.

I think the main difference is that Eglot and Emacs (and ElDoc, for that 
matter) all have different release cadences, so it would be helpful to 
have some functionality to help us manage that. With Emacs itself, we 
can ensure that every package that ships in the tarball is works well 
with each other; when we distribute a package on ELPA, this gets trickier.

>> One alternative would be for packages to be able to *recommend*
>> dependencies. Then, Eglot could recommend newer versions of ElDoc, but
>> they wouldn't actually be required.
> 
> This is probably needed, but it requires non-trivial support from
> package.el, to let informed users select the updates that fit their
> stability requirements and feature needs.

Yeah, that's where this gets tricky for Emacs 29: it's probably a bit 
late to add major new features like this to package.el for 29. However, 
we might be able to accept an imperfect solution for Eglot today while 
working to provide a better way for Emacs 30.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Tue, 18 Apr 2023 12:36:12 -0700
> Cc: dmitry@gutov.dev, joaotavora@gmail.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> >> * Stable: the version of a package included in the latest Emacs tarball
> >> * Latest: the latest version on GNU ELPA (etc)
> >> * Devel: the latest version on GNU-devel ELPA (etc)
> > 
> > I think we need only two.  Stable can move to the next version, since
> > packages are released more frequently than Emacs.
> 
> At least from a user POV, I think it makes sense to distinguish among 
> all three of these. For example, I might use the version of Eglot in 
> Emacs 29, or I might install it from GNU ELPA, or I could even install 
> it from GNU-devel ELPA. I might even switch back and forth depending on 
> what my needs are (and in fact, that's exactly what I do with Eglot; 
> while I usually prefer to stick on the GNU ELPA version, sometimes I 
> switch to GNU-devel ELPA if there's a fix for a bug I find very bothersome).

From a user POV, what is the qualitative difference between "Latest"
and "Stable"?  As soon as some newer version on ELPA becomes stable
enough, why should the user assign any significance to the previous
version included in the last released Emacs tarball?  Please keep in
mind that the version you call "Stable" is just one that was ready in
time for the last Emacs release, and is otherwise no different from
what you call "Latest".  IOW, assuming that versions in Emacs and on
ELPA are declared "Stable" using the same criteria and the same
procedures, they are not really different from the stability POV.
Moreover, the version in Emacs could (hopefully, very infrequently)
have some bug that went unnoticed during the pretest, and that bug
could now be fixed in the ELPA version.

Unless, that is, you somehow trust the decision-making process for an
Emacs release much more than you trust the same process for ELPA
releases.  But in my book, those should be the same processes,
especially if we ever get to the situation where such packages are not
in emacs.git anymore.

> I think this set of three levels also makes it easier - at least for me 
> - to reason about what to do with something like ElDoc. If a user 
> installs Eglot from GNU ELPA (i.e. the user gets "Eglot latest"), should 
> it automatically install ElDoc from GNU ELPA ("ElDoc latest") or should 
> it use the ElDoc from the Emacs tarball ("ElDoc stable")?

My answer is NO, but that is a separate issue.

> If there were only two levels - latest and devel - then I think the 
> answer to the Eglot/ElDoc problem would simply be: installing Eglot 
> latest should pull in ElDoc latest.

By default, perhaps.  But the user should still be able to say not to
upgrade any dependencies unless they _must_ be upgrade, i.e. the
package the user wants to upgrade _requires_ a newer version of
another package.

And again: this is a separate issue.

> Since there's no higher "stability 
> gradation" than latest, we wouldn't really be able to say that 
> ElDoc-from-Emacs is better/stabler than ElDoc-from-ELPA. (Well, we can 
> still *say* that, but I think it helps to embed our reasoning for it 
> into these stability gradations.)

It isn't our decision, it is up to the user.  _We_ could recommend to
upgrade to the latest version (if we will indeed get to the position
where we can do that in good faith), but the user could decide he/she
doesn't believe us, and doesn't want any additional risks.

> > How is this different from what we have in Emacs?  An exciting new
> > feature is sometimes deferred to the next major release if the release
> > branch is close enough to a release.  There's nothing new here, just
> > the fact that sometimes useful new features could destabilize Emacs,
> > so one needs to choose which one it wants more.
> 
> I think the main difference is that Eglot and Emacs (and ElDoc, for that 
> matter) all have different release cadences

But this difference, if it is real, should really disappear, right?
the periods of time when an Emacs release branch is closed to unsafe
changes is nowadays relatively short, so there should be no problem
for the core package to adhere to the same basic routine, i.e. keep
the next "candidate for latest" closed to risky changes for a couple
of months, before it actually becomes "latest".  And then the
difference will all but disappear.

> With Emacs itself, we can ensure that every package that ships in
> the tarball is works well with each other; when we distribute a
> package on ELPA, this gets trickier.

This "trickier" part will need to be fully resolved as part of getting
our act together for the brave new world where core packages are
maintained only on ELPA.  So we might as well start now, because I see
no reason why this difference should even exist -- it's not like we do
in Emacs something too complicated and expensive.

Btw, does package.el even allow to "downgrade" back to the version
that came with Emacs?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 7:57 PM Jim Porter <jporterbugs@gmail.com> wrote:
>
> On 4/18/2023 5:57 AM, Eli Zaretskii wrote:
> > So this again goes back to the main issue: how should the stability
> > considerations affect development of core packages and their
> > "stability gradation"?
>
> It sounds to me like there are 3 or 4 levels (depending on how you count):
>
> * Stable: the version of a package included in the latest Emacs tarball
> * Latest: the latest version on GNU ELPA (etc)
> * Devel: the latest version on GNU-devel ELPA (etc)
>
> You could possibly add:
>
> * Core(?): the version of a package included in the tarball of the
> user's *current* Emacs installation

These are interesting levels, but I was under the impression
that the goal is to partition the set of :core packages according
to some kind of gradation, what Eli called "stability gradation".
The set can be found in the variable package--builtins.

In contrast, you seem to be proposing to partition the much larger
set of pairs (CORE-PACKAGE . VERSION) (where VERSION is the version
of CORE-PACKAGE, not Emacs).

These two exercises are potentially interesting, but merely as
academic exercises they are not particularly useful.

For the first exercise, the useful thing that can be extracted
from it is the optimal behaviour w.r.t to stability of our existing
tools when confronted with an argument from that set.

Such existing tools include package-install, use-package,
package-update, etc. For example.

A. The effects of the form (package-install 'eglot) moving from
Emacs 26,27,28 to Emacs 29 are drastically different.
That's bad stability.

B. The effects of, say, the form (package-install 'xref) moving
from Emacs 28 to Emacs 29 are the same.  That's good stability.

What I'd like to know if anyone else thinks, like me, that this is
a problem.  We can discuss how bad of a problem (I think it's
moderately serious).

But, more importantly, am I the only one that sees an easy
fix to this we can fix case A and keep case B untouched?

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2175 bytes --]


João Távora <joaotavora@gmail.com> writes:

> A. The effects of the form (package-install 'eglot) moving from
> Emacs 26,27,28 to Emacs 29 are drastically different.
> That's bad stability.
>
> B. The effects of, say, the form (package-install 'xref) moving
> from Emacs 28 to Emacs 29 are the same.  That's good stability.
>
> What I'd like to know if anyone else thinks, like me, that this is
> a problem.  We can discuss how bad of a problem (I think it's
> moderately serious).

To me any breakage in behavior when updating to a newer version is
serious. I use Emacs for many scenarios, and some of those I only touch
twice a year (when I give a lecture and update slides for it), while
others I might not use for 3 years until I go back into that old Python
package.

Usually when I get in there and someting is broken in my Emacs that I
didn’t see otherwise, that’s very painful, because I’m often under time
pressure (like “the lecture is tomorrow and I just realized I have to
change these slides, but with the new version I’ve been using for half a
year the layout of just exactly these slides is broken and I still have
to make dinner for the family and actually get some sleep afterwards!”).

But there are too many different tasks to actually check them all every
time I update. And I usually don’t choose to do an Emacs update, but
just update when my distro ships the new version.

Emacs has been mostly really good at just working after an update
(except for some rare cases where I used a fringe org-feature or where
electric indent got enabled in org-mode), and to me it is essential that
Emacs with moderate customization (I have around 1900 lines of
custom-set-variables — mostly org-capture-templates, org-latex-classes,
and safe-local-variable-values — and 1000 lines of customization — most
of those via use-package) stays stable and does not become volatile.

To some argumentation why that’s not just my preference but essential:
https://stevelosh.com/blog/2012/04/volatile-software/

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
> Cc: Jim Porter <jporterbugs@gmail.com>, Eli Zaretskii <eliz@gnu.org>, Dmitry
>  Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> Date: Wed, 19 Apr 2023 14:13:40 +0200
> 
> João Távora <joaotavora@gmail.com> writes:
> 
> > A. The effects of the form (package-install 'eglot) moving from
> > Emacs 26,27,28 to Emacs 29 are drastically different.
> > That's bad stability.
> >
> > B. The effects of, say, the form (package-install 'xref) moving
> > from Emacs 28 to Emacs 29 are the same.  That's good stability.
> >
> > What I'd like to know if anyone else thinks, like me, that this is
> > a problem.  We can discuss how bad of a problem (I think it's
> > moderately serious).
> 
> To me any breakage in behavior when updating to a newer version is
> serious. I use Emacs for many scenarios, and some of those I only touch
> twice a year (when I give a lecture and update slides for it), while
> others I might not use for 3 years until I go back into that old Python
> package.
> 
> Usually when I get in there and someting is broken in my Emacs that I
> didn’t see otherwise, that’s very painful, because I’m often under time
> pressure (like “the lecture is tomorrow and I just realized I have to
> change these slides, but with the new version I’ve been using for half a
> year the layout of just exactly these slides is broken and I still have
> to make dinner for the family and actually get some sleep afterwards!”).
> 
> But there are too many different tasks to actually check them all every
> time I update. And I usually don’t choose to do an Emacs update, but
> just update when my distro ships the new version.

I don't think anyone will argue that breaking changes are good.

But that is not what was difficult about the particular case João is
talking about.  In that case, any solution that was proposed had a
potential to break someone's workflow.

Specifically, users of Emacs 28 and older, who had Eglot installed,
and expect Eglot to be automatically updated upon Emacs startup
whenever a new Eglot version is available, will now have their
expectations broken after they upgrade to Emacs 29, because Eglot is
now a built-in package, and package.el won't by default upgrade a
built-in package.

OTOH, the proposal to change package.el so it automatically updates
built-in packages as well would break the workflows of people who
don't expect built-in packages to be updated: they would suddenly see
packages like ElDoc or Xref or Project (and others) being updated
automatically at startup instead of staying at the version shipped
with Emacs.

So there's a dilemma here: which of the two groups of users to break?

And that is the real issue here, at least from my POV.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 6:02 PM Eli Zaretskii <eliz@gnu.org> wrote:

> OTOH, the proposal to change package.el so it automatically updates
> built-in packages as well would break the workflows of people who
> don't expect built-in packages to be updated: they would suddenly see
> packages like ElDoc or Xref or Project (and others) being updated
> automatically at startup instead of staying at the version shipped
> with Emacs.

OK. In my initial evaluation I disagreed that this was a problem,
because that has always happened in Emacs 28.

But I acknowledge that if some user has

  (package-install always-has-been-a-core-package)

in her Emacs 26/27/28 config, then some behaviour would have been
changed with some of my previous solutions.  I give you that, it
doesn't need contesting.

And that is why I later I gave a solution that _doesn't_,
I repeat, _doesn't_ let that happen.  That solution is in

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#467

It's a minimal patch, much smaller than the one that did make
it in.  I've linked to it many times now, but it has always
been ignored by all except Philip and Dmitry, who said it was a
"fine" solution and +1, respectively.

It will _not_ update Xref and ElDoc _unless_ the user specifically
asked for Eglot to be installed.  Which is, of course, exactly what
happened already in Emacs 28. For example, if the user has

  (package-install 'project)

It _won't_ be updated. And neither will its dependency Xref. Again, just like
in Emacs 28.

Why are we ignoring my patch? What's really wrong with it? In
technical terms, if possible.

> So there's a dilemma here: which of the two groups of users to break?

There's no dilemma.  There's no need for the heavy-handed solution
you chose that breaks one of the groups of users.  Presumably it was
the one you judged least numerous/valuable.  I don't even want to
dispute that judgement, because there's no need!  We can have
perfectly backward-compatible behavior, so if we value it, let's
have it.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 18:21:23 +0100
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>, jporterbugs@gmail.com, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> But I acknowledge that if some user has
> 
>   (package-install always-has-been-a-core-package)
> 
> in her Emacs 26/27/28 config, then some behaviour would have been
> changed with some of my previous solutions.  I give you that, it
> doesn't need contesting.
> 
> And that is why I later I gave a solution that _doesn't_,
> I repeat, _doesn't_ let that happen.  That solution is in
> 
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#467
> 
> It's a minimal patch, much smaller than the one that did make
> it in.  I've linked to it many times now, but it has always
> been ignored by all except Philip and Dmitry, who said it was a
> "fine" solution and +1, respectively.
> 
> It will _not_ update Xref and ElDoc _unless_ the user specifically
> asked for Eglot to be installed.  Which is, of course, exactly what
> happened already in Emacs 28. For example, if the user has
> 
>   (package-install 'project)
> 
> It _won't_ be updated. And neither will its dependency Xref. Again, just like
> in Emacs 28.
> 
> Why are we ignoring my patch? What's really wrong with it? In
> technical terms, if possible.

It has similar problems: it will automatically update packages
mentioned in package--safely-upgradeable-builtins, which might not be
what users want for built-in packages.  You assume that everyone will
want Eglot and use-package automatically updated, but this assumption
has no real basis.  I object in general to any feature that
unexpectedly installs some software without the explicit user's
say-so.

Plus, it adds to the maintenance burden of maintaining this
(internal? not a defcustom??) variable for good.

Those issues were enough for me to reject the proposal.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 21:07, Eli Zaretskii wrote:
> It has similar problems: it will automatically update packages
> mentioned in package--safely-upgradeable-builtins, which might not be
> what users want for built-in packages.

IMO that kind of choice could be deferred to the maintainer of each 
individual package. Or make it a defcustom if you're really worried.

BTW, even choosing that patch where this user option is a defcustom 
defaulting to nil would make more sense to me than the patch we 
currently decided to install.

> You assume that everyone will
> want Eglot and use-package automatically updated, but this assumption
> has no real basis.

People don't call 'M-x package-install' automatically, nor do they put 
those calls in their init files automatically. The only cases where 
Eglot would be automatically updated, even with the proposed patch, is 
when the user previously declared their intention to have Eglot updated 
at least once, in some manual fashion (e.g. using 'package-install' 
interactively).

> I object in general to any feature that
> unexpectedly installs some software without the explicit user's
> say-so.

No disagreement here.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 21:14:06 +0300
> Cc: arne_bab@web.de, jporterbugs@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 19/04/2023 21:07, Eli Zaretskii wrote:
> > It has similar problems: it will automatically update packages
> > mentioned in package--safely-upgradeable-builtins, which might not be
> > what users want for built-in packages.
> 
> IMO that kind of choice could be deferred to the maintainer of each 
> individual package.

No, it cannot, and this and the sibling discussions show why: the
package maintainers are biased in favor of their packages.  That
(completely understandable and expected) bias prevents them from
seeing the overall picture objectively.

> Or make it a defcustom if you're really worried.

That doesn't change the picture, unless the default for the defcustom
will be nil.  Which I expect João to object to, because he wants Eglot
to be updated by default and automatically.  Whereas I think the
compromise, whereby the user should say just once that he/she wants
Eglot to be automatically updated, is a good compromise given the
constraints in this case.  Not ideal, but a good-enough compromise.

> BTW, even choosing that patch where this user option is a defcustom 
> defaulting to nil would make more sense to me than the patch we 
> currently decided to install.

See above.

> > You assume that everyone will
> > want Eglot and use-package automatically updated, but this assumption
> > has no real basis.
> 
> People don't call 'M-x package-install' automatically, nor do they put 
> those calls in their init files automatically.

That's factually incorrect, AFAIU.  Moreover, the cases that bothered
João (again, AFAIU) were exactly those which you say don't exist.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 7:32 PM Eli Zaretskii <eliz@gnu.org> wrote:

> No, it cannot, and this and the sibling discussions show why: the
> package maintainers are biased in favor of their packages.  That
> (completely understandable and expected) bias prevents them from
> seeing the overall picture objectively.

What is the part of the picture that I'm not seeing?  Who am I
ignoring, who am I missing? What are these people that I'm
(IYO unwillingly, fortunately) biased against.  What exactly are
do you conjecture or know they are doing right now in Emacs 28 that
my patch will prevent, affect, break, make difficult?  Just one
example, please.

> > Or make it a defcustom if you're really worried.
> That doesn't change the picture, unless the default for the defcustom
> will be nil.  Which I expect João to object to, because he wants Eglot
> to be updated by default and automatically.

And _only_ Eglot.  Because, that's what happened to it in Emacs 28.

> > > You assume that everyone will
> > > want Eglot and use-package automatically updated, but this assumption
> > > has no real basis.
> >
> > People don't call 'M-x package-install' automatically, nor do they put
> > those calls in their init files automatically.
>
> That's factually incorrect, AFAIU.  Moreover, the cases that bothered
> João (again, AFAIU) were exactly those which you say don't exist.

Yes, you're right Eli.  I pointed to numerous issues collected over
the years, where I found proof that users have this kind of form
int heir configurations:

(use-package eglot :ensure t :config() )

Even ChatGPT suggests this. Just ask it "How do I configure the Eglot
Emacs extension for the <Your favourite language> LSP server"?

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[tomas]

[-- Attachment #1: Type: text/plain, Size: 178 bytes --]

On Wed, Apr 19, 2023 at 08:33:44PM +0100, João Távora wrote:

[...]

> Even ChatGPT suggests this [...]

OK. Now I know I don't want an LSP server ;-P

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 21:32, Eli Zaretskii wrote:
>> Date: Wed, 19 Apr 2023 21:14:06 +0300
>> Cc: arne_bab@web.de, jporterbugs@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> On 19/04/2023 21:07, Eli Zaretskii wrote:
>>> It has similar problems: it will automatically update packages
>>> mentioned in package--safely-upgradeable-builtins, which might not be
>>> what users want for built-in packages.
>>
>> IMO that kind of choice could be deferred to the maintainer of each
>> individual package.
> 
> No, it cannot, and this and the sibling discussions show why: the
> package maintainers are biased in favor of their packages.  That
> (completely understandable and expected) bias prevents them from
> seeing the overall picture objectively.

I don't know, I never really minded that project.el and xref aren't 
upgraded too often.

Though to be honest I would still expect that, when the user types 'M-x 
package-install', they'd be able to upgrade project or xref to the 
latest version. Simply because that looks like an unambiguous request 
for this exact outcome.

>> Or make it a defcustom if you're really worried.
> 
> That doesn't change the picture, unless the default for the defcustom
> will be nil.  Which I expect João to object to, because he wants Eglot
> to be updated by default and automatically.

Only when their init script contains

   (package-install 'eglot)

or

   (use-package 'eglot :force t)

right?

> Whereas I think the
> compromise, whereby the user should say just once that he/she wants
> Eglot to be automatically updated, is a good compromise given the
> constraints in this case.  Not ideal, but a good-enough compromise.

Might be. In saying the above I'm thinking about a different issue: user 
options should be logical (as much as we possibly manage). When they 
are, they are easier to find and enable.

So if we absolutely can't resolve the problem using the solution 
previously recommended by Philip, Joao and I, we should at least choose 
the most logical among approaches containing a new user option.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

Wed, Apr 19, 2023 at 8:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> >> Or make it a defcustom if you're really worried.
> >
> > That doesn't change the picture, unless the default for the defcustom
> > will be nil.  Which I expect João to object to, because he wants Eglot
> > to be updated by default and automatically.
>
> Only when their init script contains
>
>    (package-install 'eglot)
>
> or
>
>    (use-package 'eglot :force t)
>
> right?

Yes, at least that.  I've given up on interactive M
-x package-install offering Eglot in the completions.  It's also a
backward incompatibility but at least that will alarm users and they
will get some notice that something changed from Emacs 28 to
Emacs 29.

Whereas the silent noop of the non-interactive case is much
worse.

And it's ":ensure", not ":force" by the way. Just ask ChatGPT,
it'll tell you ;-)


Curiously, after many many iterations Philip's patch up to the
very end had this minimally backward compatible behaviour for
the non-interactive calls.  But alas, the final version forbade it too.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 22:46, João Távora wrote:
> Wed, Apr 19, 2023 at 8:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>>>> Or make it a defcustom if you're really worried.
>>>
>>> That doesn't change the picture, unless the default for the defcustom
>>> will be nil.  Which I expect João to object to, because he wants Eglot
>>> to be updated by default and automatically.
>>
>> Only when their init script contains
>>
>>     (package-install 'eglot)
>>
>> or
>>
>>     (use-package 'eglot :force t)
>>
>> right?
> 
> Yes, at least that.  I've given up on interactive M
> -x package-install offering Eglot in the completions.  It's also a
> backward incompatibility but at least that will alarm users and they
> will get some notice that something changed from Emacs 28 to
> Emacs 29.
> 
> Whereas the silent noop of the non-interactive case is much
> worse.
> 
> And it's ":ensure", not ":force" by the way. Just ask ChatGPT,
> it'll tell you ;-)

I try not to talk to robots in my spare time ;-)

So... are you sure that (use-package 'eglot :ensure t) upgrades/upgraded 
the package?

 From what I'm reading in use-package-ensure-elpa, all 'package-install' 
calls are wrapped in

   (unless (package-installed-p package)
     ...

That seems to mean that, as long as the package is installed already 
(whether by being bundled, or because of a previous manual 
installation), it shouldn't get upgraded when this form is executed.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 9:50 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> So... are you sure that (use-package 'eglot :ensure t) upgrades/upgraded
> the package?
>
>  From what I'm reading in use-package-ensure-elpa, all 'package-install'
> calls are wrapped in
>
>    (unless (package-installed-p package)
>      ...
>
> That seems to mean that, as long as the package is installed already
> (whether by being bundled, or because of a previous manual
> installation), it shouldn't get upgraded when this form is executed.

That's true.  Exactly the same as package-install. It will not
reinstall over another one.

But that doesn't change anything.  In Emacs 26/27/28 from scratch
package-install and use-package _will_ rev up Eglot to whatever is the
newest version.  In Emacs 29 it won't.

In fact deleting Eglot and restarting Emacs the config run again is what
I suppose the most common upgrade method (since there is no
package-update in Emacs 28).

Other than that, think CI scripts, dockerfiles, VMs, or just the
casual user who trashes the packages dir to get a fresh set when
looking for a bug (like I do, and multiple people I've interacted with).

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 1:57 PM, João Távora wrote:
> But that doesn't change anything.  In Emacs 26/27/28 from scratch
> package-install and use-package _will_ rev up Eglot to whatever is the
> newest version.  In Emacs 29 it won't.

But in that case, the user simply ends up with, at best, whatever 
version of Eglot was current at the time they ran Emacs with their 
config for the first time on that machine. The status quo is already 
that users are likely on an older version of Eglot, unless they wipe 
their elpa/ directory regularly (which seems an awkward way to update 
your packages to me).

> In fact deleting Eglot and restarting Emacs the config run again is what
> I suppose the most common upgrade method (since there is no
> package-update in Emacs 28).

As mentioned elsewhere in this thread, I just use the *Packages* buffer 
and 'package-menu-mark-upgrades' (bound to "U"). That works on older 
Emacsen too. In 29, 'package-update' would make it easier to update a 
single package though, and 'package-update-all' provides a programmatic 
way to do what I do interactively in 28.

(Though now I wonder why we use both "update" and "upgrade"...)

> Other than that, think CI scripts, dockerfiles, VMs, or just the
> casual user who trashes the packages dir to get a fresh set when
> looking for a bug (like I do, and multiple people I've interacted with).

If a user hasn't done anything to specify the release channel they want, 
then I think the current behavior is correct. "(package-install 'eglot)" 
just means "make sure I have Eglot". However, if the user pinned Eglot 
to GNU ELPA, I think it would make some sense for 'package-install' to 
install Eglot from GNU ELPA (i.e. it now means "make sure I have Eglot 
*from GNU ELPA*").

I know this isn't automatic; a user would need to pin Eglot to GNU ELPA 
to get this behavior, but it's fairly easy to do. use-package already 
supports pinning as well. We'd just need to tweak 'package-install' and 
'use-package :ensure' to handle pinning as described above.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 10:58 PM Jim Porter <jporterbugs@gmail.com> wrote:
>

> As mentioned elsewhere in this thread, I just use the *Packages* buffer
> and 'package-menu-mark-upgrades' (bound to "U"). That works on older
> Emacsen too. In 29, 'package-update' would make it easier to update a
> single package though, and 'package-update-all' provides a programmatic
> way to do what I do interactively in 28.

No. Both are absent in Emacs 28.  I can't say what the the most
common method is, but M-x package-delete + restart + letting your
config run again is the method I use.

I didn't know about "U" until recently.  And that's because
the package menu is a very slow and unreliable command IME.

And besides, "U" _also does different things in Emacs 28 and Emacs 29.
 In 28 it _will_ upgrade Eglot to the latest (so they tell me). But
not in Emacs 29.

This _also_ deserved a fix, but is IMO, not as important as the
simple non-interactice package-install case.

And, for good measure, the new M-x package-update and M-x
package-update-all in Emacs 29 _also_ won't update Eglot.

> (Though now I wonder why we use both "update" and "upgrade"...)

There's a separate bug for that.

> > Other than that, think CI scripts, dockerfiles, VMs, or just the
> > casual user who trashes the packages dir to get a fresh set when
> > looking for a bug (like I do, and multiple people I've interacted with).
>
> If a user hasn't done anything to specify the release channel they want,
> then I think the current behavior is correct. "(package-install 'eglot)"
> just means "make sure I have Eglot". However, if the user pinned Eglot
> to GNU ELPA, I think it would make some sense for 'package-install' to
> install Eglot from GNU ELPA (i.e. it now means "make sure I have Eglot
> *from GNU ELPA*").

By default, Emacs comes with GNU ELPA configured only, I think.
And Eglot has always existed there.  I don't know what you mean
by "pin".  I do know some users _also_ had Eglot on MELPA to get
an even newer version, but I've never really recommended that.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 3:29 PM, João Távora wrote:
> On Wed, Apr 19, 2023 at 10:58 PM Jim Porter <jporterbugs@gmail.com> wrote:
>> As mentioned elsewhere in this thread, I just use the *Packages* buffer
>> and 'package-menu-mark-upgrades' (bound to "U"). That works on older
>> Emacsen too. In 29, 'package-update' would make it easier to update a
>> single package though, and 'package-update-all' provides a programmatic
>> way to do what I do interactively in 28.
> 
> No. Both are absent in Emacs 28.

Yes, but the "U" keybinding is in Emacs 28, which is what I meant there.

> And besides, "U" _also does different things in Emacs 28 and Emacs 29.
>   In 28 it _will_ upgrade Eglot to the latest (so they tell me). But
> not in Emacs 29.

That depends. If I had previously installed Eglot in 28, and then 
download Emacs 29 and upgrade my packages (without deleting my elpa/ 
dir), I'll still get new versions of Eglot from GNU ELPA. I think that 
part works as expected.

> By default, Emacs comes with GNU ELPA configured only, I think.
> And Eglot has always existed there.  I don't know what you mean
> by "pin".  I do know some users _also_ had Eglot on MELPA to get
> an even newer version, but I've never really recommended that.

You can "pin" a package (see 'package-pinned-packages' or the ':pin' 
keyword in use-package) so that you always get updates from a particular 
archive, which is useful if you use a package that rarely gets releases, 
among other things. Then you can set the devel archives to a lower 
priority (see 'package-archive-priorities') and pin that package to the 
devel archive without having to live on the edge for every other 
package. This also means that if I wipe out my elpa/ dir, I get my 
packages from the archive I want.

I think this should also work as a way of declaring that you want Eglot 
from GNU ELPA even if it's built-in, but that part doesn't work right now.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 11:42 PM Jim Porter <jporterbugs@gmail.com> wrote:

> > No. Both are absent in Emacs 28.
>
> Yes, but the "U" keybinding is in Emacs 28, which is what I meant there.
>
> > And besides, "U" _also does different things in Emacs 28 and Emacs 29.
> >   In 28 it _will_ upgrade Eglot to the latest (so they tell me). But
> > not in Emacs 29.
>
> That depends. If I had previously installed Eglot in 28, and then
> download Emacs 29 and upgrade my packages (without deleting my elpa/
> dir), I'll still get new versions of Eglot from GNU ELPA. I think that
> part works as expected.

Yes it does.  Come on, not _everything_ is broken :-)

The bug#62720, reported by me, listed the only workaround
that works identically in Emacs 2*.  Just go to the package
menu and press 'I' on the package you want to install.  Boom, there
go the ancient safeguards against updating builtin packages.

If you look a bit at the package.el code, you'll see that
all this code goes through different flows etc.  All these
safeguards and existing behaviour are more likely to be
be accidental than intentional.

Nevertheless,  it is possible to pick one of them to
maintain the same behaviour.


João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 23:57, João Távora wrote:
> On Wed, Apr 19, 2023 at 9:50 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> 
>> So... are you sure that (use-package 'eglot :ensure t) upgrades/upgraded
>> the package?
>>
>>   From what I'm reading in use-package-ensure-elpa, all 'package-install'
>> calls are wrapped in
>>
>>     (unless (package-installed-p package)
>>       ...
>>
>> That seems to mean that, as long as the package is installed already
>> (whether by being bundled, or because of a previous manual
>> installation), it shouldn't get upgraded when this form is executed.
> That's true.  Exactly the same as package-install. It will not
> reinstall over another one.

OK then, I think have to re-evaluate my position on this. Previously, I 
guess, I made some hasty conclusions from how the discussion went on 
without refreshing the exact details about package.el and use-package 
(the latter I never knew to begin with). Apologies.

Eli, let me know if we should take this back to the bug tracker instead.

> But that doesn't change anything.  In Emacs 26/27/28 from scratch
> package-install and use-package_will_  rev up Eglot to whatever is the
> newest version.  In Emacs 29 it won't.

I think one of the conclusions to be made here is that even if 
(package-install 'eglot) now installs the newest version of Eglot in 
Emacs 29,

   (use-package 'eglot :ensure t)

still won't do that in Emacs 29 because (package-installed-p 'eglot) 
returns t still. So the commit 580d8278c5f48 doesn't help with your 
"most common upgrade method" cited below, if they rely on use-package 
instead of calling 'package-install' directly.

The patch I +1'd here https://debbugs.gnu.org/62720#467 wouldn't help 
with (use-package 'eglot :ensure t) either, IIUC.

But it would help in case the user put (package-install 'eglot) inside 
their init script, or if the CI script contains that. But the CI scripts 
could be fixed anyway (there must be a lot fewer of them out there than 
there are user installations).

Next: 'package-install' isn't supposed to install a new version when 
some version is already installed. The docstring says so, and I just 
tried this out: had older 'vertico' installed (with a newer one 
available in the repo), evaluated (package-install 'vertico), got back 
the message "already installed".

Do we want to change the semantics of 'package-install' just so the 
(minor) fraction of users who call this function directly will get Eglot 
upgraded when starting over from an empty config with Emacs 29? And/or 
change (package-installed-p 'eglot) to return nil when the appropriate 
user option is set? I'm not sure that's wise, certainly not the latter.

> In fact deleting Eglot and restarting Emacs the config run again is what
> I suppose the most common upgrade method (since there is no
> package-update in Emacs 28).

I've never considered this particular scenario, but as explained above, 
it's still not "fixed" by either of the proposed approaches. Not 
entirely, at least.

The most common upgrade method, though, is hopefully 
'package-menu-mark-upgrades'. Or:

   M-x list-packages
   U
   x

And that one also doesn't work for Eglot and other builtin packages. 
Unlike 'package-install', this behavior is not obviously correct, 
although it's "always been like this" so we're probably not going to 
change what it does by default at this point in time.

Another function which is supposed to upgrade packages in 'M-x 
package-update' (why aren't people commenting on bug#62750? the naming's 
gonna drive me crazy). This one is obviously incorrect because upgrades 
exist for Eglot, and yet it doesn't suggest any.

So I would suggest to focus on functions that don't work as intended. 
Namely:

- Add a user option for the list of builtin packages which would be 
upgraded automatically by 'package-menu-mark-upgrades' and 'M-x 
package-upgrade-all' (nee package-update-all). Maybe make it nil by 
default, or maybe add 'eglot' to it. I don't have a strong opinion.

- Fix 'M-x package-upgrade' (nee package-update) to suggest Eglot as one 
of the options and actually perform the upgrade. That shouldn't require 
changes to 'package-install' because, as we already know, the user can 
already install a newer version of Eglot using the 'list-packages' menu 
(and picking the exact version manually). That execution path is going 
to go through 'package-install' as well, so it must be suitable already.

- Revert 580d8278c5f48 because it creates odd semantics (upgrading 
certain packages that are already installed but not others) and it 
doesn't solve the issue with (use-package 'eglot :ensure t) anyway. We 
could keep it, but seems like a half-measure that didn't make anyone 
happy anyway. OTOH, it could minimize the rewrites of CI scripts.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 3:06 PM, Dmitry Gutov wrote:
> - Add a user option for the list of builtin packages which would be 
> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x 
> package-upgrade-all' (nee package-update-all). Maybe make it nil by 
> default, or maybe add 'eglot' to it. I don't have a strong opinion.

I think (with some fixes), 'package-pinned-packages' should work for 
this. If you want to get Eglot from GNU ELPA, then you can pin it as 
such, and it should do the right thing.

Unfortunately, it doesn't *actually* do the right thing, but I think 
that's pretty clearly a bug.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 01:21, Jim Porter wrote:
> On 4/19/2023 3:06 PM, Dmitry Gutov wrote:
>> - Add a user option for the list of builtin packages which would be 
>> upgraded automatically by 'package-menu-mark-upgrades' and 'M-x 
>> package-upgrade-all' (nee package-update-all). Maybe make it nil by 
>> default, or maybe add 'eglot' to it. I don't have a strong opinion.
> 
> I think (with some fixes), 'package-pinned-packages' should work for 
> this. If you want to get Eglot from GNU ELPA, then you can pin it as 
> such, and it should do the right thing.
> 
> Unfortunately, it doesn't *actually* do the right thing, but I think 
> that's pretty clearly a bug.

It might work, too (though I suppose altering this var might be a bit 
harder for an average use than changing a new option to t or adding 
'eglot to it).

Depends on how complex the fix will be, for the "clear bug" you mention.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 3:27 PM, Dmitry Gutov wrote:
> It might work, too (though I suppose altering this var might be a bit 
> harder for an average use than changing a new option to t or adding 
> 'eglot to it).

use-package users should have an easy time of it, at least. You can just 
add ":pin gnu" to your use-package declaration.

That said, I think there should be a way to do it from the *Packages* 
buffer, just like installing a package.

> Depends on how complex the fix will be, for the "clear bug" you mention.

Yeah, we'll have to see. I'll take a look if/when we agree that this is 
a solution to the problem.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

Not sure about the reason emacs-devel was lost from Cc. That could be my 
fault: I removed it before sending the first version of the grandparent 
email, but then I thought I managed to abort that delivery and resend it 
properly.

Anyway, adding it back and replying with a full quote.

On 20/04/2023 01:49, João Távora wrote:
> On Wed, Apr 19, 2023 at 11:01 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> I think one of the conclusions to be made here is that even if
>> (package-install 'eglot) now installs the newest version of Eglot in
>> Emacs 29,
>>
>>     (use-package 'eglot :ensure t)
>>
>> still won't do that in Emacs 29 because (package-installed-p 'eglot)
>> returns t still. So the commit 580d8278c5f48 doesn't help with your
>> "most common upgrade method" cited below, if they rely on use-package
>> instead of calling 'package-install' directly.
> 
> Right, that's likely.
> 
>> The patch I +1'd here https://debbugs.gnu.org/62720#467 wouldn't help
>> with (use-package 'eglot :ensure t) either, IIUC.
> 
> That's also likely.  So we'd need this:
> 
> diff --git a/lisp/use-package/use-package-ensure.el
> b/lisp/use-package/use-package-ensure.el
> index e0ea982594e..95e6a9e95d5 100644
> --- a/lisp/use-package/use-package-ensure.el
> +++ b/lisp/use-package/use-package-ensure.el
> @@ -160,7 +160,9 @@ use-package-ensure-elpa
>           (when (consp package)
>             (use-package-pin-package (car package) (cdr package))
>             (setq package (car package)))
> -        (unless (package-installed-p package)
> +        (when (or (and (memq package package--safely-upgradeable-builtins)
> +                       (not (assoc 'eglot (package--alist))))
> +                  (not (package-installed-p package)))
>             (condition-case-unless-debug err
>                 (progn
>                   (when (assoc package (bound-and-true-p

All right, this part would "fix" (use-package eglot :ensure t).

>> Do we want to change the semantics of 'package-install' just so the
>> (minor) fraction of users who call this function directly will get Eglot
>> upgraded when starting over from an empty config with Emacs 29? And/or
>> change (package-installed-p 'eglot) to return nil when the appropriate
>> user option is set? I'm not sure that's wise, certainly not the latter.
> 
> We don't _have_ to change the semantics.  We can get exactly the same
> semantics if we want to.  Patch below:

What kind of semantics do we get with it?

1. (use-package eglot :ensure t) considers select builtin packages to be 
"not installed" for the purposes of ":ensure t".
2. 'M-x package-install' allows installing them. It doesn't allow 
installing any other package for which (package-installed-p 'xxx) 
returns t, but allows installing (essentially upgrading) these ones 
(either just eglot, or both eglot and use-package).
3. 'M-x package-update RET eglot RET' still doesn't work unless eglot 
has been "upgraded" at least once via other means.

Is that logical? Is even just 1+2 logical?

And what about capabilities that we lose that way? I guess one of the 
reasons to bundle ELPA packages is to make sure they can be used without 
additional installation. E.g. in some Internes-less network, or one 
that's firewalled off. Let's also imagine that (for example) clangd is 
already installed through other means, which is also within the realm of 
possibility.

And take use-package. Which some people position as the new way to write 
the Emacs configuration.

The user puts the snippet which they saw on the Internet

   (use-package eglot :ensure t)

either because they think it's a good idea, or because they intend to 
add some actual config in there, restart Emacs and... startup fails 
because the package can't be installed (no connection). Should they 
remove ":ensure t"? Perhaps. But the documentation says that that option 
checks that the package is "installed automatically if not already 
present on your system". Seems legit, right? Why would startup fail when 
Eglot is already present on the system?

Or, to put it another way, why did we decide to bundle Eglot with Emacs 
if the first thing we're going to do is to try to download it anyway?

So... I understand the problem, but I think we shouldn't change the 
functions in a way that makes them conflict with documentation or with 
each other.

> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index 685f983e285..850f4ad3a7a 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -652,6 +652,12 @@ package--builtins
>   name (a symbol) and DESC is a `package--bi-desc' structure.")
>   (put 'package--builtins 'risky-local-variable t)
> 
> +(defvar package--safely-upgradeable-builtins '(eglot use-package))
> +
> +(defun package--safely-upgradeable-builtin (p)
> +  (and (memq p package--safely-upgradeable-builtins) ; whitelisted
> +       (not (assoc p (package--alist))))) ; not installed already
> +
>   (defvar package-alist nil
>     "Alist of all packages available for activation.
>   Each element has the form (PKG . DESCS), where PKG is a package
> @@ -2201,14 +2207,18 @@ package-install
>        (package--archives-initialize)
>        (list (intern (completing-read
>                       "Install package: "
> +                    (append
>                        (delq nil
>                              (mapcar (lambda (elt)
>                                        (unless (package-installed-p (car elt))
>                                          (symbol-name (car elt))))
>                                      package-archive-contents))
> +                     package--safely-upgradeable-builtins)
>                       nil t))
>              nil)))
>     (package--archives-initialize)
> +  (when (package--safely-upgradeable-builtin pkg)
> +    (setq pkg (cadr (assoc pkg package-archive-contents))))
>     (add-hook 'post-command-hook #'package-menu--post-refresh)
>     (let ((name (if (package-desc-p pkg)
>                     (package-desc-name pkg)
> diff --git a/lisp/use-package/use-package-ensure.el
> b/lisp/use-package/use-package-ensure.el
> index e0ea982594e..cfa10f453d9 100644
> --- a/lisp/use-package/use-package-ensure.el
> +++ b/lisp/use-package/use-package-ensure.el
> @@ -160,7 +160,8 @@ use-package-ensure-elpa
>           (when (consp package)
>             (use-package-pin-package (car package) (cdr package))
>             (setq package (car package)))
> -        (unless (package-installed-p package)
> +        (when (or (package--safely-upgradeable-builtin package)
> +                  (not (package-installed-p package)))
>             (condition-case-unless-debug err
>                 (progn
>                   (when (assoc package (bound-and-true-p

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 12:25 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
>
> Anyway, adding it back and replying with a full quote.

Thanks, I also didn't notice emacs-devel was lost there.

> On 20/04/2023 01:49, João Távora wrote:
> What kind of semantics do we get with it?

The closest possible to Emacs 28 for _all_ users (Eglot users and
the Emacs users that are vulnerable to furtive upgrades and which
Eli is thinking of)

> 1. (use-package eglot :ensure t) considers select builtin packages to be
> "not installed" for the purposes of ":ensure t".
> 2. 'M-x package-install' allows installing them. It doesn't allow
> installing any other package for which (package-installed-p 'xxx)
> returns t, but allows installing (essentially upgrading) these ones
> (either just eglot, or both eglot and use-package).
> 3. 'M-x package-update RET eglot RET' still doesn't work unless eglot
> has been "upgraded" at least once via other means.
>
> Is that logical? Is even just 1+2 logical?

Neither is.  IMO we have to think in terms of expectations for each
package, not in terms of a heavy handed blunt class o packages like
"builtin".  Because Eglot wasn't a builtin and now is.  So this is
why my patch includes that predicate `package--safely-upgradeable-builtin`
The name is not good, I admit.  It should be
`package--hitherto-casually-upgradeable-builtin` maybe.

> because the package can't be installed (no connection). Should they
> remove ":ensure t"? Perhaps. But the documentation says that that option
> checks that the package is "installed automatically if not already
> present on your system". Seems legit, right? Why would startup fail when
> Eglot is already present on the system?

Because we want it to do the same it did in Emacs 28?  But I see what
you mean.  Maybe the :ensure for packages in that particular subclass
should mean "try a bit harder to upgrade these, but don't blow up if
you can't -- just try next time".

> Or, to put it another way, why did we decide to bundle Eglot with Emacs
> if the first thing we're going to do is to try to download it anyway?

Who is "we"?  I know for a fact some people are.  The enthusiasts are,
the feature seekers, are immediately.  Others will experiment, I
assume.  Very often one experimentation that happens is you just
save all your .emacs.d to the side safely and let er rip with
whatever init.el someone (trustworthy presumably) gave you.

( What, you're going to say you've never tried the spacemacs
  or doom emacs or nano emacs just to see what all the hype
  is about??? :-) )

But others, like me from time to time when working on VMs via ssh
machines,  will be very happy with sudo apt install emacs in
said machines (if not done so already) and use a vanilla
configurationless emacs.  And if you add a "clangd" to that
invocation and Emacs happens to be 29 you'll have nice LSP
for the first time in many years.  Personally, for _my_ quick
hacks in those situations, I don't bother with M-x package-install
latest-and-greatest, just as I don't bother bringing my heavy
config via ssh to that machine.  I can't really, I'm frequently
tmate'ing with other people there, we have to settle on Emacs -Q
(+ electric-pair-mode and a few obvious others).

But your question also has a completely different answer:
A very good reason I wanted Eglot to be bundled with Emacs is now
actually being put into question in this very thread.  I wanted
to simplify Eglot's development as it advances with other :core
packages.  For example, I wanted to split off its external
completion style easily into a separate core package, as I did
with Stefan earlier this year.  I want to be able to add a feature
to ElDoc and use it in Eglot in the same patch, etc.  That part
has worked fine.  It wasn't impossible with Eglot in GitHub, but
it was harder.  Hopefully Eli is not asking for that bliss to end
I understand hope he's asking to be circumspect in what
is added, and not do gratuitous things like bump dependencies
without reason.  I already do that, so no problem.

Finally, if you're taking all this in the direction of "I told
you so, don't you regret bringing Eglot into Emacs 29?", I don't
blame you.  But I have to say: no, not yet.  I think having the
beginnings of an LSP library in Emacs for major-modes to work
with is excellent.  I like the attention and the high-standards
other seasoned developers hold my code up to.  I like the Info
manual and I like that emacs -Q bla.c -f eglot works just fine.

This upgrade situation sucks, is true, and I should have foreseen
it.  Spent too much time in my emacs-master worktree I guess.

Who knows, maybe this will all be OK after all?? Half the
users are using "straight.el" anyway (and I really doubt
it is hampered by these minutiae about dependencies, seeing
as MELPA-land is teeming with way more "dependency hell").

If nothing more happens on Emacs 29, which is the most likely
outcome, I think I will just start recommending "straight.el"
more, maybe learn it myself.  I had problems with it in the
past but it's probably matured now.

> So... I understand the problem, but I think we shouldn't change the
> functions in a way that makes them conflict with documentation or with
> each other.

Does the latest patch introduce any such conflict besides the
"no internet" case you mentioned?  Not that it's really a
conflict IMO: it's just what Emacs 28 did with Eglot AFAICT.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 03:13, João Távora wrote:

>> On 20/04/2023 01:49, João Távora wrote:
>> What kind of semantics do we get with it?
> 
> The closest possible to Emacs 28 for _all_ users (Eglot users and
> the Emacs users that are vulnerable to furtive upgrades and which
> Eli is thinking of)

As if Eglot wasn't bundled in the first place, you mean.

>> 1. (use-package eglot :ensure t) considers select builtin packages to be
>> "not installed" for the purposes of ":ensure t".
>> 2. 'M-x package-install' allows installing them. It doesn't allow
>> installing any other package for which (package-installed-p 'xxx)
>> returns t, but allows installing (essentially upgrading) these ones
>> (either just eglot, or both eglot and use-package).
>> 3. 'M-x package-update RET eglot RET' still doesn't work unless eglot
>> has been "upgraded" at least once via other means.
>>
>> Is that logical? Is even just 1+2 logical?
> 
> Neither is.  IMO we have to think in terms of expectations for each
> package, not in terms of a heavy handed blunt class o packages like
> "builtin".  Because Eglot wasn't a builtin and now is.  So this is
> why my patch includes that predicate `package--safely-upgradeable-builtin`
> The name is not good, I admit.  It should be
> `package--hitherto-casually-upgradeable-builtin` maybe.

This kind of option still makes sense, but I'd put in a different place: 
it would affect which packages are upgraded by 'M-x package-update-all' 
or 'U' in the packages list.

>> because the package can't be installed (no connection). Should they
>> remove ":ensure t"? Perhaps. But the documentation says that that option
>> checks that the package is "installed automatically if not already
>> present on your system". Seems legit, right? Why would startup fail when
>> Eglot is already present on the system?
> 
> Because we want it to do the same it did in Emacs 28?  But I see what
> you mean.  Maybe the :ensure for packages in that particular subclass
> should mean "try a bit harder to upgrade these, but don't blow up if
> you can't -- just try next time".

... maybe.

Though a meticulous user (some of them, maybe not others) might still 
wonder why an upgrade if attempted at all, even though the package is 
already available.

>> Or, to put it another way, why did we decide to bundle Eglot with Emacs
>> if the first thing we're going to do is to try to download it anyway?
> 
> Who is "we"?

You/we/emacs-devel as a whole.

> I know for a fact some people are.  The enthusiasts are,
> the feature seekers, are immediately.  Others will experiment, I
> assume.  Very often one experimentation that happens is you just
> save all your .emacs.d to the side safely and let er rip with
> whatever init.el someone (trustworthy presumably) gave you.

I agree that upgrading it should be made as easy as possible. But it 
should require explicit actions.

> ( What, you're going to say you've never tried the spacemacs
>    or doom emacs or nano emacs just to see what all the hype
>    is about??? :-) )
> 
> But others, like me from time to time when working on VMs via ssh
> machines,  will be very happy with sudo apt install emacs in
> said machines (if not done so already) and use a vanilla
> configurationless emacs.  And if you add a "clangd" to that
> invocation and Emacs happens to be 29 you'll have nice LSP
> for the first time in many years.  Personally, for _my_ quick
> hacks in those situations, I don't bother with M-x package-install
> latest-and-greatest, just as I don't bother bringing my heavy
> config via ssh to that machine.  I can't really, I'm frequently
> tmate'ing with other people there, we have to settle on Emacs -Q
> (+ electric-pair-mode and a few obvious others).

All good reasons. But then you add 'use-package :ensure t' to the mix, 
and something unexpected starts to happen. Maybe expected to you, but 
not to somebody other like you.

Maybe you'll put the recommended use-package form somewhere in the Eglot 
manual, right? It will be published on the web as well. Will it contain 
":ensure t" or not? If not, the users of Emacs 28 might fail to get the 
package installed. If it will, then the offline users might have 
problems, as described previously.

> But your question also has a completely different answer:
> A very good reason I wanted Eglot to be bundled with Emacs is now
> actually being put into question in this very thread.  I wanted
> to simplify Eglot's development as it advances with other :core
> packages.  For example, I wanted to split off its external
> completion style easily into a separate core package, as I did
> with Stefan earlier this year.  I want to be able to add a feature
> to ElDoc and use it in Eglot in the same patch, etc.  That part
> has worked fine.  It wasn't impossible with Eglot in GitHub, but
> it was harder.  Hopefully Eli is not asking for that bliss to end
> I understand hope he's asking to be circumspect in what
> is added, and not do gratuitous things like bump dependencies
> without reason.  I already do that, so no problem.

It seems like most of this could have worked out if Eglot moved to the 
emacs.git repository, but was removed from the distro before packaging? 
Just a thought, not that I advocate for doing that.

> Finally, if you're taking all this in the direction of "I told
> you so, don't you regret bringing Eglot into Emacs 29?", I don't
> blame you.  But I have to say: no, not yet.  I think having the
> beginnings of an LSP library in Emacs for major-modes to work
> with is excellent.  I like the attention and the high-standards
> other seasoned developers hold my code up to.  I like the Info
> manual and I like that emacs -Q bla.c -f eglot works just fine.

I like my told-you-so's as much as the next guy (just kidding; 
definitely more than the next guy), but that's not the point I was 
making. Just that it's here now, and it seems odd to avoid using it and 
hurry up to install the next version right away.

> This upgrade situation sucks, is true, and I should have foreseen
> it.  Spent too much time in my emacs-master worktree I guess.
> 
> Who knows, maybe this will all be OK after all?? Half the
> users are using "straight.el" anyway (and I really doubt
> it is hampered by these minutiae about dependencies, seeing
> as MELPA-land is teeming with way more "dependency hell").

I'm pretty sure the users on straight.el are straight-up illusion. It's 
the latest fashion on reddit all right, but it has its own problems, and 
the silent majority are still using the older methods.

Not to say it's a bad project (I'll try to migrate my authored packages 
to it one day, since it seems to make that natural), but for an average 
user? I don't think so.

> If nothing more happens on Emacs 29, which is the most likely
> outcome, I think I will just start recommending "straight.el"
> more, maybe learn it myself.  I had problems with it in the
> past but it's probably matured now.

I would suggest to start recommending ways to perform an upgrade 
explicitly (somewhere in the README, in the manual, and so on). Like 
'M-x package-upgrade', if we manage to get it fixed in Emacs 29.

And, okay, having a separate section in README about Emacs 29 might suck 
at first, but then the majority of the users move to it, and support 
will get easier, more uniform.

>> So... I understand the problem, but I think we shouldn't change the
>> functions in a way that makes them conflict with documentation or with
>> each other.
> 
> Does the latest patch introduce any such conflict besides the
> "no internet" case you mentioned?  Not that it's really a
> conflict IMO: it's just what Emacs 28 did with Eglot AFAICT.

That's the only hard breakage I managed to come up with. The others are 
of logical kind (functions and commands doing things somewhat against 
their documented behavior), but that's also not so great because people 
like their tools to make sense.

To name an example: 'package-install' will upgrade Eglot even though 
(package-installed-p 'eglot) already returns t.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 2:13 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> >> because the package can't be installed (no connection). Should they
> >> remove ":ensure t"? Perhaps. But the documentation says that that option
> >> checks that the package is "installed automatically if not already
> >> present on your system". Seems legit, right? Why would startup fail when
> >> Eglot is already present on the system?
> >
> > Because we want it to do the same it did in Emacs 28?  But I see what
> > you mean.  Maybe the :ensure for packages in that particular subclass
> > should mean "try a bit harder to upgrade these, but don't blow up if
> > you can't -- just try next time".
>
> ... maybe.

I just looked in straight.el.  I haven't actually tried it, but I noticed
that the only place where it cares about if a package is built-in or not is
in straight--convert-recipe.  And to my (pleasant?) surprise, it is using
it to decide whether to report an error or not if the package is not found.
If it's builtin, the error it not thrown.

> Maybe you'll put the recommended use-package form somewhere in the Eglot
> manual, right? It will be published on the web as well. Will it contain
> ":ensure t" or not? If not, the users of Emacs 28 might fail to get the
> package installed. If it will, then the offline users might have
> problems, as described previously.

To be honest, I'm thinking of either not recommending a package manager
at all, or just recommending a third-party one that I think rhymes
with what Eglot users expect.  Might very well be straight.el or
elpaca.el.  One things is for sure, my previous recommendation
of "use package.el" is not good.

> > to ElDoc and use it in Eglot in the same patch, etc.  That part
> > has worked fine.  It wasn't impossible with Eglot in GitHub, but
> > it was harder.  Hopefully Eli is not asking for that bliss to end
> > I understand hope he's asking to be circumspect in what
> > is added, and not do gratuitous things like bump dependencies
> > without reason.  I already do that, so no problem.
>
> It seems like most of this could have worked out if Eglot moved to the
> emacs.git repository, but was removed from the distro before packaging?
> Just a thought, not that I advocate for doing that.

Then emacs -Q bla.py -f eglot would not work.

And major modes in Emacs could not use eglot.el as a library.

> > Finally, if you're taking all this in the direction of "I told
> > you so, don't you regret bringing Eglot into Emacs 29?", I don't
> > blame you.  But I have to say: no, not yet.  I think having the
> > beginnings of an LSP library in Emacs for major-modes to work
> > with is excellent.  I like the attention and the high-standards
> > other seasoned developers hold my code up to.  I like the Info
> > manual and I like that emacs -Q bla.c -f eglot works just fine.
>
> I like my told-you-so's as much as the next guy (just kidding;
> definitely more than the next guy), but that's not the point I was
> making. Just that it's here now, and it seems odd to avoid using it and
> hurry up to install the next version right away.

I agree.  I don't want users to hurry, but I don't want them to not
hurry either :-).  It depends on what they want.  Eglot 1.12.29 in
Emacs 29 is pretty good.

> > This upgrade situation sucks, is true, and I should have foreseen
> > it.  Spent too much time in my emacs-master worktree I guess.
> >
> > Who knows, maybe this will all be OK after all?? Half the
> > users are using "straight.el" anyway (and I really doubt
> > it is hampered by these minutiae about dependencies, seeing
> > as MELPA-land is teeming with way more "dependency hell").
>
> I'm pretty sure the users on straight.el are straight-up illusion. It's
> the latest fashion on reddit all right, but it has its own problems, and
> the silent majority are still using the older methods.
>
> Not to say it's a bad project (I'll try to migrate my authored packages
> to it one day, since it seems to make that natural), but for an average
> user? I don't think so.

I don't know.  I see a lot of people using it, and have seen that
for a long time.  The average user googles things and lands on reddits
hits.  ChatGPT teaches you "straight.el" too (if you ask it).

> > If nothing more happens on Emacs 29, which is the most likely
> > outcome, I think I will just start recommending "straight.el"
> > more, maybe learn it myself.  I had problems with it in the
> > past but it's probably matured now.
>
> I would suggest to start recommending ways to perform an upgrade
> explicitly (somewhere in the README, in the manual, and so on). Like
> 'M-x package-upgrade', if we manage to get it fixed in Emacs 29.

That's a big if.  My worry here is how to clearly control this
messaging, especially when dealing with cleanly deterministic
bug reports.  I have to know exactly the version that he user
is running.  Recently, I've become adept at doing:

HOME=`mktemp -d` emacs -l recipe.el

It's a very good way to establish sanity.  And until now recipe.el
could have just `(package-install 'eglot)` and I would know exactly
what packages the user has installed.  The answer will now
be different in Emacs 28 vs Emacs 29.  Mind you, I will still
know, of course, but the thing installed on Emacs 28 will be
wildly different than the one installed on Emacs 29.  And,
as I said before and everybody understands, it will get worse with
time.  So what form to give users??

I suggested adding a 'eglot-update' function to eglot.el only --
the only code I normally have control over.  A four-line long function
that I would put it Emacs 29.  It would have this property of
being absolutely consistent in whatever Emacs you're on.
But Eli called it "the worst of all worlds" and flatly refused it.
Maybe he could reconsider, who knows, if all this writing that
has happened since has provided some more perspective about
the problems.

> And, okay, having a separate section in README about Emacs 29 might suck
> at first, but then the majority of the users move to it, and support
> will get easier, more uniform.

Yeah, I guess.  BTW the tea leaves and some comments in the Philip-Eli
interaction tell me that the behaviour is going to again be different in
Emacs 30.  That will suck for a longer time.

> That's the only hard breakage I managed to come up with. The others are
> of logical kind (functions and commands doing things somewhat against
> their documented behavior), but that's also not so great because people
> like their tools to make sense.
>
> To name an example: 'package-install' will upgrade Eglot even though
> (package-installed-p 'eglot) already returns t.

Even with today's patch in Emacs 29 that already happens if you tweak
the new "heavy-handed" variable.  So I guess that isn't a problem.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 04:49, João Távora wrote:

>> Not to say it's a bad project (I'll try to migrate my authored packages
>> to it one day, since it seems to make that natural), but for an average
>> user? I don't think so.
> 
> I don't know.  I see a lot of people using it, and have seen that
> for a long time.  The average user googles things and lands on reddits
> hits.  ChatGPT teaches you "straight.el" too (if you ask it).

I've had problems reported because of outdated straight recipes on 
users' machines. It seems like a fine tool for a hacker, though.

>> I would suggest to start recommending ways to perform an upgrade
>> explicitly (somewhere in the README, in the manual, and so on). Like
>> 'M-x package-upgrade', if we manage to get it fixed in Emacs 29.
> 
> That's a big if.  My worry here is how to clearly control this
> messaging, especially when dealing with cleanly deterministic
> bug reports.  I have to know exactly the version that he user
> is running.  Recently, I've become adept at doing:
> 
> HOME=`mktemp -d` emacs -l recipe.el
> 
> It's a very good way to establish sanity.  And until now recipe.el
> could have just `(package-install 'eglot)` and I would know exactly
> what packages the user has installed.  The answer will now
> be different in Emacs 28 vs Emacs 29.  Mind you, I will still
> know, of course, but the thing installed on Emacs 28 will be
> wildly different than the one installed on Emacs 29.  And,
> as I said before and everybody understands, it will get worse with
> time.  So what form to give users??

If we go for the pinning approach, as Jim suggested, and the fix for 
that one makes it in in time, the form could like like

   (use-package eglot :ensure t :pin gnu)

and hopefully work across 28, 29 and later.

>> To name an example: 'package-install' will upgrade Eglot even though
>> (package-installed-p 'eglot) already returns t.
> 
> Even with today's patch in Emacs 29 that already happens if you tweak
> the new "heavy-handed" variable.  So I guess that isn't a problem.

I'm not sure I like that one either.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 7:07 PM Eli Zaretskii <eliz@gnu.org> wrote:

> It has similar problems: it will automatically update packages
> mentioned in package--safely-upgradeable-builtins, which might not be
> what users want for built-in packages.

It is what they want, by definition, this is why I named it
"safely-upgradeable-builtins".  These are the users:

  Specifically, users of Emacs 28 and older, who had Eglot installed,
  and expect Eglot to be automatically updated upon Emacs startup
  whenever a new Eglot version is available, will now have their
  expectations broken after they upgrade to Emacs 29, because Eglot is
  now a built-in package, and package.el won't by default upgrade a
  built-in package.

Recognize this writing? It is yours!

> You assume that everyone will
> want Eglot and use-package automatically updated, but this assumption
> has no real basis.

First, of course it has real statistical basis! Didn't I send you
links to tens and tens of issues were users reported their configurations
and one can actually see what users are doing to install Eglot?

Secondly, it has the theoretical basis of what you wrote yourself
barely 1 hour ago!  It shows you understand the problem that is
new in Emacs 29.

Using your language, we want to not "break those user's expectations".
if we can.  And we can, if you want to.  You want to, right? You want to
break as few user's expectations as possible, ideally 0.

And the code does exactly that! It avoids bothering that set of
users while also avoiding bothering the other set of users that
you mentioned.

And, for good measure, the set of users who had Eglot installed
and expect Eglot NOT to be updated when package-install is found
is the empty set.  Surely this is evident.

So there's no "dilemma".  There is rather some kind of spectacular
misunderstanding here.  There has to be, because I'm drawing these
conclusions from nothing more than elementary facts from set theory

> Plus, it adds to the maintenance burden of maintaining this
> (internal? not a defcustom??) variable for good.

We can make it a defcustom if you want.  That's exactly
Jim's idea.  I was just trying to make the simplest thing
possible.  This is just for Emacs 29.  You asked for little
code and this is much less code (less cyclomatic complexity,
etc) than what you eventually accepted.  So I was just trying to
cover that front too.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 20:15:20 +0100
> Cc: arne_bab@web.de, jporterbugs@gmail.com, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> On Wed, Apr 19, 2023 at 7:07 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > It has similar problems: it will automatically update packages
> > mentioned in package--safely-upgradeable-builtins, which might not be
> > what users want for built-in packages.
> 
> It is what they want, by definition, this is why I named it
> "safely-upgradeable-builtins".  These are the users:
> 
>   Specifically, users of Emacs 28 and older, who had Eglot installed,
>   and expect Eglot to be automatically updated upon Emacs startup
>   whenever a new Eglot version is available, will now have their
>   expectations broken after they upgrade to Emacs 29, because Eglot is
>   now a built-in package, and package.el won't by default upgrade a
>   built-in package.
> 
> Recognize this writing? It is yours!

Yes.  Of course, you conveniently omitted the next paragraph I wrote,
which described a different group of users, whose expectations would
be broken by the changes you proposed.  That was also "my writing".

> > You assume that everyone will
> > want Eglot and use-package automatically updated, but this assumption
> > has no real basis.
> 
> First, of course it has real statistical basis! Didn't I send you
> links to tens and tens of issues were users reported their configurations
> and one can actually see what users are doing to install Eglot?

Since when "tens" and "everyone" are the same thing?

> Secondly, it has the theoretical basis of what you wrote yourself
> barely 1 hour ago!  It shows you understand the problem that is
> new in Emacs 29.

Of course, I understand the problem.  But understanding the problem
doesn't mean agreement with the solutions you propose, or any other
solution, for that matter.  An acceptable solution should solve the
problem without causing other problems, and in this case it also must
solve the problem in a safe enough manner to be eligible for Emacs
29.1.

> Using your language, we want to not "break those user's expectations".
> if we can.  And we can, if you want to.  You want to, right? You want to
> break as few user's expectations as possible, ideally 0.
> 
> And the code does exactly that! It avoids bothering that set of
> users while also avoiding bothering the other set of users that
> you mentioned.
> 
> And, for good measure, the set of users who had Eglot installed
> and expect Eglot NOT to be updated when package-install is found
> is the empty set.  Surely this is evident.
> 
> So there's no "dilemma".  There is rather some kind of spectacular
> misunderstanding here.  There has to be, because I'm drawing these
> conclusions from nothing more than elementary facts from set theory

It is clear that you like the solution you proposed, and see no
problems with it.  But I disagree, and at this point I have explained
my disagreement enough times.  If you still think I'm wrong, please,
just let's leave this disagreement as it is.  Repeating the same
arguments over and over again just wastes bandwidth and our time.

> > Plus, it adds to the maintenance burden of maintaining this
> > (internal? not a defcustom??) variable for good.
> 
> We can make it a defcustom if you want.  That's exactly
> Jim's idea.  I was just trying to make the simplest thing
> possible.  This is just for Emacs 29.  You asked for little
> code and this is much less code (less cyclomatic complexity,
> etc) than what you eventually accepted.  So I was just trying to
> cover that front too.

I said I'm okay with a defcustom if its default value is nil.

Of course, we already added a defcustom to allow built-in packages to
be updated, so I'm not sure another one is needed.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 10:38 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > It is what they want, by definition, this is why I named it
> > "safely-upgradeable-builtins".  These are the users:
> >
> >   Specifically, users of Emacs 28 and older, who had Eglot installed,
> >   and expect Eglot to be automatically updated upon Emacs startup
> >   whenever a new Eglot version is available, will now have their
> >   expectations broken after they upgrade to Emacs 29, because Eglot is
> >   now a built-in package, and package.el won't by default upgrade a
> >   built-in package.
> >
> > Recognize this writing? It is yours!
>
> Yes.  Of course, you conveniently omitted the next paragraph I wrote,
> which described a different group of users, whose expectations would
> be broken by the changes you proposed.  That was also "my writing".

The code I'm trying to write should appease both.  But you should
help characterize these users.

> > > You assume that everyone will
> > > want Eglot and use-package automatically updated, but this assumption
> > > has no real basis.
> >
> > First, of course it has real statistical basis! Didn't I send you
> > links to tens and tens of issues were users reported their configurations
> > and one can actually see what users are doing to install Eglot?
>
> Since when "tens" and "everyone" are the same thing?

Come on, you know that "everyone" is impossible to prove.  Aren't tens
(actually hundreds, I think) a good data point.  Can you show even
one issue where someone was surprised/harmed by furtive unintented
updates of dependencies?

> > Secondly, it has the theoretical basis of what you wrote yourself
> > barely 1 hour ago!  It shows you understand the problem that is
> > new in Emacs 29.
>
> Of course, I understand the problem.  But understanding the problem
> doesn't mean agreement with the solutions you propose, or any other
> solution, for that matter.  An acceptable solution should solve the
> problem without causing other problems, and in this case it also must
> solve the problem in a safe enough manner to be eligible for Emacs
> 29.1.

And that's what I'm looking for.

> > Using your language, we want to not "break those user's expectations".
> > if we can.  And we can, if you want to.  You want to, right? You want to
> > break as few user's expectations as possible, ideally 0.
> >
> > And the code does exactly that! It avoids bothering that set of
> > users while also avoiding bothering the other set of users that
> > you mentioned.
> >
> > And, for good measure, the set of users who had Eglot installed
> > and expect Eglot NOT to be updated when package-install is found
> > is the empty set.  Surely this is evident.
> >
> > So there's no "dilemma".  There is rather some kind of spectacular
> > misunderstanding here.  There has to be, because I'm drawing these
> > conclusions from nothing more than elementary facts from set theory
>
> It is clear that you like the solution you proposed, and see no
> problems with it.  But I disagree, and at this point I have explained
> my disagreement enough times.

No you haven't, sorry.  You've not said "João, if your patch goes in,
at least user X will have this specific problem Y when she does Z, and
I think Y is bad because reasons".  Dmitry is doing that, critiquing
actual code.  I don't think that's time wasted: it's the only way the
discussion can advance.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 10:48:03 +0100
> Cc: arne_bab@web.de, jporterbugs@gmail.com, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> > > Recognize this writing? It is yours!
> >
> > Yes.  Of course, you conveniently omitted the next paragraph I wrote,
> > which described a different group of users, whose expectations would
> > be broken by the changes you proposed.  That was also "my writing".
> 
> The code I'm trying to write should appease both.  But you should
> help characterize these users.

I have.

> > > > You assume that everyone will
> > > > want Eglot and use-package automatically updated, but this assumption
> > > > has no real basis.
> > >
> > > First, of course it has real statistical basis! Didn't I send you
> > > links to tens and tens of issues were users reported their configurations
> > > and one can actually see what users are doing to install Eglot?
> >
> > Since when "tens" and "everyone" are the same thing?
> 
> Come on, you know that "everyone" is impossible to prove.

I didn't expect you to prove it.  My point is that your proposed
solution is only correct if "everyone" indeed wants to update Eglot,
and if not, your solution helps one group and breaks another.  I
cannot agree to such lopsided solutions.

> Aren't tens (actually hundreds, I think) a good data point.

Not enough to justify breaking the rest.

> Can you show even one issue where someone was surprised/harmed by
> furtive unintented updates of dependencies?

You just heard one of them in this thread.

> > It is clear that you like the solution you proposed, and see no
> > problems with it.  But I disagree, and at this point I have explained
> > my disagreement enough times.
> 
> No you haven't, sorry.

Surer, I did.  You just don't like the explanation, and so you pretend
it doesn't exist.

> I don't think that's time wasted: it's the only way the discussion
> can advance.

I don't think this discussion with you can make any progress.  That
hope was lost long ago, when I first said we should agree to disagree
and leave it at that.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 12:47 PM Eli Zaretskii <eliz@gnu.org> wrote:
> Not enough to justify breaking the rest

But there is no characterization of this rest.

> > Can you show even one issue where someone was surprised/harmed by
> > furtive unintented updates of dependencies?
>
> You just heard one of them in this thread.

I must have missed this. Maybe my spam filter?  Who is this user
and can I interview them to understand how my patch would hurt
them?  Dmitry? Arne? Jim?  The only person who actively looked
at my patch and read it and explained real-world implications
was Dmitry.

> I don't think this discussion with you can make any progress.
> That hope was lost long ago

I'm beginning to agree.  I've also kind of lost hope.  I'll
take the damage and cut the losses with eglot-update.
Thank you for you efforts anyway.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 20 Apr 2023 13:00:23 +0100
> Cc: arne_bab@web.de, jporterbugs@gmail.com, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> Thank you for you efforts anyway.

I cannot thank you, unfortunately.  Not in this discussion.  I'm
deeply hurt by your conduct.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Thu, Apr 20, 2023 at 1:16 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 20 Apr 2023 13:00:23 +0100
> > Cc: arne_bab@web.de, jporterbugs@gmail.com, dmitry@gutov.dev,
> >       emacs-devel@gnu.org
> >
> > Thank you for you efforts anyway.
>
> I cannot thank you, unfortunately.  Not in this discussion.  I'm
> deeply hurt by your conduct.

I spent a lot of hours in this discussion, conducted experiments,
proposed almost countless patches, interviewed people, etc.

So I don't understand but that's OK, can't argue feelings.
I'm very sorry I hurt you somehow, is all I can say.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[John Yates]

On Wed, Apr 19, 2023 at 1:03 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Specifically, users of Emacs 28 and older, who had Eglot installed,
> and expect Eglot to be automatically updated upon Emacs startup
> whenever a new Eglot version is available, will now have their
> expectations broken after they upgrade to Emacs 29, because Eglot is
> now a built-in package, and package.el won't by default upgrade a
> built-in package.
>
> OTOH, the proposal to change package.el so it automatically updates
> built-in packages as well would break the workflows of people who
> don't expect built-in packages to be updated: they would suddenly see
> packages like ElDoc or Xref or Project (and others) being updated
> automatically at startup instead of staying at the version shipped
> with Emacs.

Full disclosure: jumping in without having read much of the prior thread.

Perhaps package.el should allow a user to declare whether (s)he wants
a stable experience or a "rolling release" experience.  In that way the
issue is no longer a function of whether a package is :core or not.

My initial thought was that this would be a single, global toggle.  But I can
also imagine more granular choices (stable, rolling or unspecified) per
package archive and per individual package.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 6:35 PM John Yates <john@yates-sheets.org> wrote:

> My initial thought was that this would be a single, global toggle.
> But I can also imagine more granular choices (stable, rolling or
> unspecified) per package archive and per individual package.

Logical! That global toggle exists now, added very recently.

But, exactly as you say, it should be more granular.  And by
default the value of the switch for Eglot should be "yes, install and
upgrade its dependencies like it happened in Emacs 28".  Whether that is
because Eglot is listed under a special category or just listed
specifically, I don't think is very important.

I already proposed this (but without the toggle) before the
heavy-handed global toggle that forcibly breaks a part of
the users was introduced.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: John Yates <john@yates-sheets.org>
> Date: Wed, 19 Apr 2023 13:35:36 -0400
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>, joaotavora@gmail.com, jporterbugs@gmail.com, 
> 	dmitry@gutov.dev, emacs-devel@gnu.org
> 
> Perhaps package.el should allow a user to declare whether (s)he wants
> a stable experience or a "rolling release" experience.  In that way the
> issue is no longer a function of whether a package is :core or not.

I agree.  But such a change in package.el was impossible when the
pretest already started.  It should, however, be a necessary feature
of package.el.

> My initial thought was that this would be a single, global toggle.  But I can
> also imagine more granular choices (stable, rolling or unspecified) per
> package archive and per individual package.

There's a lot of new turf here to be explored.  I hope we get it done
in time for Emacs 30.  But it wasn't a reasonable alternative for
Emacs 29, because this issue was raised too late, and I don't think it
would be TRT to delay Emacs 29.1 by another 6 months or so.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 10:03 AM, Eli Zaretskii wrote:
> Specifically, users of Emacs 28 and older, who had Eglot installed,
> and expect Eglot to be automatically updated upon Emacs startup
> whenever a new Eglot version is available, will now have their
> expectations broken after they upgrade to Emacs 29, because Eglot is
> now a built-in package, and package.el won't by default upgrade a
> built-in package.

I have some thoughts on this but before I go further, I want to be sure 
I understand the problem. How is the user upgrading their packages in 
this scenario? I installed Eglot 1.14 on Emacs 28 and then tried loading 
up Emacs 29. Then I did the following:

1. Add gnu-devel to 'package-archives' (this way, I can be sure there's 
a newer Eglot to upgrade to in one of the archives)
2. M-x list-packages
3. U  ;; package-menu-mark-upgrades
4. x  ;; package-menu-execute

When Emacs tells me what packages it will upgrade, Eglot is in the list. 
However, ERC (which is in ELPA, but I didn't install via package.el) is 
*not* in the list. Isn't this the behavior we want?[1]

[1] For now, ignore the part where this upgrade changes where we get 
Eglot from GNU ELPA to GNU-devel ELPA. (I think this should be fixed, 
but I don't think it poses an immediate issue for Eglot's problem, since 
Emacs doesn't enable GNU-devel by default.)

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 11:04:50 -0700
> Cc: joaotavora@gmail.com, dmitry@gutov.dev, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 4/19/2023 10:03 AM, Eli Zaretskii wrote:
> > Specifically, users of Emacs 28 and older, who had Eglot installed,
> > and expect Eglot to be automatically updated upon Emacs startup
> > whenever a new Eglot version is available, will now have their
> > expectations broken after they upgrade to Emacs 29, because Eglot is
> > now a built-in package, and package.el won't by default upgrade a
> > built-in package.
> 
> I have some thoughts on this but before I go further, I want to be sure 
> I understand the problem. How is the user upgrading their packages in 
> this scenario? I installed Eglot 1.14 on Emacs 28 and then tried loading 
> up Emacs 29. Then I did the following:
> 
> 1. Add gnu-devel to 'package-archives' (this way, I can be sure there's 
> a newer Eglot to upgrade to in one of the archives)
> 2. M-x list-packages
> 3. U  ;; package-menu-mark-upgrades
> 4. x  ;; package-menu-execute
> 
> When Emacs tells me what packages it will upgrade, Eglot is in the list. 
> However, ERC (which is in ELPA, but I didn't install via package.el) is 
> *not* in the list. Isn't this the behavior we want?[1]

AFAIU, this is not the scenario that João was bothered about.  But I
let him respond.

But if this is the scenario, then there's no problem, AFAIU what you
are saying.  So what exactly would you like to add to this discussion?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 11:34 AM, Eli Zaretskii wrote:
>> Date: Wed, 19 Apr 2023 11:04:50 -0700
>> Cc: joaotavora@gmail.com, dmitry@gutov.dev, emacs-devel@gnu.org
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> 1. Add gnu-devel to 'package-archives' (this way, I can be sure there's
>> a newer Eglot to upgrade to in one of the archives)
>> 2. M-x list-packages
>> 3. U  ;; package-menu-mark-upgrades
>> 4. x  ;; package-menu-execute
>>
>> When Emacs tells me what packages it will upgrade, Eglot is in the list.
>> However, ERC (which is in ELPA, but I didn't install via package.el) is
>> *not* in the list. Isn't this the behavior we want?[1]
> 
> AFAIU, this is not the scenario that João was bothered about.  But I
> let him respond.
> 
> But if this is the scenario, then there's no problem, AFAIU what you
> are saying.  So what exactly would you like to add to this discussion?

Two main things (once I hear back from João to confirm):

1) If there are any package-upgrade actions that *don't* work in the way 
I described, we should fix them, using the behavior of 
'package-menu-mark-upgrades' for guidance. As far as I can tell, that's 
the behavior everyone wants, but there could be other scenarios where it 
does something else.

2) More-generally, there's the question of "stability gradations". 
Elsewhere, you suggested listing these in the *Packages* buffer with 
values like "alpha", "current", "stable", etc. We can already do 
something similar to this with additional package archives (e.g. GNU 
ELPA vs GNU-devel ELPA). However, package.el doesn't automatically keep 
track of which channel you used to install a package, so you have to go 
through a fair amount of extra effort to pin your packages to particular 
release channels.

I think (1) is the immediate concern though, and it might be best to 
have a resolution for that before going too far into general solutions 
like (2).

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 12:35:15 -0700
> Cc: arne_bab@web.de, joaotavora@gmail.com, dmitry@gutov.dev,
>  emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> > But if this is the scenario, then there's no problem, AFAIU what you
> > are saying.  So what exactly would you like to add to this discussion?
> 
> Two main things (once I hear back from João to confirm):
> 
> 1) If there are any package-upgrade actions that *don't* work in the way 
> I described, we should fix them, using the behavior of 
> 'package-menu-mark-upgrades' for guidance. As far as I can tell, that's 
> the behavior everyone wants, but there could be other scenarios where it 
> does something else.
> 
> 2) More-generally, there's the question of "stability gradations". 
> Elsewhere, you suggested listing these in the *Packages* buffer with 
> values like "alpha", "current", "stable", etc. We can already do 
> something similar to this with additional package archives (e.g. GNU 
> ELPA vs GNU-devel ELPA). However, package.el doesn't automatically keep 
> track of which channel you used to install a package, so you have to go 
> through a fair amount of extra effort to pin your packages to particular 
> release channels.
> 
> I think (1) is the immediate concern though, and it might be best to 
> have a resolution for that before going too far into general solutions 
> like (2).

These are serious issues we need to discuss and solve.  But they will
probably require significant changes in package.el, and so are stuff
for Emacs 30, not for the emacs-29 branch.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1821 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:

>> But there are too many different tasks to actually check them all every
>> time I update. And I usually don’t choose to do an Emacs update, but
>> just update when my distro ships the new version.
>
> I don't think anyone will argue that breaking changes are good.

I’ve seen that more than once, but I’m glad that that’s not on the table :-)

> Specifically, users of Emacs 28 and older, who had Eglot installed,
> and expect Eglot to be automatically updated upon Emacs startup
> whenever a new Eglot version is available, will now have their
> expectations broken after they upgrade to Emacs 29, because Eglot is
> now a built-in package, and package.el won't by default upgrade a
> built-in package.
…
> So there's a dilemma here: which of the two groups of users to break?

Not updating eglot until the next Emacs release shouldn’t cause breakage
in any other packages, right?

Except if a more modern eglot is a dependency of a non-built-in package.

I think that’s what I would prefer: I would treat being pulled into
Emacs as a stabilization step that switches the package from being on
the latest version to being at the version in Emacs or the minimum
version required by dependencies — if that version is higher than the
version in Emacs. Basically minimize the distance from the Emacs
release.

But that’s just my position — thank you for working to find a solution
that works best for most!

> And that is the real issue here, at least from my POV.

Thank you for the clarification! I had thought that I had read enough of
the thread to understand the context … thank you for pulling me in
nonetheless!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, dmitry@gutov.dev,
>  emacs-devel@gnu.org
> Date: Wed, 19 Apr 2023 21:40:35 +0200
> 
> > Specifically, users of Emacs 28 and older, who had Eglot installed,
> > and expect Eglot to be automatically updated upon Emacs startup
> > whenever a new Eglot version is available, will now have their
> > expectations broken after they upgrade to Emacs 29, because Eglot is
> > now a built-in package, and package.el won't by default upgrade a
> > built-in package.
> …
> > So there's a dilemma here: which of the two groups of users to break?
> 
> Not updating eglot until the next Emacs release shouldn’t cause breakage
> in any other packages, right?

No, it shouldn't.  With the obvious exception of the breakage that is
already part of the current Emacs release, which we somehow failed to
detect and fix before releasing it.

> Except if a more modern eglot is a dependency of a non-built-in package.

Right.

> I think that’s what I would prefer: I would treat being pulled into
> Emacs as a stabilization step that switches the package from being on
> the latest version to being at the version in Emacs or the minimum
> version required by dependencies — if that version is higher than the
> version in Emacs. Basically minimize the distance from the Emacs
> release.

AFAIU, that is what will happen with Emacs 29 when it is released.
But things might change in Emacs 30, as we are currently discussing
what needs to be done to better support updating core packages.

Re: Stability of core packages

[emacs]

I have been following this thread from a meta perspective.

The essence of the difficulty is that we are
following an established pattern that has various
weaknesses and we are now trying to adjust that pattern.

What I am suggesting breaks the existing
model. Let me start with some context.

Because we use git for everything and because it
is possible to take complete git snap shots of all
relevant packages, it is possible to easily
provide stability for the user based on snapshots.

Consistent upgrades are then possible for new
consistent git snapshots. So, based on the snapshots
the packages are not stale and remain consistent
while evolving.

This is what doomemacs does. For every package,
there is a git tag which doomemacs keeps with the
adoption of the package. The tags are guaranteed
to be consistent. With Blee (ByStar Libre Emacs
Environment) I do the same but instead of keeping
that tag with the packages, I keep central
manifests for emacs releases. So, emacs-28.1 has
its own packages manifest which can be upgraded
from time to time. All upgrades go to the
appropriate repo and get the appropriate tag based
on the manifest. I do this across all emacs
package archives -- which I see as collections of
git repositories.

If you want to go with something like this, you
have to revisit the package retrieval model.

For every release of emacs, you maintain a
separate manifest. And you keep updating that at
new releases and even in between releases. Since
all emacs archives are git based, it is possible
to apply this strategy to all of them -- where
each take care of its own consistency.

In practice, this has already happened for layers
that sit on top of emacs (doomemacs, blee). It is
a matter of adopting the higher layer model in the
core. There are already many variations on the
model of packages.el.

If you want to go with this model, much needs to
be revisited. But much will be gained.

Again, these comments only deal with the meta
topic not the at hand  Eglot topic. But, it shows
how eglot upgrades could have rolled back to
emacs-28.

...Mohsen


Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
>> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, dmitry@gutov.dev,
>>  emacs-devel@gnu.org
>> Date: Wed, 19 Apr 2023 21:40:35 +0200
>>
>> > Specifically, users of Emacs 28 and older, who had Eglot installed,
>> > and expect Eglot to be automatically updated upon Emacs startup
>> > whenever a new Eglot version is available, will now have their
>> > expectations broken after they upgrade to Emacs 29, because Eglot is
>> > now a built-in package, and package.el won't by default upgrade a
>> > built-in package.
>> …
>> > So there's a dilemma here: which of the two groups of users to break?
>>
>> Not updating eglot until the next Emacs release shouldn’t cause breakage
>> in any other packages, right?
>
> No, it shouldn't.  With the obvious exception of the breakage that is
> already part of the current Emacs release, which we somehow failed to
> detect and fix before releasing it.
>
>> Except if a more modern eglot is a dependency of a non-built-in package.
>
> Right.
>
>> I think that’s what I would prefer: I would treat being pulled into
>> Emacs as a stabilization step that switches the package from being on
>> the latest version to being at the version in Emacs or the minimum
>> version required by dependencies — if that version is higher than the
>> version in Emacs. Basically minimize the distance from the Emacs
>> release.
>
> AFAIU, that is what will happen with Emacs 29 when it is released.
> But things might change in Emacs 30, as we are currently discussing
> what needs to be done to better support updating core packages.

Re: Stability of core packages

[Eli Zaretskii]

> From: emacs@mohsen.1.banan.byname.net
> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>,  joaotavora@gmail.com,
>   jporterbugs@gmail.com,  dmitry@gutov.dev,  emacs-devel@gnu.org
> Date: Fri, 28 Apr 2023 22:21:46 -0700
> 
> 
> I have been following this thread from a meta perspective.

Thanks for chiming in.

> Because we use git for everything and because it
> is possible to take complete git snap shots of all
> relevant packages, it is possible to easily
> provide stability for the user based on snapshots.
> 
> Consistent upgrades are then possible for new
> consistent git snapshots. So, based on the snapshots
> the packages are not stale and remain consistent
> while evolving.
> 
> This is what doomemacs does. For every package,
> there is a git tag which doomemacs keeps with the
> adoption of the package. The tags are guaranteed
> to be consistent. With Blee (ByStar Libre Emacs
> Environment) I do the same but instead of keeping
> that tag with the packages, I keep central
> manifests for emacs releases. So, emacs-28.1 has
> its own packages manifest which can be upgraded
> from time to time. All upgrades go to the
> appropriate repo and get the appropriate tag based
> on the manifest. I do this across all emacs
> package archives -- which I see as collections of
> git repositories.
> 
> If you want to go with something like this, you
> have to revisit the package retrieval model.
> 
> For every release of emacs, you maintain a
> separate manifest. And you keep updating that at
> new releases and even in between releases. Since
> all emacs archives are git based, it is possible
> to apply this strategy to all of them -- where
> each take care of its own consistency.

I've read this several times, and I admit I still don't really
understand the suggestion.  Maybe it's because you are using some
terminology without explaining it in enough details, perhaps because
you assume prior knowledge of what it means.

Here are some questions for which I didn't find answers, and which
precluded me from following your suggestions:

  . what is a "git snapshot"? what does it include in the context of
    this discussion?
  . what is meant by "upgrades for new consistent git snapshots"? how
    is such an upgrade done, and how does it work in practice?
  . what are those git tags which doomemacs keeps with the packages?
    what does each such tag indicate, and how are these tags
    maintained and updated?
  . what does it mean for a git tag to be "consistent"?
  . what are those "central manifests" you describe for Blee? what
    does each manifest contain and how is it used?
  . wrt your proposal of keeping separate manifests for each Emacs
    release, how will a manifest for a specific release be updated,
    and where will it be kept and made available for users?

Re: Stability of core packages

[Mohsen BANAN]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: emacs@mohsen.1.banan.byname.net
>> Cc: "Dr. Arne Babenhauserheide" <arne_bab@web.de>,  joaotavora@gmail.com,
>>   jporterbugs@gmail.com,  dmitry@gutov.dev,  emacs-devel@gnu.org
>> Date: Fri, 28 Apr 2023 22:21:46 -0700
>>
>>
>> I have been following this thread from a meta perspective.
>
> Thanks for chiming in.
>
>> Because we use git for everything and because it
>> is possible to take complete git snap shots of all
>> relevant packages, it is possible to easily
>> provide stability for the user based on snapshots.
>>
>> Consistent upgrades are then possible for new
>> consistent git snapshots. So, based on the snapshots
>> the packages are not stale and remain consistent
>> while evolving.
>>
>> This is what doomemacs does. For every package,
>> there is a git tag which doomemacs keeps with the
>> adoption of the package. The tags are guaranteed
>> to be consistent. With Blee (ByStar Libre Emacs
>> Environment) I do the same but instead of keeping
>> that tag with the packages, I keep central
>> manifests for emacs releases. So, emacs-28.1 has
>> its own packages manifest which can be upgraded
>> from time to time. All upgrades go to the
>> appropriate repo and get the appropriate tag based
>> on the manifest. I do this across all emacs
>> package archives -- which I see as collections of
>> git repositories.
>>
>> If you want to go with something like this, you
>> have to revisit the package retrieval model.
>>
>> For every release of emacs, you maintain a
>> separate manifest. And you keep updating that at
>> new releases and even in between releases. Since
>> all emacs archives are git based, it is possible
>> to apply this strategy to all of them -- where
>> each take care of its own consistency.
>
> I've read this several times, and I admit I still don't really
> understand the suggestion.  Maybe it's because you are using some
> terminology without explaining it in enough details, perhaps because
> you assume prior knowledge of what it means.

Sorry about that.

In fact, that is the case. I was assuming that you
and others would be familiar with the lessons
learned from the "straight.el" and doom emacs
efforts over the past many years. My apologies.

To better set the context, let me start with a
quote from:
https://github.com/doomemacs/doomemacs/blob/master/docs/faq.org

-----------
Why does Doom use straight.el and not package.el?

package.el simply doesn’t cut it. Its flaws become
apparent the more packages you manage, the more
complex your config becomes, and how often those
packages see updates:

- No Disaster recovery
...

- Rolling release or bust

package.el installs the latest version of every
package, or none at all. There’s no rollback, no
pinning to a stable ref, and no git history to
peek into. It offers little reproducibility;
wiping/losing/copying your config becomes a gamble
if you aren’t constantly backing up your packages
(then recompiling them if you’re up/downgrading
Emacs). melpa-stable was well intentioned, but it
offers no guarantees or standards on how/when
maintainers tag their projects. I’d rather the
user decide for themselves, because they’re the
ones in the trenches.

...

Package management needs to be easier, because
Emacs is hard enough already. Doom fills these
gaps with straight.el’s help, and beyond.
------------

I am not saying that everything that Henrik
Lissner is saying is correct. But I also think
that the model of package.el is inadequate and
that it should be replaced by something like
straight.el.

The readme of
https://github.com/radian-software/straight.el
has a Guiding principles section.

As João Távora observes, straight.el can be a
solution for the eglot use case:

João Távora> Who knows, maybe this will all be OK after all?? Half the
João Távora> users are using "straight.el" anyway (and I really doubt
João Távora> it is hampered by these minutiae about dependencies, seeing
João Távora> as MELPA-land is teeming with way more "dependency hell").

>
> Here are some questions for which I didn't find answers, and which
> precluded me from following your suggestions:
>
>   . what is a "git snapshot"? what does it include in the context of
>     this discussion?

By a "git snapshot" I mean a collection of Git
commit IDs which can be used as pins.

In the doom model, with streight.el and with the
eglot use case, consider:

(package! eglot :pin "e501275e06952889056268dabe08ccd0dbaf23e5")

https://github.com/doomemacs/doomemacs/blob/master/modules/tools/lsp/packages.el

That is an example of a git snapshot.


>   . what is meant by "upgrades for new consistent git snapshots"? how
>     is such an upgrade done, and how does it work in practice?

New collection of Git commit IDs can then become
basis for upgrades.

>   . what are those git tags which doomemacs keeps with the packages?
>     what does each such tag indicate, and how are these tags
>     maintained and updated?

In the context of evolution of say ELPA, I am
suggesting use of the model of streight.el.

And its use based on the manifest model -- not
doom's package adoption model.

For each release of emacs, emacs{28,29,30}
the archive will maintain a manifest-{28,29,30}.

Each of those files enumerates a list of Git
commit IDs for each of the packages in the
archive. An update to the Git commit ID becomes a
basis for a potential upgrade.

>   . what does it mean for a git tag to be "consistent"?

It means that it is known to work well with all
other Git commit IDs in that manifest.

>   . what are those "central manifests" you describe for Blee? what
>     does each manifest contain and how is it used?

See manifest-{28,29,30} above.

>   . wrt your proposal of keeping separate manifests for each Emacs
>     release, how will a manifest for a specific release be updated,
>     and where will it be kept and made available for users?

The manifests are an internal part of the archive.
Users can see them, but they need not.

Maintenance of a manifests is the work of the ELPA
maintainers. They need to be subjected to
regression tests/verification's that guarantee
consistent sets.

I think what I am saying above reflects current
ad-doc practices of many straight.el users and
reflects a natural evolution direction for
package.el.

Eli, if you have not used straight.el, I suggest
experimenting with it. And again sorry about
having been overly cryptic.

...Mohsen

Re: Stability of core packages

[Eli Zaretskii]

> From: Mohsen BANAN <emacs@mohsen.1.banan.byname.net>
> Cc: emacs-devel@gnu.org, João Távora
>  <joaotavora@gmail.com>
> Date: Sat, 29 Apr 2023 14:47:51 -0700
> 
> >
> > I've read this several times, and I admit I still don't really
> > understand the suggestion.  Maybe it's because you are using some
> > terminology without explaining it in enough details, perhaps because
> > you assume prior knowledge of what it means.
> 
> Sorry about that.
> 
> In fact, that is the case. I was assuming that you
> and others would be familiar with the lessons
> learned from the "straight.el" and doom emacs
> efforts over the past many years. My apologies.
> [...]

Thanks.  I think I understand now what you are proposing.

However, this is not what the current discussion thread was about.
You are describing the technical means of tracking the collections of
versions/snapshots/commits of packages that are known to work well
with a given release of Emacs.  By contrast, the issue I raised, which
I think is a prerequisite for selecting any such technical means, was:
what are the levels of "package stability" we want to support, and
what are the criteria for each supported level?  Once we have that
figured out and agreed-upon, we can proceed to discussing how to
support those stability levels from the technical POV.

> Eli, if you have not used straight.el, I suggest
> experimenting with it.

Sorry, I cannot afford that: not enough time left, after all my other
duties here are done.

But I don't think I need to do that in order to understand what you
are proposing.  For that matter, I don't really see a significant
difference between explicit versioning of package releases and using
commits/tags for the same purpose.  After all, a version is just a tag
in the repository, i.e. a label of a commit.  So they are all
equivalent.  And since in practice the only source of stability data
of a package is the maintainer(s) of that package, the reason for
using commits instead of release version tags seems weak to me;
moreover, it assumes some significant team of people unrelated to any
package that keeps track of stability and dependencies of a large
number of packages, something that IMO is impractical for Emacs.

But as I said: these are secondary issues, from my POV.  We are not
yet at a point where these issues are of any practical importance for
our package system.  And one more thing to keep in mind: these issues
are from my POV most important for the so-called "core" packages,
which are those that need to be shipped as part of the Emacs release
tarball, but are developed and maintained outside of emacs.git.  None
of the package systems you mention have ever faced and handled this
situation (with the possible exception of XEmacs, years ago, but we
all know where that went...).

Re: Stability of core packages

[Philip Kaludercic]

Mohsen BANAN <emacs@mohsen.1.banan.byname.net> writes:

[...]

> Eli, if you have not used straight.el, I suggest
> experimenting with it. And again sorry about
> having been overly cryptic.

I have never used straight.el either (mainly because of the need
aggressive incompatibility with package.el and my dislike of having a
big bootstrap-blob at the beginning of my init), but I have been working
on package-vc that a lot of people have apparently been using as a
replacement for package.el.  Recently there has been some work on
supporting arbitrary build-commands that could be executed alongside the
package compilation.  I have been told that this is a major feature
difference between the current state of package-vc and the
straight/elpaca family of package managers.  As you seem to have some
experience with the topic, do you believe this could be helpful to
address your issue?  I am uncertain if this is something worth investing
an effort into, as the issues that Emacs lisp packaging struggles with
are not only those of finding a set of compatible revisions.

Re: Stability of core packages

[Corwin Brust]

On Sat, Apr 29, 2023 at 4:47 PM Mohsen BANAN
<emacs@mohsen.1.banan.byname.net> wrote:
>
> >> This is what doomemacs does. For every package,
> >> there is a git tag which doomemacs keeps with the
> >> adoption of the package. The tags are guaranteed
> >> to be consistent. With Blee (ByStar Libre Emacs
> >> Environment) I do the same but instead of keeping
> >> that tag with the packages, I keep central
> >> manifests for emacs releases. So, emacs-28.1 has
> >> its own packages manifest which can be upgraded
> >> from time to time. All upgrades go to the
> >> appropriate repo and get the appropriate tag based
> >> on the manifest. I do this across all emacs
> >> package archives -- which I see as collections of
> >> git repositories.
> >>
> >> If you want to go with something like this, you
> >> have to revisit the package retrieval model.

[SNIP]

> -----------
> Why does Doom use straight.el and not package.el?
>
> package.el simply doesn’t cut it. Its flaws become
> apparent the more packages you manage, the more
> complex your config becomes, and how often those
> packages see updates:

This is the same doomemacs project[1] that does all development
against HEAD of master, correct?    I track three Emacs branches, all
of which see frequent pushes. Work is regularly merged back to trunk.
I don't see anything like this level of practice maturity from doom
(or straight), so I think the lesson here isn't technical, but
something more like:

1. For powerusers, package.el isn't better than manually tracking
package versions.
2. Tracking package versions is so difficult that Emacs
re-distributers find it worthwhile to hard-code working combinations.

I think the issue is the packaging landscape has matured significantly
thus it is time to reconsider the uses-cases Emacs should support -
and that's the point of this thread.

Meanwhile, I think the practical upshot of the approach taken by doom
(and straight.el, in general) is to push tracking what versions of
which packages work together to individual uses.  I feel it would be a
step backward for Emacs to adopt this approach as the default,
expecting all users to learn to do it.  User-managed explicit package
version pinning does seem like a cool thing to support -- I hope
package-vc helps pave the way for core support for that use-case.  But
it should not be the default: I would be sad if we decide to expect
new users to master package pinning to try out new features.

[1] https://github.com/doomemacs/doomemacs/branches

Re: Stability of core packages

[Mohsen BANAN]

Corwin Brust <corwin@bru.st> writes:

> On Sat, Apr 29, 2023 at 4:47 PM Mohsen BANAN
> <emacs@mohsen.1.banan.byname.net> wrote:
>> ...
>> >> If you want to go with something like this, you
>> >> have to revisit the package retrieval model.
>
> [SNIP]
>
>> -----------
>> Why does Doom use straight.el and not package.el?
>>
>> package.el simply doesn’t cut it. Its flaws become
>> apparent the more packages you manage, the more
>> complex your config becomes, and how often those
>> packages see updates:

> This is the same doomemacs project[1] that does all development
> against HEAD of master, correct?    I track three Emacs branches, all
> of which see frequent pushes. Work is regularly merged back to trunk.

My mentioning doomemacs was not an endorsement of
doomemacs ...

> so I think the lesson here isn't technical, but
> something more like:
>
> 1. For powerusers, package.el isn't better than manually tracking
> package versions.
> 2. Tracking package versions is so difficult that Emacs
> re-distributers find it worthwhile to hard-code working combinations.
>
> I think the issue is the packaging landscape has matured significantly
> thus it is time to reconsider the uses-cases Emacs should support -
> and that's the point of this thread.

Exactly.

> Meanwhile, I think the practical upshot of the approach taken by doom
> (and straight.el, in general) is to push tracking what versions of
> which packages work together to individual uses.  I feel it would be a
> step backward for Emacs to adopt this approach as the default,
> expecting all users to learn to do it.  User-managed explicit package
> version pinning does seem like a cool thing to support -- I hope
> package-vc helps pave the way for core support for that use-case.  But
> it should not be the default: I would be sad if we decide to expect
> new users to master package pinning to try out new features.

That is not what I am saying or suggesting.

Over the past several years, we have learned
various things from:

 1) the packaging landscape
 2) package archive providers
 3) Emacs re-distributers

Right now, in practice, Emacs re-distributers
maintain the stable manifest of the matching sets
for each release of emacs.

I am suggesting two things:

1) package.el should evolve or be replaced with
   something that is the equivalent of
   straight.el.

2) The responsibility for maintaining a stable
matching set of packages for each release of emacs
should come down to package archive providers (not
the Emacs re-distributers). Package archive
providers should maintain stable and consistent
manifests. Emacs re-distributers and emacs
end-users can selectively overwrite these, but
that won't be the default.

The challenge here is that in the existing emacs
ecosystem, basic emacs (emacs+elpa), package
archive providers and emacs re-distributers have
no structure to facilitate convergence.

If basic emacs (emacs+elpa) could solve this
properly, the rest could have a pattern to
emulate. For that to happen, basic emacs needs to
learn the lessons of emacs re-distributers. That
was why I mentioned that experimenting with
straight.el is a good idea. Not as a capability,
but as a basis for existing policies and
practices.

...Mohsen

Re: Stability of core packages

[David Masterson]

emacs@mohsen.1.banan.byname.net writes:

> What I am suggesting breaks the existing
> model. Let me start with some context.
>
> Because we use git for everything and because it
> is possible to take complete git snap shots of all
> relevant packages, it is possible to easily
> provide stability for the user based on snapshots.
>
> Consistent upgrades are then possible for new
> consistent git snapshots. So, based on the snapshots
> the packages are not stale and remain consistent
> while evolving.
>
> This is what doomemacs does. For every package,
> there is a git tag which doomemacs keeps with the
> adoption of the package. The tags are guaranteed
> to be consistent. With Blee (ByStar Libre Emacs
> Environment) I do the same but instead of keeping
> that tag with the packages, I keep central
> manifests for emacs releases. So, emacs-28.1 has
> its own packages manifest which can be upgraded
> from time to time. All upgrades go to the
> appropriate repo and get the appropriate tag based
> on the manifest. I do this across all emacs
> package archives -- which I see as collections of
> git repositories.
>
> If you want to go with something like this, you
> have to revisit the package retrieval model.
>
> For every release of emacs, you maintain a
> separate manifest. And you keep updating that at
> new releases and even in between releases. Since
> all emacs archives are git based, it is possible
> to apply this strategy to all of them -- where
> each take care of its own consistency.
>
> In practice, this has already happened for layers
> that sit on top of emacs (doomemacs, blee). It is
> a matter of adopting the higher layer model in the
> core. There are already many variations on the
> model of packages.el.
>
> If you want to go with this model, much needs to
> be revisited. But much will be gained.
>
> Again, these comments only deal with the meta
> topic not the at hand  Eglot topic. But, it shows
> how eglot upgrades could have rolled back to
> emacs-28.
>
> ...Mohsen
>
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
>>> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, dmitry@gutov.dev,
>>>  emacs-devel@gnu.org
>>> Date: Wed, 19 Apr 2023 21:40:35 +0200
>>>
>>> > Specifically, users of Emacs 28 and older, who had Eglot installed,
>>> > and expect Eglot to be automatically updated upon Emacs startup
>>> > whenever a new Eglot version is available, will now have their
>>> > expectations broken after they upgrade to Emacs 29, because Eglot is
>>> > now a built-in package, and package.el won't by default upgrade a
>>> > built-in package.
>>> …
>>> > So there's a dilemma here: which of the two groups of users to break?
>>>
>>> Not updating eglot until the next Emacs release shouldn’t cause breakage
>>> in any other packages, right?
>>
>> No, it shouldn't.  With the obvious exception of the breakage that is
>> already part of the current Emacs release, which we somehow failed to
>> detect and fix before releasing it.
>>
>>> Except if a more modern eglot is a dependency of a non-built-in package.
>>
>> Right.
>>
>>> I think that’s what I would prefer: I would treat being pulled into
>>> Emacs as a stabilization step that switches the package from being on
>>> the latest version to being at the version in Emacs or the minimum
>>> version required by dependencies — if that version is higher than the
>>> version in Emacs. Basically minimize the distance from the Emacs
>>> release.
>>
>> AFAIU, that is what will happen with Emacs 29 when it is released.
>> But things might change in Emacs 30, as we are currently discussing
>> what needs to be done to better support updating core packages.
>

-- 
David Masterson

Re: Stability of core packages

[David Masterson]

emacs@mohsen.1.banan.byname.net writes:

> What I am suggesting breaks the existing
> model. Let me start with some context.
>
> Because we use git for everything and because it
> is possible to take complete git snap shots of all
> relevant packages, it is possible to easily
> provide stability for the user based on snapshots.
>
> Consistent upgrades are then possible for new
> consistent git snapshots. So, based on the snapshots
> the packages are not stale and remain consistent
> while evolving.

[...]

> If you want to go with something like this, you
> have to revisit the package retrieval model.
>
> For every release of emacs, you maintain a
> separate manifest. And you keep updating that at
> new releases and even in between releases. Since
> all emacs archives are git based, it is possible
> to apply this strategy to all of them -- where
> each take care of its own consistency.
>
> In practice, this has already happened for layers
> that sit on top of emacs (doomemacs, blee). It is
> a matter of adopting the higher layer model in the
> core. There are already many variations on the
> model of packages.el.

How would Emacs deal with those system layers.  In theory, Emacs
packages could depend on packages outside of the Emacs layer (system
libraries or other language tools).  My thought is that you'd either
have to be explicit all the way up the tree (making it a logistical
nightmare) or you'd have to prune the tree (making the definition of a
release intentionally fuzzy).

Currently, the model is that each package (supposedly) calls out it's
package version dependencies that it has been tested with and assumes
that later versions are compatible.  Isn't the above approach going to
devolve into this?

How does doom handle it?

-- 
David Masterson

Re: Stability of core packages

[David Masterson]

David Masterson <dsmasterson@gmail.com> writes:

oops ignore this 

> emacs@mohsen.1.banan.byname.net writes:
>
>> What I am suggesting breaks the existing
>> model. Let me start with some context.
>>
>> Because we use git for everything and because it
>> is possible to take complete git snap shots of all
>> relevant packages, it is possible to easily
>> provide stability for the user based on snapshots.
>>
>> Consistent upgrades are then possible for new
>> consistent git snapshots. So, based on the snapshots
>> the packages are not stale and remain consistent
>> while evolving.
>>
>> This is what doomemacs does. For every package,
>> there is a git tag which doomemacs keeps with the
>> adoption of the package. The tags are guaranteed
>> to be consistent. With Blee (ByStar Libre Emacs
>> Environment) I do the same but instead of keeping
>> that tag with the packages, I keep central
>> manifests for emacs releases. So, emacs-28.1 has
>> its own packages manifest which can be upgraded
>> from time to time. All upgrades go to the
>> appropriate repo and get the appropriate tag based
>> on the manifest. I do this across all emacs
>> package archives -- which I see as collections of
>> git repositories.
>>
>> If you want to go with something like this, you
>> have to revisit the package retrieval model.
>>
>> For every release of emacs, you maintain a
>> separate manifest. And you keep updating that at
>> new releases and even in between releases. Since
>> all emacs archives are git based, it is possible
>> to apply this strategy to all of them -- where
>> each take care of its own consistency.
>>
>> In practice, this has already happened for layers
>> that sit on top of emacs (doomemacs, blee). It is
>> a matter of adopting the higher layer model in the
>> core. There are already many variations on the
>> model of packages.el.
>>
>> If you want to go with this model, much needs to
>> be revisited. But much will be gained.
>>
>> Again, these comments only deal with the meta
>> topic not the at hand  Eglot topic. But, it shows
>> how eglot upgrades could have rolled back to
>> emacs-28.
>>
>> ...Mohsen
>>
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: "Dr. Arne Babenhauserheide" <arne_bab@web.de>
>>>> Cc: joaotavora@gmail.com, jporterbugs@gmail.com, dmitry@gutov.dev,
>>>>  emacs-devel@gnu.org
>>>> Date: Wed, 19 Apr 2023 21:40:35 +0200
>>>>
>>>> > Specifically, users of Emacs 28 and older, who had Eglot installed,
>>>> > and expect Eglot to be automatically updated upon Emacs startup
>>>> > whenever a new Eglot version is available, will now have their
>>>> > expectations broken after they upgrade to Emacs 29, because Eglot is
>>>> > now a built-in package, and package.el won't by default upgrade a
>>>> > built-in package.
>>>> …
>>>> > So there's a dilemma here: which of the two groups of users to break?
>>>>
>>>> Not updating eglot until the next Emacs release shouldn’t cause breakage
>>>> in any other packages, right?
>>>
>>> No, it shouldn't.  With the obvious exception of the breakage that is
>>> already part of the current Emacs release, which we somehow failed to
>>> detect and fix before releasing it.
>>>
>>>> Except if a more modern eglot is a dependency of a non-built-in package.
>>>
>>> Right.
>>>
>>>> I think that’s what I would prefer: I would treat being pulled into
>>>> Emacs as a stabilization step that switches the package from being on
>>>> the latest version to being at the version in Emacs or the minimum
>>>> version required by dependencies — if that version is higher than the
>>>> version in Emacs. Basically minimize the distance from the Emacs
>>>> release.
>>>
>>> AFAIU, that is what will happen with Emacs 29 when it is released.
>>> But things might change in Emacs 30, as we are currently discussing
>>> what needs to be done to better support updating core packages.
>>

-- 
David Masterson

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 09:50:12 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, Dmitry Gutov <dmitry@gutov.dev>,
>  emacs-devel@gnu.org
> 
> On Tue, Apr 18, 2023 at 7:57 PM Jim Porter <jporterbugs@gmail.com> wrote:
> >
> > * Stable: the version of a package included in the latest Emacs tarball
> > * Latest: the latest version on GNU ELPA (etc)
> > * Devel: the latest version on GNU-devel ELPA (etc)
> >
> > You could possibly add:
> >
> > * Core(?): the version of a package included in the tarball of the
> > user's *current* Emacs installation
> 
> These are interesting levels, but I was under the impression
> that the goal is to partition the set of :core packages according
> to some kind of gradation, what Eli called "stability gradation".
> The set can be found in the variable package--builtins.

No you've misunderstood what I mean by "stability gradation".  It has
nothing to do with the value of package--builtins, even when that is
non-nil.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 1:55 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 19 Apr 2023 09:50:12 +0100
> > Cc: Eli Zaretskii <eliz@gnu.org>, Dmitry Gutov <dmitry@gutov.dev>,
> >  emacs-devel@gnu.org
> >
> > On Tue, Apr 18, 2023 at 7:57 PM Jim Porter <jporterbugs@gmail.com> wrote:
> > >
> > > * Stable: the version of a package included in the latest Emacs tarball
> > > * Latest: the latest version on GNU ELPA (etc)
> > > * Devel: the latest version on GNU-devel ELPA (etc)
> > >
> > > You could possibly add:
> > >
> > > * Core(?): the version of a package included in the tarball of the
> > > user's *current* Emacs installation
> >
> > These are interesting levels, but I was under the impression
> > that the goal is to partition the set of :core packages according
> > to some kind of gradation, what Eli called "stability gradation".
> > The set can be found in the variable package--builtins.
>
> No you've misunderstood what I mean by "stability gradation".  It has
> nothing to do with the value of package--builtins, even when that is
> non-nil.

OK, but you wrote:

  IOW, shouldn't packages have some "stability gradation" that is
  visible when users look at the list of packages via package.el?

By "have some" I interpreted that you wanted to map a given
characteristic to each package.   And by "packages" I interpreted
that you meant the packages that the Emacs project has control
over.  And those packages are, by definition, the ":core" or
"builtin" packages (although if the Emacs project is taken to
include ELPA .git, it will also include all the packages there).

If I've misunderstood, I think you could give examples of
real or made up packages and assign a "stability gradation" to
them and define what that gradation means.

Then explain what you want to do with that stability gradation
in Emacs (if anything, perhaps you just want to show it).

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 19 Apr 2023 14:18:21 +0100
> Cc: jporterbugs@gmail.com, dmitry@gutov.dev, emacs-devel@gnu.org
> 
> > > These are interesting levels, but I was under the impression
> > > that the goal is to partition the set of :core packages according
> > > to some kind of gradation, what Eli called "stability gradation".
> > > The set can be found in the variable package--builtins.
> >
> > No you've misunderstood what I mean by "stability gradation".  It has
> > nothing to do with the value of package--builtins, even when that is
> > non-nil.
> 
> OK, but you wrote:
> 
>   IOW, shouldn't packages have some "stability gradation" that is
>   visible when users look at the list of packages via package.el?

Yes.

> By "have some" I interpreted that you wanted to map a given
> characteristic to each package.   And by "packages" I interpreted
> that you meant the packages that the Emacs project has control
> over.  And those packages are, by definition, the ":core" or
> "builtin" packages (although if the Emacs project is taken to
> include ELPA .git, it will also include all the packages there).
> 
> If I've misunderstood, I think you could give examples of
> real or made up packages and assign a "stability gradation" to
> them and define what that gradation means.

In the "list-packages" display, instead of

  eglot       1.14    available  gnu   The Emacs Client for LSP servers

I envisioned seeing something like

  eglot       2.01    alpha      gnu   The Emacs Client for LSP servers
  eglot       1.20    current    gnu   The Emacs Client for LSP servers
  eglot       1.15    stable     gnu   The Emacs Client for LSP servers
  eglot       1.14    previous   gnu   The Emacs Client for LSP servers
  eglot       1.12    built-in   gnu   The Emacs Client for LSP servers

where the 3rd column is the "stability gradation" I had in mind.

I also envisioned some user option, say, package-preferred-stabilty,
which users could set to a value such as 'stable', and then package.el
will only automatically update a package to a newer version when the
new version satisfies the stability criteria per the value of that
option.

> Then explain what you want to do with that stability gradation
> in Emacs (if anything, perhaps you just want to show it).

If the above doesn't explain this, I don't think I understand the
question.  What do you mean by "do with that in Emacs"?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 2:44 PM Eli Zaretskii <eliz@gnu.org> wrote:

> I envisioned seeing something like
>
>   eglot       2.01    alpha      gnu   The Emacs Client for LSP servers
>   eglot       1.20    current    gnu   The Emacs Client for LSP servers
>   eglot       1.15    stable     gnu   The Emacs Client for LSP servers
>   eglot       1.14    previous   gnu   The Emacs Client for LSP servers
>   eglot       1.12    built-in   gnu   The Emacs Client for LSP servers
>
> where the 3rd column is the "stability gradation" I had in mind.
>
> I also envisioned some user option, say, package-preferred-stabilty,
> which users could set to a value such as 'stable', and then package.el
> will only automatically update a package to a newer version when the
> new version satisfies the stability criteria per the value of that
> option.
>
> > Then explain what you want to do with that stability gradation
> > in Emacs (if anything, perhaps you just want to show it).
>
> If the above doesn't explain this, I don't think I understand the
> question.

Thank you, it explains what you mean.  Seems like you want distribution
channels like Debian has.  No objection here, but IMO seems to be
solving problems that I've never personally seen happening in
Emacs packages.  And it also seems like a lot of work.

I really can't remember seeing a bug report where someone was
discontent with an upgrade of some package to a higher version
that was accidently buggy, let alone a "furtive" upgrade related to
dependencies.  Doesn't mean they don't exist, I've just never seen
them.

I remember quite well that upgrading versions of dependencies _fixes_
bugs.  One of the first code packages was cl-lib and its first versions
were off, the later versions were progressively better.

> What do you mean by "do with that in Emacs"?

I meant how is Emacs's code to react to this gradation.  You've more
or less suggested that "package-preferred-stability" would be
consulted by package-install to know which one to pick, so that also
answers my question.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 18/04/2023 15:57, Eli Zaretskii wrote:

>>>> On 14/04/2023 22:28, Eli Zaretskii wrote:
>>>>> If, OTOH,
>>>>> you think that it's imperative to allow_all_  users of Eglot with
>>>>> Emacs 29 to upgrade to Eglot 1.14 (and 1.15, 1.16, etc., when those
>>>>> become available), then we should release Emacs 29 with 1.14.
>>>>
>>>> Was this question about stability only?
>>>
>>> It was about the criteria for which versions of core packages to ship
>>> with a release.
>>
>> I don't think we can get a single set of criteria across core packages.
> 
> We don't need to have just one set.  Packages are different in both
> their complexity, their dependence on other packages, and dependence
> of other packages on them.  So one set is unlikely to fit the bill,
> indeed.
> 
> But that doesn't mean we shouldn't have criteria at all.  We should
> strive to have a small number of them, and we should know which set is
> applicable to which class of packages.

It's a good question, but not a deciding one WRT what we should do with 
Eglot for Emacs 29, I think. I hope I'll be able to explain that below.

> This will be one of the serious issues if we ever move to having some
> packages only in elpa.git, and will then bundle them when preparing an
> Emacs release tarball.  It will be imperative to know at that time
> which version/branch of each such package to take as part of preparing
> a release.  We must have a solution by then, so this is as good time
> as any to start discussing the issue.

Sure. Please keep in mind, however, that very few of external packages 
have separate branches for releases and development. I guess Org does, 
but I'm not sure if others exist. One of the reasons for that is that 
the ELPA repositories only package the very latest released version 
anyway and quickly delete the older ones. And package.el is the foremost 
distribution method for Elisp code.

For :core package, we also have Emacs's own development and release 
branches. But only master branch has versioned releases cut from it. 
It's hard to say whether the time the code has spent in a given 
'emacs-xy' branch brings maturity to it, and how much.

So every time a new version if tagged, it's a value judgment on the part 
of the maintainer: whether enough time has passed since the most recent 
big feature was added, whether there were bug reports or not.

The MELPA project has helped a lot with this over the years because it 
popularized snapshot versions. So when a package is feature on MELPA, 
the author could be fairly sure that a lot of users have downloaded the 
latest snapshot and gave it a try, and the lack of new reports means 
it's probably okay. MELPA has the problem that the "Stable" repository 
is not very popular, though. We here have it in the reverse: I'm not 
sure how many people make use of elpa-devel (GNU-devel ELPA). Probably 
not many.

>> E.g. Org is developed externally, has its own community of significant
>> size, and does split off release branches (with additional testing, I',m
>> guessing).
>>
>> Eglot, OTOH, is developed only here, with no additional release workflow
>> other than what MELPA/GNU ELPA historically provided: collect up some
>> features/fixes, bump the Version header, and push a new release out to
>> the users. The lack of extended testing period is made up for with the
>> capability to push out a new fixed version overnight. That's why the
>> difficulty in upgrading to the latest version (for Emacs 29 users) is
>> going to hurt.
> 
> If some core package is not tested enough before it gets a new
> version, then why are we okay with telling users of a stable Emacs to
> update the package willy-nilly as soon as another version is on ELPA?
> Shouldn't we at least warn them, or, better, somehow indicate that
> this version is not yet considered stable?

I have a different answer from all that had been presented here: because 
the user can uninstall it.

Uninstall any newer installed version and stay with that one that had 
been bundled with the Emacs release. As long as *that* one has been 
picked well enough to be stable, and can be reverted to, we can afford 
not to worry too much about the user installing another version.

It's like reverting to a more stable distribution channel, to use the 
analogy from the bug discussion.

> IOW, shouldn't packages have some "stability gradation" that is
> visible when users look at the list of packages via package.el?
> Shouldn't we allow users to tell package.el which stability they want
> to download, so that unstable packages don't inadvertently get
> installed and mess up their production environment?

We have elpa and elpa-devel. The latter is "explicitly unstable" and the 
former is "checkpoint releases".

Neither keeps the previous versions around, so it's not like the user at 
any point could make a choice to install a minor version update instead 
of going up a full major version. The only choice is to update to the 
latest, or not (by using elpa-devel, though, the user might opt into 
having "the latest" mean "the latest commit in master").

>> BTW, if you recall the threads before Eglot was added, I was against
>> that, and one of the things I cited is an LSP client has inherently high
>> development velocity. Maybe the LSP community will settle/mature/stop
>> adding features one day, but it's not there yet.
> 
> I don't see what development pace has to do with this issue.  If a
> package is being developed at a high pace, it might mean that the
> stable version will lag more.  But what does this change in principle?

It matters when we're talking not about simple additional, optional 
features, but about changes that keep pace with the evolution of the LSP 
standard, with new LSP servers that arrive, new changes in existing 
protocols. Things that users come to expect to be able to use now, and 
not in 3 years.

Taking a conservative stance can work when the ecosystem supports it 
too. E.g. if every LSP server that we support now had an implicit 
promise to be properly maintained for 3 years since, at least. I don't 
think that's the case. Almost every one could be deprecated in that time 
frame, with community being recommended to switch to a newer one.

>>>> Because since we've decided in favor of stability of package.el, and
>>>> against eglot's easy upgradability, I would suggest to backport Eglot
>>>> 1.14 to emacs-29.
>>>
>>> I won't object.  In fact, I asked up front why not.
>>
>> Note that that suggestion comes with a fix to eldoc which you so far
>> have rejected for emacs-29.
> 
> You mean, the change in ElDoc that avoids the "jumping mode line"
> issue?  If so, it is unfortunate for Eglot to depend on that, because
> it means users of Emacs 29 will be unable to upgrade to Eglot 1.15
> without by side effect installing a newer and potentially less stable
> ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)

Sorry if that was unclear: yes, the fix for "jumping mode-line and 
blinking text", but not as a bump in dependency. Just to backport the 
fix to eldoc.el inside emacs-29.

But Joao has explained that that idea is impractical because Eglot 1.14 
depends on Eldoc 1.14.0. And I just wanted a particular fix from it. So 
backporting the thing I wanted from Eglot 1.14 (taking up much less 
space in the echo area) is not really feasible because it depends on a 
fairly recent addition to eldoc.

To clarify: I made that suggestion from the premise that we do expect 
and encourage a fair fraction of the users to use just the versions of 
Eglot and Eldoc that come with Emacs 29, and never upgrade to the latest 
ones.

> So this again goes back to the main issue: how should the stability
> considerations affect development of core packages and their
> "stability gradation"?  Developers of core packages should keep these
> aspects in mind at all times, and, for example, avoid dependencies on
> other packages that aren't absolutely necessary.

I agree with that stance, in general.

And as per above explanation, Eglot 0.14 depends on Eldoc 0.14.0 not 
because it's the very latest and spiffiest version, but because it needs 
a particular feature from it.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Tue, Apr 18, 2023 at 11:11 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> And as per above explanation, Eglot 0.14 depends on Eldoc 0.14.0 not
> because it's the very latest and spiffiest version, but because it needs
> a particular feature from it.

Of course.  And if there are doubts as to when and why Eglot's
"Package-Require"'s are updated, one can always M-x vc-region-history
on that single line of lisp/progmodes/eglot.el.  For example, recently
Xref requirement was bumped because:

  commit f72a394716f4373dbbdc79ad0816da90bdb032a1
  Author: Basil L. Contovounesios <contovob@tcd.ie>
  Date:   Fri Jan 27 00:27:26 2023 +0000

      Work around package.el transitive dependency bug

      Eglot already depends transitively on Xref 1.4.0 via Project,
      but package.el doesn't pick up on this in Emacs 28
      (which has Xref 1.3.0).

      * lisp/progmodes/eglot.el (Version): Bump to 1.11.
      (Package-Requires): Explicitly require Xref 1.4.0, which is
      the version already required by Project, for the benefit of
      Emacs 28 (bug#61048).

If you find some dependency is set unnecessarily too high, I
don't see any problem in downgrading it.

João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 01:10:56 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > This will be one of the serious issues if we ever move to having some
> > packages only in elpa.git, and will then bundle them when preparing an
> > Emacs release tarball.  It will be imperative to know at that time
> > which version/branch of each such package to take as part of preparing
> > a release.  We must have a solution by then, so this is as good time
> > as any to start discussing the issue.
> 
> Sure. Please keep in mind, however, that very few of external packages 
> have separate branches for releases and development. I guess Org does, 
> but I'm not sure if others exist. One of the reasons for that is that 
> the ELPA repositories only package the very latest released version 
> anyway and quickly delete the older ones. And package.el is the foremost 
> distribution method for Elisp code.

Then maybe ELPA will have to change as well.

> So every time a new version if tagged, it's a value judgment on the part 
> of the maintainer: whether enough time has passed since the most recent 
> big feature was added, whether there were bug reports or not.

Similar judgment calls are necessary for Emacs.  So I see nothing here
that brings some fundamentally new issues.

Moreover, I don't see how this affects the need to have good criteria
for stability, and the importance of applying those criteria when
deciding which versions to bundle with Emacs.

> > If some core package is not tested enough before it gets a new
> > version, then why are we okay with telling users of a stable Emacs to
> > update the package willy-nilly as soon as another version is on ELPA?
> > Shouldn't we at least warn them, or, better, somehow indicate that
> > this version is not yet considered stable?
> 
> I have a different answer from all that had been presented here: because 
> the user can uninstall it.

User can also downgrade to a previous version of the package, don't
they?  Or if they cannot, they should be able to do that.  Once that
is possible, what exactly is the difference between these two kinds of
packages?  Downgrading to an older version when a newer one is buggy
or otherwise unsatisfactory should be supported for all packages.

Also, does package.el support "downgrading" to the bundled version?
Did anyone actually try that?  In particular, what happens with the
dependencies the user upgraded together with the package being
"uninstalled", due to the minimum requirements of that package?

> > IOW, shouldn't packages have some "stability gradation" that is
> > visible when users look at the list of packages via package.el?
> > Shouldn't we allow users to tell package.el which stability they want
> > to download, so that unstable packages don't inadvertently get
> > installed and mess up their production environment?
> 
> We have elpa and elpa-devel. The latter is "explicitly unstable" and the 
> former is "checkpoint releases".

So what is the stability measure of "elpa", again?  Is it on par with
the Emacs's release branch?  Could it be?  If not, why not?

> Neither keeps the previous versions around, so it's not like the user at 
> any point could make a choice to install a minor version update instead 
> of going up a full major version.

That in itself is a serious deficiency, IMO, and we should fix it by
providing the "downgrade" option.

> > I don't see what development pace has to do with this issue.  If a
> > package is being developed at a high pace, it might mean that the
> > stable version will lag more.  But what does this change in principle?
> 
> It matters when we're talking not about simple additional, optional 
> features, but about changes that keep pace with the evolution of the LSP 
> standard, with new LSP servers that arrive, new changes in existing 
> protocols. Things that users come to expect to be able to use now, and 
> not in 3 years.

Users who must have these features (presumably because they must use
servers which require that) will have to give up some stability
expectations.  Exactly like users who must have some new enough
feature of Emacs which is only available on the master branch.

So once again: what is fundamentally different or new about packages
which develop at fast pace, in the context of this discussion?  Why do
we need to bring up those details here?

> Taking a conservative stance can work when the ecosystem supports it 
> too. E.g. if every LSP server that we support now had an implicit 
> promise to be properly maintained for 3 years since, at least. I don't 
> think that's the case. Almost every one could be deprecated in that time 
> frame, with community being recommended to switch to a newer one.

How does this help us move forward with the discussion and with
handling this issue?  Conservative doesn't mean stupid; if some
external factor changes outside of our control and forces us to make
potentially destabilizing changes, what else can we do that requires a
discussion?  And how does it impact what we should do about
categorizing package releases according to their stability and about
the decisions which version to bundle with Emacs?

> To clarify: I made that suggestion from the premise that we do expect 
> and encourage a fair fraction of the users to use just the versions of 
> Eglot and Eldoc that come with Emacs 29, and never upgrade to the latest 
> ones.

AFAIU, João is of the opposite opinion: he thinks that most Eglot
users will want the latest version on ELPA, or at least a version that
is newer than the one bundled with Emacs.

> And as per above explanation, Eglot 0.14 depends on Eldoc 0.14.0 not 
> because it's the very latest and spiffiest version, but because it needs 
> a particular feature from it.

And there's no reasonable way of adding to Eglot 1.14 some fallback or
compatibility shim to remove that dependency?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 5:47 AM, Eli Zaretskii wrote:
> Also, does package.el support "downgrading" to the bundled version?
> Did anyone actually try that?  In particular, what happens with the
> dependencies the user upgraded together with the package being
> "uninstalled", due to the minimum requirements of that package?

I've done this in the past and everything works pretty much as expected 
from my recollection: you can uninstall a package that you got from 
ELPA, so afterwards, you'd just get the bundled version (you might need 
to restart; I always do).

In addition, any automatically-installed dependent packages are marked 
with the status "dependency". You can remove no-longer-needed deps via 
'package-autoremove'. When uninstalling a package interactively in the 
*Packages* buffer, it will even suggest that you remove unneeded deps 
when appropriate (see 'package-menu-execute').

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 11:22:10 -0700
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 4/19/2023 5:47 AM, Eli Zaretskii wrote:
> > Also, does package.el support "downgrading" to the bundled version?
> > Did anyone actually try that?  In particular, what happens with the
> > dependencies the user upgraded together with the package being
> > "uninstalled", due to the minimum requirements of that package?
> 
> I've done this in the past and everything works pretty much as expected 
> from my recollection: you can uninstall a package that you got from 
> ELPA, so afterwards, you'd just get the bundled version (you might need 
> to restart; I always do).
> 
> In addition, any automatically-installed dependent packages are marked 
> with the status "dependency". You can remove no-longer-needed deps via 
> 'package-autoremove'. When uninstalling a package interactively in the 
> *Packages* buffer, it will even suggest that you remove unneeded deps 
> when appropriate (see 'package-menu-execute').

IMO, downgrading to the bundled version should be much simpler and by
default should remove all the dependencies without asking and without
any need for manual user actions.

In any case, I don't think this use case was considered or tried
enough for us to consider it a solved issue.  I'm quite sure there's
more here than meets the eye, simply because this is rarely if ever
done.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Jim Porter]

On 4/19/2023 11:37 AM, Eli Zaretskii wrote:
> IMO, downgrading to the bundled version should be much simpler and by
> default should remove all the dependencies without asking and without
> any need for manual user actions.

For what it's worth, Apt (the Debian package manager) works the same 
way: if you remove a package, it will just inform you that there are 
unneeded deps and that you can use 'apt autoremove' to clean them up. I 
think that makes sense, since you might really want the new version of 
some package, and you just don't realize that it's marked as a dep 
instead of an explicitly-installed package.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Lynn Winebarger]

On Wed, Apr 19, 2023 at 2:37 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Wed, 19 Apr 2023 11:22:10 -0700
> > Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> > From: Jim Porter <jporterbugs@gmail.com>
> >
> > On 4/19/2023 5:47 AM, Eli Zaretskii wrote:
> > > Also, does package.el support "downgrading" to the bundled version?
> > > Did anyone actually try that?  In particular, what happens with the
> > > dependencies the user upgraded together with the package being
> > > "uninstalled", due to the minimum requirements of that package?
> >
> > I've done this in the past and everything works pretty much as expected
> > from my recollection: you can uninstall a package that you got from
> > ELPA, so afterwards, you'd just get the bundled version (you might need
> > to restart; I always do).
> >
> > In addition, any automatically-installed dependent packages are marked
> > with the status "dependency". You can remove no-longer-needed deps via
> > 'package-autoremove'. When uninstalling a package interactively in the
> > *Packages* buffer, it will even suggest that you remove unneeded deps
> > when appropriate (see 'package-menu-execute').
>
> IMO, downgrading to the bundled version should be much simpler and by
> default should remove all the dependencies without asking and without
> any need for manual user actions.
>
> In any case, I don't think this use case was considered or tried
> enough for us to consider it a solved issue.  I'm quite sure there's
> more here than meets the eye, simply because this is rarely if ever
> done.

One of the design points of the unboxing package-management-wrapper
I'm writing is maintaining distinct package areas with a notion of
scoping.  My purpose was to enable separate management of
site-packages and user packages.  A site administrator might have one
set of packages, and a user might want different versions of those
packages.  So, the dependency calculation for user packages should
allow for dependencies to be satisfied by site packages, but not vice
versa (that's the scoping).  The "delta" of the package set installed
in the user area has to be enough to make the set of libraries "seen"
by the user coherent.  That is, say package A and B both depend on
package C and all are in the site-level packaging area.  Then the user
wants a different version of package A, which requires a different
version of package C.  Coherence means that if the version of package
B installed at the site level is incompatible with the desired version
of package C, then a compatible version of package B will also have to
be installed in the user packaging area.  That set has to be
transitively closed.

But the separation into those two is just a default configuration.
Arbitrary package areas can be defined, each with their own set of
"in-scope" areas.  One easily reversible way to add a new version of
eglot would be to set up a new package area for it, with whatever
other package areas are loaded being put in its scope, then install
the minimum coherent set of packages into the new area.  Then rolling
back the update would be trivial, and it could be "committed" to the
usual package area once the user was satisfied.

My only other thought on this discussion, as a package (ab)user
running multiple versions of emacs, is that package releases should be
dependent on the emacs version installing it.  What I see as available
packages/upgrades when I list packages while running emacs 28 should
probably be different (for some packages) than what I see while
running emacs 25 or emacs 29.  It seems like the release philosophy
after version 24 has been conservative about changes to the run-time
semantics per major release, so the versioning probably doesn't need
to be conditioned on the minor version of the Emacs binary.

Lynn

Re: history of ELPA packages and dependencies (was: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot))

[Lynn Winebarger]

On Wed, Apr 19, 2023 at 6:51 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> The "delta" of the package set installed
> in the user area has to be enough to make the set of libraries "seen"
> by the user coherent.  That is, say package A and B both depend on
> package C and all are in the site-level packaging area.  Then the user
> wants a different version of package A, which requires a different
> version of package C.  Coherence means that if the version of package
> B installed at the site level is incompatible with the desired version
> of package C, then a compatible version of package B will also have to
> be installed in the user packaging area.  That set has to be
> transitively closed.

Calculating the set of sets of package/version pairs satsifying the
"coherence" constraint requires a lot of data from each published
version of each package in an ELPA.

Looking at the Savannah repos for GNU/nonGNU ELPAs, some package
branches appear to have multiple elisp files with package header
fields.  For example, externals/corfu in the GNU ELPA repo has a
subdirectory "extensions" in which some elisp files appear to contain
package definitions.  However, the GNU ELPA only has one "corfu"
package.  Unfortunately, I can't tell from that example if elpa-admin
completely ignores the package-requires in the other elisp files, or
determines the set of requirements based on fields from all included
elisp files.

So, can I determine package-requires of each package commit by looking
only at the main elisp file for the package after checking out that
commit?

I'm sure MELPA will be more challenging, but building a history for
GNU/nonGNU ELPA packages will be a good starting point.

Thanks,
Lynn

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Lynn Winebarger]

On Wed, Apr 19, 2023 at 6:51 PM Lynn Winebarger <owinebar@gmail.com> wrote:
> On Wed, Apr 19, 2023 at 2:37 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > In any case, I don't think this use case was considered or tried
> > enough for us to consider it a solved issue.  I'm quite sure there's
> > more here than meets the eye, simply because this is rarely if ever
> > done.

> My only other thought on this discussion, as a package (ab)user
> running multiple versions of emacs, is that package releases should be
> dependent on the emacs version installing it.  What I see as available
> packages/upgrades when I list packages while running emacs 28 should
> probably be different (for some packages) than what I see while
> running emacs 25 or emacs 29.  It seems like the release philosophy
> after version 24 has been conservative about changes to the run-time
> semantics per major release, so the versioning probably doesn't need
> to be conditioned on the minor version of the Emacs binary.

Another point in favor of maintaining separate dependencies based on
emacs major version (and possibly more) are addition of features like
the builtin sqlite support.  Some packages might require the sqlite
module for versions prior to 29, or the sqlite-builtin module for 29+,
but the latter dependency is incompatible with version 28 and below.

Lynn

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 19/04/2023 15:47, Eli Zaretskii wrote:

>>> This will be one of the serious issues if we ever move to having some
>>> packages only in elpa.git, and will then bundle them when preparing an
>>> Emacs release tarball.  It will be imperative to know at that time
>>> which version/branch of each such package to take as part of preparing
>>> a release.  We must have a solution by then, so this is as good time
>>> as any to start discussing the issue.
>>
>> Sure. Please keep in mind, however, that very few of external packages
>> have separate branches for releases and development. I guess Org does,
>> but I'm not sure if others exist. One of the reasons for that is that
>> the ELPA repositories only package the very latest released version
>> anyway and quickly delete the older ones. And package.el is the foremost
>> distribution method for Elisp code.
> 
> Then maybe ELPA will have to change as well.

I don't think that should be, or can be reasonably required. But that 
can be put it to a separate discussion. With Stefan and Philip involved, 
probably.

>> So every time a new version if tagged, it's a value judgment on the part
>> of the maintainer: whether enough time has passed since the most recent
>> big feature was added, whether there were bug reports or not.
> 
> Similar judgment calls are necessary for Emacs.  So I see nothing here
> that brings some fundamentally new issues.
> 
> Moreover, I don't see how this affects the need to have good criteria
> for stability, and the importance of applying those criteria when
> deciding which versions to bundle with Emacs.

Ok, if the question is just "how to choose the version of :core package 
to bundle with an Emacs release", then just see the criterion "N weeks 
have passed without issue" that I just described in the bug thread.

This one doesn't need any changes to ELPA, and I've been using it, more 
or less, for some years now.

>>> If some core package is not tested enough before it gets a new
>>> version, then why are we okay with telling users of a stable Emacs to
>>> update the package willy-nilly as soon as another version is on ELPA?
>>> Shouldn't we at least warn them, or, better, somehow indicate that
>>> this version is not yet considered stable?
>>
>> I have a different answer from all that had been presented here: because
>> the user can uninstall it.
> 
> User can also downgrade to a previous version of the package, don't
> they?  Or if they cannot, they should be able to do that.

Even if there was such possibility, they wouldn't have a single stable 
version to go back to (they'd have to make a choice). When we bundle a 
version, that's one fewer question to answer.

> Once that
> is possible, what exactly is the difference between these two kinds of
> packages?  Downgrading to an older version when a newer one is buggy
> or otherwise unsatisfactory should be supported for all packages.

Then the difference would be that we hand-picked a set of particular 
versions of packages, whereas for third-party packages that was done (or 
failed to be done) by other people we have no responsibility over.

> Also, does package.el support "downgrading" to the bundled version?
> Did anyone actually try that?  In particular, what happens with the
> dependencies the user upgraded together with the package being
> "uninstalled", due to the minimum requirements of that package?

It should work by "uninstalling" the package. An when you uninstall a 
package, package.el warns you about any dependencies that would be 
broken this way. Someone should test it just in case, of course.

But the important part is that the bundled package stays installed. You 
asked why we can encourage people to upgrade said packages freely. One 
answer is that they have a safety net.

>>> IOW, shouldn't packages have some "stability gradation" that is
>>> visible when users look at the list of packages via package.el?
>>> Shouldn't we allow users to tell package.el which stability they want
>>> to download, so that unstable packages don't inadvertently get
>>> installed and mess up their production environment?
>>
>> We have elpa and elpa-devel. The latter is "explicitly unstable" and the
>> former is "checkpoint releases".
> 
> So what is the stability measure of "elpa", again?  Is it on par with
> the Emacs's release branch?  Could it be?

It's more stable than 'git clone' but less stable than using a release.

And this will stay that way while we're using it to help stabilize 
package versions, too.

 > If not, why not?

I don't think we want ELPA to have the same frequency of package 
releases as Emacs itself.

We could add a new archive, though. Like elpa-very-stable. Which would 
be pushed, for example, the same versions of bundled packages that go 
into Emacs releases. Or something like that.

>> Neither keeps the previous versions around, so it's not like the user at
>> any point could make a choice to install a minor version update instead
>> of going up a full major version.
> 
> That in itself is a serious deficiency, IMO, and we should fix it by
> providing the "downgrade" option.

Doing that for every package will mean more work for everybody around, 
and it'll also increase the costs of hosting packages considerably.

>>> I don't see what development pace has to do with this issue.  If a
>>> package is being developed at a high pace, it might mean that the
>>> stable version will lag more.  But what does this change in principle?
>>
>> It matters when we're talking not about simple additional, optional
>> features, but about changes that keep pace with the evolution of the LSP
>> standard, with new LSP servers that arrive, new changes in existing
>> protocols. Things that users come to expect to be able to use now, and
>> not in 3 years.
> 
> Users who must have these features (presumably because they must use
> servers which require that) will have to give up some stability
> expectations.  Exactly like users who must have some new enough
> feature of Emacs which is only available on the master branch.
> 
> So once again: what is fundamentally different or new about packages
> which develop at fast pace, in the context of this discussion?  Why do
> we need to bring up those details here?

It's an issue of whether a "stable" Eglot could actually be useful 2 
years later for most people. If not (I admit I don't know for certain), 
we should choose to focus on other scenarios and needs, and less on 
stability in this particular case.

>> Taking a conservative stance can work when the ecosystem supports it
>> too. E.g. if every LSP server that we support now had an implicit
>> promise to be properly maintained for 3 years since, at least. I don't
>> think that's the case. Almost every one could be deprecated in that time
>> frame, with community being recommended to switch to a newer one.
> 
> How does this help us move forward with the discussion and with
> handling this issue?  Conservative doesn't mean stupid; if some
> external factor changes outside of our control and forces us to make
> potentially destabilizing changes, what else can we do that requires a
> discussion?

No, I don't think that conservative means stupid. I am fairly 
conservative in some things, just not in others.

Or take Joao: he's at the moment being conservative about how the users 
install Eglot and have had it upgraded over time when using all previous 
releases of Emacs.

> And how does it impact what we should do about
> categorizing package releases according to their stability and about
> the decisions which version to bundle with Emacs?

No, that's a different question. One I had hopefully addressed above.

>> To clarify: I made that suggestion from the premise that we do expect
>> and encourage a fair fraction of the users to use just the versions of
>> Eglot and Eldoc that come with Emacs 29, and never upgrade to the latest
>> ones.
> 
> AFAIU, João is of the opposite opinion: he thinks that most Eglot
> users will want the latest version on ELPA, or at least a version that
> is newer than the one bundled with Emacs.

I am of the above opposite opinion as well, but I can appreciate your 
POV as well, I think. And, well, if the ultimate decision is made using 
the latter principle, we still could make the most of it.

>> And as per above explanation, Eglot 0.14 depends on Eldoc 0.14.0 not
>> because it's the very latest and spiffiest version, but because it needs
>> a particular feature from it.
> 
> And there's no reasonable way of adding to Eglot 1.14 some fallback or
> compatibility shim to remove that dependency?

Everything's possible. It might not even be too hard.

But if we're talking about the improvement I was eyeing (better 
eglot<->eldoc behavior all around), then a simple shim won't cut it.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[João Távora]

On Wed, Apr 19, 2023 at 8:25 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > AFAIU, João is of the opposite opinion: he thinks that most Eglot
> > users will want the latest version on ELPA, or at least a version that
> > is newer than the one bundled with Emacs.
> I am of the above opposite opinion as well, but I can appreciate your

From my experience, yes.  Overwhelmingly so.  People get excited.
Reddit posts I didn't order appear out of some anonymous user
enthusiasm annoucing the feature :-) But I can also, like Dmitry,
appreciate your opinion, Eli.  Users that don't  want the latest version
on ELPA don't have to get it, by any means.  Of course not.

Here is, taken from the sibling discussion, myrecommendation to
Eglot users:

 "Dear user,

  Emacs 29 will come with the most stable version I can offer, and I
  can tell you, prospective user, that it has been through as much
  of the Emacs 29 pretest period as possible. If you, prospective user,
  are interested in the features listed in etc/EGLOT-NEWS, you may try
  upgrading to the latest version (I hope it will be easy, good luck!)
  In the unlikely event that things go sour for your particular
  LSP server and use case, you have my deepest apologies, and you
  should revert back to the version bundled with Emacs 29.

  Thank you,
  Your maintainer"


João

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Wed, 19 Apr 2023 22:25:08 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> >> I have a different answer from all that had been presented here: because
> >> the user can uninstall it.
> > 
> > User can also downgrade to a previous version of the package, don't
> > they?  Or if they cannot, they should be able to do that.
> 
> Even if there was such possibility, they wouldn't have a single stable 
> version to go back to (they'd have to make a choice). When we bundle a 
> version, that's one fewer question to answer.
> 
> > Once that
> > is possible, what exactly is the difference between these two kinds of
> > packages?  Downgrading to an older version when a newer one is buggy
> > or otherwise unsatisfactory should be supported for all packages.
> 
> Then the difference would be that we hand-picked a set of particular 
> versions of packages, whereas for third-party packages that was done (or 
> failed to be done) by other people we have no responsibility over.
> 
> > Also, does package.el support "downgrading" to the bundled version?
> > Did anyone actually try that?  In particular, what happens with the
> > dependencies the user upgraded together with the package being
> > "uninstalled", due to the minimum requirements of that package?
> 
> It should work by "uninstalling" the package. An when you uninstall a 
> package, package.el warns you about any dependencies that would be 
> broken this way. Someone should test it just in case, of course.
> 
> But the important part is that the bundled package stays installed. You 
> asked why we can encourage people to upgrade said packages freely. One 
> answer is that they have a safety net.

All those differences disappear when we are discussing core packages.
Which is my only focus in this discussion.

> >> We have elpa and elpa-devel. The latter is "explicitly unstable" and the
> >> former is "checkpoint releases".
> > 
> > So what is the stability measure of "elpa", again?  Is it on par with
> > the Emacs's release branch?  Could it be?
> 
> It's more stable than 'git clone' but less stable than using a release.
> 
> And this will stay that way while we're using it to help stabilize 
> package versions, too.

I very much hope it doesn't stay that way for core packages.

>  > If not, why not?
> 
> I don't think we want ELPA to have the same frequency of package 
> releases as Emacs itself.

This is a red herring.  Emacs releases are less frequent because Emacs
is a large collection of packages, and thus the period required for
its stabilization is much longer.  A single core package can apply
stability criteria and still not go anywhere near the Emacs release
frequency.  It does need more effort on the part of the package
developers, but the effect on the release frequency will be minor.

> > Users who must have these features (presumably because they must use
> > servers which require that) will have to give up some stability
> > expectations.  Exactly like users who must have some new enough
> > feature of Emacs which is only available on the master branch.
> > 
> > So once again: what is fundamentally different or new about packages
> > which develop at fast pace, in the context of this discussion?  Why do
> > we need to bring up those details here?
> 
> It's an issue of whether a "stable" Eglot could actually be useful 2 
> years later for most people.

"Two years" because you think the release frequency of Eglot will be
slowed down to such abysmally low value?  As explained above, I think
this is not going to happen, so there's no need to assume this will be
the outcome.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 12:47, Eli Zaretskii wrote:

>>> Also, does package.el support "downgrading" to the bundled version?
>>> Did anyone actually try that?  In particular, what happens with the
>>> dependencies the user upgraded together with the package being
>>> "uninstalled", due to the minimum requirements of that package?
>>
>> It should work by "uninstalling" the package. An when you uninstall a
>> package, package.el warns you about any dependencies that would be
>> broken this way. Someone should test it just in case, of course.
>>
>> But the important part is that the bundled package stays installed. You
>> asked why we can encourage people to upgrade said packages freely. One
>> answer is that they have a safety net.
> 
> All those differences disappear when we are discussing core packages.
> Which is my only focus in this discussion.

Disappear, why? The core packages are the only kind of "bundled" 
packages we have (for now), and so they are the only ones that can be 
safely reverted to the bundled version via uninstallation.

>>>> We have elpa and elpa-devel. The latter is "explicitly unstable" and the
>>>> former is "checkpoint releases".
>>>
>>> So what is the stability measure of "elpa", again?  Is it on par with
>>> the Emacs's release branch?  Could it be?
>>
>> It's more stable than 'git clone' but less stable than using a release.
>>
>> And this will stay that way while we're using it to help stabilize
>> package versions, too.
> 
> I very much hope it doesn't stay that way for core packages.

Let me try again: one of the uses of ELPA is to push out the package 
release "into the real world" so that more people can test it.

So that "N weeks later" we can consider is "stable" and be content to 
have it in an Emacs release.

By the very definition, the same release wasn't "stable" N weeks ago, 
just as it was published. So ELPA cannot be considered to have the same 
level of stability because we also use it for testing this way (among 
other things).

>>   > If not, why not?
>>
>> I don't think we want ELPA to have the same frequency of package
>> releases as Emacs itself.
> 
> This is a red herring.  Emacs releases are less frequent because Emacs
> is a large collection of packages, and thus the period required for
> its stabilization is much longer.  A single core package can apply
> stability criteria and still not go anywhere near the Emacs release
> frequency.  It does need more effort on the part of the package
> developers, but the effect on the release frequency will be minor.

I guess you're right.

But note that as per above, if we wanted ELPA to be the "stable" source, 
we'd need some other testing ground before pushing releases to it. E.g. 
we could just wait for people using the master branch to report 
problems. But since there are fewer of them than there are ELPA users, 
it stands to reason that the stabilization periods must become longer. 
Maybe not two years, but longer than with ELPA.

>>> Users who must have these features (presumably because they must use
>>> servers which require that) will have to give up some stability
>>> expectations.  Exactly like users who must have some new enough
>>> feature of Emacs which is only available on the master branch.
>>>
>>> So once again: what is fundamentally different or new about packages
>>> which develop at fast pace, in the context of this discussion?  Why do
>>> we need to bring up those details here?
>>
>> It's an issue of whether a "stable" Eglot could actually be useful 2
>> years later for most people.
> 
> "Two years" because you think the release frequency of Eglot will be
> slowed down to such abysmally low value?  As explained above, I think
> this is not going to happen, so there's no need to assume this will be
> the outcome.

No, no. I'm talking the scenario with users who are going to stay with 
the version of Eglot that comes with Emacs 29, for two years without 
upgrading. That might not be wise in a lot of cases (admittedly, this is 
just a guess at this point, looking at the speed of changes around the 
community), so we should make it easy to upgrade.

I don't believe the upgrade must be made automatic, however.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 16:03:16 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> On 20/04/2023 12:47, Eli Zaretskii wrote:
> 
> >>> Also, does package.el support "downgrading" to the bundled version?
> >>> Did anyone actually try that?  In particular, what happens with the
> >>> dependencies the user upgraded together with the package being
> >>> "uninstalled", due to the minimum requirements of that package?
> >>
> >> It should work by "uninstalling" the package. An when you uninstall a
> >> package, package.el warns you about any dependencies that would be
> >> broken this way. Someone should test it just in case, of course.
> >>
> >> But the important part is that the bundled package stays installed. You
> >> asked why we can encourage people to upgrade said packages freely. One
> >> answer is that they have a safety net.
> > 
> > All those differences disappear when we are discussing core packages.
> > Which is my only focus in this discussion.
> 
> Disappear, why? The core packages are the only kind of "bundled" 
> packages we have (for now), and so they are the only ones that can be 
> safely reverted to the bundled version via uninstallation.

Exactly.  That answers your 'why?" question, AFAICT.

> Let me try again: one of the uses of ELPA is to push out the package 
> release "into the real world" so that more people can test it.
> 
> So that "N weeks later" we can consider is "stable" and be content to 
> have it in an Emacs release.
> 
> By the very definition, the same release wasn't "stable" N weeks ago, 
> just as it was published. So ELPA cannot be considered to have the same 
> level of stability because we also use it for testing this way (among 
> other things).

Perhaps there's some kind of misunderstanding here.  To me, declaring
a version of a package "stable" just means some label in its metadata,
which is exposed to package.el and to the users.  So when the package
developer decides that "N weeks have passed", he or she will add that
label, thus making the corresponding version be considered "stable".
Next time users check for updates they will see that, and could then
upgrade to the "next stable" version if they so desire.

IOW, the version on ELPA that wasn't "stable" before, and was used for
testing, now becomes "stable".

Am I missing something here?

> But note that as per above, if we wanted ELPA to be the "stable" source, 
> we'd need some other testing ground before pushing releases to it.

For core packages, that testing ground is the master branch.  Exactly
as for code in Emacs itself.  Once a version is declared "stable", its
changes will be cherry-picked to the Emacs release branch, thus making
it part of the future bug-fix release.

> E.g. 
> we could just wait for people using the master branch to report 
> problems. But since there are fewer of them than there are ELPA users, 
> it stands to reason that the stabilization periods must become longer. 
> Maybe not two years, but longer than with ELPA.

Why do you assume that people who use the Emacs master branch don't
use core packages? or that there are fewer of them than those who use
the release branch?  And, since the code of the core package that is
on master is also on ELPA, I don't see how we can lose any testing,
because reports from users who use non-"stable" versions from ELPA are
as good as from those who use the package via the builds of the Emacs
master branch.  So nothing is lost here, not that I could see.

> >>> Users who must have these features (presumably because they must use
> >>> servers which require that) will have to give up some stability
> >>> expectations.  Exactly like users who must have some new enough
> >>> feature of Emacs which is only available on the master branch.
> >>>
> >>> So once again: what is fundamentally different or new about packages
> >>> which develop at fast pace, in the context of this discussion?  Why do
> >>> we need to bring up those details here?
> >>
> >> It's an issue of whether a "stable" Eglot could actually be useful 2
> >> years later for most people.
> > 
> > "Two years" because you think the release frequency of Eglot will be
> > slowed down to such abysmally low value?  As explained above, I think
> > this is not going to happen, so there's no need to assume this will be
> > the outcome.
> 
> No, no. I'm talking the scenario with users who are going to stay with 
> the version of Eglot that comes with Emacs 29, for two years without 
> upgrading. That might not be wise in a lot of cases (admittedly, this is 
> just a guess at this point, looking at the speed of changes around the 
> community), so we should make it easy to upgrade.
> 
> I don't believe the upgrade must be made automatic, however.

Users of Emacs 29 that need a newer Eglot don't need to stay with the
version we shipped.  First, if the machinery to promote Eglot versions
to "stable" status is in place, bug-fix 29.x releases could have newer
versions of Eglot bundled with them; our rate of putting out bug-fix
releases is much higher than once every 2 years.  Users who cannot
wait until the next bug-fix release, but still want stability, will be
able to upgrade their Eglot to a newer version, provided that we mark
some newer version on ELPA "stable" after "N weeks" or so.  Finally,
users who want newer Eglot badly and are willing to sacrifice some
stability will update to the latest-and-greatest version on ELPA, even
if it is not "stable" yet.

So I don't see how this can lose, if we indeed have the above system,
or something like it, in place.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 17:03, Eli Zaretskii wrote:
> Perhaps there's some kind of misunderstanding here.  To me, declaring
> a version of a package "stable" just means some label in its metadata,
> which is exposed to package.el and to the users.  So when the package
> developer decides that "N weeks have passed", he or she will add that
> label, thus making the corresponding version be considered "stable".
> Next time users check for updates they will see that, and could then
> upgrade to the "next stable" version if they so desire.
> 
> IOW, the version on ELPA that wasn't "stable" before, and was used for
> testing, now becomes "stable".

Okay. That can work if:

- We're able to force at least some older versions of the package to 
stay around in the repository. And we'll probably need some new UI in 
package.el to be able to choose among the versions.

- We're able to retroactively add some metadata to already published 
versions, like calling some older one "stable" after some time has passed.

> Users of Emacs 29 that need a newer Eglot don't need to stay with the
> version we shipped.  First, if the machinery to promote Eglot versions
> to "stable" status is in place, bug-fix 29.x releases could have newer
> versions of Eglot bundled with them; our rate of putting out bug-fix
> releases is much higher than once every 2 years.

That's fair.

> Users who cannot
> wait until the next bug-fix release, but still want stabil
> 
> ity, will be
> able to upgrade their Eglot to a newer version, provided that we mark
> some newer version on ELPA "stable" after "N weeks" or so.  Finally,
> users who want newer Eglot badly and are willing to sacrifice some
> stability will update to the latest-and-greatest version on ELPA, even
> if it is not "stable" yet.

So we seem to agree that the ability to upgrade is important as well.

Whether we're able to transition to a new system with stability tags, 
etc, is yet to be seen.

> So I don't see how this can lose, if we indeed have the above system,
> or something like it, in place.

The main downsides of this are probably obvious:

- The development work required.
- The additional ongoing package maintenance work that is implied by 
this design.

Whether it's really worth it in the end, I don't know. E.g. Joao seems 
to think that additional stable releases for Eglot won't have much of an 
audience (which seems sensible to me), but Arne's messages seem to 
indicate a preference for stability. And the larger community seems to 
be using MELPA quite happily, which is as unstable as it gets.

Maybe it won't be used much by Eglot, but Org devs take a liking to it. 
On my part, I would maybe tag a "stable" release once a year or so.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 17:22:03 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> > Users who cannot
> > wait until the next bug-fix release, but still want stabil
> > 
> > ity, will be
> > able to upgrade their Eglot to a newer version, provided that we mark
> > some newer version on ELPA "stable" after "N weeks" or so.  Finally,
> > users who want newer Eglot badly and are willing to sacrifice some
> > stability will update to the latest-and-greatest version on ELPA, even
> > if it is not "stable" yet.
> 
> So we seem to agree that the ability to upgrade is important as well.

Yes.

> Whether we're able to transition to a new system with stability tags, 
> etc, is yet to be seen.

Indeed.  In fact, it remains to be seen even if we have a broad-enough
agreement to consider transitioning to such a system a Good Thing.

> The main downsides of this are probably obvious:
> 
> - The development work required.
> - The additional ongoing package maintenance work that is implied by 
> this design.

Agreed.  I just don't see how will we be ever able to move to keeping
core packages only on ELPA if we don't transition to such a system, or
something like it.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 17:42, Eli Zaretskii wrote:
>> The main downsides of this are probably obvious:
>>
>> - The development work required.
>> - The additional ongoing package maintenance work that is implied by
>> this design.
> Agreed.  I just don't see how will we be ever able to move to keeping
> core packages only on ELPA if we don't transition to such a system, or
> something like it.

When you say "keeping core packages only on ELPA", what do you mean exactly?

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Eli Zaretskii]

> Date: Thu, 20 Apr 2023 18:30:47 +0300
> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
> 
> When you say "keeping core packages only on ELPA", what do you mean exactly?

That core packages will be only in elpa.git, not in emacs.git.  Then
we'd "bundle" them with Emacs when preparing the release tarballs.

I believe this is a long-term plan wrt core packages.  It was even
discussed a couple of times here.

Re: Stability of core packages

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 20 Apr 2023 18:30:47 +0300
>> Cc: joaotavora@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>> 
>> When you say "keeping core packages only on ELPA", what do you mean exactly?
>
> That core packages will be only in elpa.git, not in emacs.git.  Then
> we'd "bundle" them with Emacs when preparing the release tarballs.
>
> I believe this is a long-term plan wrt core packages.  It was even
> discussed a couple of times here.

For people building Emacs from source, would this mean that they
would have to manually install core packages?  Also, this sounds like it
would make it impossible for package.el to be distributed on ELPA, as
proposed in another thread.

Re: Stability of core packages

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Dmitry Gutov <dmitry@gutov.dev>,  joaotavora@gmail.com,
>   emacs-devel@gnu.org
> Date: Thu, 20 Apr 2023 17:26:57 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That core packages will be only in elpa.git, not in emacs.git.  Then
> > we'd "bundle" them with Emacs when preparing the release tarballs.
> >
> > I believe this is a long-term plan wrt core packages.  It was even
> > discussed a couple of times here.
> 
> For people building Emacs from source, would this mean that they
> would have to manually install core packages?

No.  We'd need to change either the build process or the structure of
emacs.git (and consequently what happens when you say "git pull") so
that manual installation will not be required.

This was all discussed in the past, and this aspect was mentioned as
one of the issues that would have to be solved.

> Also, this sounds like it would make it impossible for package.el to
> be distributed on ELPA, as proposed in another thread.

It's possible, yes.  But no one has yet got this far into designing
such a system as to figure all that out.  So other alternatives could
be workable.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov]

On 20/04/2023 18:49, Eli Zaretskii wrote:
>> Date: Thu, 20 Apr 2023 18:30:47 +0300
>> Cc:joaotavora@gmail.com,emacs-devel@gnu.org
>> From: Dmitry Gutov<dmitry@gutov.dev>
>>
>> When you say "keeping core packages only on ELPA", what do you mean exactly?
> That core packages will be only in elpa.git, not in emacs.git.  Then
> we'd "bundle" them with Emacs when preparing the release tarballs.

Okay then, though I'd call them just "bundled" packages from that point. 
Since "core" is an ELPA term for packages developed inside emacs.git.

That separation could also be considered a win, saving time for some of 
the developers. Which could make up for the extra overhead with managing 
stable tags.

> I believe this is a long-term plan wrt core packages.  It was even
> discussed a couple of times here.

Note that to do the above, we don't really have to implement this stuff 
as a feature in Emacs (as a UI in package.el), or in elpa-admin. At 
least not right away.

As long as we've agreed on a principle to choose releases to tag as 
stable, we could just write those git revision down somewhere (using git 
tags, or some text file in the Emacs repo, or a text file on a server 
somewhere, ...). And then use that info when bundling the packages.

Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Lynn Winebarger]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]

On Thu, Apr 20, 2023, 11:50 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Thu, 20 Apr 2023 18:30:47 +0300
> > Cc: joaotavora@gmail.com, emacs-devel@gnu.org
> > From: Dmitry Gutov <dmitry@gutov.dev>
> >
> > When you say "keeping core packages only on ELPA", what do you mean
> exactly?
>
> That core packages will be only in elpa.git, not in emacs.git.  Then
> we'd "bundle" them with Emacs when preparing the release tarballs.
>
> I believe this is a long-term plan wrt core packages.  It was even
> discussed a couple of times here.
>

The search function for the emacs-devel archive is still broken.  For
example, the first result (in chronological ordering) for "concurrency" is
from 2020.

Lynn

[-- Attachment #2: Type: text/html, Size: 1588 bytes --]

Re: What is :core? (was: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot))

[Lynn Winebarger]

On Tue, Apr 18, 2023 at 8:58 AM Eli Zaretskii <eliz@gnu.org> wrote:
> This discussion no longer belongs to the bug tracker, so I'm moving it
> to emacs-devel and changing its Subject.  Please reply here, not
> there.

Is there any documentation for what ":core" refers to?  Executing
"grep -rn ':core' *" in the emacs source yields the following, which
doesn't include any explanation:


ChangeLog.3:81561: Turn Eldoc, Xref and Project into GNU ELPA :core packages
ChangeLog.3:236132: * lisp/emacs-lisp/let-alist.el: Now an Elpa :core package
ChangeLog.4:12044: project.el is a GNU ELPA :core package, so this
kind of trick isn't
lisp/emacs-lisp/eldoc.el:11:;; This is a GNU ELPA :core package.
Avoid functionality that is not
lisp/emacs-lisp/let-alist.el:12:;; This is an Elpa :core package.
Don't use functionality that is not
lisp/jsonrpc.el:10:;; This is a GNU ELPA :core package.  Avoid
functionality that is not
lisp/progmodes/project.el:7:;; This is a GNU ELPA :core package.
Avoid using functionality that
lisp/progmodes/flymake.el:11:;; This is a GNU ELPA :core package.
Avoid functionality that is not
lisp/progmodes/xref.el:7:;; This is a GNU ELPA :core package.  Avoid
functionality that is not
lisp/progmodes/eglot.el:12:;; This is a GNU ELPA :core package.  Avoid
adding functionality
lisp/progmodes/eglot.el:57:;;   available as GNU ELPA :core packages.
Historically, a number of
lisp/progmodes/eglot.el:58:;;   :core packages were added or reworked
in Emacs to make this
test/lisp/progmodes/eglot-tests.el:39:;; IMPORTANT: Since Eglot is a
:core ELPA package, these tests are

Lynn

Re: What is :core? (was: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot))

[João Távora]

No one knows, apparently, :-D

But what it means in practice, is that some files (or sets of files)
in the Emacs.git repository on the master branch are watched daily by the
scripts that govern the distribution of packages  in https://elpa.gnu.org.
If those scripts notice a change in the ";; Version:" header of said files,
a packaged-up version of that code is tar'ed into a file named
package-<version>.tar.lz and archived somewhere in that site.

Then the tar.lz file can be installed in Emacs via 'package-install',
'use-package', or various other third-party means.  Crucially (!) the
package can be installed even in older (sometimes quite a bit older)
versions of Emacs, depending on the value of "Emacs" in the
";; Package-Requires" header.  When installed in these Emacs versions,
the package will dutifully  perform the job _as if_ you were using it
as part of Emacs master version at the commit where the ";; Version:"
header was updated.

The ";; Version:" header of the Emacs.git repository files is
updated when the maintainers of this package deem the code and its
dependencies sufficiently stable and ready to enter this distribution
channel.

All in all, it is, IMHO, the best recent development we have had
to escape the glacially slow frequency of rolling out new features
in Emacs to the general public.  As a user, you don't have to
compile a master Emacs, and -- frequently -- you don't even need to
upgrade your Emacs.

Entire "Emacs distributions" are built around this.
Doom Emacs, for example, requires just Emacs 27.2.  But it gets the
user all the new stuff via ELPA (and MELPA, a cousin of ELPA).

There is an issue that no one foresaw, and that is bug#62720 where
this all started: When a package that wasn't :core becomes :core (it
had never happened but it did shortly before Emacs 29 was cut), then
some of the installation methods some people were using (notably Emacs's
built-in package manager) will start doing unexpected things when given
the same package to install.

João

On Wed, Apr 19, 2023 at 1:31 PM Lynn Winebarger <owinebar@gmail.com> wrote:
>
> On Tue, Apr 18, 2023 at 8:58 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > This discussion no longer belongs to the bug tracker, so I'm moving it
> > to emacs-devel and changing its Subject.  Please reply here, not
> > there.
>
> Is there any documentation for what ":core" refers to?  Executing
> "grep -rn ':core' *" in the emacs source yields the following, which
> doesn't include any explanation:
>
>
> ChangeLog.3:81561: Turn Eldoc, Xref and Project into GNU ELPA :core packages
> ChangeLog.3:236132: * lisp/emacs-lisp/let-alist.el: Now an Elpa :core package
> ChangeLog.4:12044: project.el is a GNU ELPA :core package, so this
> kind of trick isn't
> lisp/emacs-lisp/eldoc.el:11:;; This is a GNU ELPA :core package.
> Avoid functionality that is not
> lisp/emacs-lisp/let-alist.el:12:;; This is an Elpa :core package.
> Don't use functionality that is not
> lisp/jsonrpc.el:10:;; This is a GNU ELPA :core package.  Avoid
> functionality that is not
> lisp/progmodes/project.el:7:;; This is a GNU ELPA :core package.
> Avoid using functionality that
> lisp/progmodes/flymake.el:11:;; This is a GNU ELPA :core package.
> Avoid functionality that is not
> lisp/progmodes/xref.el:7:;; This is a GNU ELPA :core package.  Avoid
> functionality that is not
> lisp/progmodes/eglot.el:12:;; This is a GNU ELPA :core package.  Avoid
> adding functionality
> lisp/progmodes/eglot.el:57:;;   available as GNU ELPA :core packages.
> Historically, a number of
> lisp/progmodes/eglot.el:58:;;   :core packages were added or reworked
> in Emacs to make this
> test/lisp/progmodes/eglot-tests.el:39:;; IMPORTANT: Since Eglot is a
> :core ELPA package, these tests are
>
> Lynn



-- 
João Távora

Re: What is :core? (was: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot))

[Eli Zaretskii]

> From: Lynn Winebarger <owinebar@gmail.com>
> Date: Wed, 19 Apr 2023 08:31:16 -0400
> Cc: Dmitry Gutov <dmitry@gutov.dev>,
>  João Távora <joaotavora@gmail.com>, 
>  emacs-devel <emacs-devel@gnu.org>
> 
> Is there any documentation for what ":core" refers to?  Executing
> "grep -rn ':core' *" in the emacs source yields the following, which
> doesn't include any explanation:

It is ELPA terminology, so the answer is in ELPA's file elpa-packages:

  ;; List of packages that are maintained externally.
  ;; The list is made of elements of the form (NAME KIND URL OPTS...).
  ;; See `admin/README' for further documentation about the format.
  ;;
  ;; Where NAME is the name of the package;
  ;;
  ;; KIND can be one of:
  ;;  :url      = kept in a separate `externals/<name>' branch.
  ;;  :core     = part of GNU Emacs repository.