Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

["João Távora" <joaotavora@gmail.com>]

On Tue, Apr 18, 2023 at 3:47 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Tue, 18 Apr 2023 15:02:01 +0100
> > Cc: Dmitry Gutov <dmitry@gutov.dev>, emacs-devel@gnu.org
> >
> > On Tue, Apr 18, 2023 at 1:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > > without by side effect installing a newer and potentially less stable
> > > ElDoc.  (I also am surprised that change is a must for Eglot 1.15.)
> >
> > It's not a "must".  Eglot can work without it.  The problem happens in
> > ElDoc and doesn't affect its interface.
>
> Then what Dmitry said about Eglot 1.15 being dependent on that change
> in ElDoc is not relevant to the issue at hand, which is whether Eglot
> 1.15 could be bundled with Emacs 29.1.

It is quite relevant.

Eglot 1.15 depends on many other things in ElDoc.  That particular
bugfix might not -- or might indeed -- included, depending on what
other non-bugfix things  Eglot will require of ElDoc at the time.
ElDoc is now 1.14 but it could be 1.15 at the time motivated by
Eglot 1.15/16/17.

And even in ElDoc 1.14 there are already things (the :echo display
option) that are not in Emacs 29. And Eglot 1.14 directly depends
on those things.  It relies on them to do a good job.

I would think that if you oppose a bugfix backport you would also
oppose a _feature_ backport, which usually is (and is indeed in
this case) a much more complicated, "scary", bug-prone development.

> > But Eglot relies on ElDoc as a whole.  In general you can't expect
> > to have new features in Eglot without some advancing of the
> > infrastructure that Eglot depends on.
>
> Understood.  My point is that if you want Eglot users to be able to
> upgrade to a newer versions, you need to have compatibility layers, to
> avoid the need to upgrade too many other packages, which might hamper
> stability.  Otherwise, we cannot in good faith recommend that users of
> stable Emacs update their core packages without a second thought.

Yes, and this is why each released version of Eglot specifies exactly
the _released_ versions of its dependencies that it depends on. The
dependency language isn't very elaborated (there is no way to say
Eglot 1.1234  depends on ElDoc between 1.23 and 1.56), but it's been
enough.  It's not much different from what is found in numerous
other package systems.

Another matter is what package.el does with this info and the
implied graph. Transitive dependencies are known not to matter.
And if a package requires ElDoc 1.14 but 1.99 is already available
package.el will just go ahead and install the latest.  Always has,
I'm afraid (anyone can correct me on this if I'm mistaken, which
I'd love to be).

I think the experience of the straight.el and elpaca.el package
manager authors could be useful here to us.  If someone knows them
by heart, please do tell me.  Who knows, maybe what we want
is to bring one of these into Emacs and kill package.el. straight.el
is what a lot of people seem to be using these days anyway (and the cooler
kids elpaca.el).

> > So there _isn't_ a way to partially upgrade a package and not its
> > dependencies.
>
> Updating a package P1 should require update of as few packages P2...Pn
> as possible.  Ideally, none at all.

And very often that does happen, I suppose.  Not every Eglot release
_requires_ installation of new versions of its dependencies.  But some
do.

> Users should be able to decide
> whether they want or don't want to update any single package without
> also needing to decide whether they are okay with updating half a
> dozen of others.  This should be our goal, because otherwise updating
> a package will be unsafe if you use any Emacs except master (and thus
> don't care much about stability anyway).

I don't know how you can meet that goal in general.  We should
indeed work to minimize dependencies and do things that don't affect
interfaces and don't require changes other's interfaces.  As much as
possible, I agree.  But in general programs rely on other programs.
Dependencies exist.  Like in many other package managers, users
should be presented with the consequences of their wishes, when
that is feasible and when we can do so without breaking their
configs.

That's my idea of stability anyways.

> > "there shouldn't be a single set of criteria governing core
> > packages".  Then we can teach Emacs's upgrade mechanisms to
> > deal with each set differently, carefully examining the
> > requirements for each set.
>
> Sure, but we need to have these sets, actually.  Right now, we don't,
> not for every core package out there anyway.

[ Just a note that :core packages are not "out there", they're
"in here" -- by definition.  That's their main selling point
and precisely what we should take advantage of :-) ]

I propose two main sets of :core packages to start with.

Set 1 - :core packages that have always been core, i.e. they started
their life in the code

Set 2 - :core packages that started their life somewhere else
(GNU ELPA, Github, etc), were installable through ELPA interfaces
and now are :core (which means Git-versioned in Emacs.git but
still installable via ELPA).  Two known such packages are Eglot
and Use-Package.  Both depend on _other_ :core packages.  Eglot
on many of these, Use-Package only on "bind-key", which is
also new, but didn't seem to be installable independently
before Use-Package appeared.

This isn't the end of the analysis, of course.  I'm just
providing these two sets to see if it rings a bell with participants,
because from the opinions I collected in that very long bug, they
seem to make and map neatly onto the requirements that each
party put forth:

- That members of set 1 shouldn't be upgradable "willy nilly" to
maintain exact backward-compatibility.

- That members of set 2 should be upgraded in much easier fashion
because that's what guarantees that people's configs already
doing so won't break.

> > I'd like you, if possible, to also respond (here, if you prefer) to the
> > points I raised in my own reply to Dmitry's message you're replying
> > to.  This is the message;:
> >
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62720#529
>
> I didn't respond because I had nothing to say to that which I didn't
> already say in response to Dmitry's message.

I proposed a simple bugfix to bug#62720, based precisely on
the above idea of separation of sets.  A dead-simple 7-line
fix.

That patch was +1'ed by Philip and Dmitry and hasn't been
addressed by you.  Philip, by the way, tried until the end to
give you a patch that also didn't have the non-interactive
package-install/use-package lockout, but you insisted.  Why? Why
keep it for those members of Set 1?  What is there to be gained?
Do you acknowledge what there is to be lost?

But moving on from that minor tragedy, it's nevertheless easy
to note that bug#62720 conflates the two sets.  We should work
to improve that, hopefully in time to avert damage.  The code
required to express these sets in Elisp is, of course, quite trivial.

João