Re: Stability of core packages (was: Not easy at all to upgrade :core packages like Eglot)

[Dmitry Gutov <dmitry@gutov.dev>]

On 20/04/2023 03:13, João Távora wrote:

>> On 20/04/2023 01:49, João Távora wrote:
>> What kind of semantics do we get with it?
> 
> The closest possible to Emacs 28 for _all_ users (Eglot users and
> the Emacs users that are vulnerable to furtive upgrades and which
> Eli is thinking of)

As if Eglot wasn't bundled in the first place, you mean.

>> 1. (use-package eglot :ensure t) considers select builtin packages to be
>> "not installed" for the purposes of ":ensure t".
>> 2. 'M-x package-install' allows installing them. It doesn't allow
>> installing any other package for which (package-installed-p 'xxx)
>> returns t, but allows installing (essentially upgrading) these ones
>> (either just eglot, or both eglot and use-package).
>> 3. 'M-x package-update RET eglot RET' still doesn't work unless eglot
>> has been "upgraded" at least once via other means.
>>
>> Is that logical? Is even just 1+2 logical?
> 
> Neither is.  IMO we have to think in terms of expectations for each
> package, not in terms of a heavy handed blunt class o packages like
> "builtin".  Because Eglot wasn't a builtin and now is.  So this is
> why my patch includes that predicate `package--safely-upgradeable-builtin`
> The name is not good, I admit.  It should be
> `package--hitherto-casually-upgradeable-builtin` maybe.

This kind of option still makes sense, but I'd put in a different place: 
it would affect which packages are upgraded by 'M-x package-update-all' 
or 'U' in the packages list.

>> because the package can't be installed (no connection). Should they
>> remove ":ensure t"? Perhaps. But the documentation says that that option
>> checks that the package is "installed automatically if not already
>> present on your system". Seems legit, right? Why would startup fail when
>> Eglot is already present on the system?
> 
> Because we want it to do the same it did in Emacs 28?  But I see what
> you mean.  Maybe the :ensure for packages in that particular subclass
> should mean "try a bit harder to upgrade these, but don't blow up if
> you can't -- just try next time".

... maybe.

Though a meticulous user (some of them, maybe not others) might still 
wonder why an upgrade if attempted at all, even though the package is 
already available.

>> Or, to put it another way, why did we decide to bundle Eglot with Emacs
>> if the first thing we're going to do is to try to download it anyway?
> 
> Who is "we"?

You/we/emacs-devel as a whole.

> I know for a fact some people are.  The enthusiasts are,
> the feature seekers, are immediately.  Others will experiment, I
> assume.  Very often one experimentation that happens is you just
> save all your .emacs.d to the side safely and let er rip with
> whatever init.el someone (trustworthy presumably) gave you.

I agree that upgrading it should be made as easy as possible. But it 
should require explicit actions.

> ( What, you're going to say you've never tried the spacemacs
>    or doom emacs or nano emacs just to see what all the hype
>    is about??? :-) )
> 
> But others, like me from time to time when working on VMs via ssh
> machines,  will be very happy with sudo apt install emacs in
> said machines (if not done so already) and use a vanilla
> configurationless emacs.  And if you add a "clangd" to that
> invocation and Emacs happens to be 29 you'll have nice LSP
> for the first time in many years.  Personally, for _my_ quick
> hacks in those situations, I don't bother with M-x package-install
> latest-and-greatest, just as I don't bother bringing my heavy
> config via ssh to that machine.  I can't really, I'm frequently
> tmate'ing with other people there, we have to settle on Emacs -Q
> (+ electric-pair-mode and a few obvious others).

All good reasons. But then you add 'use-package :ensure t' to the mix, 
and something unexpected starts to happen. Maybe expected to you, but 
not to somebody other like you.

Maybe you'll put the recommended use-package form somewhere in the Eglot 
manual, right? It will be published on the web as well. Will it contain 
":ensure t" or not? If not, the users of Emacs 28 might fail to get the 
package installed. If it will, then the offline users might have 
problems, as described previously.

> But your question also has a completely different answer:
> A very good reason I wanted Eglot to be bundled with Emacs is now
> actually being put into question in this very thread.  I wanted
> to simplify Eglot's development as it advances with other :core
> packages.  For example, I wanted to split off its external
> completion style easily into a separate core package, as I did
> with Stefan earlier this year.  I want to be able to add a feature
> to ElDoc and use it in Eglot in the same patch, etc.  That part
> has worked fine.  It wasn't impossible with Eglot in GitHub, but
> it was harder.  Hopefully Eli is not asking for that bliss to end
> I understand hope he's asking to be circumspect in what
> is added, and not do gratuitous things like bump dependencies
> without reason.  I already do that, so no problem.

It seems like most of this could have worked out if Eglot moved to the 
emacs.git repository, but was removed from the distro before packaging? 
Just a thought, not that I advocate for doing that.

> Finally, if you're taking all this in the direction of "I told
> you so, don't you regret bringing Eglot into Emacs 29?", I don't
> blame you.  But I have to say: no, not yet.  I think having the
> beginnings of an LSP library in Emacs for major-modes to work
> with is excellent.  I like the attention and the high-standards
> other seasoned developers hold my code up to.  I like the Info
> manual and I like that emacs -Q bla.c -f eglot works just fine.

I like my told-you-so's as much as the next guy (just kidding; 
definitely more than the next guy), but that's not the point I was 
making. Just that it's here now, and it seems odd to avoid using it and 
hurry up to install the next version right away.

> This upgrade situation sucks, is true, and I should have foreseen
> it.  Spent too much time in my emacs-master worktree I guess.
> 
> Who knows, maybe this will all be OK after all?? Half the
> users are using "straight.el" anyway (and I really doubt
> it is hampered by these minutiae about dependencies, seeing
> as MELPA-land is teeming with way more "dependency hell").

I'm pretty sure the users on straight.el are straight-up illusion. It's 
the latest fashion on reddit all right, but it has its own problems, and 
the silent majority are still using the older methods.

Not to say it's a bad project (I'll try to migrate my authored packages 
to it one day, since it seems to make that natural), but for an average 
user? I don't think so.

> If nothing more happens on Emacs 29, which is the most likely
> outcome, I think I will just start recommending "straight.el"
> more, maybe learn it myself.  I had problems with it in the
> past but it's probably matured now.

I would suggest to start recommending ways to perform an upgrade 
explicitly (somewhere in the README, in the manual, and so on). Like 
'M-x package-upgrade', if we manage to get it fixed in Emacs 29.

And, okay, having a separate section in README about Emacs 29 might suck 
at first, but then the majority of the users move to it, and support 
will get easier, more uniform.

>> So... I understand the problem, but I think we shouldn't change the
>> functions in a way that makes them conflict with documentation or with
>> each other.
> 
> Does the latest patch introduce any such conflict besides the
> "no internet" case you mentioned?  Not that it's really a
> conflict IMO: it's just what Emacs 28 did with Eglot AFAICT.

That's the only hard breakage I managed to come up with. The others are 
of logical kind (functions and commands doing things somewhat against 
their documented behavior), but that's also not so great because people 
like their tools to make sense.

To name an example: 'package-install' will upgrade Eglot even though 
(package-installed-p 'eglot) already returns t.