Re: Reconsider defaults for use-package-vc-prefer-newest

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

Now that we're on Emacs 30.0.91, I feel the need to call this out as a fairly release-critical bug. Perhaps I should have emailed bug-gnu-emacs, but now that'd split the discussion over two places, so I'll keep it here.

If this is the kind of setting we can flip in a patch release, then there's no rush. Maybe someone can weigh in about that.

Otherwise it would remain as a source of instability for many years, and conscientious devs would have to insert a check for Emacs 30 that reminds the user that the package may be terribly outdated.

At least one other developer agrees the current setting is "fragile" (https://github.com/melpa/melpa/pull/9133#issuecomment-2351653325).

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: "Martin Edström" <meedstrom@runbox.eu>
> CC: "emacs-devel" <emacs-devel@gnu.org>
> Date: Sun, 15 Sep 2024 19:38:55 +0200 (CEST)
> 
> Now that we're on Emacs 30.0.91, I feel the need to call this out as a fairly release-critical bug. Perhaps I should have emailed bug-gnu-emacs, but now that'd split the discussion over two places, so I'll keep it here.
> 
> If this is the kind of setting we can flip in a patch release, then there's no rush. Maybe someone can weigh in about that.
> 
> Otherwise it would remain as a source of instability for many years, and conscientious devs would have to insert a check for Emacs 30 that reminds the user that the package may be terribly outdated.
> 
> At least one other developer agrees the current setting is "fragile" (https://github.com/melpa/melpa/pull/9133#issuecomment-2351653325).

I've read this thread, and I must confess that I'm not convinced.
Besides being waaay too late for such changes in Emacs 30, I also
don't think we have enough experience at this time to make such
decisions.  Both package-vc and use-package are relatively very new in
Emacs, and we have yet to collect enough user experience and
understanding of what are the expectations and how they should work
together.  Moreover, use-package-vc-prefer-newest is a new option in
Emacs 30, and we rarely make new optional behaviors take effect by
default as soon as they are introduced, to avoid behavior changes
which will surprise and perhaps annoy.  We made it an option to make
it easily adaptable to user preferences and needs.

Some of the rationale (like outdated Package-Version) also sounds like
it isn't our problem to solve.  It should be easy for the respective
package developers to get their act together in this matter.  It is
also TRT for them.  When a user uses use-package, he/she doesn't
necessarily want the latest commit, so the change isn't as obvious as
you make it sound, IMO.

Bottom line: I think no catastrophe happens if Emacs 30 is released
with the current default value of use-package-vc-prefer-newest.  It's
a user option, so customizing it is easy for people who want the
latest commit.  Even if we are making a mistake (for reasons we will
learn later), first time you do something there are always some rough
edges, so we can be excused for not realizing something due to lack of
experience.

Thanks.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edstrom]

> > From: "Martin Edström" <meedstrom@runbox.eu>
> > CC: "emacs-devel" <emacs-devel@gnu.org>
> > Date: Sun, 15 Sep 2024 19:38:55 +0200 (CEST)
> > 
> > Now that we're on Emacs 30.0.91, I feel the need to call this out as a fairly release-critical bug. Perhaps I should have emailed bug-gnu-emacs, but now that'd split the discussion over two places, so I'll keep it here.
> > 
> > If this is the kind of setting we can flip in a patch release, then there's no rush. Maybe someone can weigh in about that.
> > 
> > Otherwise it would remain as a source of instability for many years, and conscientious devs would have to insert a check for Emacs 30 that reminds the user that the package may be terribly outdated.
> > 
> > At least one other developer agrees the current setting is "fragile" (https://github.com/melpa/melpa/pull/9133#issuecomment-2351653325).
> 
> I've read this thread, and I must confess that I'm not convinced.
> Besides being waaay too late for such changes in Emacs 30, I also
> don't think we have enough experience at this time to make such
> decisions.  Both package-vc and use-package are relatively very new in
> Emacs, and we have yet to collect enough user experience and
> understanding of what are the expectations and how they should work
> together.  Moreover, use-package-vc-prefer-newest is a new option in
> Emacs 30, and we rarely make new optional behaviors take effect by
> default as soon as they are introduced, to avoid behavior changes
> which will surprise and perhaps annoy.  We made it an option to make
> it easily adaptable to user preferences and needs.
> 
> Some of the rationale (like outdated Package-Version) also sounds like
> it isn't our problem to solve.  It should be easy for the respective
> package developers to get their act together in this matter.  It is
> also TRT for them.  When a user uses use-package, he/she doesn't
> necessarily want the latest commit, so the change isn't as obvious as
> you make it sound, IMO.
> 
> Bottom line: I think no catastrophe happens if Emacs 30 is released
> with the current default value of use-package-vc-prefer-newest.  It's
> a user option, so customizing it is easy for people who want the
> latest commit.  Even if we are making a mistake (for reasons we will
> learn later), first time you do something there are always some rough
> edges, so we can be excused for not realizing something due to lack of
> experience.
> 
> Thanks.

Thanks for engaging!

(What is TRT?)

While the option is a new thing in Emacs 30, use-package :vc itself is a new thing in Emacs 30, so it is not as if there is an old behavior to emulate.  I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!  So, that setting has already been tested by lots of users on Emacs <29, and it is Emacs 30 that will change things.

Anyway. If that's not good enough, maybe we can test the new setting in Emacs 31, and backport to Emacs 30.2 in the future.

I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.

As for making devs get their act together, sure, they could do that. But three problems with that attitude:

1. It makes sense to impose requirements on devs who are submitting packages to NonGNU Elpa, but this setting affects everyone, including those who have not opted in to such requirements.
2. Not every dev will get the memo, naturally, and the ones who get hurt in the meantime are users, who believe that the dev's package is broken when it is not (and the dev should not be punished for being out of the loop).
3. The devs who do get the memo, and were previously content with a frozen Package-Version, will resent GNU for forcing what they perceive as a workaround. I do not think that is worth it.  Like it or not, MELPA has allowed many of us to taste of the convenience of git/hg tags, and it didn't use to be a problem... until Emacs 30. Now they have to adopt some toolchain like sisyphus.el and pollute the commit log with two extra commits for every new version, to solve what used to be a non-issue.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: "Martin Edstrom" <meedstrom@runbox.eu>
> CC: "emacs-devel" <emacs-devel@gnu.org>
> Date: Sun, 15 Sep 2024 21:46:32 +0200 (CEST)
> 
> > Bottom line: I think no catastrophe happens if Emacs 30 is released
> > with the current default value of use-package-vc-prefer-newest.  It's
> > a user option, so customizing it is easy for people who want the
> > latest commit.  Even if we are making a mistake (for reasons we will
> > learn later), first time you do something there are always some rough
> > edges, so we can be excused for not realizing something due to lack of
> > experience.
> > 
> > Thanks.
> 
> Thanks for engaging!
> 
> (What is TRT?)

The Right Thing

> While the option is a new thing in Emacs 30, use-package :vc itself is a new thing in Emacs 30, so it is not as if there is an old behavior to emulate.  I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!  So, that setting has already been tested by lots of users on Emacs <29, and it is Emacs 30 that will change things.

Yes, you said that in your original message, and I understand that
argument.  But as I responded, I'm not convinced the two options must
always be in sync, as they are used differently for different purposes.

> Anyway. If that's not good enough, maybe we can test the new setting in Emacs 31, and backport to Emacs 30.2 in the future.

Yes, that would be one way forward.  Especially if enough users come
to us asking to flip the default, and explain why.

> I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.

There's no reason to do this for user options, since customizing an
option if the user doesn't like its default is easy.

> As for making devs get their act together, sure, they could do that. But three problems with that attitude:
> 
> 1. It makes sense to impose requirements on devs who are submitting packages to NonGNU Elpa, but this setting affects everyone, including those who have not opted in to such requirements.
> 2. Not every dev will get the memo, naturally, and the ones who get hurt in the meantime are users, who believe that the dev's package is broken when it is not (and the dev should not be punished for being out of the loop).
> 3. The devs who do get the memo, and were previously content with a frozen Package-Version, will resent GNU for forcing what they perceive as a workaround. I do not think that is worth it.  Like it or not, MELPA has allowed many of us to taste of the convenience of git/hg tags, and it didn't use to be a problem... until Emacs 30. Now they have to adopt some toolchain like sisyphus.el and pollute the commit log with two extra commits for every new version, to solve what used to be a non-issue.

I consider advancing Package-Version when significant changes are made
a simple and uncontroversial thing to do for any package developer.  I
cannot imagine why someone who wants their package in good shape to
regard this as some annoyance.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

On Mon, 16 Sep 2024 14:34:55 +0300, Eli Zaretskii <eliz@gnu.org> wrote:

> > While the option is a new thing in Emacs 30, use-package :vc itself is a new thing in Emacs 30, so it is not as if there is an old behavior to emulate.  I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!  So, that setting has already been tested by lots of users on Emacs <29, and it is Emacs 30 that will change things.
> 
> Yes, you said that in your original message, and I understand that
> argument.  But as I responded, I'm not convinced the two options must
> always be in sync, as they are used differently for different purposes.

Okay, I'll accept that for the command package-vc-install as I have not thought much about its use cases.  I want to call attention to the old use-package :vc, shippedwhich had the confusing vc-use-package [1].  The Emacs 30 use-package :vc is a drop-in 1:1 replacement  This thing:    It had the opposite default from the one Emacs 30 is about to have.

> 
> > Anyway. If that's not good enough, maybe we can test the new setting in Emacs 31, and backport to Emacs 30.2 in the future.
> 
> Yes, that would be one way forward.  Especially if enough users come
> to us asking to flip the default, and explain why.
> 
> > I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.
> 
> There's no reason to do this for user options, since customizing an
> option if the user doesn't like its default is easy.
> 
> > As for making devs get their act together, sure, they could do that. But three problems with that attitude:
> > 
> > 1. It makes sense to impose requirements on devs who are submitting packages to NonGNU Elpa, but this setting affects everyone, including those who have not opted in to such requirements.
> > 2. Not every dev will get the memo, naturally, and the ones who get hurt in the meantime are users, who believe that the dev's package is broken when it is not (and the dev should not be punished for being out of the loop).
> > 3. The devs who do get the memo, and were previously content with a frozen Package-Version, will resent GNU for forcing what they perceive as a workaround. I do not think that is worth it.  Like it or not, MELPA has allowed many of us to taste of the convenience of git/hg tags, and it didn't use to be a problem... until Emacs 30. Now they have to adopt some toolchain like sisyphus.el and pollute the commit log with two extra commits for every new version, to solve what used to be a non-issue.
> 
> I consider advancing Package-Version when significant changes are made
> a simple and uncontroversial thing to do for any package developer.  I
> cannot imagine why someone who wants their package in good shape to
> regard this as some annoyance.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

Sorry the last email was incomplete, my keyboard has a bug.

On Mon, 16 Sep 2024 14:34:55 +0300, Eli Zaretskii <eliz@gnu.org> wrote:

> > While the option is a new thing in Emacs 30, use-package :vc itself is a new thing in Emacs 30, so it is not as if there is an old behavior to emulate.  I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!  So, that setting has already been tested by lots of users on Emacs <29, and it is Emacs 30 that will change things.
> 
> Yes, you said that in your original message, and I understand that
> argument.  But as I responded, I'm not convinced the two options must
> always be in sync, as they are used differently for different purposes.

Okay, I'll accept that for the command package-vc-install, as I have not thought much about its use cases.  

I want to call attention to the old use-package :vc, shipped under the confusing name vc-use-package [1], that people could install on Emacs 29 and earlier.  The Emacs 30 use-package :vc is a drop-in almost 1:1 replacement for this thing.  It had the opposite default behavior from the default Emacs 30 is about to have.  That change needs justification, not the other way around!

[1] https://github.com/slotThe/vc-use-package

> > I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.
> 
> There's no reason to do this for user options, since customizing an
> option if the user doesn't like its default is easy.

You cannot rely on users to configure this kind of thing for themselves, most of them would not know the reason why things broke. It is not a matter of "liking" the default, since breakage in one direction would be much more severe than is possible to encounter in the other direction (years out of date vs. slightly too new).

It is absolutely Emacs' responsibility, at least as long as stability for users is desired. Isn't it the reason to favor conservative defaults? Avoiding breakage? I get this new default comes from a good place, intending stability, but it is a radical departure from what has been the norm established by Straight/Quelpa/el-get and their use-package integrations, and that will work against stability.  

> 
> > As for making devs get their act together, sure, they could do that. But three problems with that attitude:
> > 
> > 1. It makes sense to impose requirements on devs who are submitting packages to NonGNU Elpa, but this setting affects everyone, including those who have not opted in to such requirements.
> > 2. Not every dev will get the memo, naturally, and the ones who get hurt in the meantime are users, who believe that the dev's package is broken when it is not (and the dev should not be punished for being out of the loop).
> > 3. The devs who do get the memo, and were previously content with a frozen Package-Version, will resent GNU for forcing what they perceive as a workaround. I do not think that is worth it.  Like it or not, MELPA has allowed many of us to taste of the convenience of git/hg tags, and it didn't use to be a problem... until Emacs 30. Now they have to adopt some toolchain like sisyphus.el and pollute the commit log with two extra commits for every new version, to solve what used to be a non-issue.
> 
> I consider advancing Package-Version when significant changes are made
> a simple and uncontroversial thing to do for any package developer.  I
> cannot imagine why someone who wants their package in good shape to
> regard this as some annoyance.

This email discussion risks getting long, but let me know if you want me to describe why I do not find it simple.  For now, I believe the relevant thing from Emacs' perspective is that there does exist developers who will not advance Package-Version, nevermind why.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: "Martin Edström" <meedstrom@runbox.eu>
> CC: "emacs-devel" <emacs-devel@gnu.org>
> Date: Mon, 16 Sep 2024 18:15:45 +0200 (CEST)
> 
> Okay, I'll accept that for the command package-vc-install, as I have not thought much about its use cases.  
> 
> I want to call attention to the old use-package :vc, shipped under the confusing name vc-use-package [1], that people could install on Emacs 29 and earlier.  The Emacs 30 use-package :vc is a drop-in almost 1:1 replacement for this thing.  It had the opposite default behavior from the default Emacs 30 is about to have.  That change needs justification, not the other way around!
> 
> [1] https://github.com/slotThe/vc-use-package

That's a different package, so I don't see why we should necessarily
follow its decisions.  When users migrate to use-package :vc, they
should expect some changes in the default behavior.

> > > I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.
> > 
> > There's no reason to do this for user options, since customizing an
> > option if the user doesn't like its default is easy.
> 
> You cannot rely on users to configure this kind of thing for themselves, most of them would not know the reason why things broke. It is not a matter of "liking" the default, since breakage in one direction would be much more severe than is possible to encounter in the other direction (years out of date vs. slightly too new).

Nowadays an Internet search will bring the answers very quickly and
efficiently.  And if that's not enough, there are enough user-level
help forums where people could ask questions and get useful answers.

IOW, "not a catastrophe".

> It is absolutely Emacs' responsibility, at least as long as stability for users is desired. Isn't it the reason to favor conservative defaults? Avoiding breakage? I get this new default comes from a good place, intending stability, but it is a radical departure from what has been the norm established by Straight/Quelpa/el-get and their use-package integrations, and that will work against stability.  

We can only be responsible for stability when our own implementation
and decisions are concerned.  We cannot be expected to blindly follow
someone else's decision in the name of "stability" for users who
switch from other packages, this kind of expectation is completely
unreasonable.

> > I consider advancing Package-Version when significant changes are made
> > a simple and uncontroversial thing to do for any package developer.  I
> > cannot imagine why someone who wants their package in good shape to
> > regard this as some annoyance.
> 
> This email discussion risks getting long, but let me know if you want me to describe why I do not find it simple.  For now, I believe the relevant thing from Emacs' perspective is that there does exist developers who will not advance Package-Version, nevermind why.

Well, simple or not, it is their act to get together.  Not ours.

Also, if we want to go an extra mile, I suggested a couple more
measures that would delineate the issue to unsuspecting users.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

On Mon, 16 Sep 2024 20:57:15 +0300, Eli Zaretskii <eliz@gnu.org> wrote:

> > [1] https://github.com/slotThe/vc-use-package
> 
> That's a different package, so I don't see why we should necessarily
> follow its decisions.  When users migrate to use-package :vc, they
> should expect some changes in the default behavior.
> 
> > You cannot rely on users to configure this kind of thing for themselves, most of them would not know the reason why things broke. It is not a matter of "liking" the default, since breakage in one direction would be much more severe than is possible to encounter in the other direction (years out of date vs. slightly too new).
> 
> Nowadays an Internet search will bring the answers very quickly and
> efficiently.  And if that's not enough, there are enough user-level
> help forums where people could ask questions and get useful answers.
> 
> IOW, "not a catastrophe".
> 
> > It is absolutely Emacs' responsibility, at least as long as stability for users is desired. Isn't it the reason to favor conservative defaults? Avoiding breakage? I get this new default comes from a good place, intending stability, but it is a radical departure from what has been the norm established by Straight/Quelpa/el-get and their use-package integrations, and that will work against stability.  
> 
> We can only be responsible for stability when our own implementation
> and decisions are concerned.  We cannot be expected to blindly follow
> someone else's decision in the name of "stability" for users who
> switch from other packages, this kind of expectation is completely
> unreasonable.

I am sorry.  I use the word "responsibility" perhaps different from many people, I think of responsibility as basically good, a nice thing to have, but as words go it's a powder-keg with different interpretations. I did not mean to be entitled nor expect the Emacs volunteers to do any services for me or anyone else.

I also have not contributed any core Emacs code (yet), and I am very grateful to those who have, such as yourself, Eli.

We are currently enjoying a Golden Age of Emacs, and I see that as being a lot about the Emacs Lisp ecosystem wrapped around the Emacs core, and of course, Emacs core doing what it can to be friendly to that ecosystem.  To me, Emacs *is* the ecosystem.

That frames my opinions about this new default setting.  It does not seem to me like the theoretical benefits outweigh the costs to GNU reputation each time a GitHub issue is raised to a developer who preferred to just do rolling releases (e.g. just today I had to warn someone at https://github.com/JuliaEditorSupport/julia-emacs/issues/212).

I believe I've said all I had to say, but to be very clear for the record, I consider this default to be radical, not conservative, at least from the perspective of causing least breakage.  I also do not think it is good stewardship of the ecosystem to say that each user can just Google the solution once they face breakage.  It affects developers too, who may not have learned about this default, and lose users as a consequence.

Anyhoo. The concept of versioned releases is good, and if git/hg tags are checked, that's great. Maybe it's not the end of the world that devs who want pure "rolling releases" are forced to adopt some sort of automated toolchain that pretends to cut releases for every commit, or insert a warning message to their users to upgrade to the master branch. But novice devs and experimental packages frequently fit into this box so I have mixed feelings about that.

Possible improvements in packaging (was: Reconsider defaults for use-package-vc-prefer-newest)

[Suhail Singh]

"Martin Edström" <meedstrom@runbox.eu> writes:

> Maybe it's not the end of the world that devs who want pure "rolling
> releases" are forced to adopt some sort of automated toolchain that
> pretends to cut releases for every commit, or insert a warning message
> to their users to upgrade to the master branch.

I believe that users benefit from installing the "recommended" version.
However, what that is can vary from package to package as well as from
user to user.

Ideally, upstream would provide sufficient information for different
users to be able to decide what is best for their needs.

Part of that information, which today isn't explicitly captured
anywhere, is the release model that is being followed.  If the package
spec allowed for this to be captured and package authors / maintainers
started using it, I believe it would be helpful.  Regardless, having
package authors / maintainers explicitly specify this in any way would
be an improvement.

Additionally, it is, perhaps, not that well known that while MELPA
Stable ELPA uses tags, GNU ELPA and NonGNU ELPA use the version
specified in the package header.  If there were some way to distinguish
between these (and/or when these versions aren't consistent), that too
would be helpful (IMO).  To its credit, MELPA (as opposed to MELPA
Stable) makes it quite clear (based on the assigned versions) that it's
using an altogether different way of versioning.

-- 
Suhail