Reconsider defaults for use-package-vc-prefer-newest

Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

Previous discussion: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=67955

Currently, the setting of `use-package-vc-prefer-newest` defaults to nil.
I want to bring up some problems with that, and suggest fixes.


PROBLEM 1. MISMATCH WITH PACKAGE-VC

Now use-package :vc is not in line with the behavior of the
package-vc-install command, which fetches the newest commit.

All other package managers I know of also fetch the newest commit.
It is the new use-package :vc which is the odd one out in the
ecosystem, and that can cause stability issues, which I go into
more deeply on this Reddit comment:

https://old.reddit.com/r/emacs/comments/1f8ok7c/do_you_set_packageinstallupgradebuiltin_to_t/llimwe0/


PROBLEM 2. "LAST RELEASE" CANNOT BE ASSUMED TO BE THE LAST RELEASE

I see many packages that have not bumped their Package-Version
header for years.  Perhaps they bump by git tags, or just stopped
bumping versions.  I understand them, since it is a lot of manual
labor to bump it, cut a release commit, then bump again to append a
"-pre" or a ".0.50-git", and to remember to do this everywhere the
version string may occur, like in docs.

I'm also guessing there's a segment of developers that have sensed
that this labor does not end up helping users, so they are not
motivated to keep doing it.  Versions only work as intended if
everyone is versioning regularly, and that is possible to enforce
in a curated repo like GNU Elpa or Debian's repackaged Emacs
packages, but as soon as you also install packages outside of that
repo (and well, most packages are outside), there's no longer any
guarantee and in fact the outdated versions in the curated repo are
likely to cause breakage.

I have personally had to give up on things like ELPA, Melpa Stable,
Debian packages etc.  It always caused instability.  By contrast,
installing 200-400 packages off Melpa, in other words the latest
dev snapshots, has caused me no issues in ~7 years.  So there are
two islands of stability:

1. Stay inside a curated repo entirely
2. Leave the curated repo entirely and install only dev snapshots

Now consider what use-package :vc is for.  It is for installing
things OUTSIDE any curated repo.  I leave it to you to pick a
default setting for `use-package-vc-prefer-newest`.


PROBLEM 3. FORCED REPO REWRITES

I have been forced to use git-filter-repo to rewrite my package's
commit history, to erase all instances of the Package-Version
header.

That's because I had been content to leave the header permanently
at "0.6.0.50-git".  If I kept doing that with the new use-package
:vc in existence, it would cause users to download a horribly
outdated version of my package. If I understand correctly.

I foresee one of two things will happen in the future:

1. More and more package authors carry out the same rewrite I did
2. (more likely) Increased instability as package authors do NOT do
   this rewrite, and users end up with ancient versions all over
   the place


SUGGESTED FIXES

A few possibilities:

1. Change `use-package-vc-prefer-newest` to default to t.

2. Change package-vc so when it walks the commit history to find
   the "last release" or any specific version, it should target the
   newest commit that has the given Package-Version, not the oldest
   commit.

   - That would allow devs to continue having a frozen
     Package-Version without breaking users' setups.

   - Of course, this is effectively the same as option 1, but the
     semantic distinction might be usable in the future to
     e.g. ignore version strings that have a "-pre" inside.

3. Roll out a strategy for moving the entire community onto the
   norm of versioning packages prior to distribution, ideally to
   the extent that Melpa is no longer necessary and can go offline
   in favor of Melpa Stable.  (Until then, Melpa Stable is best
   avoided: https://old.reddit.com/r/emacs/comments/etikbz/)

   Obviously difficult, but as I've said, versioning only works if
   everyone does it.  Specifically, it gives us stability if every
   developer is running the "released version" of most packages
   other than their own.  Currently we have an ecosystem where most
   developers are running the dev snapshot of most packages, and
   any half-measures to change this will only destroy the islands
   of stability that we do currently enjoy.  The topic of this
   email is one such half-measure.

4. Have Emacs ship an automation tool like
   https://github.com/magit/sisyphus, recommend it everywhere, then
   have package.el, package-vc and use-package :vc all throw an
   error on finding more than one commit with a given
   Package-Version string and refuse to install due to ambiguity.

5. Deprecate the Package-Version header and instead maybe let
   package.el perform an automatic string substitution from the
   git/hg tag, if it is still desirable to see the version in the
   source file.  The version could also be substituted into Info
   docs and other places the version string should occur.

   - This is effectively a convenient alternative to option 4.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

Now that we're on Emacs 30.0.91, I feel the need to call this out as a fairly release-critical bug. Perhaps I should have emailed bug-gnu-emacs, but now that'd split the discussion over two places, so I'll keep it here.

If this is the kind of setting we can flip in a patch release, then there's no rush. Maybe someone can weigh in about that.

Otherwise it would remain as a source of instability for many years, and conscientious devs would have to insert a check for Emacs 30 that reminds the user that the package may be terribly outdated.

At least one other developer agrees the current setting is "fragile" (https://github.com/melpa/melpa/pull/9133#issuecomment-2351653325).

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: "Martin Edström" <meedstrom@runbox.eu>
> CC: "emacs-devel" <emacs-devel@gnu.org>
> Date: Sun, 15 Sep 2024 19:38:55 +0200 (CEST)
> 
> Now that we're on Emacs 30.0.91, I feel the need to call this out as a fairly release-critical bug. Perhaps I should have emailed bug-gnu-emacs, but now that'd split the discussion over two places, so I'll keep it here.
> 
> If this is the kind of setting we can flip in a patch release, then there's no rush. Maybe someone can weigh in about that.
> 
> Otherwise it would remain as a source of instability for many years, and conscientious devs would have to insert a check for Emacs 30 that reminds the user that the package may be terribly outdated.
> 
> At least one other developer agrees the current setting is "fragile" (https://github.com/melpa/melpa/pull/9133#issuecomment-2351653325).

I've read this thread, and I must confess that I'm not convinced.
Besides being waaay too late for such changes in Emacs 30, I also
don't think we have enough experience at this time to make such
decisions.  Both package-vc and use-package are relatively very new in
Emacs, and we have yet to collect enough user experience and
understanding of what are the expectations and how they should work
together.  Moreover, use-package-vc-prefer-newest is a new option in
Emacs 30, and we rarely make new optional behaviors take effect by
default as soon as they are introduced, to avoid behavior changes
which will surprise and perhaps annoy.  We made it an option to make
it easily adaptable to user preferences and needs.

Some of the rationale (like outdated Package-Version) also sounds like
it isn't our problem to solve.  It should be easy for the respective
package developers to get their act together in this matter.  It is
also TRT for them.  When a user uses use-package, he/she doesn't
necessarily want the latest commit, so the change isn't as obvious as
you make it sound, IMO.

Bottom line: I think no catastrophe happens if Emacs 30 is released
with the current default value of use-package-vc-prefer-newest.  It's
a user option, so customizing it is easy for people who want the
latest commit.  Even if we are making a mistake (for reasons we will
learn later), first time you do something there are always some rough
edges, so we can be excused for not realizing something due to lack of
experience.

Thanks.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edstrom]

> > From: "Martin Edström" <meedstrom@runbox.eu>
> > CC: "emacs-devel" <emacs-devel@gnu.org>
> > Date: Sun, 15 Sep 2024 19:38:55 +0200 (CEST)
> > 
> > Now that we're on Emacs 30.0.91, I feel the need to call this out as a fairly release-critical bug. Perhaps I should have emailed bug-gnu-emacs, but now that'd split the discussion over two places, so I'll keep it here.
> > 
> > If this is the kind of setting we can flip in a patch release, then there's no rush. Maybe someone can weigh in about that.
> > 
> > Otherwise it would remain as a source of instability for many years, and conscientious devs would have to insert a check for Emacs 30 that reminds the user that the package may be terribly outdated.
> > 
> > At least one other developer agrees the current setting is "fragile" (https://github.com/melpa/melpa/pull/9133#issuecomment-2351653325).
> 
> I've read this thread, and I must confess that I'm not convinced.
> Besides being waaay too late for such changes in Emacs 30, I also
> don't think we have enough experience at this time to make such
> decisions.  Both package-vc and use-package are relatively very new in
> Emacs, and we have yet to collect enough user experience and
> understanding of what are the expectations and how they should work
> together.  Moreover, use-package-vc-prefer-newest is a new option in
> Emacs 30, and we rarely make new optional behaviors take effect by
> default as soon as they are introduced, to avoid behavior changes
> which will surprise and perhaps annoy.  We made it an option to make
> it easily adaptable to user preferences and needs.
> 
> Some of the rationale (like outdated Package-Version) also sounds like
> it isn't our problem to solve.  It should be easy for the respective
> package developers to get their act together in this matter.  It is
> also TRT for them.  When a user uses use-package, he/she doesn't
> necessarily want the latest commit, so the change isn't as obvious as
> you make it sound, IMO.
> 
> Bottom line: I think no catastrophe happens if Emacs 30 is released
> with the current default value of use-package-vc-prefer-newest.  It's
> a user option, so customizing it is easy for people who want the
> latest commit.  Even if we are making a mistake (for reasons we will
> learn later), first time you do something there are always some rough
> edges, so we can be excused for not realizing something due to lack of
> experience.
> 
> Thanks.

Thanks for engaging!

(What is TRT?)

While the option is a new thing in Emacs 30, use-package :vc itself is a new thing in Emacs 30, so it is not as if there is an old behavior to emulate.  I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!  So, that setting has already been tested by lots of users on Emacs <29, and it is Emacs 30 that will change things.

Anyway. If that's not good enough, maybe we can test the new setting in Emacs 31, and backport to Emacs 30.2 in the future.

I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.

As for making devs get their act together, sure, they could do that. But three problems with that attitude:

1. It makes sense to impose requirements on devs who are submitting packages to NonGNU Elpa, but this setting affects everyone, including those who have not opted in to such requirements.
2. Not every dev will get the memo, naturally, and the ones who get hurt in the meantime are users, who believe that the dev's package is broken when it is not (and the dev should not be punished for being out of the loop).
3. The devs who do get the memo, and were previously content with a frozen Package-Version, will resent GNU for forcing what they perceive as a workaround. I do not think that is worth it.  Like it or not, MELPA has allowed many of us to taste of the convenience of git/hg tags, and it didn't use to be a problem... until Emacs 30. Now they have to adopt some toolchain like sisyphus.el and pollute the commit log with two extra commits for every new version, to solve what used to be a non-issue.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

> I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!

At least, I infer that it had the opposite default, going by its README (https://github.com/slotThe/vc-use-package), since it sounds like one had to explicitly pass :last-release to get it to download an old commit.

Re: Reconsider defaults for use-package-vc-prefer-newest

[chad]

[-- Attachment #1: Type: text/plain, Size: 223 bytes --]

Please forgive me if I misunderstand the situation, but I wonder if it
wouldn't be possible to suggest a patch that leaves the conservative
default but detects the potential "catastrophe" and warns the user?

Thanks,
~Chad

[-- Attachment #2: Type: text/html, Size: 308 bytes --]

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edstrom]

> Please forgive me if I misunderstand the situation, but I wonder if it
> wouldn't be possible to suggest a patch that leaves the conservative
> default but detects the potential "catastrophe" and warns the user?
> 
> Thanks,
> ~Chad

The "catastrophe" would be a situation such as:

- In 2020, Developer releases Package for the first time
- In 2021, Developer tires of bumping Package-Version, leaves it at 0.9
- In 2024, Package is now at 2.2 according to the convenient git/hg tag, or maybe it has no official version beyond just "0.9.0.50-git"
- User installs Package using (use-package :vc)
- User gets the version from 2021
- It doesn't seem to work
- 10 such users give up on Package
- The 11th user files the bug reports
- Bug reports are strange and make no sense
- User and Developer finally figure out that it's because the user used (use-package :vc) which fetched the 2021 version

To counteract this would amount to a heuristic that looks up the package repository online and compares the age of recent commits with the commit fetched by (use-package :vc), but what is the threshold that you would set to trigger the warning? I don't think it is realistically doable.

Re: Reconsider defaults for use-package-vc-prefer-newest

[chad]

[-- Attachment #1: Type: text/plain, Size: 2179 bytes --]

On Sun, Sep 15, 2024 at 5:09 PM Martin Edstrom <meedstrom@runbox.eu> wrote:

> The "catastrophe" would be a situation such as:
>
> - In 2020, Developer releases Package for the first time
> - In 2021, Developer tires of bumping Package-Version, leaves it at 0.9
> - In 2024, Package is now at 2.2 according to the convenient git/hg tag,
> or maybe it has no official version beyond just "0.9.0.50-git"
> - User installs Package using (use-package :vc)
> - User gets the version from 2021
> [...]


You're describing the situation in terms of theoretical user results, but
that's not how the code works, obviously. At some point, the code checks
the user's value of use-package-vs-prefer-newest, sees whether it's the
default or has been customized, and decides to fetch the new package
contents or not. My rough understanding is, *in the non-VC case*, it
downloads new archive-contents for the (each) entire archive regardless,
and then I'm guessing that it currently uses the setting to determine
whether to get a new version or not.

I don't know how the code path in the (use-package :vc) works, but
presumably there's a clear point where the two behaviors (that is,
vc-use-package's supposed former update-by-default versus use-package :vc's
...prefer-newer 'nil lack of update) diverge. I'm asking if there's a
reasonable way to change the default behavior from the supposed-old
yes-update and the current default wait-for-Package-version into something
that leaves the package alone but warns the user, perhaps via *Messages*.

Put another way, your concern seems to be that people might not _realize_
that they're "stuck" on an older Package-Version revision. The alternative
package "manager" code for emacs suggests that there is desire both for
version pinning and direct-from-vcs-packages, so it's likely that there are
users who prefer both alternatives, and emacs is often default-conservative
in such matters, but it seems like it might be possible to have use-package
do a little extra work in the default case to address the "not realize"
part, rather than changing the updating behavior itself.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 2670 bytes --]

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edstrom]

On Sun, 15 Sep 2024 18:12:00 -0400, chad <yandros@gmail.com> wrote:

> I don't know how the code path in the (use-package :vc) works, but
> presumably there's a clear point where the two behaviors (that is,
> vc-use-package's supposed former update-by-default versus use-package :vc's
> ...prefer-newer 'nil lack of update) diverge. I'm asking if there's a
> reasonable way to change the default behavior from the supposed-old
> yes-update and the current default wait-for-Package-version into something
> that leaves the package alone but warns the user, perhaps via *Messages*.

If I understand correctly, you suggest that a message be printed when there is any new commit since the last Package-Version?

Something like:

"310 packages installed, 237 have new commits"

But I think people would just learn to ignore this message...

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: "Martin Edstrom" <meedstrom@runbox.eu>
> CC: "emacs-devel" <emacs-devel@gnu.org>
> Date: Sun, 15 Sep 2024 21:46:32 +0200 (CEST)
> 
> > Bottom line: I think no catastrophe happens if Emacs 30 is released
> > with the current default value of use-package-vc-prefer-newest.  It's
> > a user option, so customizing it is easy for people who want the
> > latest commit.  Even if we are making a mistake (for reasons we will
> > learn later), first time you do something there are always some rough
> > edges, so we can be excused for not realizing something due to lack of
> > experience.
> > 
> > Thanks.
> 
> Thanks for engaging!
> 
> (What is TRT?)

The Right Thing

> While the option is a new thing in Emacs 30, use-package :vc itself is a new thing in Emacs 30, so it is not as if there is an old behavior to emulate.  I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!  So, that setting has already been tested by lots of users on Emacs <29, and it is Emacs 30 that will change things.

Yes, you said that in your original message, and I understand that
argument.  But as I responded, I'm not convinced the two options must
always be in sync, as they are used differently for different purposes.

> Anyway. If that's not good enough, maybe we can test the new setting in Emacs 31, and backport to Emacs 30.2 in the future.

Yes, that would be one way forward.  Especially if enough users come
to us asking to flip the default, and explain why.

> I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.

There's no reason to do this for user options, since customizing an
option if the user doesn't like its default is easy.

> As for making devs get their act together, sure, they could do that. But three problems with that attitude:
> 
> 1. It makes sense to impose requirements on devs who are submitting packages to NonGNU Elpa, but this setting affects everyone, including those who have not opted in to such requirements.
> 2. Not every dev will get the memo, naturally, and the ones who get hurt in the meantime are users, who believe that the dev's package is broken when it is not (and the dev should not be punished for being out of the loop).
> 3. The devs who do get the memo, and were previously content with a frozen Package-Version, will resent GNU for forcing what they perceive as a workaround. I do not think that is worth it.  Like it or not, MELPA has allowed many of us to taste of the convenience of git/hg tags, and it didn't use to be a problem... until Emacs 30. Now they have to adopt some toolchain like sisyphus.el and pollute the commit log with two extra commits for every new version, to solve what used to be a non-issue.

I consider advancing Package-Version when significant changes are made
a simple and uncontroversial thing to do for any package developer.  I
cannot imagine why someone who wants their package in good shape to
regard this as some annoyance.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: "Martin Edstrom" <meedstrom@runbox.eu>
> CC: "emacs-devel" <emacs-devel@gnu.org>
> Date: Sun, 15 Sep 2024 23:09:05 +0200 (CEST)
> 
> The "catastrophe" would be a situation such as:
> 
> - In 2020, Developer releases Package for the first time
> - In 2021, Developer tires of bumping Package-Version, leaves it at 0.9
> - In 2024, Package is now at 2.2 according to the convenient git/hg tag, or maybe it has no official version beyond just "0.9.0.50-git"

If the problem is that there's a tag with the correct version, but the
Package-Version heading was not updated, we could perhaps have
use-package :vc detect that and either display a warning or even
automatically use the commit with the tag.  Crucially, this is NOT the
latest commit in the Git repository, it is the commit which has the
latest release tag -- a far cry from what you suggested originally
(apologies if I misunderstood you back then.)

> - User installs Package using (use-package :vc)
> - User gets the version from 2021
> - It doesn't seem to work
> - 10 such users give up on Package
> - The 11th user files the bug reports
> - Bug reports are strange and make no sense
> - User and Developer finally figure out that it's because the user used (use-package :vc) which fetched the 2021 version

In any case, I wonder why the above mess you describe is not the fault
of the package developer, but of Emacs?  It makes little sense to be,
TBH.

> To counteract this would amount to a heuristic that looks up the package repository online and compares the age of recent commits with the commit fetched by (use-package :vc), but what is the threshold that you would set to trigger the warning? I don't think it is realistically doable.

Why not?  Detecting such suspicious "last versions" could be a good
idea and a good service to users, and is not that hard to implement.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

On Mon, 16 Sep 2024 14:34:55 +0300, Eli Zaretskii <eliz@gnu.org> wrote:

> > While the option is a new thing in Emacs 30, use-package :vc itself is a new thing in Emacs 30, so it is not as if there is an old behavior to emulate.  I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!  So, that setting has already been tested by lots of users on Emacs <29, and it is Emacs 30 that will change things.
> 
> Yes, you said that in your original message, and I understand that
> argument.  But as I responded, I'm not convinced the two options must
> always be in sync, as they are used differently for different purposes.

Okay, I'll accept that for the command package-vc-install as I have not thought much about its use cases.  I want to call attention to the old use-package :vc, shippedwhich had the confusing vc-use-package [1].  The Emacs 30 use-package :vc is a drop-in 1:1 replacement  This thing:    It had the opposite default from the one Emacs 30 is about to have.

> 
> > Anyway. If that's not good enough, maybe we can test the new setting in Emacs 31, and backport to Emacs 30.2 in the future.
> 
> Yes, that would be one way forward.  Especially if enough users come
> to us asking to flip the default, and explain why.
> 
> > I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.
> 
> There's no reason to do this for user options, since customizing an
> option if the user doesn't like its default is easy.
> 
> > As for making devs get their act together, sure, they could do that. But three problems with that attitude:
> > 
> > 1. It makes sense to impose requirements on devs who are submitting packages to NonGNU Elpa, but this setting affects everyone, including those who have not opted in to such requirements.
> > 2. Not every dev will get the memo, naturally, and the ones who get hurt in the meantime are users, who believe that the dev's package is broken when it is not (and the dev should not be punished for being out of the loop).
> > 3. The devs who do get the memo, and were previously content with a frozen Package-Version, will resent GNU for forcing what they perceive as a workaround. I do not think that is worth it.  Like it or not, MELPA has allowed many of us to taste of the convenience of git/hg tags, and it didn't use to be a problem... until Emacs 30. Now they have to adopt some toolchain like sisyphus.el and pollute the commit log with two extra commits for every new version, to solve what used to be a non-issue.
> 
> I consider advancing Package-Version when significant changes are made
> a simple and uncontroversial thing to do for any package developer.  I
> cannot imagine why someone who wants their package in good shape to
> regard this as some annoyance.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

Sorry the last email was incomplete, my keyboard has a bug.

On Mon, 16 Sep 2024 14:34:55 +0300, Eli Zaretskii <eliz@gnu.org> wrote:

> > While the option is a new thing in Emacs 30, use-package :vc itself is a new thing in Emacs 30, so it is not as if there is an old behavior to emulate.  I should also point out that the module it was inspired by, vc-use-package, actually had the opposite default!  So, that setting has already been tested by lots of users on Emacs <29, and it is Emacs 30 that will change things.
> 
> Yes, you said that in your original message, and I understand that
> argument.  But as I responded, I'm not convinced the two options must
> always be in sync, as they are used differently for different purposes.

Okay, I'll accept that for the command package-vc-install, as I have not thought much about its use cases.  

I want to call attention to the old use-package :vc, shipped under the confusing name vc-use-package [1], that people could install on Emacs 29 and earlier.  The Emacs 30 use-package :vc is a drop-in almost 1:1 replacement for this thing.  It had the opposite default behavior from the default Emacs 30 is about to have.  That change needs justification, not the other way around!

[1] https://github.com/slotThe/vc-use-package

> > I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.
> 
> There's no reason to do this for user options, since customizing an
> option if the user doesn't like its default is easy.

You cannot rely on users to configure this kind of thing for themselves, most of them would not know the reason why things broke. It is not a matter of "liking" the default, since breakage in one direction would be much more severe than is possible to encounter in the other direction (years out of date vs. slightly too new).

It is absolutely Emacs' responsibility, at least as long as stability for users is desired. Isn't it the reason to favor conservative defaults? Avoiding breakage? I get this new default comes from a good place, intending stability, but it is a radical departure from what has been the norm established by Straight/Quelpa/el-get and their use-package integrations, and that will work against stability.  

> 
> > As for making devs get their act together, sure, they could do that. But three problems with that attitude:
> > 
> > 1. It makes sense to impose requirements on devs who are submitting packages to NonGNU Elpa, but this setting affects everyone, including those who have not opted in to such requirements.
> > 2. Not every dev will get the memo, naturally, and the ones who get hurt in the meantime are users, who believe that the dev's package is broken when it is not (and the dev should not be punished for being out of the loop).
> > 3. The devs who do get the memo, and were previously content with a frozen Package-Version, will resent GNU for forcing what they perceive as a workaround. I do not think that is worth it.  Like it or not, MELPA has allowed many of us to taste of the convenience of git/hg tags, and it didn't use to be a problem... until Emacs 30. Now they have to adopt some toolchain like sisyphus.el and pollute the commit log with two extra commits for every new version, to solve what used to be a non-issue.
> 
> I consider advancing Package-Version when significant changes are made
> a simple and uncontroversial thing to do for any package developer.  I
> cannot imagine why someone who wants their package in good shape to
> regard this as some annoyance.

This email discussion risks getting long, but let me know if you want me to describe why I do not find it simple.  For now, I believe the relevant thing from Emacs' perspective is that there does exist developers who will not advance Package-Version, nevermind why.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

On Mon, 16 Sep 2024 14:50:51 +0300, Eli Zaretskii <eliz@gnu.org> wrote:
 
> If the problem is that there's a tag with the correct version, but the
> Package-Version heading was not updated, we could perhaps have
> use-package :vc detect that and either display a warning or even
> automatically use the commit with the tag.  Crucially, this is NOT the
> latest commit in the Git repository, it is the commit which has the
> latest release tag -- a far cry from what you suggested originally
> (apologies if I misunderstood you back then.)

I agree it's a good way forward if you are open to look up both the Package-Version and the git tag and basically choose the highest-looking version between the two.  I had somehow ruled out that option.

What I suggested was, technically, to get the latest commit *with a given Package-Version*.

Which need not be the same thing as the latest commit. 

The latest commit could have a pre-release version like "0.6-pre", and so (I assume...? Not sure) Emacs would walk back until finding the commit with "0.6" and check out that one.

It still looks necessary to me to do the change I suggested, since there exists a population of developers who either believe that's how versions work, or who do not really bump versions anyway. Since you can't educate everyone, this is a situation for picking defaults that handle this reality.

At worst, a package can have just ";; Package-Version: 0" since the very first commit, and no tag. I do not think that in this situation, the correct move for use-package :vc is to install literally the first commit, when it would be the very first package manager to behave in such a way.  

It would be a deliberate introduction of breakage in the ecosystem, mainly harming users. It does not matter if the packagers are not doing The Right Thing, you gotta protect the users.

Please consider what use-package :vc is for in the first place. It wouldn't be used for things that exist on [Non]GNU ELPA, but for what exists off it, so its behavior should not be modeled on what you can expect from packages hosted there. On the contrary, its behavior can't expect any norms to be followed. It's not necessarily incompetence nor disagreement about procedure; many novice developers also take their first baby steps in this wilderness.

Of course it is possible to use use-package :vc to install what's on the ELPA, but the crowd that cares about running stable releases are going to be the last ones to choose to do that.

I can see an user switching to use-package :vc globally, but it'd be precisely because they want the bleeding edge version of everything, including the stuff on [Non]GNU ELPA.  If they don't want it, they can simply elect not do so for the stuff on [Non]GNU ELPA.

> > - User installs Package using (use-package :vc)
> > - User gets the version from 2021
> > - It doesn't seem to work
> > - 10 such users give up on Package
> > - The 11th user files the bug reports
> > - Bug reports are strange and make no sense
> > - User and Developer finally figure out that it's because the user used (use-package :vc) which fetched the 2021 version
> 
> In any case, I wonder why the above mess you describe is not the fault
> of the package developer, but of Emacs?  It makes little sense to be,
> TBH.

I don't believe in blaming, but Emacs definitely has some of the responsibility for the simple reason that this didn't use to happen and people do not expect it to happen.

A house of cards falls when someone shakes the table, but it isn't the fault of the one who put it together, it is who shook the table when they could instead have permitted that artwork to stand.

After knowing how this new Emacs builtin behaves, you can consider it to be the fault of the developer who did not take it into account, at least if it is reasonable to expect all developers to be professionals that know how every package manager works instead of only most of them, having missed this one, but even then a crowd will resent this addition to their workload, as I have explained prior. 

My own packaging experience would be unharmed if git tags become a valid version identifier, but that just affects my packages.  I'd like the ecosystem to have guarantees in place, and blaming devs because the fault is theirs will not make them do The Right Thing.  The right thing is that Emacs just decline to shake the table.

> 
> > To counteract this would amount to a heuristic that looks up the package repository online and compares the age of recent commits with the commit fetched by (use-package :vc), but what is the threshold that you would set to trigger the warning? I don't think it is realistically doable.
> 
> Why not?  Detecting such suspicious "last versions" could be a good
> idea and a good service to users, and is not that hard to implement.

But how would you do it? All it takes is a single commit to introduce changes on which another package will rely.  It can be as little as one minute between the commits in question.  Or conversely, a package may be mostly unchanged for years but that does not mean there is a problem. So that brings us to a bunch of UX issues to answer.  Should users "dismiss" each warning individually?  (Bear in mind there could be hundreds.)  Should these dismissals be recorded in `custom-file` or some such location?  It seems like a lot of effort to do well compared to my other suggestions, but that's just my impression.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: "Martin Edström" <meedstrom@runbox.eu>
> CC: "emacs-devel" <emacs-devel@gnu.org>
> Date: Mon, 16 Sep 2024 18:15:45 +0200 (CEST)
> 
> Okay, I'll accept that for the command package-vc-install, as I have not thought much about its use cases.  
> 
> I want to call attention to the old use-package :vc, shipped under the confusing name vc-use-package [1], that people could install on Emacs 29 and earlier.  The Emacs 30 use-package :vc is a drop-in almost 1:1 replacement for this thing.  It had the opposite default behavior from the default Emacs 30 is about to have.  That change needs justification, not the other way around!
> 
> [1] https://github.com/slotThe/vc-use-package

That's a different package, so I don't see why we should necessarily
follow its decisions.  When users migrate to use-package :vc, they
should expect some changes in the default behavior.

> > > I should also point out that the catastrophe occurs not at release time, but years afterwards, when we're on Emacs 31, 32, 33... but devs still want to support Emacs 30, getting worse with time. So I do hope that it will be possible to change a setting like this with Emacs 30.2 or some other "bugfix" release.
> > 
> > There's no reason to do this for user options, since customizing an
> > option if the user doesn't like its default is easy.
> 
> You cannot rely on users to configure this kind of thing for themselves, most of them would not know the reason why things broke. It is not a matter of "liking" the default, since breakage in one direction would be much more severe than is possible to encounter in the other direction (years out of date vs. slightly too new).

Nowadays an Internet search will bring the answers very quickly and
efficiently.  And if that's not enough, there are enough user-level
help forums where people could ask questions and get useful answers.

IOW, "not a catastrophe".

> It is absolutely Emacs' responsibility, at least as long as stability for users is desired. Isn't it the reason to favor conservative defaults? Avoiding breakage? I get this new default comes from a good place, intending stability, but it is a radical departure from what has been the norm established by Straight/Quelpa/el-get and their use-package integrations, and that will work against stability.  

We can only be responsible for stability when our own implementation
and decisions are concerned.  We cannot be expected to blindly follow
someone else's decision in the name of "stability" for users who
switch from other packages, this kind of expectation is completely
unreasonable.

> > I consider advancing Package-Version when significant changes are made
> > a simple and uncontroversial thing to do for any package developer.  I
> > cannot imagine why someone who wants their package in good shape to
> > regard this as some annoyance.
> 
> This email discussion risks getting long, but let me know if you want me to describe why I do not find it simple.  For now, I believe the relevant thing from Emacs' perspective is that there does exist developers who will not advance Package-Version, nevermind why.

Well, simple or not, it is their act to get together.  Not ours.

Also, if we want to go an extra mile, I suggested a couple more
measures that would delineate the issue to unsuspecting users.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: "Martin Edström" <meedstrom@runbox.eu>
> CC: "yandros" <yandros@gmail.com>, "emacs-devel" <emacs-devel@gnu.org>
> Date: Mon, 16 Sep 2024 18:46:48 +0200 (CEST)
> 
> On Mon, 16 Sep 2024 14:50:51 +0300, Eli Zaretskii <eliz@gnu.org> wrote:
>  
> > If the problem is that there's a tag with the correct version, but the
> > Package-Version heading was not updated, we could perhaps have
> > use-package :vc detect that and either display a warning or even
> > automatically use the commit with the tag.  Crucially, this is NOT the
> > latest commit in the Git repository, it is the commit which has the
> > latest release tag -- a far cry from what you suggested originally
> > (apologies if I misunderstood you back then.)
> 
> I agree it's a good way forward if you are open to look up both the Package-Version and the git tag and basically choose the highest-looking version between the two.  I had somehow ruled out that option.
> 
> What I suggested was, technically, to get the latest commit *with a given Package-Version*.
> 
> Which need not be the same thing as the latest commit. 

It is also not necessarily a stable commit.  It could have bugs.
Installing the commit with a Package-Version heading means we don't
subject users to the risk of getting a bad version, on the assumption
that bumping a version means the developer announces the code as more
or less stable.  Looking for a commit with a release tag has the same
advantage, whereas what you suggest doesn't.

> The latest commit could have a pre-release version like "0.6-pre", and so (I assume...? Not sure) Emacs would walk back until finding the commit with "0.6" and check out that one.

This is indeed a risk, and this is why we prefer to use
Package-Version.  But we could also try to look at release tags, at
least as an option.

> > > To counteract this would amount to a heuristic that looks up the package repository online and compares the age of recent commits with the commit fetched by (use-package :vc), but what is the threshold that you would set to trigger the warning? I don't think it is realistically doable.
> > 
> > Why not?  Detecting such suspicious "last versions" could be a good
> > idea and a good service to users, and is not that hard to implement.
> 
> But how would you do it?

By looking at commit dates and showing a warning or informational
message.

> All it takes is a single commit to introduce changes on which another package will rely.  It can be as little as one minute between the commits in question.

One minute ago is not outdated in any sense of the word.  In any case,
this is just a warning, meant to draw the user's attention to
something that he/she should look into.  What to do with that is
entirely up to the user.

> Or conversely, a package may be mostly unchanged for years but that does not mean there is a problem. So that brings us to a bunch of UX issues to answer.  Should users "dismiss" each warning individually?  (Bear in mind there could be hundreds.)  Should these dismissals be recorded in `custom-file` or some such location?  It seems like a lot of effort to do well compared to my other suggestions, but that's just my impression.

Showing a message with a 1-sec sit-for is the usual technique.  The
messages are logged in *Messages*, so users could look them up later.
I don't see a significant problem here, sorry.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

> If the problem is that there's a tag with the correct version, but the
> Package-Version heading was not updated, we could perhaps have
> use-package :vc detect that and either display a warning or even
> automatically use the commit with the tag.

FWIW, a situation I have encountered is when the commit corresponding to
a release tag is not the same commit that introduced the version in the
package header.  This can come up, for instance, when a package is
available both in melpa-stable and nongnu.  The former uses git tags for
versions whereas the latter uses the package header.  However, while the
version of the packages may nominally be the same, they may correspond
to different commits.  I.e., having two different sources of version
information can result in ambiguity.

When the two notions of version don't correspond to the same commit, it
would be helpful if the user could be notified in some way.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: "Martin Edstrom" <meedstrom@runbox.eu>,  yandros@gmail.com,
>   emacs-devel@gnu.org
> Date: Mon, 16 Sep 2024 16:16:03 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If the problem is that there's a tag with the correct version, but the
> > Package-Version heading was not updated, we could perhaps have
> > use-package :vc detect that and either display a warning or even
> > automatically use the commit with the tag.
> 
> FWIW, a situation I have encountered is when the commit corresponding to
> a release tag is not the same commit that introduced the version in the
> package header.  This can come up, for instance, when a package is
> available both in melpa-stable and nongnu.  The former uses git tags for
> versions whereas the latter uses the package header.  However, while the
> version of the packages may nominally be the same, they may correspond
> to different commits.  I.e., having two different sources of version
> information can result in ambiguity.
> 
> When the two notions of version don't correspond to the same commit, it
> would be helpful if the user could be notified in some way.

Yes, that'd be another reason to alert the user.

Patches to that effect are welcome.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

On Mon, 16 Sep 2024 20:57:15 +0300, Eli Zaretskii <eliz@gnu.org> wrote:

> > [1] https://github.com/slotThe/vc-use-package
> 
> That's a different package, so I don't see why we should necessarily
> follow its decisions.  When users migrate to use-package :vc, they
> should expect some changes in the default behavior.
> 
> > You cannot rely on users to configure this kind of thing for themselves, most of them would not know the reason why things broke. It is not a matter of "liking" the default, since breakage in one direction would be much more severe than is possible to encounter in the other direction (years out of date vs. slightly too new).
> 
> Nowadays an Internet search will bring the answers very quickly and
> efficiently.  And if that's not enough, there are enough user-level
> help forums where people could ask questions and get useful answers.
> 
> IOW, "not a catastrophe".
> 
> > It is absolutely Emacs' responsibility, at least as long as stability for users is desired. Isn't it the reason to favor conservative defaults? Avoiding breakage? I get this new default comes from a good place, intending stability, but it is a radical departure from what has been the norm established by Straight/Quelpa/el-get and their use-package integrations, and that will work against stability.  
> 
> We can only be responsible for stability when our own implementation
> and decisions are concerned.  We cannot be expected to blindly follow
> someone else's decision in the name of "stability" for users who
> switch from other packages, this kind of expectation is completely
> unreasonable.

I am sorry.  I use the word "responsibility" perhaps different from many people, I think of responsibility as basically good, a nice thing to have, but as words go it's a powder-keg with different interpretations. I did not mean to be entitled nor expect the Emacs volunteers to do any services for me or anyone else.

I also have not contributed any core Emacs code (yet), and I am very grateful to those who have, such as yourself, Eli.

We are currently enjoying a Golden Age of Emacs, and I see that as being a lot about the Emacs Lisp ecosystem wrapped around the Emacs core, and of course, Emacs core doing what it can to be friendly to that ecosystem.  To me, Emacs *is* the ecosystem.

That frames my opinions about this new default setting.  It does not seem to me like the theoretical benefits outweigh the costs to GNU reputation each time a GitHub issue is raised to a developer who preferred to just do rolling releases (e.g. just today I had to warn someone at https://github.com/JuliaEditorSupport/julia-emacs/issues/212).

I believe I've said all I had to say, but to be very clear for the record, I consider this default to be radical, not conservative, at least from the perspective of causing least breakage.  I also do not think it is good stewardship of the ecosystem to say that each user can just Google the solution once they face breakage.  It affects developers too, who may not have learned about this default, and lose users as a consequence.

Anyhoo. The concept of versioned releases is good, and if git/hg tags are checked, that's great. Maybe it's not the end of the world that devs who want pure "rolling releases" are forced to adopt some sort of automated toolchain that pretends to cut releases for every commit, or insert a warning message to their users to upgrade to the master branch. But novice devs and experimental packages frequently fit into this box so I have mixed feelings about that.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

>> When the two notions of version don't correspond to the same commit, it
>> would be helpful if the user could be notified in some way.
>
> Yes, that'd be another reason to alert the user.
>
> Patches to that effect are welcome.

I suppose if the output of M-x describe-package mentioned the commit
hash in addition to the version the user would at least be aware when
the same version numbers from different archives differ.  Out of
curiosity, is that what you had in mind, or something else?

Displaying the commit date in addition to the commit hash could also be
helpful.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  meedstrom@runbox.eu,
>  yandros@gmail.com,  emacs-devel@gnu.org
> Date: Wed, 18 Sep 2024 23:38:08 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> When the two notions of version don't correspond to the same commit, it
> >> would be helpful if the user could be notified in some way.
> >
> > Yes, that'd be another reason to alert the user.
> >
> > Patches to that effect are welcome.
> 
> I suppose if the output of M-x describe-package mentioned the commit
> hash in addition to the version the user would at least be aware when
> the same version numbers from different archives differ.  Out of
> curiosity, is that what you had in mind, or something else?

No, I meant exactly what you described: to detect the situation where
the commit which introduced the version number is not the same as the
commit with the version tag.  They are supposed to be the same commit.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

"Martin Edström" <meedstrom@runbox.eu> writes:

> Previous discussion: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=67955
>
> Currently, the setting of `use-package-vc-prefer-newest` defaults to nil.
> I want to bring up some problems with that, and suggest fixes.

Sorry for the late response, I'm responsible for this decision and would
like to defend it here.  Feel free to CC me in future discussions
pertaining to package-vc.  I have also added Tony to the CCs, as he has
a better understanding of the use-package side of this.

> PROBLEM 1. MISMATCH WITH PACKAGE-VC
>
> Now use-package :vc is not in line with the behavior of the
> package-vc-install command, which fetches the newest commit.
>
> All other package managers I know of also fetch the newest commit.

The main reference is package.el, which by default uses GNU ELPA and
NonGNU ELPA.  Both distribute explicitly released packages, as indicated
by the version header.  To me this has a much greater weight than what
third-party package managers and archives decide to do.

> It is the new use-package :vc which is the odd one out in the
> ecosystem, and that can cause stability issues, which I go into
> more deeply on this Reddit comment:
>
> https://old.reddit.com/r/emacs/comments/1f8ok7c/do_you_set_packageinstallupgradebuiltin_to_t/llimwe0/

>
> PROBLEM 2. "LAST RELEASE" CANNOT BE ASSUMED TO BE THE LAST RELEASE
>
> I see many packages that have not bumped their Package-Version
> header for years.  Perhaps they bump by git tags, or just stopped
> bumping versions.  I understand them, since it is a lot of manual
> labor to bump it, cut a release commit, then bump again to append a
> "-pre" or a ".0.50-git", and to remember to do this everywhere the
> version string may occur, like in docs.
>
> I'm also guessing there's a segment of developers that have sensed
> that this labor does not end up helping users, so they are not
> motivated to keep doing it.  Versions only work as intended if
> everyone is versioning regularly, and that is possible to enforce
> in a curated repo like GNU Elpa or Debian's repackaged Emacs
> packages, but as soon as you also install packages outside of that
> repo (and well, most packages are outside), there's no longer any
> guarantee and in fact the outdated versions in the curated repo are
> likely to cause breakage.
>
> I have personally had to give up on things like ELPA, Melpa Stable,
> Debian packages etc.  It always caused instability.  By contrast,
> installing 200-400 packages off Melpa, in other words the latest
> dev snapshots, has caused me no issues in ~7 years.  So there are
> two islands of stability:
>
> 1. Stay inside a curated repo entirely
> 2. Leave the curated repo entirely and install only dev snapshots
>
> Now consider what use-package :vc is for.  It is for installing
> things OUTSIDE any curated repo.  I leave it to you to pick a
> default setting for `use-package-vc-prefer-newest`.

No, this is an unjustified assumption.  FWIW my intention of writing
package-vc was to make contributing to packages easier.  The reason the
"install latest release" option exists for package-vc is that it can
make it easier to fix a bug with the current version.

The thing is that with use-package, having a more deterministic
installation is of interest, especially since use-package is used to
automatically set up an Emacs environment.  I do not share your
experience that non-stable MELPA is more stable than explicitly released
ELPA packages.

Regarding the main point of uncurrated packages: Users can circumvent
this by just upgrading the packages by hand, so I don't think it is that
bad.  That being said, I do think that the pressure to maintain packages
and cut versions from time to time is a good thing.  Here again, I don't
think that we should feel pressured by third-party repositories like
MELPA, that have been ignoring guidelines like in (elisp) Simple
Packages, that plainly state

        The version number comes from the ‘Package-Version’ header, if
        it exists, or from the ‘Version’ header otherwise.  One or the
        other _must_ be present.

Package-vc is related to ELPA and follows ELPA's practices.  The reason
that package specifications look the way they do is to encourage
contributing packages to ELPA.

> PROBLEM 3. FORCED REPO REWRITES
>
> I have been forced to use git-filter-repo to rewrite my package's
> commit history, to erase all instances of the Package-Version
> header.
>
> That's because I had been content to leave the header permanently
> at "0.6.0.50-git".  If I kept doing that with the new use-package
> :vc in existence, it would cause users to download a horribly
> outdated version of my package. If I understand correctly.
>
> I foresee one of two things will happen in the future:
>
> 1. More and more package authors carry out the same rewrite I did
> 2. (more likely) Increased instability as package authors do NOT do
>    this rewrite, and users end up with ancient versions all over
>    the place

It is well known that overwriting published git commits is bad practice,
which is why git push --force (or variations) is a thing.

The third option is that package maintainers become aware of the issue,
and start updating the header to mean something.

A feature we can add in the future is to support ELPA's :rolling-release
attribute, so that packages that wish to ensure that the default branch
is stable with every commit can do so without having to do tricks like
custom time-stamp file local variables that update every time the file
is modified.

>
> SUGGESTED FIXES
>
> A few possibilities:
>
> 1. Change `use-package-vc-prefer-newest` to default to t.
>
> 2. Change package-vc so when it walks the commit history to find
>    the "last release" or any specific version, it should target the
>    newest commit that has the given Package-Version, not the oldest
>    commit.
>
>    - That would allow devs to continue having a frozen
>      Package-Version without breaking users' setups.
>
>    - Of course, this is effectively the same as option 1, but the
>      semantic distinction might be usable in the future to
>      e.g. ignore version strings that have a "-pre" inside.

I am afraid I don't understand what you are suggesting here.  The
current approach is to find the newest commit that touches a version
header and use that.

> 3. Roll out a strategy for moving the entire community onto the
>    norm of versioning packages prior to distribution, ideally to
>    the extent that Melpa is no longer necessary and can go offline
>    in favor of Melpa Stable.  (Until then, Melpa Stable is best
>    avoided: https://old.reddit.com/r/emacs/comments/etikbz/)
>
>    Obviously difficult, but as I've said, versioning only works if
>    everyone does it.  Specifically, it gives us stability if every
>    developer is running the "released version" of most packages
>    other than their own.  Currently we have an ecosystem where most
>    developers are running the dev snapshot of most packages, and
>    any half-measures to change this will only destroy the islands
>    of stability that we do currently enjoy.  The topic of this
>    email is one such half-measure.

Everyone on {GNU,NonGNU} ELPA does it and it works fine.  There is
little we can do about third-party projects, so I don't concern myself
about them.  This is Emacs after all, you have the freedom to break
anything you want, and "THERE IS NO WARRANTY FOR THE PROGRAM" as the GPL
says.

> 4. Have Emacs ship an automation tool like
>    https://github.com/magit/sisyphus, recommend it everywhere, then
>    have package.el, package-vc and use-package :vc all throw an
>    error on finding more than one commit with a given
>    Package-Version string and refuse to install due to ambiguity.

Again, I am afraid we are talking past one another.  Are you suggesting
that the version number should be changed from commit to commit?

> 5. Deprecate the Package-Version header and instead maybe let
>    package.el perform an automatic string substitution from the
>    git/hg tag, if it is still desirable to see the version in the
>    source file.  The version could also be substituted into Info
>    docs and other places the version string should occur.
>
>    - This is effectively a convenient alternative to option 4.

IIRC this is not possible as to limitations with git tags that doesn't
allow us to mirror them in elpa.git.

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Eli Zaretskii <eliz@gnu.org> writes:

> No, I meant exactly what you described: to detect the situation where
> the commit which introduced the version number is not the same as the
> commit with the version tag.  They are supposed to be the same commit.

Two questions:

1. When would it make sense for this check to be triggered?

2. Who should be notified (and how) of this when this discrepancy is
   detected?

Ideally, the package authors / maintainers would be notified directly
via the GNU and NonGNU ELPA archives.  Or was your recommendation to
notify the users of the package upon install time, instead?  If former,
are there already some set of checks / tests run by the archives, where
such an additional check could be added?

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  meedstrom@runbox.eu,
>   yandros@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 19 Sep 2024 08:08:13 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No, I meant exactly what you described: to detect the situation where
> > the commit which introduced the version number is not the same as the
> > commit with the version tag.  They are supposed to be the same commit.
> 
> Two questions:
> 
> 1. When would it make sense for this check to be triggered?

I guess when we see version tags in the repository?

> 2. Who should be notified (and how) of this when this discrepancy is
>    detected?

The user (for example, by issuing a warning).

> Ideally, the package authors / maintainers would be notified directly
> via the GNU and NonGNU ELPA archives.  Or was your recommendation to
> notify the users of the package upon install time, instead?  If former,
> are there already some set of checks / tests run by the archives, where
> such an additional check could be added?

I meant the users.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip, I am summarizing below some thoughts / suggestions that I have
shared elsewhere in this discussion for your benefit.  If you're already
aware and are caught up with the discussion, please ignore (and excuse
the noise).

Philip Kaludercic <philipk@posteo.net> writes:

> The current approach is to find the newest commit that touches a
> version header and use that.

Would it be possible to reflect the commit corresponding to the version
in the "Other versions:" field in the output of M-x describe-package ?
Doing so would make it easier for the user to check when a package that
is available, say, in both MELPA Stable and NonGNU ELPA have the same
claimed version, but differ in their corresponding commits.

Additionally, if it were possible to reflect the commit date as well,
that would be helpful as well.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Philip, I am summarizing below some thoughts / suggestions that I have
> shared elsewhere in this discussion for your benefit.  If you're already
> aware and are caught up with the discussion, please ignore (and excuse
> the noise).

A single message rarely constitutes noise, so don't worry about it :)

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> The current approach is to find the newest commit that touches a
>> version header and use that.
>
> Would it be possible to reflect the commit corresponding to the version
> in the "Other versions:" field in the output of M-x describe-package ?

Why under "Other version:" (which links to other packages).  IIRC the
commit of a VC package should appear under the commit header.  There is
the possibility of using a symbolic name if available.

I now notice that this is not the case as I had assumed... I'll try to
fix it as soon as possible.

> Doing so would make it easier for the user to check when a package that
> is available, say, in both MELPA Stable and NonGNU ELPA have the same
> claimed version, but differ in their corresponding commits.

I know that ELPA annotates packages with the commit used to produce the
package.  When the above issue is fixed, then you should be able to
compare the two commits by following the link under "Other version:".

> Additionally, if it were possible to reflect the commit date as well,
> that would be helpful as well.

I don't think there is a nice vc-generic way to extract the date of a
commit, at least without extending the vc interface any further.  I
might be mistaken though.

The other issues is that if you only have the revision string from the
ELPA server, without any checkout, it isn't easy/cheap to determine the
date even if you just wanted to support Git.

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

Philip Kaludercic <philipk@posteo.net> writes:


[...]

>>> The current approach is to find the newest commit that touches a
>>> version header and use that.
>>
>> Would it be possible to reflect the commit corresponding to the version
>> in the "Other versions:" field in the output of M-x describe-package ?
>
> Why under "Other version:" (which links to other packages).  IIRC the
> commit of a VC package should appear under the commit header.  There is
> the possibility of using a symbolic name if available.
>
> I now notice that this is not the case as I had assumed... I'll try to
> fix it as soon as possible.

This is the fix:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Insert-correct-commit-data-into-VC-package-descripti.patch --]
[-- Type: text/x-diff, Size: 2071 bytes --]

From 3a3cd2d4e6915d02bfb66680c2c00f3e179c7e0f Mon Sep 17 00:00:00 2001
From: Philip Kaludercic <philipk@posteo.net>
Date: Thu, 19 Sep 2024 22:27:26 +0200
Subject: [PATCH] Insert correct commit data into VC package descriptions

* lisp/emacs-lisp/package-vc.el (package-vc-commit): Rename
argument from PKG to PKG-DESC.
(package-vc--generate-description-file): Update the "extras"
section of the package description with the revision string at
generation time.
---
 lisp/emacs-lisp/package-vc.el | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/lisp/emacs-lisp/package-vc.el b/lisp/emacs-lisp/package-vc.el
index 34610102aa0..e168096e153 100644
--- a/lisp/emacs-lisp/package-vc.el
+++ b/lisp/emacs-lisp/package-vc.el
@@ -265,13 +265,13 @@ package-vc--download-and-read-archives
 
 (add-hook 'package-read-archive-hook     #'package-vc--read-archive-data 20)
 
-(defun package-vc-commit (pkg)
-  "Return the last commit of a development package PKG."
-  (cl-assert (package-vc-p pkg))
+(defun package-vc-commit (pkg-desc)
+  "Return the last commit of a development package PKG-DESC."
+  (cl-assert (package-vc-p pkg-desc))
   ;; FIXME: vc should be extended to allow querying the commit of a
   ;; directory (as is possible when dealing with git repositories).
   ;; This should be a fallback option.
-  (cl-loop with dir = (package-desc-dir pkg)
+  (cl-loop with dir = (package-desc-dir pkg-desc)
            for file in (directory-files dir t "\\.el\\'" t)
            when (vc-working-revision file) return it
            finally return "unknown"))
@@ -359,7 +359,11 @@ package-vc--generate-description-file
                          requires))))
           (list :kind 'vc)
           (package--alist-to-plist-args
-           (package-desc-extras pkg-desc))))
+           (let ((extras (copy-alist (package-desc-extras pkg-desc))))
+             (setf (alist-get :commit extras)
+                   (package-vc-commit pkg-desc))
+             extras)
+           )))
         "\n")
        nil pkg-file nil 'silent))))
 
-- 
2.45.2


[-- Attachment #3: Type: text/plain, Size: 90 bytes --]


Would anyone mind me pushing this to emacs-30?

[...]


-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip Kaludercic <philipk@posteo.net> writes:

>> Would it be possible to reflect the commit corresponding to the version
>> in the "Other versions:" field in the output of M-x describe-package ?
>
> Why under "Other version:" (which links to other packages).

For a package that is available under both MELPA Stable and NonGNU ELPA
one can encounter the situation where the versions look the same, but
instead point to different commits.  They don't necessarily have to
point to different commits, and in fact when they do I would consider it
a bug.  However, as of today, a user only becomes aware of this if they
follow the links in the "Other version" and then contrast the "Commit"
information in the subsequent package's description with the one from
the prior page.

An example of a package where this was observed in the wild (probably
not the only case) was julia-mode.  The 0.4 version of that package on
NonGNU corresponds to commit 6408b96c1c97e41bc2af060d661afee4f7b22e89
whereas the version 0.4 on MELPA Stable corresponds to commit
8bfc709716a257521cb386f20b8932e83db930a9 .

Since the information is available via M-x button-describe on the links
to the other packages, hopefully it could be made visible (perhaps via
some customization).

> IIRC the commit of a VC package should appear under the commit header.

That necessitates additional clicks.  If those clicks or button pushes
could be avoided it would improve user convenience.

> There is the possibility of using a symbolic name if available.

Could you please give an example?

>> Additionally, if it were possible to reflect the commit date as well,
>> that would be helpful as well.
>
> I don't think there is a nice vc-generic way to extract the date of a
> commit, at least without extending the vc interface any further.  I
> might be mistaken though.

I may be mistaken, but my understanding was that if `annotate-command'
is defined for a backend then this may be possible via `annotate-time'.

> The other issues is that if you only have the revision string from the
> ELPA server, without any checkout, it isn't easy/cheap to determine the
> date even if you just wanted to support Git.

Ah.  That is, indeed, a problem.  Out of curiosity, what is an example
of such a revision string (say, for the julia-mode package from NonGNU
ELPA)?

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Tony Zorman]

On Thu, Sep 19 2024 11:49, Philip Kaludercic wrote:
> "Martin Edström" <meedstrom@runbox.eu> writes:
>
>> Previous discussion: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=67955
>>
>> Currently, the setting of `use-package-vc-prefer-newest` defaults to nil.
>> I want to bring up some problems with that, and suggest fixes.
>
> Sorry for the late response, I'm responsible for this decision and would
> like to defend it here.  Feel free to CC me in future discussions
> pertaining to package-vc.  I have also added Tony to the CCs, as he has
> a better understanding of the use-package side of this.

Thanks!

>> PROBLEM 1. MISMATCH WITH PACKAGE-VC
>>
>> Now use-package :vc is not in line with the behavior of the
>> package-vc-install command, which fetches the newest commit.
>>
>> All other package managers I know of also fetch the newest commit.
>
> The main reference is package.el, which by default uses GNU ELPA and
> NonGNU ELPA.  Both distribute explicitly released packages, as indicated
> by the version header.  To me this has a much greater weight than what
> third-party package managers and archives decide to do.

I will say that I also find it surprising behaviour that, by default,

    (package-vc-install "«URL»")

and

    (use-package blah
      :vc (:url "«URL»"))

have different semantics.

Ultimately, different people have different preferences and needs—for
example, "distros" such as DOOM Emacs manually pin all of their packages
to specific commits in order to ensure coherence. As you already know, I
personally would be in favour of setting use-package-vc-prefer-newest to
t by default, but I also only install packages that are either not on a
package archive or that I plan to contribute to with package-vc.el, the
rest is managed by my operating system.

>> It is the new use-package :vc which is the odd one out in the
>> ecosystem, and that can cause stability issues, which I go into
>> more deeply on this Reddit comment:
>>
>> https://old.reddit.com/r/emacs/comments/1f8ok7c/do_you_set_packageinstallupgradebuiltin_to_t/llimwe0/
>
>>
>> PROBLEM 2. "LAST RELEASE" CANNOT BE ASSUMED TO BE THE LAST RELEASE
>>
>> I see many packages that have not bumped their Package-Version
>> header for years.  Perhaps they bump by git tags, or just stopped
>> bumping versions.  I understand them, since it is a lot of manual
>> labor to bump it, cut a release commit, then bump again to append a
>> "-pre" or a ".0.50-git", and to remember to do this everywhere the
>> version string may occur, like in docs.
>>
>> I'm also guessing there's a segment of developers that have sensed
>> that this labor does not end up helping users, so they are not
>> motivated to keep doing it.  Versions only work as intended if
>> everyone is versioning regularly, and that is possible to enforce
>> in a curated repo like GNU Elpa or Debian's repackaged Emacs
>> packages, but as soon as you also install packages outside of that
>> repo (and well, most packages are outside), there's no longer any
>> guarantee and in fact the outdated versions in the curated repo are
>> likely to cause breakage.
>>
>> I have personally had to give up on things like ELPA, Melpa Stable,
>> Debian packages etc.  It always caused instability.  By contrast,
>> installing 200-400 packages off Melpa, in other words the latest
>> dev snapshots, has caused me no issues in ~7 years.  So there are
>> two islands of stability:
>>
>> 1. Stay inside a curated repo entirely
>> 2. Leave the curated repo entirely and install only dev snapshots
>>
>> Now consider what use-package :vc is for.  It is for installing
>> things OUTSIDE any curated repo.  I leave it to you to pick a
>> default setting for `use-package-vc-prefer-newest`.
>
> No, this is an unjustified assumption.  FWIW my intention of writing
> package-vc was to make contributing to packages easier.  The reason the
> "install latest release" option exists for package-vc is that it can
> make it easier to fix a bug with the current version.
>
> The thing is that with use-package, having a more deterministic
> installation is of interest, especially since use-package is used to
> automatically set up an Emacs environment.

I feel like the only way to actually guarantee a deterministic
installation procedure is to use outside means (that is, to ping
packages to specific versions/commits, either manually or with nix
et.al). This can certainly be done with package-vc.el as well, but
that's not what happens when one just uses :last-release.

  Tony

-- 
Tony Zorman | https://tony-zorman.com

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: Martin Edström <meedstrom@runbox.eu>,  "emacs-devel"
>  <emacs-devel@gnu.org>,  Tony Zorman <tonyzorman@mailbox.org>
> Date: Thu, 19 Sep 2024 20:31:25 +0000
> 
> >>> The current approach is to find the newest commit that touches a
> >>> version header and use that.
> >>
> >> Would it be possible to reflect the commit corresponding to the version
> >> in the "Other versions:" field in the output of M-x describe-package ?
> >
> > Why under "Other version:" (which links to other packages).  IIRC the
> > commit of a VC package should appear under the commit header.  There is
> > the possibility of using a symbolic name if available.
> >
> > I now notice that this is not the case as I had assumed... I'll try to
> > fix it as soon as possible.
> 
> This is the fix:

Can you explain in plain English what this does?

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: Martin Edström <meedstrom@runbox.eu>,  "emacs-devel"
>>  <emacs-devel@gnu.org>,  Tony Zorman <tonyzorman@mailbox.org>
>> Date: Thu, 19 Sep 2024 20:31:25 +0000
>> 
>> >>> The current approach is to find the newest commit that touches a
>> >>> version header and use that.
>> >>
>> >> Would it be possible to reflect the commit corresponding to the version
>> >> in the "Other versions:" field in the output of M-x describe-package ?
>> >
>> > Why under "Other version:" (which links to other packages).  IIRC the
>> > commit of a VC package should appear under the commit header.  There is
>> > the possibility of using a symbolic name if available.
>> >
>> > I now notice that this is not the case as I had assumed... I'll try to
>> > fix it as soon as possible.
>> 
>> This is the fix:
>
> Can you explain in plain English what this does?

Sure, each package has a description object (search for "(cl-defstruct
(package-desc" in package.el), with some fixed attributes like name,
version, dependencies and others that are stored in an alist.  The
commit used to create a package is one such attribute stored in the
"extras" slot of a package-desc.  My chance modifies the package
description given by the ELPA server and updates the :commit field to
store the commit that was checked out when the package was built
locally.  I hope that is clear enough?

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>   tonyzorman@mailbox.org
> Date: Fri, 20 Sep 2024 15:14:26 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> This is the fix:
> >
> > Can you explain in plain English what this does?
> 
> Sure, each package has a description object (search for "(cl-defstruct
> (package-desc" in package.el), with some fixed attributes like name,
> version, dependencies and others that are stored in an alist.  The
> commit used to create a package is one such attribute stored in the
> "extras" slot of a package-desc.  My chance modifies the package
> description given by the ELPA server and updates the :commit field to
> store the commit that was checked out when the package was built
> locally.  I hope that is clear enough?

Yes, thanks.  What was the :commit field storing before the change?

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>>   tonyzorman@mailbox.org
>> Date: Fri, 20 Sep 2024 15:14:26 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> This is the fix:
>> >
>> > Can you explain in plain English what this does?
>> 
>> Sure, each package has a description object (search for "(cl-defstruct
>> (package-desc" in package.el), with some fixed attributes like name,
>> version, dependencies and others that are stored in an alist.  The
>> commit used to create a package is one such attribute stored in the
>> "extras" slot of a package-desc.  My chance modifies the package
>> description given by the ELPA server and updates the :commit field to
>> store the commit that was checked out when the package was built
>> locally.  I hope that is clear enough?
>
> Yes, thanks.  What was the :commit field storing before the change?

The commit string of the last release, i.e. the revision used to build
the package on the server side.

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>   tonyzorman@mailbox.org
> Date: Fri, 20 Sep 2024 16:56:45 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Sure, each package has a description object (search for "(cl-defstruct
> >> (package-desc" in package.el), with some fixed attributes like name,
> >> version, dependencies and others that are stored in an alist.  The
> >> commit used to create a package is one such attribute stored in the
> >> "extras" slot of a package-desc.  My chance modifies the package
> >> description given by the ELPA server and updates the :commit field to
> >> store the commit that was checked out when the package was built
> >> locally.  I hope that is clear enough?
> >
> > Yes, thanks.  What was the :commit field storing before the change?
> 
> The commit string of the last release, i.e. the revision used to build
> the package on the server side.

Then I'm not really sure the change is TRT.  Why would we prefer the
local commit?

Re: Reconsider defaults for use-package-vc-prefer-newest

[Martin Edström]

On Fri, 20 Sep 2024 06:57:40 +0200, Tony Zorman <tonyzorman@mailbox.org> wrote:
> > The thing is that with use-package, having a more deterministic
> > installation is of interest, especially since use-package is used to
> > automatically set up an Emacs environment.
> 
> I feel like the only way to actually guarantee a deterministic
> installation procedure is to use outside means (that is, to ping
> packages to specific versions/commits, either manually or with nix
> et.al). This can certainly be done with package-vc.el as well, but
> that's not what happens when one just uses :last-release.
> 
>   Tony

Yea, I've recently thought that when you have an ecosystem that grew ad-hoc and has not very rigid curating, the solution isn't necessarily to start trying to pressure everyone into a reliable release cadence and break innocents' initfiles on the way. This isn't corporate, we're all lisping for free, and have to lisp smart, not hard. A Pareto solution would seem to be lockfiles. That is, an autogenerated file specifying pinned versions or commits.

Say there's a file "~/.emacs.d/versions.toml" containing lines such as

vertico = "1.9.0.33"
consult = "1.8.0.2"
ess = "24.0.20"
elpaca = "0.0.2.0.34"

and every time you upgrade, you can have the option of trying to upgrade everything at once, rolling back after you find it breaks, only upgrade some of them etc.  (Rolling back is a pretty important part)

This would be deterministic and everything.  Actually it sort of already happens with use-package, nevermind whether it fetches the latest snapshot or a "release version", because it's not as if it will auto-update every time you start Emacs, only when you manually tell it to update. The missing piece is for the user to know how to check their `elpa/` subdirectory into a git repo that they can rollback, but that's asking a lot for most users.

You might have noticed in the mockup above that I appended a ".0.X" number to all the versions.  That's because commit pinning is ugly and unsemantic. Here X is just the number of commits since the last "release version", no matter how long ago that was.  So Vertico 1.9.0.33 is the snapshot that's 33 commits after Vertico 1.9. It's totally possible to do something like that (got the idea from https://github.com/melpa/package-build/pull/81).

If a developer has no Package-Version and no git tag, it could just be forever at 0.0.X.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>> Would it be possible to reflect the commit corresponding to the version
>>> in the "Other versions:" field in the output of M-x describe-package ?
>>
>> Why under "Other version:" (which links to other packages).
>
> For a package that is available under both MELPA Stable and NonGNU ELPA
> one can encounter the situation where the versions look the same, but
> instead point to different commits.  They don't necessarily have to
> point to different commits, and in fact when they do I would consider it
> a bug.  However, as of today, a user only becomes aware of this if they
> follow the links in the "Other version" and then contrast the "Commit"
> information in the subsequent package's description with the one from
> the prior page.
>
> An example of a package where this was observed in the wild (probably
> not the only case) was julia-mode.  The 0.4 version of that package on
> NonGNU corresponds to commit 6408b96c1c97e41bc2af060d661afee4f7b22e89
> whereas the version 0.4 on MELPA Stable corresponds to commit
> 8bfc709716a257521cb386f20b8932e83db930a9 .
>
> Since the information is available via M-x button-describe on the links
> to the other packages, hopefully it could be made visible (perhaps via
> some customization).

One concern I have would be that an entire revision string can get very
cluttered, and is usually not interesting (as I don't use third-party
archives, I don't have duplicates of the kind that you describe).  I
wonder if indicating a "commit mismatch" for remote packages might be
interesting (we explicitly don't want this for local packages,
e.g. packages installed via package-vc).

Could you perhaps elaborate on why you consider this to be a bug?

>> IIRC the commit of a VC package should appear under the commit header.
>
> That necessitates additional clicks.  If those clicks or button pushes
> could be avoided it would improve user convenience.

True, but there are many things that could be made more convenient that
only interest few people.  I am trying to understand why this is
something that would interest everyone.

>> There is the possibility of using a symbolic name if available.
>
> Could you please give an example?

What I had in mind was using `vc-git-symbolic-commit'.  So in the case
of Git: If the commit
is following the "master" branch, then it would print "master" instead
a commit hash, likewise if the package is checked out on some tag, it
could display that.

>>> Additionally, if it were possible to reflect the commit date as well,
>>> that would be helpful as well.
>>
>> I don't think there is a nice vc-generic way to extract the date of a
>> commit, at least without extending the vc interface any further.  I
>> might be mistaken though.
>
> I may be mistaken, but my understanding was that if `annotate-command'
> is defined for a backend then this may be possible via `annotate-time'.

From what I see, that would only give us "the time of the next line of
annotation at or after point, as a floating point fractional number
of days.", and not something we can cleanly use to get an exact date.  I
mean you could try to multiply it with (* 60 60 24) to get the value
that was probably passed to `vc-annotate-convert-time', but that feels
unreliable to me.  But I too might be mistaken.

>> The other issues is that if you only have the revision string from the
>> ELPA server, without any checkout, it isn't easy/cheap to determine the
>> date even if you just wanted to support Git.
>
> Ah.  That is, indeed, a problem.  Out of curiosity, what is an example
> of such a revision string (say, for the julia-mode package from NonGNU
> ELPA)?

In the case of Git it is just a commit hash, like the ones you gave above?

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>>   tonyzorman@mailbox.org
>> Date: Fri, 20 Sep 2024 15:14:26 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> This is the fix:
>> >
>> > Can you explain in plain English what this does?
>> 
>> Sure, each package has a description object (search for "(cl-defstruct
>> (package-desc" in package.el), with some fixed attributes like name,
>> version, dependencies and others that are stored in an alist.  The
>> commit used to create a package is one such attribute stored in the
>> "extras" slot of a package-desc.  My chance modifies the package
>> description given by the ELPA server and updates the :commit field to
>> store the commit that was checked out when the package was built
>> locally.  I hope that is clear enough?
>
> Yes, thanks.  What was the :commit field storing before the change?

The commit the ELPA build server used to release a new version of the
package.  M-x list-packages lists the commit of the local checkout for
VC packages, btw.  I could have sworn that M-x describe-package did the
same.

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

"Martin Edström" <meedstrom@runbox.eu> writes:

> On Fri, 20 Sep 2024 06:57:40 +0200, Tony Zorman <tonyzorman@mailbox.org> wrote:
>> > The thing is that with use-package, having a more deterministic
>> > installation is of interest, especially since use-package is used to
>> > automatically set up an Emacs environment.
>> 
>> I feel like the only way to actually guarantee a deterministic
>> installation procedure is to use outside means (that is, to ping
>> packages to specific versions/commits, either manually or with nix
>> et.al). This can certainly be done with package-vc.el as well, but
>> that's not what happens when one just uses :last-release.
>> 
>>   Tony
>
> Yea, I've recently thought that when you have an ecosystem that grew
> ad-hoc and has not very rigid curating, the solution isn't necessarily
> to start trying to pressure everyone into a reliable release cadence
> and break innocents' initfiles on the way. This isn't corporate, we're
> all lisping for free, and have to lisp smart, not hard. A Pareto
> solution would seem to be lockfiles. That is, an autogenerated file
> specifying pinned versions or commits.

As mentioned before, both ELPA repositories requires releases to be
tagged, and it is not a problem.

> Say there's a file "~/.emacs.d/versions.toml" containing lines such as
>
> vertico = "1.9.0.33"
> consult = "1.8.0.2"
> ess = "24.0.20"
> elpaca = "0.0.2.0.34"
>
> and every time you upgrade, you can have the option of trying to
> upgrade everything at once, rolling back after you find it breaks,
> only upgrade some of them etc.  (Rolling back is a pretty important
> part)

Modulo the part of having separate files in a non-sexp format, this is a
good idea, that I don't think anyone would object to in principle.

> This would be deterministic and everything.  Actually it sort of
> already happens with use-package, nevermind whether it fetches the
> latest snapshot or a "release version", because it's not as if it will
> auto-update every time you start Emacs, only when you manually tell it
> to update. The missing piece is for the user to know how to check
> their `elpa/` subdirectory into a git repo that they can rollback, but
> that's asking a lot for most users.
>
> You might have noticed in the mockup above that I appended a ".0.X"
> number to all the versions.  That's because commit pinning is ugly and
> unsemantic. Here X is just the number of commits since the last
> "release version", no matter how long ago that was.  So Vertico
> 1.9.0.33 is the snapshot that's 33 commits after Vertico 1.9. It's
> totally possible to do something like that (got the idea from
> https://github.com/melpa/package-build/pull/81).
>
> If a developer has no Package-Version and no git tag, it could just be forever at 0.0.X.
>

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip Kaludercic <philipk@posteo.net> writes:

> I wonder if indicating a "commit mismatch" for remote packages might
> be interesting (we explicitly don't want this for local packages,
> e.g. packages installed via package-vc).

Depending on the logic of "commit mismatch" detection, that may be
sufficient.  Could you describe what you had in mind?

> Could you perhaps elaborate on why you consider this to be a bug?

To be clear I meant that it's a bug in the remote package.
Specifically, in the case of julia-mode, it was a bug for it to have
introduced the 0.4 package header in a commit that was different from
the one that was tagged as 0.4.

I don't consider this to be a bug in package.el.  However, if package.el
had made it obvious that the 0.4 version from melpa-stable isn't the
same as the 0.4 version from nongnu, it would have saved me some
headache.

For my part, I know better than to take version numbers at face value
now, especially when they are from different archives.

>>> IIRC the commit of a VC package should appear under the commit header.
>>
>> That necessitates additional clicks.  If those clicks or button pushes
>> could be avoided it would improve user convenience.
>
> True, but there are many things that could be made more convenient that
> only interest few people.  I am trying to understand why this is
> something that would interest everyone.

I don't believe this would interest _everyone_.  I also don't believe
that that's the necessary bar for a feature.  One might argue that this
is too niche a feature.  I don't believe so, but I also am not sure if
my opinion on the matter has any significance.

The people to whom this would be of interest would be users who have
melpa-stable enabled and who use some packages that are available both
on melpa-stable and either nongnu or gnu.

> What I had in mind was using `vc-git-symbolic-commit'.  So in the case
> of Git: If the commit
> is following the "master" branch, then it would print "master" instead
> a commit hash, likewise if the package is checked out on some tag, it
> could display that.

At least for the scenario I am considering, symbolic names would not be
very useful.  Commits are useful because they are unique (more or less).

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[chad]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Sat, Sep 21, 2024 at 2:53 AM Suhail Singh <suhailsingh247@gmail.com>
wrote:

> The people to whom this would be of interest would be users who have
> melpa-stable enabled and who use some packages that are available both
> on melpa-stable and either nongnu or gnu.


Isn't melpa-stable still considered harmful/to-be-avoided? That was my
understanding
years ago, and a quick web-search suggests that it's still basically true.

~Chad

[-- Attachment #2: Type: text/html, Size: 829 bytes --]

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>   tonyzorman@mailbox.org
> Date: Fri, 20 Sep 2024 20:43:27 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Philip Kaludercic <philipk@posteo.net>
> >> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
> >>   tonyzorman@mailbox.org
> >> Date: Fri, 20 Sep 2024 15:14:26 +0000
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> This is the fix:
> >> >
> >> > Can you explain in plain English what this does?
> >> 
> >> Sure, each package has a description object (search for "(cl-defstruct
> >> (package-desc" in package.el), with some fixed attributes like name,
> >> version, dependencies and others that are stored in an alist.  The
> >> commit used to create a package is one such attribute stored in the
> >> "extras" slot of a package-desc.  My chance modifies the package
> >> description given by the ELPA server and updates the :commit field to
> >> store the commit that was checked out when the package was built
> >> locally.  I hope that is clear enough?
> >
> > Yes, thanks.  What was the :commit field storing before the change?
> 
> The commit the ELPA build server used to release a new version of the
> package.

That is not right, I agree.

> M-x list-packages lists the commit of the local checkout for
> VC packages, btw.

What is "the local checkout" in this context?

> I could have sworn that M-x describe-package did the same.

Shouldn't they all be consistent in what commit they show?

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

chad <yandros@gmail.com> writes:

> Isn't melpa-stable still considered harmful/to-be-avoided? That was my
> understanding
> years ago, and a quick web-search suggests that it's still basically true.

The utility of melpa-stable depends on a package maintainer's diligence.
Consider the casual suite.  It is available on melpa-stable and the
author takes care to make meaningful releases.  As a user, using
melpa-stable for those packages has utility.

The fact that a large number of authors / maintainers of melpa packages
don't show a similar level of care for their users is certainly
unfortunate.  However, that's not the fault of melpa-stable.

One might argue that melpa-stable's use of a versioning mechanism that
deviates from that of GNU and NonGNU ELPA is problematic.  I agree with
that.  However, that's not the argument being made when people label
melpa-stable as "harmful".

In any case, as long as a package author is making regular tagged
releases melpa-stable is no worse than melpa.  In fact, it has some
utility over melpa in that it presents version numbers that are suitable
for use by the package manager.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>>   tonyzorman@mailbox.org
>> Date: Fri, 20 Sep 2024 20:43:27 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Philip Kaludercic <philipk@posteo.net>
>> >> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>> >>   tonyzorman@mailbox.org
>> >> Date: Fri, 20 Sep 2024 15:14:26 +0000
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> >> This is the fix:
>> >> >
>> >> > Can you explain in plain English what this does?
>> >> 
>> >> Sure, each package has a description object (search for "(cl-defstruct
>> >> (package-desc" in package.el), with some fixed attributes like name,
>> >> version, dependencies and others that are stored in an alist.  The
>> >> commit used to create a package is one such attribute stored in the
>> >> "extras" slot of a package-desc.  My chance modifies the package
>> >> description given by the ELPA server and updates the :commit field to
>> >> store the commit that was checked out when the package was built
>> >> locally.  I hope that is clear enough?
>> >
>> > Yes, thanks.  What was the :commit field storing before the change?
>> 
>> The commit the ELPA build server used to release a new version of the
>> package.
>
> That is not right, I agree.
>
>> M-x list-packages lists the commit of the local checkout for
>> VC packages, btw.
>
> What is "the local checkout" in this context?

The checkout of the VC package that the *Help* buffer is describing.

>> I could have sworn that M-x describe-package did the same.
>
> Shouldn't they all be consistent in what commit they show?

Yes, that is my point.  But since the commit in the package description
is not updated when the package is installed, there is a discrepancy.

-- 
	Philip Kaludercic on icterid

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Tony Zorman <tonyzorman@mailbox.org> writes:

>>> PROBLEM 1. MISMATCH WITH PACKAGE-VC
>>>
>>> Now use-package :vc is not in line with the behavior of the
>>> package-vc-install command, which fetches the newest commit.
>>>
>>> All other package managers I know of also fetch the newest commit.
>>
>> The main reference is package.el, which by default uses GNU ELPA and
>> NonGNU ELPA.  Both distribute explicitly released packages, as indicated
>> by the version header.  To me this has a much greater weight than what
>> third-party package managers and archives decide to do.
>
> I will say that I also find it surprising behaviour that, by default,
>
>     (package-vc-install "«URL»")
>
> and
>
>     (use-package blah
>       :vc (:url "«URL»"))
>
> have different semantics.

One idea I had we that we could require :vc to take a :rev argument.
But if it really is the case that we haven't found the right solution,
perhaps we should remove the keyword from the next release, or at least
mark it as some kind of an experimental feature?

-- 
	Philip Kaludercic on icterid

Re: Reconsider defaults for use-package-vc-prefer-newest

[Tony Zorman]

On Sat, Sep 21 2024 14:44, Philip Kaludercic wrote:
> Tony Zorman <tonyzorman@mailbox.org> writes:
>
>>>> PROBLEM 1. MISMATCH WITH PACKAGE-VC
>>>>
>>>> Now use-package :vc is not in line with the behavior of the
>>>> package-vc-install command, which fetches the newest commit.
>>>>
>>>> All other package managers I know of also fetch the newest commit.
>>>
>>> The main reference is package.el, which by default uses GNU ELPA and
>>> NonGNU ELPA.  Both distribute explicitly released packages, as indicated
>>> by the version header.  To me this has a much greater weight than what
>>> third-party package managers and archives decide to do.
>>
>> I will say that I also find it surprising behaviour that, by default,
>>
>>     (package-vc-install "«URL»")
>>
>> and
>>
>>     (use-package blah
>>       :vc (:url "«URL»"))
>>
>> have different semantics.
>
> One idea I had we that we could require :vc to take a :rev argument.
> But if it really is the case that we haven't found the right solution,
> perhaps we should remove the keyword from the next release, or at least
> mark it as some kind of an experimental feature?

That seems a bit too extreme to me, and I think the feature is much too
useful for that. Defaulting to *something* when not being given an
explicit revision also seems sensible, seeing how use-package generally
tries to do "the right thing". Plus, just setting
use-package-vc-prefer-newest to t resolves this "tension" in a
satisfactory manner—at least to me.

-- 
Tony Zorman | https://tony-zorman.com

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip Kaludercic <philipk@posteo.net> writes:

> But if it really is the case that we haven't found the right solution,
> perhaps we should remove the keyword from the next release, or at least
> mark it as some kind of an experimental feature?

I have no opinion as to the default behaviour, but having the :vc
keyword as a builtin is quite useful.  It would be unfortunate for it to
be removed (and a step back IMO, unless some mechansim with equivalent
functionality is provided).

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>   tonyzorman@mailbox.org
> Date: Sat, 21 Sep 2024 14:31:21 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Philip Kaludercic <philipk@posteo.net>
> >> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
> >>   tonyzorman@mailbox.org
> >> Date: Fri, 20 Sep 2024 20:43:27 +0000
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> From: Philip Kaludercic <philipk@posteo.net>
> >> >> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
> >> >>   tonyzorman@mailbox.org
> >> >> Date: Fri, 20 Sep 2024 15:14:26 +0000
> >> >> 
> >> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> >> 
> >> >> >> This is the fix:
> >> >> >
> >> >> > Can you explain in plain English what this does?
> >> >> 
> >> >> Sure, each package has a description object (search for "(cl-defstruct
> >> >> (package-desc" in package.el), with some fixed attributes like name,
> >> >> version, dependencies and others that are stored in an alist.  The
> >> >> commit used to create a package is one such attribute stored in the
> >> >> "extras" slot of a package-desc.  My chance modifies the package
> >> >> description given by the ELPA server and updates the :commit field to
> >> >> store the commit that was checked out when the package was built
> >> >> locally.  I hope that is clear enough?
> >> >
> >> > Yes, thanks.  What was the :commit field storing before the change?
> >> 
> >> The commit the ELPA build server used to release a new version of the
> >> package.
> >
> > That is not right, I agree.
> >
> >> M-x list-packages lists the commit of the local checkout for
> >> VC packages, btw.
> >
> > What is "the local checkout" in this context?
> 
> The checkout of the VC package that the *Help* buffer is describing.
> 
> >> I could have sworn that M-x describe-package did the same.
> >
> > Shouldn't they all be consistent in what commit they show?
> 
> Yes, that is my point.  But since the commit in the package description
> is not updated when the package is installed, there is a discrepancy.

So your change should make all of them agree?  In that case, please
install, and thanks.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>>   tonyzorman@mailbox.org
>> Date: Sat, 21 Sep 2024 14:31:21 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Philip Kaludercic <philipk@posteo.net>
>> >> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>> >>   tonyzorman@mailbox.org
>> >> Date: Fri, 20 Sep 2024 20:43:27 +0000
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> >> From: Philip Kaludercic <philipk@posteo.net>
>> >> >> Cc: suhailsingh247@gmail.com,  meedstrom@runbox.eu,  emacs-devel@gnu.org,
>> >> >>   tonyzorman@mailbox.org
>> >> >> Date: Fri, 20 Sep 2024 15:14:26 +0000
>> >> >> 
>> >> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> >> 
>> >> >> >> This is the fix:
>> >> >> >
>> >> >> > Can you explain in plain English what this does?
>> >> >> 
>> >> >> Sure, each package has a description object (search for "(cl-defstruct
>> >> >> (package-desc" in package.el), with some fixed attributes like name,
>> >> >> version, dependencies and others that are stored in an alist.  The
>> >> >> commit used to create a package is one such attribute stored in the
>> >> >> "extras" slot of a package-desc.  My chance modifies the package
>> >> >> description given by the ELPA server and updates the :commit field to
>> >> >> store the commit that was checked out when the package was built
>> >> >> locally.  I hope that is clear enough?
>> >> >
>> >> > Yes, thanks.  What was the :commit field storing before the change?
>> >> 
>> >> The commit the ELPA build server used to release a new version of the
>> >> package.
>> >
>> > That is not right, I agree.
>> >
>> >> M-x list-packages lists the commit of the local checkout for
>> >> VC packages, btw.
>> >
>> > What is "the local checkout" in this context?
>> 
>> The checkout of the VC package that the *Help* buffer is describing.
>> 
>> >> I could have sworn that M-x describe-package did the same.
>> >
>> > Shouldn't they all be consistent in what commit they show?
>> 
>> Yes, that is my point.  But since the commit in the package description
>> is not updated when the package is installed, there is a discrepancy.
>
> So your change should make all of them agree?  In that case, please
> install, and thanks.

Right, will do.

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> I wonder if indicating a "commit mismatch" for remote packages might
>> be interesting (we explicitly don't want this for local packages,
>> e.g. packages installed via package-vc).
>
> Depending on the logic of "commit mismatch" detection, that may be
> sufficient.  Could you describe what you had in mind?

In your case/the case of julia-mode, something like

  Other versions: 1.2.3 (melpa-stable, COMMIT MISMATCH).

with a help annotation.

>> Could you perhaps elaborate on why you consider this to be a bug?
>
> To be clear I meant that it's a bug in the remote package.

Perhaps I am being pedantic, but this sounds like a mistake, not a /bug/
in the code itself.

> Specifically, in the case of julia-mode, it was a bug for it to have
> introduced the 0.4 package header in a commit that was different from
> the one that was tagged as 0.4.

Do you know which of the two is correct?  In cases like these, it sounds
like one should contact the maintainers to remind them that they
shouldn't repeat the same issue in the future.

> I don't consider this to be a bug in package.el.  However, if package.el
> had made it obvious that the 0.4 version from melpa-stable isn't the
> same as the 0.4 version from nongnu, it would have saved me some
> headache.
>
> For my part, I know better than to take version numbers at face value
> now, especially when they are from different archives.
>
>>>> IIRC the commit of a VC package should appear under the commit header.
>>>
>>> That necessitates additional clicks.  If those clicks or button pushes
>>> could be avoided it would improve user convenience.
>>
>> True, but there are many things that could be made more convenient that
>> only interest few people.  I am trying to understand why this is
>> something that would interest everyone.
>
> I don't believe this would interest _everyone_.  I also don't believe
> that that's the necessary bar for a feature.  One might argue that this
> is too niche a feature.  I don't believe so, but I also am not sure if
> my opinion on the matter has any significance.

You are right, I misphrased that.  It should be "always".  If two
package versions are based on the same commit, then I'd argue this
information is not interesting, as this is assumed to be the default.

> The people to whom this would be of interest would be users who have
> melpa-stable enabled and who use some packages that are available both
> on melpa-stable and either nongnu or gnu.
>
>> What I had in mind was using `vc-git-symbolic-commit'.  So in the case
>> of Git: If the commit
>> is following the "master" branch, then it would print "master" instead
>> a commit hash, likewise if the package is checked out on some tag, it
>> could display that.
>
> At least for the scenario I am considering, symbolic names would not be
> very useful.  Commits are useful because they are unique (more or less).

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> But if it really is the case that we haven't found the right solution,
>> perhaps we should remove the keyword from the next release, or at least
>> mark it as some kind of an experimental feature?
>
> I have no opinion as to the default behaviour, but having the :vc
> keyword as a builtin is quite useful.  It would be unfortunate for it to
> be removed (and a step back IMO, unless some mechansim with equivalent
> functionality is provided).

To clarify, I am not suggesting removing it all together, but if there
is a serious issue regarding the consistency between the
`package-vc-install' command and the :vc keyword for use-package,
especially regarding the question what the correct behaviour for

  (use-package ... :vc ...) 

should be by default, then it might make sense to "hide" the feature for
now, not enable it by default, remove it from the emacs-30 branch or
just defer the proper addition until Emacs 31.

I tend to agree with Tony that the current approach seems like the best
compromise, but as Eli mentioned it is pretty late to address Martin's
complaint.  As such, the only real options I see is to keep everything
as it is or to defer the addition.  Personally I am fine with either. 

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip Kaludercic <philipk@posteo.net> writes:

>>> I wonder if indicating a "commit mismatch" for remote packages might
>>> be interesting (we explicitly don't want this for local packages,
>>> e.g. packages installed via package-vc).
>>
>> Depending on the logic of "commit mismatch" detection, that may be
>> sufficient.  Could you describe what you had in mind?
>
> In your case/the case of julia-mode, something like
>
>   Other versions: 1.2.3 (melpa-stable, COMMIT MISMATCH).
>
> with a help annotation.

Yes, something like that would have worked.  Thanks in advance, if you
do implement something of that nature.  It would be useful.

>>> Could you perhaps elaborate on why you consider this to be a bug?
>>
>> To be clear I meant that it's a bug in the remote package.
>
> Perhaps I am being pedantic, but this sounds like a mistake, not a /bug/
> in the code itself.

I meant that I considered it a software defect in the packaging process.
It might have been more accurate to term it a bug in the remote
package's packaging process.  In any case, the implication wasn't that
it was a bug in the remote package's source code.  I hope it's clearer
now.

>> Specifically, in the case of julia-mode, it was a bug for it to have
>> introduced the 0.4 package header in a commit that was different from
>> the one that was tagged as 0.4.
>
> Do you know which of the two is correct?  In cases like these, it sounds
> like one should contact the maintainers to remind them that they
> shouldn't repeat the same issue in the future.

Given that the change that was added between the commit that updated the
package header and the commit that corresponded to the git tag was a
fairly important bug-fix, I believe the intended revision (for 0.4) was
the one corresponding to the git tag (i.e., the more recent commit and
the one available via melpa-stable).

In general, for packages with a git repository that have a presence
outside of GNU and NonGNU ELPA, I believe the version corresponding to
the "git tag" to be more closely aligned with the maintainer's intent.

That being said, however, based on a recent exchange, the recommended
version that the maintainer of julia-mode wishes users download is the
"rolling release" equivalent from melpa.  I believe it is for this
reason that they have not made a tagged release in the last four years.
Quoting below the maintainer's response on the issue ([1]) where I
brought this matter to their attention:

#+begin_quote
  Since we do not make stable releases (effectively, it is just rolling on
  `master`, I think we should just clarify that users should use `melpa`, and if
  possible expunge the package from `melpa-stable` etc.
#+end_quote

[1]: <https://github.com/JuliaEditorSupport/julia-emacs/issues/212#issuecomment-2328150198>

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>>> I wonder if indicating a "commit mismatch" for remote packages might
>>>> be interesting (we explicitly don't want this for local packages,
>>>> e.g. packages installed via package-vc).
>>>
>>> Depending on the logic of "commit mismatch" detection, that may be
>>> sufficient.  Could you describe what you had in mind?
>>
>> In your case/the case of julia-mode, something like
>>
>>   Other versions: 1.2.3 (melpa-stable, COMMIT MISMATCH).
>>
>> with a help annotation.
>
> Yes, something like that would have worked.  Thanks in advance, if you
> do implement something of that nature.  It would be useful.

Can you test if this works:


[-- Attachment #2: Type: text/plain, Size: 1994 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 7cae8d68bc0..d27b6b73eee 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2961,23 +2961,26 @@ describe-package-1
                              (cdr (assq name package-archive-contents))
                              (let ((bi (assq name package--builtins)))
                                (if bi (list (package--from-builtin bi))))))
-           (other-pkgs (delete desc all-pkgs)))
+           (other-pkgs (delete desc all-pkgs))
+           (commit (alist-get :commit (package-desc-extras desc))))
       (when other-pkgs
         (package--print-help-section "Other versions"
           (mapconcat (lambda (opkg)
                        (let* ((ov (package-desc-version opkg))
                               (dir (package-desc-dir opkg))
                               (from (or (package-desc-archive opkg)
-                                        (if (stringp dir) "installed" dir))))
+                                        (if (stringp dir) "installed" dir)))
+                              (ocommit (alist-get :commit (package-desc-extras opkg))))
                          (if (not ov) (format "%s" from)
-                           (format "%s (%s)"
+                           (format "%s (%s%s)"
                                    (make-text-button (package-version-join ov) nil
                                                      'font-lock-face 'link
                                                      'follow-link t
                                                      'action
                                                      (lambda (_button)
                                                        (describe-package opkg)))
-                                   from))))
+                                   from
+                                   (if (equal ocommit commit) "" ", COMMIT MISMATCH!")))))
                      other-pkgs ", ")
           ".")))
 

[-- Attachment #3: Type: text/plain, Size: 2479 bytes --]


>>>> Could you perhaps elaborate on why you consider this to be a bug?
>>>
>>> To be clear I meant that it's a bug in the remote package.
>>
>> Perhaps I am being pedantic, but this sounds like a mistake, not a /bug/
>> in the code itself.
>
> I meant that I considered it a software defect in the packaging process.
> It might have been more accurate to term it a bug in the remote
> package's packaging process.  In any case, the implication wasn't that
> it was a bug in the remote package's source code.  I hope it's clearer
> now.

No worries, I think we both understand what we mean.

>>> Specifically, in the case of julia-mode, it was a bug for it to have
>>> introduced the 0.4 package header in a commit that was different from
>>> the one that was tagged as 0.4.
>>
>> Do you know which of the two is correct?  In cases like these, it sounds
>> like one should contact the maintainers to remind them that they
>> shouldn't repeat the same issue in the future.
>
> Given that the change that was added between the commit that updated the
> package header and the commit that corresponded to the git tag was a
> fairly important bug-fix, I believe the intended revision (for 0.4) was
> the one corresponding to the git tag (i.e., the more recent commit and
> the one available via melpa-stable).
>
> In general, for packages with a git repository that have a presence
> outside of GNU and NonGNU ELPA, I believe the version corresponding to
> the "git tag" to be more closely aligned with the maintainer's intent.
>
> That being said, however, based on a recent exchange, the recommended
> version that the maintainer of julia-mode wishes users download is the
> "rolling release" equivalent from melpa.  I believe it is for this
> reason that they have not made a tagged release in the last four years.
> Quoting below the maintainer's response on the issue ([1]) where I
> brought this matter to their attention:
>
> #+begin_quote
>   Since we do not make stable releases (effectively, it is just rolling on
>   `master`, I think we should just clarify that users should use `melpa`, and if
>   possible expunge the package from `melpa-stable` etc.
> #+end_quote
>
> [1]: <https://github.com/JuliaEditorSupport/julia-emacs/issues/212#issuecomment-2328150198>

If that is so, then we can also mark the ELPA package as using a
rolling-release model, i.e. the build server prepares a new tarball
every time is synchronises new commits.

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip Kaludercic <philipk@posteo.net> writes:

>>> In your case/the case of julia-mode, something like
>>>
>>>   Other versions: 1.2.3 (melpa-stable, COMMIT MISMATCH).
>>>
>>> with a help annotation.
>>
>> Yes, something like that would have worked.  Thanks in advance, if you
>> do implement something of that nature.  It would be useful.
>
> Can you test if this works:

The patch you shared does /something/, but it's not clear to me what the
expected behaviour is supposed to be.  Could you please comment?

Specifically, in the case of casual-dired (latest available on
melpa-stable is 1.8.2 and I have 1.8.1 installed), I observe the
following when viewing the package description:

#+begin_quote
  Other versions: 1.8.1 (installed, COMMIT MISMATCH!), 20240917.401 (melpa).
#+end_quote

The commit that introduced the 1.8.1 version header isn't the same as
the one that was assigned the git tag, so the above makes sense.
The fact that the melpa version doesn't have a corresponding COMMIT
MISMATCH seems desirable (since there's no expectation for the melpa
version to correspond to the package header in any way).

However, when I view the package description for casual-suite (the
latest on melpa-stable is 1.6.0 and I have the same installed), I
observe:

#+begin_quote
  Other versions: 1.6.0 (melpa-stable).
#+end_quote

Given that the commit that introduced the 1.6.0 version in the package
header is different from the one that was tagged by the git tag, I would
have expected a COMMIT MISMATCH in the above as well.  The absence of it
seems like a bug.

The situation gets weirder when I observe the description of htmlize.

From the melpa archive:
#+begin_quote
  Other versions: 20240915.1657 (installed), 1.58 (nongnu, COMMIT MISMATCH!), 1.58 (melpa-stable, COMMIT MISMATCH!).
#+end_quote

From the melpa-stable archive:
#+begin_quote
  Other versions: 20240915.1657 (installed, COMMIT MISMATCH!), 1.58 (nongnu), 20240915.1657 (melpa, COMMIT MISMATCH!).
#+end_quote

From the nongnu archive:
#+begin_quote
  Other versions: 20240915.1657 (installed, COMMIT MISMATCH!), 1.58 (melpa-stable), 20240915.1657 (melpa, COMMIT MISMATCH!).
#+end_quote

Could you comment on what's happening above?

>> That being said, however, based on a recent exchange, the recommended
>> version that the maintainer of julia-mode wishes users download is the
>> "rolling release" equivalent from melpa.  I believe it is for this
>> reason that they have not made a tagged release in the last four years.
>> Quoting below the maintainer's response on the issue ([1]) where I
>> brought this matter to their attention:
>>
>> #+begin_quote
>>   Since we do not make stable releases (effectively, it is just rolling on
>>   `master`, I think we should just clarify that users should use `melpa`, and if
>>   possible expunge the package from `melpa-stable` etc.
>> #+end_quote
>>
>> [1]: <https://github.com/JuliaEditorSupport/julia-emacs/issues/212#issuecomment-2328150198>
>
> If that is so, then we can also mark the ELPA package as using a
> rolling-release model, i.e. the build server prepares a new tarball
> every time is synchronises new commits.

I believe that that would be desirable, and the accurate thing in the
case of julia-mode.  Could you please confirm when that change has been
made.

Additionally, could you please point me to where the "rolling-release
model" is documented?  A search for "rolling" in
<https://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README> didn't
yield any matches.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Suhail Singh <suhailsingh247@gmail.com> writes:

>>> #+begin_quote
>>>   Since we do not make stable releases (effectively, it is just rolling on
>>>   `master`, I think we should just clarify that users should use `melpa`, and if
>>>   possible expunge the package from `melpa-stable` etc.
>>> #+end_quote
>>>
>>> [1]: <https://github.com/JuliaEditorSupport/julia-emacs/issues/212#issuecomment-2328150198>
>>
>> If that is so, then we can also mark the ELPA package as using a
>> rolling-release model, i.e. the build server prepares a new tarball
>> every time is synchronises new commits.
>
> I believe that that would be desirable, and the accurate thing in the
> case of julia-mode.  Could you please confirm when that change has been
> made.

It seems the maintainers are considering switching to semantic
versioning (version 1.0.0 onwards).  See
<https://github.com/JuliaEditorSupport/julia-emacs/pull/213> for
details.  Please don't switch to a rolling-release model.

> Additionally, could you please point me to where the "rolling-release
> model" is documented?  A search for "rolling" in
> <https://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README> didn't
> yield any matches.

Regardless, please do share the reference to the documentation when you
get a chance.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 3569 bytes --]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>>> In your case/the case of julia-mode, something like
>>>>
>>>>   Other versions: 1.2.3 (melpa-stable, COMMIT MISMATCH).
>>>>
>>>> with a help annotation.
>>>
>>> Yes, something like that would have worked.  Thanks in advance, if you
>>> do implement something of that nature.  It would be useful.
>>
>> Can you test if this works:
>
> The patch you shared does /something/, but it's not clear to me what the
> expected behaviour is supposed to be.  Could you please comment?

The idea is to add a "COMMIT MISMATCH" warning whenever we detect that
two packages have different commits.  What this doesn't do yet is
eliminate false positives, such as different commits between a local
version of a package and a remote version.  I guess we are only
interested in differences between remote packages, right?

> Specifically, in the case of casual-dired (latest available on
> melpa-stable is 1.8.2 and I have 1.8.1 installed), I observe the
> following when viewing the package description:
>
> #+begin_quote
>   Other versions: 1.8.1 (installed, COMMIT MISMATCH!), 20240917.401 (melpa).
> #+end_quote

Does MELPA annotate their packages with commits?

>
> The commit that introduced the 1.8.1 version header isn't the same as
> the one that was assigned the git tag, so the above makes sense.
> The fact that the melpa version doesn't have a corresponding COMMIT
> MISMATCH seems desirable (since there's no expectation for the melpa
> version to correspond to the package header in any way).

There is no causal-dired package on ELPA, so version headers don't
matter here since MELPA-stable relies on Git tags.

> However, when I view the package description for casual-suite (the
> latest on melpa-stable is 1.6.0 and I have the same installed), I
> observe:
>
> #+begin_quote
>   Other versions: 1.6.0 (melpa-stable).
> #+end_quote
>
>
> Given that the commit that introduced the 1.6.0 version in the package
> header is different from the one that was tagged by the git tag, I would
> have expected a COMMIT MISMATCH in the above as well.  The absence of it
> seems like a bug.

As mentioned above, this is to be expected.  MELPA ignores the Version
header, and given just the tarball, we cannot detect the discrepancy you mention.

> The situation gets weirder when I observe the description of htmlize.
>
> From the melpa archive:
>
> #+begin_quote
>   Other versions: 20240915.1657 (installed), 1.58 (nongnu, COMMIT MISMATCH!), 1.58 (melpa-stable, COMMIT MISMATCH!).
> #+end_quote
>
>
> From the melpa-stable archive:
>
> #+begin_quote
>   Other versions: 20240915.1657 (installed, COMMIT MISMATCH!), 1.58 (nongnu), 20240915.1657 (melpa, COMMIT MISMATCH!).
> #+end_quote
>
>
> From the nongnu archive:
>
> #+begin_quote
>   Other versions: 20240915.1657 (installed, COMMIT MISMATCH!), 1.58 (melpa-stable), 20240915.1657 (melpa, COMMIT MISMATCH!).
> #+end_quote
>
> Could you comment on what's happening above?

The fist one is fine, because you have installed the MELPA unstable
package, which appears to be ahead of both the MELPA and the NonGNU
version.  The last two though appear to be in sync, so the commit that
bumped the version tag is also the one that was tagged for release.
That is why in the last two, you see that there is no MISMATCH between
nongnu and melpa-stable (and vice-versa).

As mentioned in my previous message, I think it might be less confusing
if we just focus on mismatches between remote packages?  Does this
change improve that?


[-- Attachment #2: Type: text/plain, Size: 2115 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 7cae8d68bc0..1c7c2f40aa7 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -2961,23 +2961,28 @@ describe-package-1
                              (cdr (assq name package-archive-contents))
                              (let ((bi (assq name package--builtins)))
                                (if bi (list (package--from-builtin bi))))))
-           (other-pkgs (delete desc all-pkgs)))
+           (other-pkgs (delete desc all-pkgs))
+           (commit (alist-get :commit (package-desc-extras desc))))
       (when other-pkgs
         (package--print-help-section "Other versions"
           (mapconcat (lambda (opkg)
                        (let* ((ov (package-desc-version opkg))
                               (dir (package-desc-dir opkg))
                               (from (or (package-desc-archive opkg)
-                                        (if (stringp dir) "installed" dir))))
+                                        (if (stringp dir) "installed" dir)))
+                              (ocommit (alist-get :commit (package-desc-extras opkg))))
                          (if (not ov) (format "%s" from)
-                           (format "%s (%s)"
+                           (format "%s (%s%s)"
                                    (make-text-button (package-version-join ov) nil
                                                      'font-lock-face 'link
                                                      'follow-link t
                                                      'action
                                                      (lambda (_button)
                                                        (describe-package opkg)))
-                                   from))))
+                                   from
+                                   (if (and (not (package-desc-dir opkg))
+                                            (equal ocommit commit))
+                                       "" ", COMMIT MISMATCH!")))))
                      other-pkgs ", ")
           ".")))
 

[-- Attachment #3: Type: text/plain, Size: 1714 bytes --]


>>> That being said, however, based on a recent exchange, the recommended
>>> version that the maintainer of julia-mode wishes users download is the
>>> "rolling release" equivalent from melpa.  I believe it is for this
>>> reason that they have not made a tagged release in the last four years.
>>> Quoting below the maintainer's response on the issue ([1]) where I
>>> brought this matter to their attention:
>>>
>>> #+begin_quote
>>>   Since we do not make stable releases (effectively, it is just rolling on
>>>   `master`, I think we should just clarify that users should use `melpa`, and if
>>>   possible expunge the package from `melpa-stable` etc.
>>> #+end_quote
>>>
>>> [1]: <https://github.com/JuliaEditorSupport/julia-emacs/issues/212#issuecomment-2328150198>
>>
>> If that is so, then we can also mark the ELPA package as using a
>> rolling-release model, i.e. the build server prepares a new tarball
>> every time is synchronises new commits.
>
> I believe that that would be desirable, and the accurate thing in the
> case of julia-mode.  Could you please confirm when that change has been
> made.

I haven't changed anything.  Are you involved in julia-mode?

> Additionally, could you please point me to where the "rolling-release
> model" is documented?  A search for "rolling" in
> <https://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README> didn't
> yield any matches.

It is documented on the elpa-admin branch:

https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/README?h=elpa-admin&id=9bd65395f1d4875915731ddbdd73a471f10d7794#n215

That being said, I still think that this is a feature that we would want
to advise package maintainers not to use.


-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Suhail Singh <suhailsingh247@gmail.com> writes:
>
>>>> #+begin_quote
>>>>   Since we do not make stable releases (effectively, it is just rolling on
>>>>   `master`, I think we should just clarify that users should use `melpa`, and if
>>>>   possible expunge the package from `melpa-stable` etc.
>>>> #+end_quote
>>>>
>>>> [1]: <https://github.com/JuliaEditorSupport/julia-emacs/issues/212#issuecomment-2328150198>
>>>
>>> If that is so, then we can also mark the ELPA package as using a
>>> rolling-release model, i.e. the build server prepares a new tarball
>>> every time is synchronises new commits.
>>
>> I believe that that would be desirable, and the accurate thing in the
>> case of julia-mode.  Could you please confirm when that change has been
>> made.
>
> It seems the maintainers are considering switching to semantic
> versioning (version 1.0.0 onwards).  See
> <https://github.com/JuliaEditorSupport/julia-emacs/pull/213> for
> details.  Please don't switch to a rolling-release model.

No problem.

>> Additionally, could you please point me to where the "rolling-release
>> model" is documented?  A search for "rolling" in
>> <https://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README> didn't
>> yield any matches.
>
> Regardless, please do share the reference to the documentation when you
> get a chance.

I think we sent our previous messages just at the same time.  But for
the sake of anyone following the archives, I'll give the link here
again:

https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/README?h=elpa-admin&id=9bd65395f1d4875915731ddbdd73a471f10d7794#n215

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip Kaludercic <philipk@posteo.net> writes:

> The idea is to add a "COMMIT MISMATCH" warning whenever we detect that
> two packages have different commits.

Which are the two packages being considered?

From my perspective, the following is the desired behaviour: whenever
package.el has evidence that the same purported package version is being
served via different commits in the various remote archives (that the
user has enabled) the user is made aware.  If the package versions
aren't the same, then no "COMMIT MISMATCH" should be shown.

I.e., the situation of interest is when versions match, but commits
don't.  If it helps, "NON-UNIQUE COMMIT" might be more accurate, but I
don't have a strong opinion on the wording.

> What this doesn't do yet is eliminate false positives, such as
> different commits between a local version of a package and a remote
> version.  I guess we are only interested in differences between remote
> packages, right?

If the package is a local :vc checkout, I don't have strong opinions on
whether it is considered or ignored.  If the package is an installed
version (via package-install), then it should be compared against any
other version that has the same version number (i.e., in the same manner
as a remote package would be compared).

> Does MELPA annotate their packages with commits?

Both MELPA and MELPA Stable seem to.  I am basing this on the assumption
that the result of button-describe comes from the archive in question.
If there is a better way to confirm, please do let me know.

> +                                   (if (and (not (package-desc-dir opkg))
> +                                            (equal ocommit commit))
> +                                       "" ", COMMIT MISMATCH!")))))

If (package-desc-dir opkg) evaluates to non-nil, then the above
evaluates to ", COMMIT MISMATCH!" which seems incorrect.

> It is documented on the elpa-admin branch:
>
> https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/README?h=elpa-admin&id=9bd65395f1d4875915731ddbdd73a471f10d7794#n215

Thanks for sharing the reference, but why is this not in the default
branch (which is the only one linked from <https://elpa.gnu.org/>) to
begin with?  Alternatively, if the elpa-admin variant is considered the
canonical version, why doesn't the link from <https://elpa.gnu.org/>
point to
<https://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README?h=elpa-admin>
instead?

The comment at the top of the file states that the two versions "differ
slightly".  However, differences in the documentation of supported
options (regardless of whether or not their use is encouraged) is not
what I would consider a "slight" difference.

> That being said, I still think that this is a feature that we would want
> to advise package maintainers not to use.

Agreed.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Suhail Singh <suhailsingh247@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> The idea is to add a "COMMIT MISMATCH" warning whenever we detect that
>> two packages have different commits.
>
> Which are the two packages being considered?

IIRC the local package you have installed, the package on NonGNU ELPA,
the package on MELPA Stable and the package on MELPA Unstable.

> From my perspective, the following is the desired behaviour: whenever
> package.el has evidence that the same purported package version is being
> served via different commits in the various remote archives (that the
> user has enabled) the user is made aware.  If the package versions
> aren't the same, then no "COMMIT MISMATCH" should be shown.

Yeah, that was my intention as well.

> I.e., the situation of interest is when versions match, but commits
> don't.  If it helps, "NON-UNIQUE COMMIT" might be more accurate, but I
> don't have a strong opinion on the wording.

NON-UNIQUE COMMIT seems more vague to me? 

>> What this doesn't do yet is eliminate false positives, such as
>> different commits between a local version of a package and a remote
>> version.  I guess we are only interested in differences between remote
>> packages, right?
>
> If the package is a local :vc checkout, I don't have strong opinions on
> whether it is considered or ignored.  If the package is an installed
> version (via package-install), then it should be compared against any
> other version that has the same version number (i.e., in the same manner
> as a remote package would be compared).

We don't track multiple version numbers by default, but only the package
as announced by a package archive.  And VC packages seem like a bit of
an headache in this case, as 

>> Does MELPA annotate their packages with commits?
>
> Both MELPA and MELPA Stable seem to.  I am basing this on the assumption
> that the result of button-describe comes from the archive in question.
> If there is a better way to confirm, please do let me know.

I could have checked that as well myself, basically I just had to load

  https://melpa.org/packages/archive-contents

and then one sees that each package description has a :commit entry.

>> +                                   (if (and (not (package-desc-dir opkg))
>> +                                            (equal ocommit commit))
>> +                                       "" ", COMMIT MISMATCH!")))))
>
> If (package-desc-dir opkg) evaluates to non-nil, then the above
> evaluates to ", COMMIT MISMATCH!" which seems incorrect.

Right, I'll try to test this myself before sending you more patches ^^

>> It is documented on the elpa-admin branch:
>>
>> https://git.savannah.gnu.org/cgit/emacs/elpa.git/tree/README?h=elpa-admin&id=9bd65395f1d4875915731ddbdd73a471f10d7794#n215
>
> Thanks for sharing the reference, but why is this not in the default
> branch (which is the only one linked from <https://elpa.gnu.org/>) to
> begin with?  Alternatively, if the elpa-admin variant is considered the
> canonical version, why doesn't the link from <https://elpa.gnu.org/>
> point to
> <https://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README?h=elpa-admin>
> instead?

No reason really, I'll try to update it soon.

> The comment at the top of the file states that the two versions "differ
> slightly".  However, differences in the documentation of supported
> options (regardless of whether or not their use is encouraged) is not
> what I would consider a "slight" difference.
>
>> That being said, I still think that this is a feature that we would want
>> to advise package maintainers not to use.
>
> Agreed.

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip Kaludercic <philipk@posteo.net> writes:

>> Which are the two packages being considered?
>
> IIRC the local package you have installed, the package on NonGNU ELPA,
> the package on MELPA Stable and the package on MELPA Unstable.

Since MELPA Unstable adds time-based versioning which is incompatible
with NonGNU and MELPA Stable, it can simply be ignored.  I.e., *only*
when NonGNU (or much less likely, but for completeness GNU) ELPA and
MELPA Stable version numbers match do we care about checking to verify
if the commits match as well.

>> From my perspective, the following is the desired behaviour: whenever
>> package.el has evidence that the same purported package version is being
>> served via different commits in the various remote archives (that the
>> user has enabled) the user is made aware.  If the package versions
>> aren't the same, then no "COMMIT MISMATCH" should be shown.
>
> Yeah, that was my intention as well.

Hm I didn't see the check to ensure that the package versions were
identical in the last patch, but perhaps that was happening implicitly
somehow.

>> I.e., the situation of interest is when versions match, but commits
>> don't.  If it helps, "NON-UNIQUE COMMIT" might be more accurate, but I
>> don't have a strong opinion on the wording.
>
> NON-UNIQUE COMMIT seems more vague to me?

Yeah, I don't have a strong opinion on how we phrase it as long as the
intended intent is "the version number of this package matches the
version number available from another source, however the commits don't
match".

> And VC packages seem like a bit of an headache in this case, as

Yeah, if you want to exclude VC packages I have no objection.
Presumably if someone is using :vc checkout they are already paying
attention to the commits.

> Right, I'll try to test this myself before sending you more patches ^^

All good; appreciate your time on this :)

>> Thanks for sharing the reference, but why is this not in the default
>> branch (which is the only one linked from <https://elpa.gnu.org/>) to
>> begin with?  Alternatively, if the elpa-admin variant is considered the
>> canonical version, why doesn't the link from <https://elpa.gnu.org/>
>> point to
>> <https://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README?h=elpa-admin>
>> instead?
>
> No reason really, I'll try to update it soon.

Thank you.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Charles Choi]

As my package `casual-dired` has been mentioned in this thread, I'd like to understand better what the recommended practice for package versioning is.

At current my development practice is to:

- Create a feature branch from the `main` branch and bump the `Version:` header.
- Make commits to the feature branch.
- Merge the feature branch back to `main`.
- Tag `main` with the `Version:` header value.

The above practice seems to violate whatever logic is being used to determine if a package is versioned correctly as it raises a COMMIT MISMATCH warning. My interpretation of this is that the `Version:` header is being used to track the build of a package which seems to me too strict. As long as the tag is in sync with the `Version:` header, I think it should be fine as a package can have many files.

Just my 2¢.

Charles

—
Charles Y. Choi, Ph.D.
kickingvegas@gmail.com

Re: Reconsider defaults for use-package-vc-prefer-newest

[Adam Porter]

> As my package `casual-dired` has been mentioned in this thread, I'd
> like to understand better what the recommended practice for package
> versioning is.
> 
> At current my development practice is to:
> 
> - Create a feature branch from the `main` branch and bump the
> `Version:` header. - Make commits to the feature branch. - Merge the
> feature branch back to `main`. - Tag `main` with the `Version:`
> header value.
> 
> The above practice seems to violate whatever logic is being used to
> determine if a package is versioned correctly as it raises a COMMIT
> MISMATCH warning. My interpretation of this is that the `Version:`
> header is being used to track the build of a package which seems to
> me too strict. As long as the tag is in sync with the `Version:`
> header, I think it should be fine as a package can have many files.
> 
> Just my 2¢.

This is described in the ELPA documentation.  See
<https://git.savannah.gnu.org/cgit/emacs/elpa.git/plain/README> and
search for occurrences of "version".

The procedure I recommend, which you can see in the commit history of my 
ELPA packages is:

1. Configure the ELPA recipe to build from a "stable" release branch. 
Develop new versions on the master branch.

2. On master, make a new "Meta:" commit that updates the "Y" in the 
version header and adds a suffix, like "X.Y-pre" (e.g. "0.1-pre").

3. Develop the next version (the "Y" version, in SemVer terms) on master.

4. When that version is ready to release, make a new "Release:" commit 
that updates the version header, removing the suffix, to "X.Y" (e.g. "0.1").

5. Merge that commit into the "stable" branch, from which ELPA will build.

6. When ready to start developing the next version, goto 2.

7. To make a bug-fix release on a stable version, apply the same steps 
but on the stable branch (i.e. a "Meta:" commit setting the version 
header to "X.Y.Z-pre", then add commits fixing bugs, then a "Release:" 
commit setting the version header to "X.Y.Z"); then push the stable 
branch so ELPA will build it; and merge the stable branch into master to 
apply the bug fixes to master as appropriate.

This workflow is simple and only requires a few moments to update the 
version headers when needed.  It clearly separates commits that update 
source code from commits that update version information.  It ensures 
that only one commit (on each branch) has a version header with a 
certain version number (so there is no confusion about which commit with 
version X.Y a user actually has).  And it results in a simple commit 
graph.

YMMV, IWFM, etc.

--Adam

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Charles Choi <kickingvegas@gmail.com> writes:

> ... I'd like to understand better what the recommended practice for
> package versioning is.

I am not an authority on the matter, so please take my comments below
with a grain of salt and for information purposes only (i.e., they are
not a recommendation).

> - Create a feature branch from the `main` branch and bump the `Version:` header.
> - Make commits to the feature branch.
> - Merge the feature branch back to `main`.

Without going into what could be possible or should be possible, when
merging the feature branch back to `main' if you perform a fast-forward
merge all will be well.

> - Tag `main` with the `Version:` header value.

Doing a fast-forward merge will ensure that when you tag `main' the
commit that gets tagged is the one that bumped the `Version:' (or
`Package-Version:') value.

> ... a package can have many files.

In that case, the desirable thing is to update the version headers in
all files in a single commit.

If a package release is made for version "a.b.c", it is worthwhile to
also make the following commit update the package headers to something
like "a.b.c.50-git".  Doing so ensures that version strings are
monotonically non-decreasing and ensures that anyone cloning the
repository where HEAD is not-yet released gets a version number that is
consistent and distinct from the latest release.

If you're using the `sisyphus' package, the below configuration may be
of interest:

#+begin_src emacs-lisp
  (use-package sisyphus :ensure t :pin melpa
    :unless noninteractive
    :after magit :demand t
    :config
    (setq sisyphus-non-release-bump-header t
          sisyphus-non-release-suffix ".50-git"))
#+end_src

With the above, I can use `M-x sisyphus-create-release` and `M-x
sisyphus-bump-post-release` as conveniences.  On a related note, if you
do decide to use `sisyphus`, you may need to rename `Version:` in the
package header to `Package-Version:` instead.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Charles Choi <kickingvegas@gmail.com>
> Date: Thu, 26 Sep 2024 16:23:22 -0700
> Cc: Philip Kaludercic <philipk@posteo.net>,
>  Martin Edström <meedstrom@runbox.eu>,
>  Tony Zorman <tonyzorman@mailbox.org>
> 
> As my package `casual-dired` has been mentioned in this thread, I'd like to understand better what the recommended practice for package versioning is.
> 
> At current my development practice is to:
> 
> - Create a feature branch from the `main` branch and bump the `Version:` header.
> - Make commits to the feature branch.
> - Merge the feature branch back to `main`.
> - Tag `main` with the `Version:` header value.
> 
> The above practice seems to violate whatever logic is being used to determine if a package is versioned correctly as it raises a COMMIT MISMATCH warning. My interpretation of this is that the `Version:` header is being used to track the build of a package which seems to me too strict. As long as the tag is in sync with the `Version:` header, I think it should be fine as a package can have many files.

The above seems to indicate there's some kind of bug in deciding
whether there's a commit mismatch, because I don't see anything wrong
with what you do.  As long as the version tag tags the commit where
the corresponding "Version:" entered the mainline, be it a
merge-commit or a regular commit, the mismatch alert should not
happen.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> The idea is to add a "COMMIT MISMATCH" warning whenever we detect that
  > >> two packages have different commits.
  > >
  > > Which are the two packages being considered?

  > IIRC the local package you have installed, the package on NonGNU ELPA,
  > the package on MELPA Stable and the package on MELPA Unstable.

Would it be possible to describe a concrete example and state
concretely what sort of problem this would aim to detect?
How do those problems arise in practice?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Reconsider defaults for use-package-vc-prefer-newest

[Philip Kaludercic]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > >> The idea is to add a "COMMIT MISMATCH" warning whenever we detect that
>   > >> two packages have different commits.
>   > >
>   > > Which are the two packages being considered?
>
>   > IIRC the local package you have installed, the package on NonGNU ELPA,
>   > the package on MELPA Stable and the package on MELPA Unstable.
>
> Would it be possible to describe a concrete example and state
> concretely what sort of problem this would aim to detect?

Suhail gave the example of julia-mode.

> How do those problems arise in practice?

Because of differences in how releases are marked.  ELPA checks if a
commit touches the "Version" header of an Elisp file, while MELPA
requires a Git tag to annotate the commit used to create the tarball.

-- 
	Philip Kaludercic on siskin

Re: Reconsider defaults for use-package-vc-prefer-newest

[Suhail Singh]

Philip Kaludercic <philipk@posteo.net> writes:

>> Would it be possible to describe a concrete example and state
>> concretely what sort of problem this would aim to detect?
>
> Suhail gave the example of julia-mode.

Specifically, for the 0.4 version of julia-mode the release on NonGNU
ELPA differed from the 0.4 release that was served on MELPA Stable.

>> How do those problems arise in practice?
>
> Because of differences in how releases are marked.  ELPA checks if a
> commit touches the "Version" header of an Elisp file, while MELPA
> requires a Git tag to annotate the commit used to create the tarball.

And in the case of julia-mode, for the 0.4 release, the commit that was
tagged was (meaningfully) distinct from the commit that updated the
"Version" header.

Users who had enabled the melpa-stable archive and installed julia-mode
version 0.4 experienced differing behavior based on whether they
installed it from melpa-stable or nongnu, due to the mistake by the
maintainer.  Specifically, the release on nongnu was buggy since it
preceded the commit where the regression was fixed.

-- 
Suhail

Re: Reconsider defaults for use-package-vc-prefer-newest

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> The above seems to indicate there's some kind of bug in deciding
> whether there's a commit mismatch, because I don't see anything wrong
> with what you do.

The problem is the order in which the steps are taken.  A git tag should
be added only to those commits that bump the "Version" header, or you
will run into problems.

In other words, this:

  1. Create a feature branch from the `main` branch.
  2. Bump the `Version:` header.
  3. Make commits to the feature branch.
  4. Merge the feature branch back to `main`.
  5. Tag `main` with the `Version:` header value.

Should be done like this instead:

  1. Create a feature branch from the `main` branch.
  2. Make commits to the feature branch.
  3. Merge the feature branch back to `main`.
  4. Bump the `Version:` header.
  5. Tag `main` with the `Version:` header value.

Re: Reconsider defaults for use-package-vc-prefer-newest

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sun, 29 Sep 2024 07:22:10 -0700
> Cc: suhailsingh247@gmail.com, emacs-devel@gnu.org, philipk@posteo.net, 
> 	meedstrom@runbox.eu, tonyzorman@mailbox.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The above seems to indicate there's some kind of bug in deciding
> > whether there's a commit mismatch, because I don't see anything wrong
> > with what you do.
> 
> The problem is the order in which the steps are taken.  A git tag should
> be added only to those commits that bump the "Version" header, or you
> will run into problems.
> 
> In other words, this:
> 
>   1. Create a feature branch from the `main` branch.
>   2. Bump the `Version:` header.
>   3. Make commits to the feature branch.
>   4. Merge the feature branch back to `main`.
>   5. Tag `main` with the `Version:` header value.

In the original description, step 2 and step 3 were in reverse order,
AFAIR, or at least I thought so.