MELPA version numbers

MELPA version numbers

[Stefan Monnier]

[ As is sadly becoming the norm nowadays, it's difficult to find
  a contact email address at http://melpa.milkbox.net/, so I send this
  here instead, hoping someone here will know where to resend it for
  me.  ]

A few times now I've heard people suffer from problems due to MELPA's
use of incomparable version numbers: MELPA builds packages straight from
a DVCS branch tip, so you get the "bleeding edge" and their build script
ignores the package's own version number and instead just slaps
a version number based on the current time, such as 20100105.123.

This system makes sense, but if the package is also available (in it's
"latest released version") via GNU ELPA or Marmalade, we have a problem
because the two version numbers can't be compared correctly, and
package.el doesn't even know it, so it just always picks MELPA's version
(since 20100105.123 is "clearly" much more recent than say 2.3).

This mess works surprisingly well in practice, since MELPA's versions
are often at least as recent as the one on GNU ELPA or Maramalde (by virtue
of being the bleeding edge from the DVCS).

But if you ever stop using MELPA, all your MELPA-installed packages will
stay non-updated for the foreseeable future since it'll take a while for
foo.el to go from version 2.3 to version 20100106.0.
And if for some reason the MELPA recipe points to an old DVCS
unmaintained branch, you're similarly out of luck.

One way out of this is to change the MELPA version numbers so that
instead of ignoring the package's "2.3" and replacing it with
"20100105.123", it should replace it with "2.3.20100105.123".

WDYT?


        Stefan

Re: MELPA version numbers

[Donald Curtis]

[-- Attachment #1: Type: text/plain, Size: 2719 bytes --]

I apologize for the difficulty of finding contact information. We try to do
most dealings via GitHub which we know doesn't work for everyone.

This would work as a solution except for that we would then break updates
for current users in the same way that it would break if someone stopped
using MELPA. It is also not clear in this case how to handle packages that
don't include a version number.

I'm happy to keep this discussion open as I know it affects many users as
well as developers.

One solution I might suggest is the approach that js2-mode has taken on the
GNU repo. It seems that their versions follow with MELPA sans the time. I
think that having arbitrary version numbers for things like extensions is a
little overkill and MELPA is at least a little evidence that versioning
based in the date of release *can* work. In this case it could be as simple
as the Emacs community endorsing a release date based versioning system.






On Thu, Aug 1, 2013 at 10:50 AM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:

> [ As is sadly becoming the norm nowadays, it's difficult to find
>   a contact email address at http://melpa.milkbox.net/, so I send this
>   here instead, hoping someone here will know where to resend it for
>   me.  ]
>
> A few times now I've heard people suffer from problems due to MELPA's
> use of incomparable version numbers: MELPA builds packages straight from
> a DVCS branch tip, so you get the "bleeding edge" and their build script
> ignores the package's own version number and instead just slaps
> a version number based on the current time, such as 20100105.123.
>
> This system makes sense, but if the package is also available (in it's
> "latest released version") via GNU ELPA or Marmalade, we have a problem
> because the two version numbers can't be compared correctly, and
> package.el doesn't even know it, so it just always picks MELPA's version
> (since 20100105.123 is "clearly" much more recent than say 2.3).
>
> This mess works surprisingly well in practice, since MELPA's versions
> are often at least as recent as the one on GNU ELPA or Maramalde (by virtue
> of being the bleeding edge from the DVCS).
>
> But if you ever stop using MELPA, all your MELPA-installed packages will
> stay non-updated for the foreseeable future since it'll take a while for
> foo.el to go from version 2.3 to version 20100106.0.
> And if for some reason the MELPA recipe points to an old DVCS
> unmaintained branch, you're similarly out of luck.
>
> One way out of this is to change the MELPA version numbers so that
> instead of ignoring the package's "2.3" and replacing it with
> "20100105.123", it should replace it with "2.3.20100105.123".
>
> WDYT?
>
>
>         Stefan
>

[-- Attachment #2: Type: text/html, Size: 3514 bytes --]

Re: MELPA version numbers

[Stefan Monnier]

> This would work as a solution except for that we would then break updates
> for current users in the same way that it would break if someone stopped
> using MELPA.

Yes, clearly, that would be a big cost (one-time cost, but fairly big).
Maybe we could try to invent a smoother transition.  E.g.:
1- use "20100105.123.2.3" instead of "2.3.20100105.123".  Rebuild all
   packages to follow this rule.  This way, most users will "upgrade"
   their package to the "newer" (identical save the revnb) packages, so
   that old-style revnb will disappear over time.
2- change package.el to recognize those special mid-style rev numbers
   and internally handle them as new-style "2.3.20100105.123".
   This way, users of old package.el get the same behavior as before,
   but users of newer ones get the new behavior.
3- after a few years (when you can expect most users to have upgraded
   their installed packages to the mid-style revnb and to use the newer
   package.el), switch to new-style "2.3.20100105.123".

> It is also not clear in this case how to handle packages that
> don't include a version number.

No version could be treated like version 0.0 ?

> One solution I might suggest is the approach that js2-mode has taken
> on the GNU repo.  It seems that their versions follow with MELPA sans
> the time.

They had the same problem: they "accidentally" used a YYYYMMDD version
numbering, and when they fixed that, they discovered that it prevented
users from upgrading, so they had to revert to YYYYMMDD.

> I think that having arbitrary version numbers for things
> like extensions is a little overkill and MELPA is at least a little
> evidence that versioning based in the date of release *can* work.
> In this case it could be as simple as the Emacs community endorsing
> a release date based versioning system.

I tend to agree that for many package, version numbers aren't that
important.  But they are used and I think there are good cognitive
reasons for them.  Ubuntu's date-based version numbers seem to strike
a reasonable balance, but I don't think this could work for MELPA since
the "resolution" needs to be much higher (for lack of control over the
rate of releases).  Tho at least for GNU ELPA, I guess we could stick to
"YY.MM" release names.

> I think, ultimately this is not an MELPA problem, but a shortcoming of
> package.el.  One of many others…

Clearly, that's another way to look at it.  After all, package evolution
is not always linear, so we may/will hit similar problems sooner or
later, kind of like what happened back then with Ispell-4.

But these non-linearities are exceptions, whereas the issue here is
between different version numbers used for the very same
linear evolution.

A solution to both non-linearities cold be to add release-time as
a package-property.  So package.el could then warn the user "there's
a newer package with an older version number, you might want to look at
it".


        Stefan

Re: MELPA version numbers

[Sebastian Wiesner]

2013/8/1 Stefan Monnier <monnier@iro.umontreal.ca>:
>> I think, ultimately this is not an MELPA problem, but a shortcoming of
>> package.el.  One of many others…
>
> Clearly, that's another way to look at it.  After all, package evolution
> is not always linear, so we may/will hit similar problems sooner or
> later, kind of like what happened back then with Ispell-4.
>
> But these non-linearities are exceptions, whereas the issue here is
> between different version numbers used for the very same
> linear evolution.
>
> A solution to both non-linearities cold be to add release-time as
> a package-property.  So package.el could then warn the user "there's
> a newer package with an older version number, you might want to look at
> it".

That's not what I meant.  I don't think that the problem is in
versioning at all.  Rather the problem is the insufficient tracking of
multiple archives in package.el.

It doesn't track archives independently, and though it actually stores
the origin of a package, it doesn't make any use of this information
at all. These limitations sort of defeat the purpose of having
multiple archives imho.

Ideally, package.el would track *all* packages of *all* repositories
in "package-archive-contents" (not just the latest out of all
repositories), so that packages, which are contained in multiple
archives, also appear multiple times in M-x list-packages, once for
each archive.

The user could then choose from which archive a package should be
installed.  And package.el would remember this choice, and look for
updates only in the archive the package was installed from.  If this
archive is no longer contained in "package-archive-contents",
package.el would warn the user, and offer to look for the package in
other archives.

To explicitly “switch” archives for a single package, the user would
just explicitly install the package from another archive, implicitly
telling packages to “pin” this package to the new archive from now on.

This more sophisticated way of tracking archives would not only neatly
solve the problem of MELPA version numbers (without even touching
them), but also a number of other issues with package.el, such as the
popular demand to be able to explicitly choose for each package
whether to use a stable release from GNU ELPA or Marmalade, or a VCS
snapshot from MELPA.

And it'd give *users* control about archives and the ability to choose.

Re: MELPA version numbers

[Stefan Monnier]

> It doesn't track archives independently, and though it actually stores
> the origin of a package, it doesn't make any use of this information
> at all.

Not true any more.  When you click on a package, it lists the "other
versions" it knows about, including their respective origin.

Also you can pin a package to a particular archive.

> These limitations sort of defeat the purpose of having
> multiple archives imho.

> Ideally, package.el would track *all* packages of *all* repositories
> in "package-archive-contents" (not just the latest out of all
> repositories), so that packages, which are contained in multiple
> archives, also appear multiple times in M-x list-packages, once for
> each archive.

The current package.el does store all packages in
package-archive-contents, IIRC.  And M-x list-packages only has one
entry per package, but once you select one of those entries, it should
tell you about all the versions it knows.

If you prefer to list all the versions directly in list-packages, we
could add an option for that (patches welcome).  The 24.3 code did
sometimes list a package twice (e.g. cl-lib which is both listed as
built-in and as available from GNU ELPA), but when the two versions
don't have the same status ("built-in" vs "available"), they get sorted
by default to completely different places.  For that reason I decided to
have this two-level approach where list-package only lists a package
once and only when you select a package do you see the
available alternatives.

> The user could then choose from which archive a package should be
> installed.

That can be done now.

> And package.el would remember this choice, and look for
> updates only in the archive the package was installed from.

That indeed is not available currently.  I'm not convinced we'd want to
always restrict updates to come from the same archive.  But it could be
a useful option.  Actually, we can already do that via pinning, but it's
obviously not as streamlined as you'd want.
I guess the best way to do it would be something like:
- when the user requests to install a package, check if it's the most recent.
- if not, ask the user if he wants to "pin" that package either to the
  corresponding archive or to the specific version.
Patches welcome.

> This more sophisticated way of tracking archives would not only neatly
> solve the problem of MELPA version numbers (without even touching them),

No, it doesn't really solve the problem: most people wouldn't know when
MELPA's version is out of date.  How would you know to switch archive if
package.el doesn't tell you that there's a newer version in the
"gnu" archive?

> but also a number of other issues with package.el, such as the
> popular demand to be able to explicitly choose for each package
> whether to use a stable release from GNU ELPA or Marmalade, or a VCS
> snapshot from MELPA.

Hmm... I hadn't heard those requests, but I think I did satisfy them to
some extent in my recentish changes ;-)


        Stefan

Re: MELPA version numbers

[Steve Purcell]

On 2 Aug 2013, at 15:27, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> This more sophisticated way of tracking archives would not only neatly
>> solve the problem of MELPA version numbers (without even touching them),
> 
> No, it doesn't really solve the problem: most people wouldn't know when
> MELPA's version is out of date.  How would you know to switch archive if
> package.el doesn't tell you that there's a newer version in the
> "gnu" archive?


Can that even happen, for longer than a MELPA build cycle?

-Steve (MELPA co-maintainer)

Re: MELPA version numbers

[Sebastian Wiesner]

2013/8/2 Stefan Monnier <monnier@iro.umontreal.ca>:
>> It doesn't track archives independently, and though it actually stores
>> the origin of a package, it doesn't make any use of this information
>> at all.
>
> Not true any more.  When you click on a package, it lists the "other
> versions" it knows about, including their respective origin.
>
> Also you can pin a package to a particular archive.

Indeed, I just discovered "package-pinned-packages", which I presume
is what you are referring to.  In theory it's a nice feature, but I
really dislike the current implementation.  I am actually a bit
disappointed about the quick hack that was use to implement this
feature: When reading the package archives, package.el appears to just
ignore(!) all package versions which are pinned to another archive.

It'll not just put them in a different list, or track them somewhere
else, no, it will completely forget that these package versions even
existed! I can't say that I like this, I am actually quite
disappointed about this quick and dirty hack to implement pinning.

Essentially, "package-archive-contents" doesn't reflect the actual
archive contents anymore, but rather package archives minus some
random, possibly broken user configuration. Essentially
package-archive-contents is not at all reliable anymore, which opens
up for a wide range of hard to debug errors.

For instance, assume the user accidentally pins a package Foo to a
non-existing archive, e.g. by a spelling mistake or whatever.
Package.el hence discards all versions of this package when reading
the contents, so that "package-archive-contents" does not even contain
Foo.  Not because it does exist, no, but just because of a user error.

If the user later attempts to install a package Bar which requires
Foo, she'll be confronted with a plain and completely "package Foo is
not available" message.   Even if it wanted to, package.el could not
warn the user, simply because package.el itself doesn't know about Foo
anymore.

Ideally, pinning would be evaluated when installing a package.  In
this case, package.el could easily detect the above misconfiguration
by looking at "package-archive-contents", and warn about package Bar
being available, but pinned to another archive.  It could even warn
about packages being pinned to non-existing archives.

Please revise the implementation of this feature, and do *not* drop
pinned packages from "package-archive-contents".

>> These limitations sort of defeat the purpose of having
>> multiple archives imho.
>
>> Ideally, package.el would track *all* packages of *all* repositories
>> in "package-archive-contents" (not just the latest out of all
>> repositories), so that packages, which are contained in multiple
>> archives, also appear multiple times in M-x list-packages, once for
>> each archive.
>
> The current package.el does store all packages in
> package-archive-contents, IIRC.

Indeed, I also just discovered it. With the exception of the bad
pinning implementation, this is really nice.  We'll hopefully make use
of this in Carton soon, probably to implement our own approach to
pinning (unless pinning is improved, see above).

> And M-x list-packages only has one
> entry per package, but once you select one of those entries, it should
> tell you about all the versions it knows.
>
> If you prefer to list all the versions directly in list-packages, we
> could add an option for that (patches welcome).

I don't care for the UI, I am only concerned about the implementation
and API of package.el.  I manage my packages through Carton (, and
Carton has to

>> This more sophisticated way of tracking archives would not only neatly
>> solve the problem of MELPA version numbers (without even touching them),
>
> No, it doesn't really solve the problem: most people wouldn't know when
> MELPA's version is out of date.  How would you know to switch archive if
> package.el doesn't tell you that there's a newer version in the
> "gnu" archive?

How exactly can that happen given that MELPA *always* builds VCS
snapshots, with are *by definition* the most recent state of a
package?

Or, in other words, if a GNU ELPA package happens to be more recent
than MELPA, that's a failure in MELPA, because it's the explicit aim
of MELPA to always provide the most recent snapshots of any included
package.

Re: MELPA version numbers

[Stefan Monnier]

> Indeed, I just discovered "package-pinned-packages", which I presume
> is what you are referring to.  In theory it's a nice feature, but I
> really dislike the current implementation.  I am actually a bit
> disappointed about the quick hack that was use to implement this
> feature: When reading the package archives, package.el appears to just
> ignore(!) all package versions which are pinned to another archive.

Right, I haven't upgraded the code to follow the new style (filter
late, rather than filter early) for package-pinned-packages.
Patch welcome.

> and API of package.el.  I manage my packages through Carton (, and
> Carton has to

Ah, well if Carton has to

>> No, it doesn't really solve the problem: most people wouldn't know when
>> MELPA's version is out of date.  How would you know to switch archive if
>> package.el doesn't tell you that there's a newer version in the
>> "gnu" archive?
> How exactly can that happen given that MELPA *always* builds VCS
> snapshots, with are *by definition* the most recent state of a
> package?

Steve asks similarly:
> Can that even happen, for longer than a MELPA build cycle?

Well, it can happen (and has happened, IIRC I saw it for csv-mode, tho
it seems to be fixed now) because MELPA gives you the latest revision of
the DVCS branch it point at, but that branch may be abandoned.

> Or, in other words, if a GNU ELPA package happens to be more recent
> than MELPA, that's a failure in MELPA,

That's right.


        Stefan

Re: MELPA version numbers

[Steve Purcell]

On 2 Aug 2013, at 16:47, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>>> No, it doesn't really solve the problem: most people wouldn't know when
>>> MELPA's version is out of date.  How would you know to switch archive if
>>> package.el doesn't tell you that there's a newer version in the
>>> "gnu" archive?
>> How exactly can that happen given that MELPA *always* builds VCS
>> snapshots, with are *by definition* the most recent state of a
>> package?
> 
> Steve asks similarly:
>> Can that even happen, for longer than a MELPA build cycle?
> 
> Well, it can happen (and has happened, IIRC I saw it for csv-mode, tho
> it seems to be fixed now) because MELPA gives you the latest revision of
> the DVCS branch it point at, but that branch may be abandoned.


Yes, that would be a fault in MELPA's recipe; we've typically corrected this promptly when it has arisen previously, but I agree that there's no easy way to detect this situation.

-Steve

Re: MELPA version numbers

[Sebastian Wiesner]

2013/8/1 Stefan Monnier <monnier@iro.umontreal.ca>:
> [ As is sadly becoming the norm nowadays, it's difficult to find
>   a contact email address at http://melpa.milkbox.net/, so I send this
>   here instead, hoping someone here will know where to resend it for
>   me.  ]

There is no contact mail address.  You should report an issue to the
public issue tracker at https://github.com/milkypostman/melpa/issues.

I doubt that you'll find the maintainer's support, though, but in
either case there is no point in discussing this issue on this list.
The MELPA maintainers likely won't notice.

> WDYT?

I think, ultimately this is not an MELPA problem, but a shortcoming of
package.el.  One of many others…

Re: MELPA version numbers

[Dmitry Gutov]

Resending.

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> [ As is sadly becoming the norm nowadays, it's difficult to find
>   a contact email address at http://melpa.milkbox.net/, so I send this
>   here instead, hoping someone here will know where to resend it for
>   me.  ]

AFAICT, the discussions mostly happen on GitHub issues. That way,
they're visible to the largest amount of users.

> A few times now I've heard people suffer from problems due to MELPA's
> use of incomparable version numbers: MELPA builds packages straight from
> a DVCS branch tip, so you get the "bleeding edge" and their build script
> ignores the package's own version number and instead just slaps
> a version number based on the current time, such as 20100105.123.
>
> This system makes sense, but if the package is also available (in it's
> "latest released version") via GNU ELPA or Marmalade, we have a problem
> because the two version numbers can't be compared correctly, and
> package.el doesn't even know it, so it just always picks MELPA's version
> (since 20100105.123 is "clearly" much more recent than say 2.3).
>
> This mess works surprisingly well in practice, since MELPA's versions
> are often at least as recent as the one on GNU ELPA or Maramalde (by virtue
> of being the bleeding edge from the DVCS).
>
> But if you ever stop using MELPA, all your MELPA-installed packages will
> stay non-updated for the foreseeable future since it'll take a while for
> foo.el to go from version 2.3 to version 20100106.0.
> And if for some reason the MELPA recipe points to an old DVCS
> unmaintained branch, you're similarly out of luck.
>
> One way out of this is to change the MELPA version numbers so that
> instead of ignoring the package's "2.3" and replacing it with
> "20100105.123", it should replace it with "2.3.20100105.123".

I like this approach in abstract, but if Melpa just switches to it, all
existing users will have to know somehow to reinstall all installed
packages manually. Otherwise, they won't ever update, for the same
reason as why you're bringing up this issue.

> WDYT?

Re: MELPA version numbers

[Steve Purcell]

On 1 Aug 2013, at 21:57, Dmitry Gutov <dgutov@yandex.ru> wrote:
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
>> One way out of this is to change the MELPA version numbers so that
>> instead of ignoring the package's "2.3" and replacing it with
>> "20100105.123", it should replace it with "2.3.20100105.123".
> 
> I like this approach in abstract, but if Melpa just switches to it, all
> existing users will have to know somehow to reinstall all installed
> packages manually. Otherwise, they won't ever update, for the same
> reason as why you're bringing up this issue.


I'm personally open to switching MELPA over to a different numbering scheme if practicable.

I'm concerned, though, that the suggested scheme has significant potential for problems.

Firstly, when there *is* an author-assigned version, it must be visible in the checked-in code for MELPA to see, and not filled in by a release script. This is conspicuously not the case for a large number of libraries which we currently package, and it's obviously no good for us to silently default to 0.0 in those cases, because that would lead to uninstallable (low-versioned) MELPA packages.

A related loophole is that non-authors can take someone's v0.1 library, hack it, and then upload the resulting package to Marmalade as v0.2. (Yes, this happens!) MELPA would then continue to build uninstallable point releases of v0.1 from the author's SCM repo until the Marmalade issue is resolved.

-Steve

Re: MELPA version numbers

[Stefan Monnier]

> Firstly, when there *is* an author-assigned version, it must be
> visible in the checked-in code for MELPA to see, and not filled in by
> a release script.

Right.  The way the GNU ELPA archive is built, we just require this info.
But I guess for MELPA you'd like to be able to get by without any
cooperation from the author.  Good point.

> A related loophole is that non-authors can take someone's v0.1 library, hack
> it, and then upload the resulting package to Marmalade as v0.2. (Yes, this
> happens!) MELPA would then continue to build uninstallable point releases of
> v0.1 from the author's SCM repo until the Marmalade issue is resolved.

This is the problem of forking, which is inherently problematic for
version numbers.

It looks like all those problems could be solved with the addition of
a "release date" info.  This way, package.el can use release-date and
version-number ordering disagreement as a hint that the automation might
not do the right thing, and warn the user about it, letting him figure
out what should happen.


        Stefan