bug#19296: [PATCH] Package archives now have priorities.

bug#19296: [PATCH] Package archives now have priorities.

[Drew Adams]

Here we seem to have a bug report that "fixes" a problem that
is not even clearly described.  Instead of a proper problem
description, we get, directly, a "solution".

Please prove the problem first.  A one-liner stating that MELPA
assigns date version numbers and you need to "fix" that is not
a problem description.

> This solves the "MELPA problem", where MELPA assigns date-based
> version numbers to packages which override all other archives.
> Giving MELPA a lower priority means packages are installed from
> MELPA only when the package is not available from other archives.

Why is that good?  Why should MELPA be given a lower priority?

> This can be overridden manually by the user.

How?

Why make users jump through extra hoops to retrieve packages
from MELPA in priority?

This sounds like a bad hack.

MELPA is, so far, the most complete repository, with the most
packages, refreshed dynamically and presenting the most recent
versions of packages.

Many users put MELPA first in their list of repositories to
retrieve from.  It doesn't sound like their use case is being
considered seriously.

Caveat: I'm going only by your description above; I have not
looked into the code.  It just smells bad, from that description.

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On Sun, 7 Dec 2014 06:19:29 -0800 (PST)
Drew Adams <drew.adams@oracle.com> wrote:

> Why is that good?  Why should MELPA be given a lower priority?

MELPA provides unstable versions of packages. To provide stable
versions of packages, there is the "MELPA Stable" repository (among
others, including GNU ELPA). As not all packages in MELPA unstable
are available in MELPA stable, users have to add both to their
archives list to get access to all packages. But due to the way MELPA
assigns version numbers, the unstable versions will always override
stable versions, even when both are available.

This patch will allow a setup that takes the newest version of a
package from GNU ELPA, MELPA stable or Marmalade, and only if there is
no version available from any of these repositories, take the one
available from MELPA unstable. No jumping through hoops required.

The current solution to this is to add all packages available from
non-MELPA unstable to `package-pinned-packages'.

The facility of priorities for repositories is widely available in
other package managers, e.g. in Debian's apt (see apt_preferences(5)).

You can read more about the problems with MELPA's versioning system
here:

http://blog.jorgenschaefer.de/2014/06/the-sorry-state-of-emacs-lisp-package.html

I hope I could answer your questions.

Regards,
Jorgen

bug#19296: [PATCH] Package archives now have priorities.

[Drew Adams]

> > Why is that good?  Why should MELPA be given a lower priority?
> 
> MELPA provides unstable versions of packages.

Baloney.  Please stop pushing this myth.

MELPA provides *packages*.  Whether a package is "stable" or
not, as far as one can tell from the outside, is only something
(optionally) claimed by its developer.

> To provide stable versions of packages, there is the "MELPA Stable"

No.  There is nothing more stable about the packages in "MELPA
Stable" (according to the creator of MELPA and MELPA "Stable").

Again, this "stability" is merely a designation by the package
maintainer, for *his* own convenience.  If a package maintainer
wants to distinguish two versions of a package, and call one
of them "stable", s?he can choose to put the latter into "MELPA
Stable".

That's the stated purpose of "MELPA Stable".  It was added
because some package maintainers wanted two, distinguishable
versions that users could choose from.

That's all - the name means nothing more than that.  It might
well be that the version put into MELPA Stable is less stable
than the one put into MELPA.

> repository (among others, including GNU ELPA). As not all
> packages in MELPA unstable

There is no "MELPA unstable".  That is a fiction.  Even the
creator of MELPA and "MELPA Stable" has said so, and that
adding MELPA Stable was maybe not a great idea (my paraphrase).
According to him, "MELPA Stable" is in maintenance mode.

> are available in MELPA stable, users have to add both to their
> archives list to get access to all packages. But due to the way
> MELPA assigns version numbers, the unstable versions will always
> override stable versions, even when both are available.

Take it up with the MELPA maintainers if you have a complaint
about the design of MELPA and its version numbers.

Don't try to lock out MELPA for all Emacs users, just because
you have a problem with MELPA.  Don't make the many MELPA users
jump through hoops to let package.el treat MELPA like any other
repository.
 
> This patch will allow a setup that takes the newest version of a
> package from GNU ELPA, MELPA stable or Marmalade, and only if
                                                        ^^^^^^^
> there is no version available from any of these repositories,
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> take the one available from MELPA unstable.

IOW, you want to favor all other repositories over MELPA.

That logic is flawed.  This certainly should not be the
hard-coded/default Emacs behavior.  "Only if there is no version
available from any of these repositories, take the one available
from MELPA unstable."  Sheesh.

Why should MELPA (there is no "MELPA unstable") be usable
ONLY if no version of the package is available from any
other repositories?

> No jumping through hoops required.

It's a joke, right?  You can get off the bus at your bus stop
only if there are no other people on the bus.  Prerequisite:
get everyone else off the bus first.  IOW, you are last.

Oh - UNLESS you bring a note from your mother saying
"Please allow my boy Jorgen to get off the bus without
waiting until he is the only passenger left."

> The current solution to this is to add all packages available
> from non-MELPA unstable to `package-pinned-packages'.

"Please allow my boy Jorgen..."  Quite a "solution".

Be fair.  If you want to require notes from mothers, then do
that for *every* package repository.  Make it so that to get
a package from *any* repository you need to pin the package
to that repository.

Then you will see what this amounts to.  Then you will have
something that is fair.  And equally cumbersome for all.

There is no reason to discriminate against MELPA, treating it
differently.

> The facility of priorities for repositories is widely available
> in other package managers, e.g. in Debian's apt (see
> apt_preferences(5)).

I couldn't care less.  Do they single out one repository like
you have singled out MELPA, hard-coding things so that to use
that repository a given package must *not be available anywhere
else*?

  "only if there is no version available from any" other
  repository, "take the one available from" THE-BAD-BOY.

Repository priorities can be expressed in a normal way,
using a list or assigning priority values - with *no built-in
prejudice* for or against any particular repository.  There
is no reason to blackball MELPA this way.  Treat repositories
the same a priori.  Let only *users* prioritize them.

> You can read more about the problems with MELPA's versioning system
> here: http://blog.jorgenschaefer.de/2014/06/
> the-sorry-state-of-emacs-lisp-package.html

You quote yourself...  Wunderbar.

You have a problem with MELPA.  You should take up your problem
with the MELPA designers & maintainers.  Please do not impose
this stuff on Emacs, trying to blackball MELPA.  A sorry state,
indeed.  MELPA is the most widely used and the most useful
repository we have, by far.  If you don't want to use it then
don't use it.  Circulez ; il n'y a rien a voir.

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On Sun, 7 Dec 2014 07:55:20 -0800 (PST)
Drew Adams <drew.adams@oracle.com> wrote:

> > > Why is that good?  Why should MELPA be given a lower priority?
> > 
> > MELPA provides unstable versions of packages.
> 
> Baloney.  Please stop pushing this myth.
>
> MELPA provides *packages*.  Whether a package is "stable" or
> not, as far as one can tell from the outside, is only something
> (optionally) claimed by its developer.

MELPA actively discourages package authors to have a "stable" branch in
the recipe:

https://github.com/milkypostman/melpa/pull/1129#issuecomment-27156539

MELPA strongly prefers "snapshot" releases, and suggests that users use
the "MELPA stable" package archive if they do want officially released
versions.

When I raised this issue in the past, I was told that the "E" in
"MELPA" originally stood for "Experimental":

https://github.com/milkypostman/melpa/pull/1129#issuecomment-27156209

> Repository priorities can be expressed in a normal way,
> using a list or assigning priority values - with *no built-in
> prejudice*

This patch does not add any built-in prejudice. As the patch
description says, this *allows* the user to set it up like that. It
does not force anyone to do anything, and does not change default
behavior at all.


Could I ask you to be a bit more courteous and civil to possible
contributors in the future? Thank you.

If you require further responses, please do specify in what official
capacity you are speaking. I seem unable to find you on the list of
maintainers for Emacs.

Regards,
Jorgen

bug#19296: [PATCH] Package archives now have priorities.

[Drew Adams]

> > MELPA provides *packages*.  Whether a package is "stable" or
> > not, as far as one can tell from the outside, is only something
> > (optionally) claimed by its developer.
> 
> MELPA actively discourages package authors to have a "stable"
> branch in the recipe:
> https://github.com/milkypostman/melpa/pull/1129#issuecomment-
> 27156539

That URL points to Donald Curtis saying just what I've been
saying here, AFAICT.  MELPA packages are packages, nothing more.
And saying this:

 "part of the reason MELPA is so popular is because it's
  automatic-doesn't rely on the developer to make releases-
  and it provides the newest features".

There is nothing there about MELPA encouraging unstable
packages or discouraging stable packages.

You don't like MELPA.  That much is clear.  That does not an
Emacs bug make.  You say (at that URL):

 "MELPA is becoming the standard package archive outside of
  GNU, and a lot of places recommend using MELPA."

That seems to be what bothers you, AFAICT.  Get over it.
Just don't use it, if you can't get along with it.

> MELPA strongly prefers "snapshot" releases, 

It allows any kind of "release" you want.  And yes, in
particular, it *allows* automatic updating.  This is a
*feature*.  Don't use it if you don't want it.

A package maintainer who uses it wants it.

Nothing requires a maintainer to modify the package source
that is pulled from, so that an automatic "update" to the
repository mirror of it actually changes something.  A
"stable", "release", "official" package can sit in its source
location unchanged for as long as it wants.  When it is
mirrored to MELPA it will remain just as "stable", "release",
and "official".

> and suggests that users use the "MELPA stable" package archive
> if they do want officially released versions.

There is nothing in the URL you cite about "officially
released versions".  How a package maintainer wants to
characterize a given occurrence of a package in a repository
is that maintainer's business.

MELPA does not recognize any "officially released versions",
and it should not.  Nor should package.el.  Who or what is
"official" here?

Releases, "official" or not; snapshots; and whatever else you
like are labels that *package maintainers* can give to their
packages.  If used, they are meaningful only to the package
itself; they are not recognized by the package manager.

They are orthogonal to any handling by ELPA repositories
and package.el (or at least they should be).

They have, and should have, nothing to do with a package
repository.  package.el should not get involved with any
particular labeling of a package by a package maintainer
wrt its use category, whether that be "beta", "snapshot",
"foobar", "official", or anything else.  Such labels do
not exist for the package manager.

Now let's see how you talk about it (same URL):

 "If MELPA wants to enable users to use the bleeding edge
  at the danger of them having unstable code, then MELPA
  needs to advertise this more."

"bleeding edge", "danger", "unstable"...  Sounds scary!

 "As is, MELPA is advertised as the one big useful lisp
  archive with no shortcomings, and that is how many
  users copy and paste the MELPA address to their
  configuration files."

And:

 "[users] are easily tricked into using MELPA" because
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  it "is establishing itself as the de facto standard
  repository for Emacs Lisp packages."

Uh, MELPA isn't really advertised, and certainly not
like that.  It sounds like you have a problem with the
fact that many users *do* think of MELPA as "the one
big useful lisp archive" and they *do* "copy and paste
the MELPA address to their configuration files."

Are they wrong to do so?  Do you want to tell them what
they should think and do?  Is that what this Emacs "bug"
is all about?  MELPA is dangerous! bleeding edge! unstable
and scary stuff - stay away from it!

What do you care what many or most users do with MELPA
or what they think of it?  If you don't want to use
MELPA then don't use it.  "Problem" solved.

> When I raised this issue in the past, I was told that the
> "E" in "MELPA" originally stood for "Experimental":

Who cares?

> https://github.com/milkypostman/melpa/pull/1129#issuecomment-
> 27156209

Clearly you have gripes with MELPA.  My advice is for you to
simply not use it.  Leave it for the many other Emacs users
who appreciate it.  It is trivial for you not to use it.
We'll all be happier.  Not a bug.

> > Repository priorities can be expressed in a normal way,
> > using a list or assigning priority values - with *no
> > built-in prejudice*
> 
> This patch does not add any built-in prejudice. As the patch
> description says, this *allows* the user to set it up like that.
> It does not force anyone to do anything, and does not change
> default behavior at all.

Given your attitude toward MELPA, I'm not convinced.  But I
suppose we shall see.

> Could I ask you to be a bit more courteous and civil to
> possible contributors in the future? Thank you.

Don't be so presumptuous.

You showed the same attitude in the URL you cited:

 "So you are saying that you simply refuse to provide the
  version of my package, which I politely asked you to
  provide?"

To which the reply was:

 "That's not what I'm saying. I hoped that we could have a 
  conversation in order to understand the trade-offs and
  jointly look for a solution which suits everyone.
  If that's not the case, then your polite request is
  technically a demand."

You are so polite, and everyone else is so uncooperative.
Funny about that.

> If you require further responses, please do specify in
> what official capacity you are speaking. I seem unable to
> find you on the list of maintainers for Emacs.

It appears that you are quite impressed by whatever you
think is "official", Herr Schaefer.

I am an Emacs *user*.  Fortunately, we are still able to
speak our minds.  When you are elected Pope, maybe things
will change in that regard.

Apparently this has all come about because this happened:

 "I did not supply an elpy recipe to MELPA (quite
  intentionally so because of this problem). Someone else
  has. Suddenly, I have users coming to me with weird bugs
  related to them unknowingly using a development version
  of elpy."

That's not MELPA's fault.  Tell your users not to do that.
Tell your users that any ELPY taken from MELPA will not
be supported by you.  Tell them to beware the evil MELPA.

You say that you:

 "have a problem with getting feedback about a snapshot by
  users not knowing they are using a snapshot, never having
  wanted to use a snapshot, and being ill-equipped to even
  debug it because they do not know Emacs Lisp at all (and
  why should they, they just use Emacs). They were
  basically tricked into using a source of software that not
            ^^^^^^^
  only accidentally exposes them to possibly broken software,
  but apparently even refuses to take steps to protect them
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  from that."

To which the polite reply was:

 "Man, you just seem mad for no reason. We're just trying to
  offer solutions that would work for all of us, but I guess
  we will have to do it your way."

Tell your users to stay away from dangerous MELPA.
Just say no to MELPA if you don't want to use it.

bug#19296: [PATCH] Package archives now have priorities.

[Ted Zlatanov]

On Sun, 7 Dec 2014 07:55:20 -0800 (PST) Drew Adams <drew.adams@oracle.com> wrote: 

DA> Take it up with the MELPA maintainers if you have a complaint
DA> about the design of MELPA and its version numbers.

DA> Don't try to lock out MELPA for all Emacs users, just because
DA> you have a problem with MELPA.  Don't make the many MELPA users
DA> jump through hoops to let package.el treat MELPA like any other
DA> repository.

What are you talking about?!!??!  Have you even looked at the code?!
MELPA is only mentioned in the commentary, there's nothing in the code
that uses it or refers to it.

The amount of effort and time you spent on that reply... wow.

DA> That logic is flawed.  This certainly should not be the
DA> hard-coded/default Emacs behavior.  "Only if there is no version
DA> available from any of these repositories, take the one available
DA> from MELPA unstable."  Sheesh.

It's not the default, it's a user option.

DA> Circulez ; il n'y a rien a voir.

French for "I just want to argue."

Ted

bug#19296: [PATCH] Package archives now have priorities.

[Stefan Monnier]

> Here we seem to have a bug report that "fixes" a problem that
> is not even clearly described.

It's a well known problem among the people who worked on package.el
(AFAIK).


        Stefan

bug#19296: [PATCH] Package archives now have priorities.

[Drew Adams]

> This solves the "MELPA problem", where MELPA assigns date-based
> version numbers to packages which override all other archives.
> Giving MELPA a lower priority means packages are installed from
> MELPA only when the package is not available from other archives.

IOW, it creates a new MELPA problem.

> This can be overridden manually by the user.

IOW, MELPA users will be made to jump through additional hoops.

And why are we choosing to do this to the greatest number of
users and for the greatest number of packages?  (MELPA has
more users, more packages, and more downloads.)

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

When installing packages by name, only packages from archives with
the highest priority are considered, before versions are compared.

This solves the "MELPA problem", where MELPA assigns date-based
version numbers to packages which override all other archives.
Giving MELPA a lower priority means packages are installed from
MELPA only when the package is not available from other archives.

This can be overridden manually by the user.
---
 lisp/emacs-lisp/package.el     |   45 ++++++++++++++++++++++++++++++++++++++--
 test/automated/package-test.el |   17 +++++++++++++++
 2 files changed, 60 insertions(+), 2 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 4e5c397..2030482 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -228,6 +228,33 @@ a package can run arbitrary code."
   :group 'package
   :version "24.1")
 
+(defcustom package-archive-default-priority 500
+  "The default priority for archives.
+
+This is used if the archive is not found in
+`package-archive-priorities'."
+  :type 'integer
+  :risky t
+  :group 'package
+  :version "25.1")
+
+(defcustom package-archive-priorities nil
+  "An alist of priorities for packages.
+
+Each element has the form (ARCHIVE-ID . PRIORITY).
+
+When installing packages, the package with the highest version
+number from the archive with the highest priority is
+selected. When higher versions are available from archives with
+lower priorities, the user has to select those manually.
+
+Archives not in this list have the priority given in
+`package-archive-default-priority'."
+  :type 'integer
+  :risky t
+  :group 'package
+  :version "25.1")
+
 (defcustom package-pinned-packages nil
   "An alist of packages that are pinned to specific archives.
 This can be useful if you have multiple package archives enabled,
@@ -1063,6 +1090,7 @@ Also, add the originating archive to the `package-desc' structure."
                         ;; Older archive-contents files have only 4
                         ;; elements here.
                         (package--ac-desc-extras (cdr package)))))
+         (archive-priority (package-archive-priority archive))
          (existing-packages (assq name package-archive-contents))
          (pinned-to-archive (assoc name package-pinned-packages)))
     (cond
@@ -1075,8 +1103,12 @@ Also, add the originating archive to the `package-desc' structure."
      (t
       (while
           (if (and (cdr existing-packages)
-                   (version-list-<
-                    version (package-desc-version (cadr existing-packages))))
+                   (or (< archive-priority
+                          (package-archive-priority
+                           (package-desc-archive (cadr existing-packages))))
+                       (version-list-<
+                        version
+                        (package-desc-version (cadr existing-packages)))))
               (setq existing-packages (cdr existing-packages))
             (push pkg-desc (cdr existing-packages))
             nil))))))
@@ -1268,6 +1300,15 @@ The file can either be a tar file or an Emacs Lisp file."
   "Return the archive containing the package NAME."
   (cdr (assoc (package-desc-archive desc) package-archives)))
 
+(defun package-archive-priority (archive)
+  "Return the priority of ARCHIVE.
+
+The archive priorities are specified in
+`package-archive-priorities' and
+`package-archive-default-priority'."
+  (or (cdr (assoc archive package-archive-priorities))
+      package-archive-default-priority))
+
 (defun package--download-one-archive (archive file)
   "Retrieve an archive file FILE from ARCHIVE, and cache it.
 ARCHIVE should be a cons cell of the form (NAME . LOCATION),
diff --git a/test/automated/package-test.el b/test/automated/package-test.el
index 6e7994a..2a337fb 100644
--- a/test/automated/package-test.el
+++ b/test/automated/package-test.el
@@ -230,6 +230,23 @@ Must called from within a `tar-mode' buffer."
     (package-refresh-contents)
     (package-install 'simple-single)))
 
+(ert-deftest package-test-install-prioritized ()
+  "Install a lower version from a higher-prioritized archive."
+  (with-package-test ()
+    (let* ((newer-version (expand-file-name "data/package/newer-versions"
+                                            package-test-file-dir))
+           (package-archives `(("older" . ,package-test-data-dir)
+                               ("newer" . ,newer-version)))
+           (package-archive-priorities '(("newer" . 100))))
+
+      (package-initialize)
+      (package-refresh-contents)
+      (package-install 'simple-single)
+
+      (let ((installed (cdr (assq 'simple-single package-alist))))
+        (should (version-list-= '(1 3)
+                                (package-desc-version installed)))))))
+
 (ert-deftest package-test-install-multifile ()
   "Check properties of the installed multi-file package."
   (with-package-test (:basedir "data/package" :install '(multi-file))
-- 
1.7.10.4

bug#19296: [PATCH] Package archives now have priorities.

[Ted Zlatanov]

On Sun, 7 Dec 2014 13:31:50 +0100 Jorgen Schaefer <forcer@forcix.cx> wrote: 

JS> When installing packages by name, only packages from archives with
JS> the highest priority are considered, before versions are compared.

JS> This solves the "MELPA problem", where MELPA assigns date-based
JS> version numbers to packages which override all other archives.
JS> Giving MELPA a lower priority means packages are installed from
JS> MELPA only when the package is not available from other archives.

I like it.

Ted

bug#19296: [PATCH] Package archives now have priorities.

[Stefan Monnier]

> When installing packages by name, only packages from archives with
> the highest priority are considered, before versions are compared.

What can this be used for (other than the MELPA case, obviously)?

> This solves the "MELPA problem", where MELPA assigns date-based
> version numbers to packages which override all other archives.
> Giving MELPA a lower priority means packages are installed from
> MELPA only when the package is not available from other archives.

I think the better way to solve the problem of versioning the "bleeding
edge package" would be to take the base version and tuck the date to it
(instead of only using the date).
I.e. file names like foo-mode-1.3.0.20141023.tar.gz where "1.3" is the
version of the last release.

Of course that requires a change on MELPA side and I have no idea how
easy/feasible that would be.  And I'm not completely sure it would
really be the best option either.

But maybe an idea along the same lines would be to treat revision
numbers of the form YYYYMMDD (or similar) as being a "bleeding edge"
release and to translate them to, "0.0.YYYYMMDD".
This would have a similar effect to setting MELPA's priority lower.

> This can be overridden manually by the user.

An important issue is what happens after the user did such an override.
In my above suggestion, the behavior would kind of suck since
package-list would then constantly recommend "upgrading" to the
official release (since 1.3 is "more uptodate" than "0.0.YYYYMMDD").


        Stefan

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On Sun, 07 Dec 2014 12:56:53 -0500
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > When installing packages by name, only packages from archives with
> > the highest priority are considered, before versions are compared.
> 
> What can this be used for (other than the MELPA case, obviously)?

Local/site-wide package archives that provide specific versions for
packages that are required by the site that should not be overridden by
external sources. (Can be emulated by pinning them.)

Adding archives that provide testing or debugging packages which should
not be installed by default, but can be installed by the user if they
want to. (Can be emulated by adding the archive only for the duration
of installing those packages.)

More generally, the use cases are either very similar to the existing
"pinning" functionality, or to adding archives only temporarily, except
the interface is easier to the user and does not require constant
manual work.

> > This solves the "MELPA problem", where MELPA assigns date-based
> > version numbers to packages which override all other archives.
> > Giving MELPA a lower priority means packages are installed from
> > MELPA only when the package is not available from other archives.
> 
> I think the better way to solve the problem of versioning the
> "bleeding edge package" would be to take the base version and tuck
> the date to it (instead of only using the date).
> I.e. file names like foo-mode-1.3.0.20141023.tar.gz where "1.3" is the
> version of the last release.
> 
> Of course that requires a change on MELPA side and I have no idea how
> easy/feasible that would be.  And I'm not completely sure it would
> really be the best option either.

This is not easy for the general MELPA, as not all packages have
version numbers at all, but certainly feasible for MELPA stable.

My patch that would have added this was rejected:

https://github.com/milkypostman/melpa/pull/1771

The version number issues has been raised a few times, and it does not
seem likely to get fixed any time soon.

> > This can be overridden manually by the user.
> 
> An important issue is what happens after the user did such an
> override. In my above suggestion, the behavior would kind of suck
> since package-list would then constantly recommend "upgrading" to the
> official release (since 1.3 is "more uptodate" than "0.0.YYYYMMDD").

Good point. The correct implementation here would likely move the
sorting by version number out of the `package--add-to-archive-contents'
function and into the various users of `package-archive-contents',
which should sort the list depending on their use case. This is a
breaking API change and likely a good deal more work.

Would a patch that does that be more acceptable?

Regards,
Jorgen

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On Sun, 7 Dec 2014 19:21:05 +0100
Jorgen Schaefer <forcer@forcix.cx> wrote:

> On Sun, 07 Dec 2014 12:56:53 -0500
> Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
> > > This can be overridden manually by the user.
> > 
> > An important issue is what happens after the user did such an
> > override. In my above suggestion, the behavior would kind of suck
> > since package-list would then constantly recommend "upgrading" to
> > the official release (since 1.3 is "more uptodate" than
> > "0.0.YYYYMMDD").
> 
> Good point. The correct implementation here would likely move the
> sorting by version number out of the
> `package--add-to-archive-contents' function and into the various
> users of `package-archive-contents', which should sort the list
> depending on their use case. This is a breaking API change and likely
> a good deal more work.

Actually, it should not suggest an upgrade in this case, because the
currently installed version is higher than the highest available one
(package-menu--find-upgrades).

Currently, that method ignores priorities, though, as it uses an
entirely different way of looking up the available packages. I'll
provide a fixed patch.

Regards,
Jorgen

bug#19296: [PATCH] Package archives now have priorities.

[Stefan Monnier]

> Actually, it should not suggest an upgrade in this case, because the
> currently installed version is higher than the highest available one
> (package-menu--find-upgrades).

So we're back with the same problem as we currently have:
if you've installed MELPA's 20120425 and the package gets into GNU ELPA
with version 1.4, you'll keep using the older version.


        Stefan

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On Sun, 07 Dec 2014 21:48:48 -0500
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > Actually, it should not suggest an upgrade in this case, because the
> > currently installed version is higher than the highest available one
> > (package-menu--find-upgrades).
> 
> So we're back with the same problem as we currently have:
> if you've installed MELPA's 20120425 and the package gets into GNU
> ELPA with version 1.4, you'll keep using the older version.

What kind of behavior do we want, then? :-)

The problem I am trying to solve is:


When I install a package by name using M-x package-install foo, I want
to install the highest version of "foo" from any repository but MELPA
if it is available, and if not, the highest version of "foo" from MELPA.

When I installed a package from another repository than MELPA, and I
ask package.el to upgrade all packages, I do not want to get an upgrade
suggestion to MELPA.


The problem you now mention is: What should happen if I installed a
package from MELPA and it is now available from another repository?

I don't think there is a good solution for that. We could come up with
an idea such as "if, at the time this repository was chosen, the
package was not available from another repository, but now it is, it
should ...", but that seems rather arcane and requires us to store a
lot more information than we currently do.

Right now, the upgrades should simply stick to MELPA. Once MELPA,
always MELPA. I think this is, while maybe not the best possible
option, an acceptable one.

Considering the two problems I described above are solved, and the
default behavior is not affected at all, I think this patch is a strict
improvement over the current situation.

Regards,
Jorgen

bug#19296: [PATCH] Package archives now have priorities.

[Stefan Monnier]

> What kind of behavior do we want, then? :-)

That's the first question, indeed.

> When I install a package by name using M-x package-install foo, I want
> to install the highest version of "foo" from any repository but MELPA
> if it is available, and if not, the highest version of "foo" from MELPA.

OK.

> When I installed a package from another repository than MELPA, and I
> ask package.el to upgrade all packages, I do not want to get an upgrade
> suggestion to MELPA.

OK.

> The problem you now mention is: What should happen if I installed a
> package from MELPA and it is now available from another repository?

Right.  Or, to more generally, as suggested by Achim:
- I've installed version V2 from repository R2
- there's a version V1 from repository R1
- V2 > V1
- yet V1 is more recent than V2 (because R1 and R2 use different
  versioning schemes).

> I don't think there is a good solution for that.

But that is the problem that causes most harm since people get stuck
with an old version.

> We could come up with an idea such as "if, at the time this repository
> was chosen, the package was not available from another repository, but
> now it is, it should ...", but that seems rather arcane and requires
> us to store a lot more information than we currently do.

Last time we discussed it, another suggestion was to supplement the
version info with a date info.  That doesn't in itself solve the
problem, but it does provide enough extra info that package.el could try
to be more clever.

BTW, there's yet another interesting situation to consider (which we've
had once in GNU ELPA for AucTeX):
- V2 is in R2, user installs it.
- Some problem is found in V2
- R2 reverts to V1
- User is never told that reverting to V1 is the recommended course of action

Now that I think about it, maybe a better solution (which will also
handle this last case) is to compare the old archive-contents for each
archive and use that diff as a basis to discover what is new (instead of
relying on version numbers).

This last approach suffers from the problem that this diff is naturally
transient, so we'd have to accumulate those diffs that can be relevant
to the user and stash them in some file until the user has properly been
told about it (and she should be allowed to do "sorry, can't deal with it
right now, please remind me later").


        Stefan

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On Mon, 08 Dec 2014 10:42:49 -0500
Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> > I don't think there is a good solution for that.
> 
> But that is the problem that causes most harm since people get stuck
> with an old version.

I don't think this is the problem that causes most harm. For people to
be stuck with an old version, the package needs to be removed from
MELPA, else the version from MELPA will always be "the newest".

The problem that I am trying to solve is that it is currently
meaningless to have e.g. both MELPA and MELPA Stable in the archive
list, because all packages in MELPA Stable are also available from
MELPA, so if you have both in your archive list, you will always get
the MELPA ones, never the MELPA Stable ones. But you do want MELPA in
the archive list, because MELPA Stable only has about a third of the
packages of MELPA.

> BTW, there's yet another interesting situation to consider (which
> we've had once in GNU ELPA for AucTeX):
> - V2 is in R2, user installs it.
> - Some problem is found in V2
> - R2 reverts to V1
> - User is never told that reverting to V1 is the recommended course
> of action

In other package archives, this is a problem of the person doing the
releases, not the archive. They should do another release V3 that is
equivalent to V1, not simply "revoke" an existing release.

> Now that I think about it, maybe a better solution

Ok, there are a number of possible approaches, all hypothetical. Let's
be more concrete.

What requirements would a patch need to fulfill that you deem it
acceptable to be applied to the Emacs repository that solves my
original problem (i.e. only install packages from a given repository if
the package is not available from other repositories)?

Regards,
Jorgen

bug#19296: [PATCH] Package archives now have priorities.

[Stefan Monnier]

>> > I don't think there is a good solution for that.
>> But that is the problem that causes most harm since people get stuck
>> with an old version.
> I don't think this is the problem that causes most harm.  For people to
> be stuck with an old version, the package needs to be removed from
> MELPA, else the version from MELPA will always be "the newest".

Experience shows that MELPA can fall behind also if the development
moves elsewhere (e.g. in GNU ELPA) and its recipe is not updated.

> The problem that I am trying to solve is that it is currently
> meaningless to have e.g. both MELPA and MELPA Stable in the archive
> list,

I see, that makes sense.

> What requirements would a patch need to fulfill that you deem it
> acceptable to be applied to the Emacs repository that solves my
> original problem (i.e. only install packages from a given repository if
> the package is not available from other repositories)?

I'd like a solution that also addresses, at least partly, the other
problem (the one where the MELPA version is superseded by a version
elsewhere such as in GNU ELPA).

It's not important to automatically upgrade the package from the MELPA
version to the GNU ELPA version, but the user should somehow be warned
at some point that the MELPA version is not the latest any more.

I think in general it would be desirable to try and remember where
a package came from so that upgrading to a version in another repository
doesn't happen automatically.


        Stefan

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On 12/15/2014 05:59 AM, Stefan Monnier wrote:

>> What requirements would a patch need to fulfill that you deem it
>> acceptable to be applied to the Emacs repository that solves my
>> original problem (i.e. only install packages from a given repository if
>> the package is not available from other repositories)?
>
> I'd like a solution that also addresses, at least partly, the other
> problem (the one where the MELPA version is superseded by a version
> elsewhere such as in GNU ELPA).
 >
> It's not important to automatically upgrade the package from the MELPA
> version to the GNU ELPA version, but the user should somehow be warned
> at some point that the MELPA version is not the latest any more.

So a warning "this package was installed from archive X, but is not 
available from there anymore" would suffice to fulfill this requirement, 
correct? Would it be enough if the package list displayed a string like 
"changed archive" or similar? (This needs a shorter name.)

> I think in general it would be desirable to try and remember where
> a package came from so that upgrading to a version in another repository
> doesn't happen automatically.

I can see two ways for this. One, we overwrite the package's -pkg.el 
file on install time with an expanded version that includes our own 
infos, e.g. :archive "name" for this 
(`package-generate-description-file' already has the code to generate 
this file, and it just needs to be called even if the file already 
exists). Alternatively, we add an .archive file to the package directory 
recording the archive this came from.

I would tend to lean to overriding the -pkg.el file, but I do not have 
any big preferences. Which way would you prefer?

In either case, we should populate the pkg-desc in `package-alist' from 
this, which should make it easy to implement the display above.

If I add this functionality, will that make the reminder of the patch 
acceptable to be included in Emacs?

Regards,
Jorgen

bug#19296: [PATCH] Package archives now have priorities.

[Ted Zlatanov]

On Mon, 15 Dec 2014 09:36:10 +0100 Jorgen Schaefer <forcer@forcix.cx> wrote: 

JS> I can see two ways for this. One, we overwrite the package's -pkg.el
JS> file on install time with an expanded version that includes our own
JS> infos, e.g. :archive "name" for this
JS> (`package-generate-description-file' already has the code to generate
JS> this file, and it just needs to be called even if the file already
JS> exists). Alternatively, we add an .archive file to the package
JS> directory recording the archive this came from.

JS> I would tend to lean to overriding the -pkg.el file, but I do not have
JS> any big preferences. Which way would you prefer?

Keep in mind the -pkg.el file can be signed by the archive maintainers, IIRC.

Ted

bug#19296: [PATCH] Package archives now have priorities.

[Stefan Monnier]

> So a warning "this package was installed from archive X, but is not
> available from there anymore" would suffice to fulfill this requirement,
> correct?

IIUC this is a very rare case.  The more common and important case is
when the package is available from several archives at the same time.

> Would it be enough if the package list displayed a string like
> "changed archive" or similar? (This needs a shorter name.)

From the UI point of view, I don't know how best to display "new version
available from other archive".  Maybe it could be a message or prompt
that shows up when you "Mark Upgradable Packages"?

> I would tend to lean to overriding the -pkg.el file, but I do not have any
> big preferences.  Which way would you prefer?

Either way seems OK.

> If I add this functionality, will that make the reminder of the patch
> acceptable to be included in Emacs?

Not sure what "remainder" you're referring to.


        Stefan

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On 12/15/2014 03:52 PM, Stefan Monnier wrote:

>> If I add this functionality, will that make the reminder of the patch
>> acceptable to be included in Emacs?
>
> Not sure what "remainder" you're referring to.

The part where M-x package-install RET foo RET will install foo-1.2.3 
over foo-20141205.315 when the latter is only available from a 
repository with a lower priority, and where "U" in the package list will 
not upgrade foo-1.2.3 to foo-20141205.315 when the latter is only 
available from a repository with a lower priority.

That is, the whole point of the original patch.

Regards,
Jorgen

bug#19296: [PATCH] Package archives now have priorities.

[Stefan Monnier]

> The part where M-x package-install RET foo RET will install foo-1.2.3 over
> foo-20141205.315 when the latter is only available from a repository
> with a lower priority, and where "U" in the package list will not upgrade
> foo-1.2.3 to foo-20141205.315 when the latter is only available
> from a repository with a lower priority.

So there are two part above:
- decide which repository to use when M-x package-install is used.
- don't switch repository for upgrade.

I don't think we need priorities for the second part once we refrain
from automatically switching from one repository to another during
upgrade.  For the first part priorities could still be useful, indeed,
tho we should simply prompt the user for those cases where there is
a choice between several repositories.

So, overall, I'm not sure having repository priorities would be really
important, but the patch is fairly small, so I think it's OK.  BTW,
please simplify the patch by removing package-archive-default-priority
(and hard code 0 as the default priority).


        Stefan

bug#19296: [PATCH] Package archives now have priorities.

[Achim Gratz]

Stefan Monnier writes:
>> When installing packages by name, only packages from archives with
>> the highest priority are considered, before versions are compared.
>
> What can this be used for (other than the MELPA case, obviously)?

Just like with package archives in the GNU/Linux world, this is useful
if the version numbering across those archives is incompatible even
though they are supposedly serving the same packages.  Priorities are a
good way for a user to describe in which order he wants multiple
archives to be searched for updates and have been used for this purpose
elsewhere.

> I think the better way to solve the problem of versioning the "bleeding
> edge package" would be to take the base version and tuck the date to it
> (instead of only using the date).

MELPA has choisen their incompatible version number scheme deliberately
and I don't think they are going to stop using it.

> I.e. file names like foo-mode-1.3.0.20141023.tar.gz where "1.3" is the
> version of the last release.

That would not help since it would still be interpreted as a higher
version than the released package and be an update candidate.
Priorities do not need coordination among package archives.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

bug#19296: [PATCH] Package archives now have priorities.

[Stefan Monnier]

>> I.e. file names like foo-mode-1.3.0.20141023.tar.gz where "1.3" is the
>> version of the last release.
> That would not help since it would still be interpreted as a higher
> version than the released package and be an update candidate.
> Priorities do not need coordination among package archives.

The main problem, AFAIK (at least that's the one that caused bug reports
and hence brought this problem to my attention) is when the MELPA recipe
points at an unmaintained branch, so you end up stuck with version
20120415 while a brand new 1.4 just came out on GNU ELPA.  This wouldn't
happen with foo-mode-1.3.0.20120415.tar.gz.


        Stefan

bug#19296: [PATCH] Package archives now have priorities.

[Achim Gratz]

Stefan Monnier writes:
> The main problem, AFAIK (at least that's the one that caused bug reports
> and hence brought this problem to my attention) is when the MELPA recipe
> points at an unmaintained branch, so you end up stuck with version
> 20120415 while a brand new 1.4 just came out on GNU ELPA.  This wouldn't
> happen with foo-mode-1.3.0.20120415.tar.gz.

Forget MELPA for the moment.  The problem this patch solves is giving
users some option to rank the archives when those package archives can't
be bothered to coordinate their package versioning or if the user
explicitly wants to install from a different archive than the one with
the latest version.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

When installing packages by name, only packages from archives with
the highest priority are considered, before versions are compared.

This solves the "MELPA problem", where MELPA assigns date-based
version numbers to packages which override all other archives.
Giving MELPA a lower priority means packages are installed from
MELPA only when the package is not available from other archives.

This can be overridden manually by the user.
---
 lisp/emacs-lisp/package.el     |  107 ++++++++++++++++++++++++++++++----------
 test/automated/package-test.el |   17 +++++++
 2 files changed, 98 insertions(+), 26 deletions(-)

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 4e5c397..844e5ea 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -228,6 +228,33 @@ a package can run arbitrary code."
   :group 'package
   :version "24.1")
 
+(defcustom package-archive-default-priority 500
+  "The default priority for archives.
+
+This is used if the archive is not found in
+`package-archive-priorities'."
+  :type 'integer
+  :risky t
+  :group 'package
+  :version "25.1")
+
+(defcustom package-archive-priorities nil
+  "An alist of priorities for packages.
+
+Each element has the form (ARCHIVE-ID . PRIORITY).
+
+When installing packages, the package with the highest version
+number from the archive with the highest priority is
+selected. When higher versions are available from archives with
+lower priorities, the user has to select those manually.
+
+Archives not in this list have the priority given in
+`package-archive-default-priority'."
+  :type 'integer
+  :risky t
+  :group 'package
+  :version "25.1")
+
 (defcustom package-pinned-packages nil
   "An alist of packages that are pinned to specific archives.
 This can be useful if you have multiple package archives enabled,
@@ -1063,23 +1090,32 @@ Also, add the originating archive to the `package-desc' structure."
                         ;; Older archive-contents files have only 4
                         ;; elements here.
                         (package--ac-desc-extras (cdr package)))))
-         (existing-packages (assq name package-archive-contents))
          (pinned-to-archive (assoc name package-pinned-packages)))
-    (cond
-     ;; Skip entirely if pinned to another archive.
-     ((and pinned-to-archive
-           (not (equal (cdr pinned-to-archive) archive)))
-      nil)
-     ((not existing-packages)
-      (push (list name pkg-desc) package-archive-contents))
-     (t
-      (while
-          (if (and (cdr existing-packages)
-                   (version-list-<
-                    version (package-desc-version (cadr existing-packages))))
-              (setq existing-packages (cdr existing-packages))
-            (push pkg-desc (cdr existing-packages))
-            nil))))))
+    ;; Skip entirely if pinned to another archive.
+    (when (not (and pinned-to-archive
+                    (not (equal (cdr pinned-to-archive) archive))))
+      (setq package-archive-contents
+            (package--add-to-alist pkg-desc package-archive-contents)))))
+
+(defun package--add-to-alist (pkg-desc alist)
+  "Add PKG-DESC to ALIST.
+
+Packages are grouped by name. The package descriptions are sorted
+by version number."
+  (let* ((name (package-desc-name pkg-desc))
+         (priority-version (package-desc-priority-version pkg-desc))
+         (existing-packages (assq name alist)))
+    (if (not existing-packages)
+        (cons (list name pkg-desc)
+              alist)
+      (while (if (and (cdr existing-packages)
+                      (version-list-< priority-version
+                                      (package-desc-priority-version
+                                       (cadr existing-packages))))
+                 (setq existing-packages (cdr existing-packages))
+               (push pkg-desc (cdr existing-packages))
+               nil))
+      alist)))
 
 (defun package-download-transaction (packages)
   "Download and install all the packages in PACKAGES.
@@ -1268,6 +1304,25 @@ The file can either be a tar file or an Emacs Lisp file."
   "Return the archive containing the package NAME."
   (cdr (assoc (package-desc-archive desc) package-archives)))
 
+(defun package-archive-priority (archive)
+  "Return the priority of ARCHIVE.
+
+The archive priorities are specified in
+`package-archive-priorities' and
+`package-archive-default-priority'."
+  (or (cdr (assoc archive package-archive-priorities))
+      package-archive-default-priority))
+
+(defun package-desc-priority-version (pkg-desc)
+  "Return the version PKG-DESC with the archive priority prepended.
+
+This allows for easy comparison of package versions from
+different archives if archive priorities are meant to be taken in
+consideration."
+  (cons (package-archive-priority
+         (package-desc-archive pkg-desc))
+        (package-desc-version pkg-desc)))
+
 (defun package--download-one-archive (archive file)
   "Retrieve an archive file FILE from ARCHIVE, and cache it.
 ARCHIVE should be a cons cell of the form (NAME . LOCATION),
@@ -1940,18 +1995,18 @@ If optional arg BUTTON is non-nil, describe its associated package."
       ;; ENTRY is (PKG-DESC [NAME VERSION STATUS DOC])
       (let ((pkg-desc (car entry))
 	    (status (aref (cadr entry) 2)))
-	(cond ((member status '("installed" "unsigned"))
-	       (push pkg-desc installed))
-	      ((member status '("available" "new"))
-	       (push (cons (package-desc-name pkg-desc) pkg-desc)
-                     available)))))
+        (cond ((member status '("installed" "unsigned"))
+               (push pkg-desc installed))
+              ((member status '("available" "new"))
+               (setq available (package--add-to-alist pkg-desc available))))))
     ;; Loop through list of installed packages, finding upgrades.
     (dolist (pkg-desc installed)
-      (let ((avail-pkg (assq (package-desc-name pkg-desc) available)))
-	(and avail-pkg
-	     (version-list-< (package-desc-version pkg-desc)
-                             (package-desc-version (cdr avail-pkg)))
-	     (push avail-pkg upgrades))))
+      (let* ((name (package-desc-name pkg-desc))
+             (avail-pkg (cadr (assq name available))))
+        (and avail-pkg
+             (version-list-< (package-desc-priority-version pkg-desc)
+                             (package-desc-priority-version avail-pkg))
+             (push (cons name avail-pkg) upgrades))))
     upgrades))
 
 (defun package-menu-mark-upgrades ()
diff --git a/test/automated/package-test.el b/test/automated/package-test.el
index 6e7994a..2a337fb 100644
--- a/test/automated/package-test.el
+++ b/test/automated/package-test.el
@@ -230,6 +230,23 @@ Must called from within a `tar-mode' buffer."
     (package-refresh-contents)
     (package-install 'simple-single)))
 
+(ert-deftest package-test-install-prioritized ()
+  "Install a lower version from a higher-prioritized archive."
+  (with-package-test ()
+    (let* ((newer-version (expand-file-name "data/package/newer-versions"
+                                            package-test-file-dir))
+           (package-archives `(("older" . ,package-test-data-dir)
+                               ("newer" . ,newer-version)))
+           (package-archive-priorities '(("newer" . 100))))
+
+      (package-initialize)
+      (package-refresh-contents)
+      (package-install 'simple-single)
+
+      (let ((installed (cdr (assq 'simple-single package-alist))))
+        (should (version-list-= '(1 3)
+                                (package-desc-version installed)))))))
+
 (ert-deftest package-test-install-multifile ()
   "Check properties of the installed multi-file package."
   (with-package-test (:basedir "data/package" :install '(multi-file))
-- 
1.7.10.4

bug#19296: [PATCH] Package archives now have priorities.

[Jorgen Schaefer]

On Sun, 7 Dec 2014 22:28:38 +0100
Jorgen Schaefer <forcer@forcix.cx> wrote:

>  lisp/emacs-lisp/package.el     |  107

This has some more reworks and abstractions, but now works well for me
for both installing new packages and upgrading existing ones.

Regards,
Jorgen