bug#63757: 29.0.91 order of package paths changed: random old versions of packages in load-path

[Philip Kaludercic <philipk@posteo.net>]

[-- Attachment #1: Type: text/plain, Size: 3614 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 28 May 2023 02:30:18 +1000
>> From:  Евгений Бойков via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> I am not confident how to classify this kind of behaviour change.
>> You can see the beginning of the discussion below.
>> 
>> Briefly, if you have different versions of packages in package-user-dir then pretest 29 loads random
>> old versions of them whereas 28 loads mostly latest versions.
>> 
>> ---------- Forwarded message ---------
>> From: Philip Kaludercic <philipk@posteo.net>
>> Date: Sat, May 27, 2023 at 6:23 PM
>> Subject: Re: emacs 04c4c578 pretest 29.0.91
>> To: Евгений Бойков <artscan@list.ru>
>> 
>> Евгений Бойков <artscan@list.ru> writes:
>> 
>> > Hello, Philip.
>> 
>> Hi,
>> 
>> > In your commit 04c4c578, package directories became NOSORT due to (fourth t
>> > argument)
>> >     (directory-files dir)
>> > replaced by
>> >     (directory-files dir t "^[^.]" t)
>> >
>> > It affects package-alist forming. For example, there are old and new
>> > versions of a package in my elpa directory:
>> >
>> > go-mode-20170726.555
>> > go-mode-20220114.2239
>> >
>> > After (package-initialize)
>> > in emacs 28 (require 'go-mode) gives 2022 version, e.g. the latest version
>> > of a package in most cases, directory-files default sorting works.
>> > In emacs 29.0.91 I get old 2017 version because of NOSORT unpredictable
>> > version order in package-alist and hence random paths in load-path.
>> >
>> > I just tried to move from 28 to 29.0.91 and got many errors due to old
>> > versions of many packages.
>> > Excuse me if it is a feature and I missed something.
>> 
>> No, this is not a feature but I think it might have uncovered a bug.
>> Relying on the ordering of file names (which IIRC is lexiographic not
>> numeric) is not a robust way to decide what packages should be preferred
>> when loading descriptors.  Could you submit a bug report to
>> bug-gnu-emacs@gnu.org so that this issue can be discussed with the other
>> maintainers?
>
> Philip, are you looking into this?  Or do you have any comments?

Евгений sent me a message with some more details, so yes I am looking at
it:

> I sent the bug report https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63757 (
> 63757@debbugs.gnu.org) and did more research.
> It turned out that the final ordering is defined (fb87d5) by

>   (let ((pkg-descs (sort (cdr (assq pkg-name package-alist))
>                          (lambda (p1 p2)
>                            (let ((v1 (package-desc-version p1))
>                                  (v2 (package-desc-version p2)))
>                              (or
>                               ;; Prefer VC packages.
>                               (package-vc-p p1)
>                               (package-vc-p p2)
>                               ;; Prefer builtin packages.
>                               (package-disabled-p p1 v1)
>                               (not (package-disabled-p p2 v2))))))))
>
> So I can fix it like this
>
> (sort (cdr (assq 'go-mode package-alist))
>                          (lambda (p1 p2)
>                            (let ((v1 (package-desc-version p1))
>                                  (v2 (package-desc-version p2)))
>                              (or
>      (not (version-list-< v1 v2))
>      ))))
>
> But I still can't understand how to fix it along with builtin and VC
> preferred ordering.

Won't this work:


[-- Attachment #2: Type: text/plain, Size: 733 bytes --]

diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
index 01826da273d..7eb185e7042 100644
--- a/lisp/emacs-lisp/package.el
+++ b/lisp/emacs-lisp/package.el
@@ -927,7 +927,9 @@ package--get-activatable-pkg
                               (package-vc-p p2)
                               ;; Prefer builtin packages.
                               (package-disabled-p p1 v1)
-                              (not (package-disabled-p p2 v2))))))))
+                              (not (package-disabled-p p2 v2))
+                              ;; Prever newer packages
+                              (version-list-< v2 v1)))))))
     ;; Check if PACKAGE is available in `package-alist'.
     (while
         (when pkg-descs