bug#63757: 29.0.91 order of package paths changed: random old versions of packages in load-path

[Philip Kaludercic <philipk@posteo.net>]

Philip Kaludercic <philipk@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Sun, 28 May 2023 02:30:18 +1000
>>> From:  Евгений Бойков via "Bug reports for GNU Emacs,
>>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>>> 
>>> I am not confident how to classify this kind of behaviour change.
>>> You can see the beginning of the discussion below.
>>> 
>>> Briefly, if you have different versions of packages in package-user-dir then pretest 29 loads random
>>> old versions of them whereas 28 loads mostly latest versions.
>>> 
>>> ---------- Forwarded message ---------
>>> From: Philip Kaludercic <philipk@posteo.net>
>>> Date: Sat, May 27, 2023 at 6:23 PM
>>> Subject: Re: emacs 04c4c578 pretest 29.0.91
>>> To: Евгений Бойков <artscan@list.ru>
>>> 
>>> Евгений Бойков <artscan@list.ru> writes:
>>> 
>>> > Hello, Philip.
>>> 
>>> Hi,
>>> 
>>> > In your commit 04c4c578, package directories became NOSORT due to (fourth t
>>> > argument)
>>> >     (directory-files dir)
>>> > replaced by
>>> >     (directory-files dir t "^[^.]" t)
>>> >
>>> > It affects package-alist forming. For example, there are old and new
>>> > versions of a package in my elpa directory:
>>> >
>>> > go-mode-20170726.555
>>> > go-mode-20220114.2239
>>> >
>>> > After (package-initialize)
>>> > in emacs 28 (require 'go-mode) gives 2022 version, e.g. the latest version
>>> > of a package in most cases, directory-files default sorting works.
>>> > In emacs 29.0.91 I get old 2017 version because of NOSORT unpredictable
>>> > version order in package-alist and hence random paths in load-path.
>>> >
>>> > I just tried to move from 28 to 29.0.91 and got many errors due to old
>>> > versions of many packages.
>>> > Excuse me if it is a feature and I missed something.
>>> 
>>> No, this is not a feature but I think it might have uncovered a bug.
>>> Relying on the ordering of file names (which IIRC is lexiographic not
>>> numeric) is not a robust way to decide what packages should be preferred
>>> when loading descriptors.  Could you submit a bug report to
>>> bug-gnu-emacs@gnu.org so that this issue can be discussed with the other
>>> maintainers?
>>
>> Philip, are you looking into this?  Or do you have any comments?
>
> Евгений sent me a message with some more details, so yes I am looking at
> it:
>
>> I sent the bug report https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63757 (
>> 63757@debbugs.gnu.org) and did more research.
>> It turned out that the final ordering is defined (fb87d5) by
>
>>   (let ((pkg-descs (sort (cdr (assq pkg-name package-alist))
>>                          (lambda (p1 p2)
>>                            (let ((v1 (package-desc-version p1))
>>                                  (v2 (package-desc-version p2)))
>>                              (or
>>                               ;; Prefer VC packages.
>>                               (package-vc-p p1)
>>                               (package-vc-p p2)
>>                               ;; Prefer builtin packages.
>>                               (package-disabled-p p1 v1)
>>                               (not (package-disabled-p p2 v2))))))))
>>
>> So I can fix it like this
>>
>> (sort (cdr (assq 'go-mode package-alist))
>>                          (lambda (p1 p2)
>>                            (let ((v1 (package-desc-version p1))
>>                                  (v2 (package-desc-version p2)))
>>                              (or
>>      (not (version-list-< v1 v2))
>>      ))))
>>
>> But I still can't understand how to fix it along with builtin and VC
>> preferred ordering.
>
> Won't this work:
>
> diff --git a/lisp/emacs-lisp/package.el b/lisp/emacs-lisp/package.el
> index 01826da273d..7eb185e7042 100644
> --- a/lisp/emacs-lisp/package.el
> +++ b/lisp/emacs-lisp/package.el
> @@ -927,7 +927,9 @@ package--get-activatable-pkg
>                                (package-vc-p p2)
>                                ;; Prefer builtin packages.
>                                (package-disabled-p p1 v1)
> -                              (not (package-disabled-p p2 v2))))))))
> +                              (not (package-disabled-p p2 v2))
> +                              ;; Prever newer packages
> +                              (version-list-< v2 v1)))))))
>      ;; Check if PACKAGE is available in `package-alist'.
>      (while
>          (when pkg-descs

Ping?