bug#49888: 28.0.50; fido-mode not sorting by recency

bug#49888: 28.0.50; fido-mode not sorting by recency

[Manuel Uberti]

In 'emacs -Q', this is what I did:

- M-x savehist-mode RET
- M-x fido-mode RET
- M-x icomplete-vertical-mode RET
- M-x emacs-init-time
- C-x p p, pick a Git-versioned directory RET
- f, pick the fourth candidate, C-x k RET
- M-x now shows pr-ps-file-preview on top
- C-x p p, pick the only available project
- C-x p f does not offer the previously selected candidate

Is it possible to make fido-mode sort the candidates by recency?


In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.18, cairo 
version 1.16.0)
  of 2021-08-05 built on hathaway
Repository revision: 50d3178a4fa6fbfaaaeee7025e8d9ad3a070aea0
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12009000
System Description: Ubuntu 20.04 LTS

Configured using:
  'configure --with-harfbuzz --with-native-compilation'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
   value of $LC_MESSAGES: en_GB.UTF-8
   value of $LC_MONETARY: it_IT.UTF-8
   value of $LC_NUMERIC: it_IT.UTF-8
   value of $LC_TIME: it_IT.UTF-8
   value of $LANG: en_US.UTF-8
   value of $XMODIFIERS: @im=ibus
   locale-coding-system: utf-8-unix

Major mode: ELisp/l

Minor modes in effect:
   icomplete-vertical-mode: t
   icomplete-mode: t
   fido-mode: t
   savehist-mode: t
   tooltip-mode: t
   global-eldoc-mode: t
   eldoc-mode: t
   electric-indent-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-composition-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   indent-tabs-mode: t
   transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs auth-source eieio eieio-core eieio-loaddefs
password-cache json map text-property-search time-date mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
thingatpt pp vc-mtn vc-hg vc-git diff-mode easy-mmode vc-bzr vc-src
vc-sccs vc-svn vc-cvs vc-rcs comp comp-cstr warnings subr-x rx cl-seq
cl-macs cl-extra help-mode vc vc-dispatcher project seq byte-opt gv
bytecomp byte-compile cconv time icomplete cl-loaddefs cl-lib cus-start
cus-load savehist iso-transl tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar
mouse jit-lock font-lock syntax font-core term/tty-colors frame
minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite charscript charprop case-table epa-hook jka-cmpr-hook help
simple abbrev obarray cl-preloaded nadvice button loaddefs faces
cus-face macroexp files window text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk x-toolkit
x multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 113322 10791)
  (symbols 48 10241 0)
  (strings 32 29042 1772)
  (string-bytes 1 917069)
  (vectors 16 18702)
  (vector-slots 8 322889 12691)
  (floats 8 37 275)
  (intervals 56 295 33)
  (buffers 992 12))

-- 
Manuel Uberti
www.manueluberti.eu

bug#49888: 28.0.50; fido-mode not sorting by recency

[João Távora]

Manuel Uberti <manuel.uberti@inventati.org> writes:

> In 'emacs -Q', this is what I did:
>
> - M-x savehist-mode RET
> - M-x fido-mode RET
> - M-x icomplete-vertical-mode RET
> - M-x emacs-init-time
> - C-x p p, pick a Git-versioned directory RET
> - f, pick the fourth candidate, C-x k RET
> - M-x now shows pr-ps-file-preview on top
> - C-x p p, pick the only available project
> - C-x p f does not offer the previously selected candidate
>
> Is it possible to make fido-mode sort the candidates by recency?

Thanks for the report.  Seems like a nice reproduction recipe.  But it
seems like it demonstrates the problem for two fido-mode-using things,
right?  M-x _and_ C-x p p RET f, right?  No problem, just checking.

Indeed, ido-mode was quited perfected in recency stuff.  I was hard and
still kind is to give up on its very nice default M-p binding that
searches the recent history with whatever you have entered as a
pattern.  I used to use that all the time, and still haven't found a
nice substitute in fido-mode or any other completer for that matter.

João

bug#49888: 28.0.50; fido-mode not sorting by recency

[Manuel Uberti]

On 15/08/21 10:22, João Távora wrote:
> Thanks for the report.  Seems like a nice reproduction recipe.  But it
> seems like it demonstrates the problem for two fido-mode-using things,
> right?  M-x _and_ C-x p p RET f, right?  No problem, just checking.

Yes, both M-x and C-x p p RET f, but the same is true for C-x p f once I am in a 
project. I used those two as examples because they are two of my most use 
commands and, among those, the ones with usually more candidates than the rest.

I feel like recency could make a difference especially with commands like these 
(i.e., commands with a lot of candidates) in terms of how quickly I can find 
what I am looking for.

> Indeed, ido-mode was quited perfected in recency stuff.  I was hard and
> still kind is to give up on its very nice default M-p binding that
> searches the recent history with whatever you have entered as a
> pattern.  I used to use that all the time, and still haven't found a
> nice substitute in fido-mode or any other completer for that matter.

True, although I still find M-p useful in fido-mode especially in M-x when I 
want to reach for the last or second-to-last command.

-- 
Manuel Uberti
www.manueluberti.eu

bug#49888: 28.0.50; fido-mode not sorting by recency

[João Távora]

Manuel Uberti <manuel.uberti@inventati.org> writes:

> In 'emacs -Q', this is what I did:
>
> - M-x savehist-mode RET
> - M-x fido-mode RET
> - M-x icomplete-vertical-mode RET
> - M-x emacs-init-time
> - C-x p p, pick a Git-versioned directory RET
> - f, pick the fourth candidate, C-x k RET
> - M-x now shows pr-ps-file-preview on top
> - C-x p p, pick the only available project
> - C-x p f does not offer the previously selected candidate
>
> Is it possible to make fido-mode sort the candidates by recency?

I've now taken a look at this.  Yes, it is certainly possible and
desirable _if_ there aren't any other overriding criteria.  This has to
do with the 'flex' completion style.  In that style, if anything has
been input that leads to "flexy" filtering, the default and correct
behaviour is to sort by completion score.

But if nothing has been input and there are basically no scores, it
makes sense to fall back to a sensible strategy like the one you
suggest.

Try this patch.  It will fix the M-x case, bugt not the C-x p f or C-x f
cases unfortunately.  Those will be harder to fix.

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 1e8e9fc624..36d95ad5f5 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3947,27 +3947,36 @@ completion--flex-adjust-metadata
       ((compose-flex-sort-fn
         (existing-sort-fn) ; wish `cl-flet' had proper indentation...
         (lambda (completions)
-          (let ((pre-sorted
-                 (if existing-sort-fn
-                     (funcall existing-sort-fn completions)
-                   completions)))
-            (cond
-             ((or (not (window-minibuffer-p))
-                  ;; JT@2019-12-23: FIXME: this is still wrong.  What
-                  ;; we need to test here is "some input that actually
-                  ;; leads to flex filtering", not "something after
-                  ;; the minibuffer prompt".  Among other
-                  ;; inconsistencies, the latter is always true for
-                  ;; file searches, meaning the next clauses will be
-                  ;; ignored.
-                  (> (point-max) (minibuffer-prompt-end)))
-              (sort
-               pre-sorted
-               (lambda (c1 c2)
-                 (let ((s1 (get-text-property 0 'completion-score c1))
-                       (s2 (get-text-property 0 'completion-score c2)))
-                   (> (or s1 0) (or s2 0))))))
-             (t pre-sorted))))))
+          (cond
+           (;; Sort by flex score whenever outside the minibuffer or
+            ;; in the minibuffer with some input.  JT@2019-12-23:
+            ;; FIXME: this is still wrong.  What we need to test here
+            ;; is "some input that actually leads to flex filtering",
+            ;; not "something after the minibuffer prompt".  Among
+            ;; other inconsistencies, the latter is always true for
+            ;; file searches, meaning the next clauses in this cond
+            ;; will be ignored.
+            (or (not (window-minibuffer-p))
+                (> (point-max) (minibuffer-prompt-end)))
+            (sort
+             (if existing-sort-fn
+                 (funcall existing-sort-fn completions)
+               completions)
+             (lambda (c1 c2)
+               (let ((s1 (get-text-property 0 'completion-score c1))
+                     (s2 (get-text-property 0 'completion-score c2)))
+                 (> (or s1 0) (or s2 0))))))
+           (;; If no existing sort fn and nothing flexy happening, use
+            ;; the default sort.
+            (not existing-sort-fn)
+            (let ((la (minibuffer--sort-by-length-alpha completions))
+                  (hist (and (minibufferp)
+                             (and (not (eq minibuffer-history-variable t))
+                                  (symbol-value minibuffer-history-variable)))))
+              ;; Sort by history position, put the default, if it
+              ;; exists, on top.
+              (if hist (minibuffer--sort-by-position hist la) la)))
+           (t completions)))))
     `(metadata
       (display-sort-function
        . ,(compose-flex-sort-fn

bug#49888: 28.0.50; fido-mode not sorting by recency

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> Manuel Uberti <manuel.uberti@inventati.org> writes:
>
>> In 'emacs -Q', this is what I did:
>>
>> - M-x savehist-mode RET
>> - M-x fido-mode RET
>> - M-x icomplete-vertical-mode RET
>> - M-x emacs-init-time
>> - C-x p p, pick a Git-versioned directory RET
>> - f, pick the fourth candidate, C-x k RET
>> - M-x now shows pr-ps-file-preview on top
>> - C-x p p, pick the only available project
>> - C-x p f does not offer the previously selected candidate
>>
>> Is it possible to make fido-mode sort the candidates by recency?
>
> I've now taken a look at this.  Yes, it is certainly possible and
> desirable _if_ there aren't any other overriding criteria.  This has to
> do with the 'flex' completion style.  In that style, if anything has
> been input that leads to "flexy" filtering, the default and correct
> behaviour is to sort by completion score.
>
> But if nothing has been input and there are basically no scores, it
> makes sense to fall back to a sensible strategy like the one you
> suggest.
>
> Try this patch.  It will fix the M-x case, bugt not the C-x p f or C-x f
> cases unfortunately.  Those will be harder to fix.

Actually, I correct that.  C-x f will be harder to fix, but C-x p f and
C-x p p RET f work just fine!

So I think I will commit this correction to the flex completion style.
It's indeed extremely useful, especially for C-x p f, which I now also
use a lot.

João

bug#49888: 28.0.50; fido-mode not sorting by recency

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> Manuel Uberti <manuel.uberti@inventati.org> writes:
>>
>>> In 'emacs -Q', this is what I did:
>>>
>>> - M-x savehist-mode RET
>>> - M-x fido-mode RET
>>> - M-x icomplete-vertical-mode RET
>>> - M-x emacs-init-time
>>> - C-x p p, pick a Git-versioned directory RET
>>> - f, pick the fourth candidate, C-x k RET
>>> - M-x now shows pr-ps-file-preview on top
>>> - C-x p p, pick the only available project
>>> - C-x p f does not offer the previously selected candidate
>>>
>>> Is it possible to make fido-mode sort the candidates by recency?
>>
>> I've now taken a look at this.  Yes, it is certainly possible and
>> desirable _if_ there aren't any other overriding criteria.  This has to
>> do with the 'flex' completion style.  In that style, if anything has
>> been input that leads to "flexy" filtering, the default and correct
>> behaviour is to sort by completion score.
>>
>> But if nothing has been input and there are basically no scores, it
>> makes sense to fall back to a sensible strategy like the one you
>> suggest.
>>
>> Try this patch.  It will fix the M-x case, bugt not the C-x p f or C-x f
>> cases unfortunately.  Those will be harder to fix.
>
> Actually, I correct that.  C-x f will be harder to fix, but C-x p f and
> C-x p p RET f work just fine!

I've just commited this fix:

    commit ab23fa4eb22f6557414724769958a63f1c59b49a (HEAD -> master, origin/master, origin/HEAD)
    Author: João Távora <joaotavora@gmail.com>
    Date:   Sun Aug 15 13:19:59 2021 +0100
     
        Sort by recency in flex completion style when no flexy stuff happening

There is still the C-x f bug, but that wasn't in your report.  You seem
to be concerned with M-x and C-x p f (as am I, by the way)

That other problem isn't even very well understood by me.  C-x f looks
for files in a single directory, but minibuffer history has whole
paths...  I think it deserves a new bug report if anyone really cares.

For now I think particular bug report can be closed.  If anyone
disagrees, let me know.

João

bug#49888: 28.0.50; fido-mode not sorting by recency

[Manuel Uberti]

On 15/08/21 14:38, João Távora wrote:
> I've just commited this fix:
> 
>      commit ab23fa4eb22f6557414724769958a63f1c59b49a (HEAD -> master, origin/master, origin/HEAD)
>      Author: João Távora <joaotavora@gmail.com>
>      Date:   Sun Aug 15 13:19:59 2021 +0100
>       
>          Sort by recency in flex completion style when no flexy stuff happening
> 
> There is still the C-x f bug, but that wasn't in your report.  You seem
> to be concerned with M-x and C-x p f (as am I, by the way)

Fantastic, it works greatly for those commands.

> That other problem isn't even very well understood by me.  C-x f looks
> for files in a single directory, but minibuffer history has whole
> paths...  I think it deserves a new bug report if anyone really cares.

Yes, I can live with C-x f not sorting by recency. I mostly move by projects, 
and when I use C-x f I don't mind typing a bit more to get where I want. So I 
agree with you, this can be a matter for a different bug report.

> For now I think particular bug report can be closed.  If anyone
> disagrees, let me know.

Thank you so much for having worked on this.

-- 
Manuel Uberti
www.manueluberti.eu