Add user customization fido-completion-styles

Add user customization fido-completion-styles

[Andrew Schwartzmeyer]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

Hi,

Users of fido-mode (like me) may want to customize the completion style it uses, so add a variable to allow them to do so.

For instance, I really like using the “initials” style so “vlm” quickly completes visual-line-mode, and then falling back to flex. There are also neat packages like orderless (https://github.com/oantolin/orderless <https://github.com/oantolin/orderless>) which add a new completion-style, which folks may want to plug-in and use. Anyway, it seems like it ought to be customizable.

Thanks,

Andy

P.S. I wish “flex” were a bit faster. On my circa 2012 MacBook, M-x is noticeably slow with “flex” for commands. Also, I don’t know what “:version” to put on the defcustom, and the docs could be improved a bit.


[-- Attachment #2.1: Type: text/html, Size: 1310 bytes --]

[-- Attachment #2.2: 0001-Add-user-customization-fido-completion-styles.patch --]
[-- Type: application/octet-stream, Size: 1316 bytes --]

From 59177be2198aa9aa9d12d730f732456caf187f9e Mon Sep 17 00:00:00 2001
From: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>
Date: Sun, 31 May 2020 13:55:13 -0700
Subject: [PATCH] Add user customization fido-completion-styles

Users of fido-mode may want to customize the completion style it uses,
so add a variable to allow them to do so.
---
 lisp/icomplete.el | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index c12f3901f..40626663f 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -134,6 +134,12 @@ icompletion is occurring."
   :type 'hook
   :group 'icomplete)
 
+(defcustom fido-completion-styles '(flex)
+  "Completion styles used by `fido-mode'.
+See `completion-styles'"
+  :type completion--styles-type
+  :group 'icomplete)
+
 
 ;;;_* Initialization
 
@@ -328,7 +334,7 @@ if that doesn't produce a completion match."
     (setq-local icomplete-tidy-shadowed-file-names t
                 icomplete-show-matches-on-no-input t
                 icomplete-hide-common-prefix nil
-                completion-styles '(flex)
+                completion-styles fido-completion-styles
                 completion-flex-nospace nil
                 completion-category-defaults nil)))
 
-- 
2.26.0


[-- Attachment #2.3: Type: text/html, Size: 226 bytes --]

Re: Add user customization fido-completion-styles

[João Távora]

Andrew Schwartzmeyer <andrew@schwartzmeyer.com> writes:

> Hi,
>
> Users of fido-mode (like me) may want to customize the completion style it uses, so add a variable to allow them to
> do so.
>
> For instance, I really like using the “initials” style so “vlm”
> quickly completes visual-line-mode, and then falling back to
> flex. There are also neat packages like orderless
> (https://github.com/oantolin/orderless) which add a new
> completion-style, which folks may want to plug-in and use. Anyway, it
> seems like it ought to be customizable.

Actually, it is, just not with custom.  These two lines should do the trick:

  (add-hook 'icomplete-minibuffer-setup-hook 'my-flex-styles)
  (defun my-flex-styles () (setq-local completion-styles '(initials flex)))

Or just use a lambda, which is even shorter.  I'd like to avoid a
duplicate version of completion-styles customization vars.  

I am however, receptive to customizing the default styles used in fido
mode.  We could come up with a "smarter" flex that values matches
following "-" a bit more.  Or maybe '(initials flex) is a good fit
already, I'll play around a bit with it.  Or we could even think about
making two styles merge "sooner", since currently we skip to the next
one only once the previous no longer matches anything.  Maybe that could
be changed, i.e. adding the matches of the secondary style once the
first one only has two or three matches.

> P.S. I wish “flex” were a bit faster. On my circa 2012 MacBook, M-x is
> noticeably slow with “flex” for commands.

Yes, flex isn't blazing fast, particularly with short patterns.  The
solution is (likely) to port parts of it in C, which isn't particularly
hard, just a bit tedious.

However, the slowness of flex shouldn't affect the ability to type fast
after M-x, since very new character typed should interrupt the lenghty
calculation.  Let me know if that is somehow not the case with your
completion experience.

> Also, I don’t know what “:version” to put on the defcustom, and the
> docs could be improved a bit.

That would depend on what version you intend the patch to target.  Since
fido-mode is a new thing in Emacs 27.1, it is my understanding (from
earlier feedback of Eli) that we can to push to emacs-27, since we
won't, by definition, introduce regressions.  But I would refrain from
doing so this late in the game, unless we are fixing severe
misbehaviour.

Also, "orderless" looks cool.  Is it faster than flex?

João

Re: Add user customization fido-completion-styles

[Dmitry Gutov]

On 01.06.2020 02:43, João Távora wrote:
>    (add-hook 'icomplete-minibuffer-setup-hook 'my-flex-styles)
>    (defun my-flex-styles () (setq-local completion-styles '(initials flex)))

I think that's not very obvious, and requires familiarity with how 
fido-mode is implemented.

And the request for being able to customize which styles are used has 
come up before.

Re: Add user customization fido-completion-styles

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 01.06.2020 02:43, João Távora wrote:
>>    (add-hook 'icomplete-minibuffer-setup-hook 'my-flex-styles)
>>    (defun my-flex-styles () (setq-local completion-styles '(initials flex)))
> I think that's not very obvious, and requires familiarity with how
> fido-mode is implemented.

I think it requires familiarity with hooks, if anything.  The manual is
rife with trivial add-hook snippets such as these and I'm going to
document it there.  As for the "obvious", I'm afraid that's another
boring "existential Emacs" discussion.

João

Re: Add user customization fido-completion-styles

[Andrew Schwartzmeyer]

[-- Attachment #1: Type: text/plain, Size: 4719 bytes --]

> Actually, it is, just not with custom.  These two lines should do the trick:
> 
>  (add-hook 'icomplete-minibuffer-setup-hook 'my-flex-styles)
>  (defun my-flex-styles () (setq-local completion-styles '(initials flex)))

Ah! Thank you so much! I didn’t realize the ordering would work out such that this would override the setq-local in icomplete--fido-mode-setup.

> Or just use a lambda, which is even shorter.  I'd like to avoid a
> duplicate version of completion-styles customization vars. 

I agree, it felt weird writing it, but I didn’t know the above trick. So thanks for that! I think we can not patch this until we come up with a cohesive customization (and documentation!) story.

> Or we could even think about
> making two styles merge "sooner", since currently we skip to the next
> one only once the previous no longer matches anything.

This seems like it would be really nice. There are a lot of good (and bad) filtering algorithms out there. Flx, Orderless, Prescient are all things to learn from. 

> Also, "orderless" looks cool.  Is it faster than flex?

So far it seems very fast, but I haven’t used it long. Prescient is also fast, but ironically this issue on GitHub https://github.com/raxod502/prescient.el/issues/58 <https://github.com/raxod502/prescient.el/issues/58> wants to use Orderless in Prescient to make it even faster.

Honestly, the strangest thing in my experience so far is that M-x with “flex” style doesn’t start with my historical items, just seemingly random (but static) commands. It seems like switching it to “initials flex” already pulls in my history as default sort, but I need to investigate further and be sure it isn’t something else. I think an integral part to any completion framework is for the initial suggestions to follow frequency/recency, to provide good context with no input, and allow immediate selection of frequently/recently used commands (buffers, files…).

Thanks,

Andy

> On May 31, 2020, at 4:43 PM, João Távora <joaotavora@gmail.com> wrote:
> 
> Andrew Schwartzmeyer <andrew@schwartzmeyer.com> writes:
> 
>> Hi,
>> 
>> Users of fido-mode (like me) may want to customize the completion style it uses, so add a variable to allow them to
>> do so.
>> 
>> For instance, I really like using the “initials” style so “vlm”
>> quickly completes visual-line-mode, and then falling back to
>> flex. There are also neat packages like orderless
>> (https://github.com/oantolin/orderless) which add a new
>> completion-style, which folks may want to plug-in and use. Anyway, it
>> seems like it ought to be customizable.
> 
> Actually, it is, just not with custom.  These two lines should do the trick:
> 
>  (add-hook 'icomplete-minibuffer-setup-hook 'my-flex-styles)
>  (defun my-flex-styles () (setq-local completion-styles '(initials flex)))
> 
> Or just use a lambda, which is even shorter.  I'd like to avoid a
> duplicate version of completion-styles customization vars.  
> 
> I am however, receptive to customizing the default styles used in fido
> mode.  We could come up with a "smarter" flex that values matches
> following "-" a bit more.  Or maybe '(initials flex) is a good fit
> already, I'll play around a bit with it.  Or we could even think about
> making two styles merge "sooner", since currently we skip to the next
> one only once the previous no longer matches anything.  Maybe that could
> be changed, i.e. adding the matches of the secondary style once the
> first one only has two or three matches.
> 
>> P.S. I wish “flex” were a bit faster. On my circa 2012 MacBook, M-x is
>> noticeably slow with “flex” for commands.
> 
> Yes, flex isn't blazing fast, particularly with short patterns.  The
> solution is (likely) to port parts of it in C, which isn't particularly
> hard, just a bit tedious.
> 
> However, the slowness of flex shouldn't affect the ability to type fast
> after M-x, since very new character typed should interrupt the lenghty
> calculation.  Let me know if that is somehow not the case with your
> completion experience.
> 
>> Also, I don’t know what “:version” to put on the defcustom, and the
>> docs could be improved a bit.
> 
> That would depend on what version you intend the patch to target.  Since
> fido-mode is a new thing in Emacs 27.1, it is my understanding (from
> earlier feedback of Eli) that we can to push to emacs-27, since we
> won't, by definition, introduce regressions.  But I would refrain from
> doing so this late in the game, unless we are fixing severe
> misbehaviour.
> 
> Also, "orderless" looks cool.  Is it faster than flex?
> 
> João
> 
> 


[-- Attachment #2: Type: text/html, Size: 6743 bytes --]

Re: Add user customization fido-completion-styles

[Andrew Schwartzmeyer]

[-- Attachment #1: Type: text/plain, Size: 1933 bytes --]

Okay so re:

> On May 31, 2020, at 5:37 PM, Andrew Schwartzmeyer <andrew@schwartzmeyer.com> wrote:
> 
> Honestly, the strangest thing in my experience so far is that M-x with “flex” style doesn’t start with my historical items, just seemingly random (but static) commands. It seems like switching it to “initials flex” already pulls in my history as default sort, but I need to investigate further and be sure it isn’t something else. I think an integral part to any completion framework is for the initial suggestions to follow frequency/recency, to provide good context with no input, and allow immediate selection of frequently/recently used commands (buffers, files…).

With just “flex” as the completion style, for some reason M-x with no input (from an emacs -q) presents the following candidates as the first:

icomplete-fido-delete-char
enable-theme
url-setup-privacy-info
dired-at-point
tab-bar-select-tab

I can’t see a pattern in these. If I then call a command, I’d want/expect (from experience with other completion frameworks) the next use of M-x to show it as the first candidate, but it doesn’t.

Now if I use the aforementioned Orderless package and the minibuffer hook to use it as the completion style in fido, somehow it does some magic such that M-x (which is still bound to execute-extended-command) now has some history sorting! (Working for switch-to-buffer too. :)

For repro:

(require 'package)
(add-to-list 'package-archives
             '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)
(package-refresh-contents)
(package-install 'orderless)
(add-hook 'icomplete-minibuffer-setup-hook
	  (lambda () (setq-local completion-styles '(orderless))))
(fido-mode)

So yeah, I’m off to look through Orderless to see what magic it does that sorts the candidates for M-x (without having do anything like what smex does!).

Cheers,

Andy

[-- Attachment #2: Type: text/html, Size: 3630 bytes --]

Re: Add user customization fido-completion-styles

[João Távora]

Andrew Schwartzmeyer <andrew@schwartzmeyer.com> writes:

> With just “flex” as the completion style, for some reason M-x with no
> input (from an emacs -q) presents the following candidates as the
> first:
>
> icomplete-fido-delete-char
> enable-theme
> url-setup-privacy-info
> dired-at-point
> tab-bar-select-tab

With no inputs to the flex completion style, it matches every candidate.

> I can’t see a pattern in these. If I then call a command, I’d
> want/expect (from experience with other completion frameworks) the
> next use of M-x to show it as the first candidate, but it doesn’t.

fido-mode (and I believe icomplete-mode, whereupon fido-mode is based)
do this for minibuffer-reading commands that have a "default", i.e. a
result that is returned immediately if the user "forces" the completion
with no input, regardless of whether the candidates are showing or not.

M-x, which maps to execute-extended-command, is not one of those
commands.

Regarding the pattern, there is none.  I believe that's the order of all
symbols as returned by the all-completions when called to return a list
of everything that is a command.  It doesn't waste any time sorting.

> Now if I use the aforementioned Orderless package and the minibuffer
> hook to use it as the completion style in fido, somehow it does some
> magic such that M-x (which is still bound to execute-extended-command)
> now has some history sorting! (Working for switch-to-buffer too. :)

You could do some sorting, but that is a a O(NlogN) operation (I think,
may be wrong, N is the number of commands times the number of elements
in the history).  It doesn't make much sense to invest in this sorting
because it is only useful if all the candidates have the same flex
score.  Which they do in the beginning situation, but that quickly
changes as the user inputs a pattern.

Alternatively, a dirty hack might be to exceptionally use the history
list itself -- and not all the candidates -- when the pattern is
empty. That'd probably fill in the space and wouldn't be functionally
wrong.

> So yeah, I’m off to look through Orderless to see what magic it does
> that sorts the candidates for M-x (without having do anything like
> what smex does!).

I haven't looked at Orderless yet.  For the empty pattern case, it might
be taking the shortcut I suggested.

But for non-empty patterns iss it doing what I think it is doing?  Will
typing the three letter pattern "foo" match "foobarbaz" "fabrobazo" and
"orzobafab"?  In which order will they be returned?

João

RE: Add user customization fido-completion-styles

[Drew Adams]

FWIW:

If you choose the sort order "by flx score" in
Icicles, and if you type no pattern to match in
the minibuffer, then the candidates (e.g. commands,
for `M-x') are sorted by `string-lessp' (but
respecting `completion-ignore-case').

In general, the predefined sort orders in Icicles
do something reasonable when the main sorting
test says two candidates are incomparable.

Some such orders have more than one fallback level,
i.e., more than one kind of sorting, the 2nd
taking over when the 1st can't compare, the 3rd
taking over when the 2nd can't compare, etc.

Often the (last) fallback is alphabetical order.
For example, for flx-score sorting, the test
predicate is this:

(defun icicle-flx-score-greater-p (s1 s2)
  "Non-nil means the `flx-score' of S1 is greater than that of S2.
That is, the cars of the `flx-score' values are compared.

If `flx-score' returns nil for either argument, then they are compared
using `icicle-case-string-less-p'.

This function requires library `flx.el'."
  (let* ((input   (if (and (icicle-file-name-input-p)  
                           insert-default-directory)
                      (file-name-nondirectory icicle-current-input)
                    icicle-current-input))
         (score1  (flx-score s1 input))
         (score2  (flx-score s2 input)))
    (if (and score1  score2)
        (> (car score1) (car score2))
      (icicle-case-string-less-p s1 s2))))

I don't know why Emacs shouldn't act similarly.

As for performance, for `M-x' there certainly is
no problem.  `M-x' with no input pattern to match
shows the 6000-some candidates immediately.

https://www.emacswiki.org/emacs/Icicles_-_Sorting_Candidates
___

FWIW2:

Icicles extends the idea of combining predicates
in the definition of a single sort predicate, in
two ways:

1. Interactively, you can apply multiple sort
   orders, sequentially.

2. A sort comparer can itself be defined as a
   list ((PRED...) FINAL-PRED), where each PRED
   and  FINAL-PRED are binary predicates.

#1 is described here:

https://www.emacswiki.org/emacs/Icicles_-_Sorting_Candidates#AddingASavedSortOrder

In #2, each PRED returns true, false, or undecided,
where undecided means that the next PRED takes over
deciding, and so on.  FINAL-PRED just returns true
or false.  (True is `(t)', false is `(nil)', and
undecided is nil.)

This way of combining yes-no-maybe predicates is
described here:

https://www.emacswiki.org/emacs/ApplesAndOranges
___

(Note: Icicles uses the flx-scoring provided by
3rd-party library `flx.el', from Le Wang (if loaded),
not the flx completing style you recently added to
vanilla Emacs.

https://github.com/lewang/flx)

Re: Add user customization fido-completion-styles

[João Távora]

Drew Adams <drew.adams@oracle.com> writes:

> As for performance, for `M-x' there certainly is
> no problem.  `M-x' with no input pattern to match
> shows the 6000-some candidates immediately.

Certainly it doesn't _show_ 6000 candidates, right? You must mean it
calculates them quickly and shows the top-sorting set quickly.  That's
great, I guess, and I join in asking if Emacs shouldn't do that.

I didnt' dig down very intensively, but my naive profiling shows the
bulk of the work to be in `all-completions' or thereabouts to be the
main sink of cpu time:

       - icomplete-post-command-hook                             2032  86%
        - icomplete-exhibit                                      2032  86%
         - icomplete-completions                                 2009  85%
          - icomplete--sorted-completions                        1986  84%
           - completion-all-sorted-completions                   1986  84%
            - completion-all-completions                         1982  84%
             - completion--nth-completion                        1982  84%
              - completion--some                                 1982  84%
               - #<compiled 0x15809f8924ad>                      1982  84%
                - completion-flex-all-completions                1965  84%
                 - completion-substring--all-completions         1935  82%
                  - completion-pcm--all-completions              1932  82%
                   - all-completions                             1929  82%
                    - #<compiled 0x1fe57a0ac19d>                 1929  82%
                     - complete-with-action                      1929  82%
                      - all-completions                            37   1%

As you know, all-completions is a C function.  The 1% is suspicious, I
don't know if I should trust it.

Anyway, I'm curious how Icycles evades this.

Anyway2, I wasn't trying this in an Emacs -Q.  When I do, and I set
icomplete-compute-delay to 0, then M-x is practically instantaneous.

Maybe the default 0.3 value of that variable is excessively
conservative.

So, to summarize: I do believe could be some shortcuts for the notable
case of no input pattern to give a speed boost, maybe showing them them
their relevant minibuffer history.  I don't believe there's some kind of
magical speed pickup to be had.

Even the C rewrite of parts of completion-pcm--all-completions
now seems dubious.

> This way of combining yes-no-maybe predicates is
> described here:
>
> https://www.emacswiki.org/emacs/ApplesAndOranges

Have you tried `fido-mode` and/or flex?  Can you think of specific ways
to improve it?  If so, patches are very welcome.

It's a bit harder, I admit, to try to learn Icicles and decide which
features to migrate.  But if you can point some laser-targeted
improvement to the flex algorithm (in efficiency of functionality) are
very welcome.

João

Re: Add user customization fido-completion-styles

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 02 Jun 2020 18:51:31 +0100
> Cc: emacs-devel@gnu.org
> 
>        - icomplete-post-command-hook                             2032  86%
>         - icomplete-exhibit                                      2032  86%
>          - icomplete-completions                                 2009  85%
>           - icomplete--sorted-completions                        1986  84%
>            - completion-all-sorted-completions                   1986  84%
>             - completion-all-completions                         1982  84%
>              - completion--nth-completion                        1982  84%
>               - completion--some                                 1982  84%
>                - #<compiled 0x15809f8924ad>                      1982  84%
>                 - completion-flex-all-completions                1965  84%
>                  - completion-substring--all-completions         1935  82%
>                   - completion-pcm--all-completions              1932  82%
>                    - all-completions                             1929  82%
>                     - #<compiled 0x1fe57a0ac19d>                 1929  82%
>                      - complete-with-action                      1929  82%
>                       - all-completions                            37   1%

This actually says that the time sink is complete-with-action, not
all-completions.

> As you know, all-completions is a C function.  The 1% is suspicious, I
> don't know if I should trust it.

You should trust the profile.  It works the same whether it's a C
function or a Lisp function.

Re: Add user customization fido-completion-styles

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Date: Tue, 02 Jun 2020 18:51:31 +0100
>> Cc: emacs-devel@gnu.org
>> 
>>        - icomplete-post-command-hook                             2032  86%
>>         - icomplete-exhibit                                      2032  86%
>>          - icomplete-completions                                 2009  85%
>>           - icomplete--sorted-completions                        1986  84%
>>            - completion-all-sorted-completions                   1986  84%
>>             - completion-all-completions                         1982  84%
>>              - completion--nth-completion                        1982  84%
>>               - completion--some                                 1982  84%
>>                - #<compiled 0x15809f8924ad>                      1982  84%
>>                 - completion-flex-all-completions                1965  84%
>>                  - completion-substring--all-completions         1935  82%
>>                   - completion-pcm--all-completions              1932  82%
>>                    - all-completions                             1929  82%
>>                     - #<compiled 0x1fe57a0ac19d>                 1929  82%
>>                      - complete-with-action                      1929  82%
>>                       - all-completions                            37   1%
>
> This actually says that the time sink is complete-with-action, not
> all-completions.
>
>> As you know, all-completions is a C function.  The 1% is suspicious, I
>> don't know if I should trust it.
>
> You should trust the profile.  It works the same whether it's a C
> function or a Lisp function.

OK that's good to know, but then, I don't know where the time is going:
Shouldn't I be seeing some callee of this function using up the time??

This is complete-with-action minus docstring, can you help me understand
what's going on?

(defun complete-with-action (action collection string predicate)
  (cond
   ((functionp collection) (funcall collection string predicate action))
   ((eq (car-safe action) 'boundaries) nil)
   ((eq action 'metadata) nil)
   (t
    (funcall
     (cond
      ((null action) 'try-completion)
      ((eq action t) 'all-completions)
      (t 'test-completion))
     string collection predicate))))

João

Re: Add user customization fido-completion-styles

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: drew.adams@oracle.com,  emacs-devel@gnu.org
> Date: Tue, 02 Jun 2020 19:24:45 +0100
> 
> This is complete-with-action minus docstring, can you help me understand
> what's going on?
> 
> (defun complete-with-action (action collection string predicate)
>   (cond
>    ((functionp collection) (funcall collection string predicate action))
>    ((eq (car-safe action) 'boundaries) nil)
>    ((eq action 'metadata) nil)
>    (t
>     (funcall
>      (cond
>       ((null action) 'try-completion)
>       ((eq action t) 'all-completions)
>       (t 'test-completion))
>      string collection predicate))))

What is 'collection'? is it a function?

Anyway, I suggest to run the same experiment after loading all the
code involved in this as *.el files (not *.elc).  That should show a
more detailed profile, which hopefully will give a clue.

Re: Add user customization fido-completion-styles

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

> What is 'collection'? is it a function?

I suppose it can be, yes.

> Anyway, I suggest to run the same experiment after loading all the
> code involved in this as *.el files (not *.elc).  That should show a
> more detailed profile, which hopefully will give a clue.

That did give some clues.  When loading minibuffer.el that section
becomes:

               - complete-with-action                          1125  75%
                  - cond                                         1125  75%
                   - funcall                                     1125  75%
                    - all-completions                              24   1%
                       #<compiled 0x15809d82b069>                  15   1%

Which shows there's an compiled function coming from somewhere else,
likely simple.el.  When I load simple.el, the profile changes
dramatically, and becomes so deep that it's impossivle to read because
M-x profiler-report truncates the names of functions at a certain point.

So then I remembered loading the compiled minibuffer.elc and loading the
uncompiled simple.el, and I got more interesting results, such as these:

                   - #<compiled 0x15809c619759>                  1264  82%
                    - completion-flex-all-completions               1233  80%
                     - completion-substring--all-completions               1212  78%
                      - completion-pcm--all-completions               1209  78%
                       - all-completions                         1209  78%
                        - #<lambda -0x5acbf46d069cde1>                318  20%
                         - let                                    318  20%
                          - complete-with-action                  318  20%
                           - all-completions                       10   0%
                            - #<lambda -0xded78e903380374>                  5   0%
                             - and                                  5   0%
                                funcall                             4   0%
                              - or                                  1   0%
                                 equal                              1   0%
                        - #<lambda 0xf61f9dacf96321f>                197  12%
                         - let                                    197  12%
                          - complete-with-action                  197  12%
                           - all-completions                        7   0%
                            - #<lambda 0x7121b4e1cc7fc8f>                  4   0%
                             - and                                  4   0%
                                funcall                             4   0%
                        - #<lambda 0xf590d1eef96321f>                180  11%
                         - let                                    180  11%
                          - complete-with-action                  180  11%
                           - all-completions                        5   0%
                            - #<lambda 0x7018618bcc7fc8f>                  4   0%
                             - and                                  4   0%
                                funcall                             4   0%
                        + #<lambda 0xf61e0170f96321f>                177  11%
                        + #<lambda 0xf61d91a2f96321f>                172  11%
                        + #<lambda -0x15745e458069cde1>                165  10%
                     + completion-pcm--hilit-commonality                 21   1%

Which are still kind of confusing, but at least kinda point to the time
wasters being in simple.el, i.e. in the "client" code of the completions
code, and _not_ in minibuffer.el, where the completion algorithm itself
lives.

But maybe I'm totally mistaken.  Stefan is probably the best person to
shed some light here.

João

Re: Add user customization fido-completion-styles

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Cc: drew.adams@oracle.com,  emacs-devel@gnu.org, monnier@iro.umontreal.ca
> Date: Tue, 02 Jun 2020 20:11:49 +0100
> 
>                - complete-with-action                          1125  75%
>                   - cond                                         1125  75%
>                    - funcall                                     1125  75%
>                     - all-completions                              24   1%
>                        #<compiled 0x15809d82b069>                  15   1%

This looks like funcall takes most of the time, so it's important to
understand which function is being called here.

>                       - completion-pcm--all-completions               1209  78%
>                        - all-completions                         1209  78%
>                         - #<lambda -0x5acbf46d069cde1>                318  20%
>                          - let                                    318  20%
>                           - complete-with-action                  318  20%
>                            - all-completions                       10   0%
>                             - #<lambda -0xded78e903380374>                  5   0%

Is this real? all-completions calls some lambda which itself calls
all-completions?

>                              - and                                  5   0%
>                                 funcall                             4   0%
>                               - or                                  1   0%
>                                  equal                              1   0%
>                         - #<lambda 0xf61f9dacf96321f>                197  12%
>                          - let                                    197  12%
>                           - complete-with-action                  197  12%
>                            - all-completions                        7   0%
>                             - #<lambda 0x7121b4e1cc7fc8f>                  4   0%
>                              - and                                  4   0%
>                                 funcall                             4   0%
>                         - #<lambda 0xf590d1eef96321f>                180  11%
>                          - let                                    180  11%
>                           - complete-with-action                  180  11%
>                            - all-completions                        5   0%
>                             - #<lambda 0x7018618bcc7fc8f>                  4   0%
>                              - and                                  4   0%
>                                 funcall                             4   0%
>                         + #<lambda 0xf61e0170f96321f>                177  11%
>                         + #<lambda 0xf61d91a2f96321f>                172  11%
>                         + #<lambda -0x15745e458069cde1>                165  10%
>                      + completion-pcm--hilit-commonality                 21   1%

What are all those lambda-functions?

Re: Add user customization fido-completion-styles

[João Távora]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: João Távora <joaotavora@gmail.com>
>> Cc: drew.adams@oracle.com,  emacs-devel@gnu.org, monnier@iro.umontreal.ca
>> Date: Tue, 02 Jun 2020 20:11:49 +0100
>> 
>>                - complete-with-action                          1125  75%
>>                   - cond                                         1125  75%
>>                    - funcall                                     1125  75%
>>                     - all-completions                              24   1%
>>                        #<compiled 0x15809d82b069>                  15   1%
>
> This looks like funcall takes most of the time, so it's important to
> understand which function is being called here.

I think it's something in simple.el, given as the profile changes
completely once I load that file versus loading simple.elc.

>>                       - completion-pcm--all-completions               1209  78%
>>                        - all-completions                         1209  78%
>>                         - #<lambda -0x5acbf46d069cde1>                318  20%
>>                          - let                                    318  20%
>>                           - complete-with-action                  318  20%
>>                            - all-completions                       10   0%
>>                             - #<lambda -0xded78e903380374>                  5   0%
>
> Is this real? all-completions calls some lambda which itself calls
> all-completions?

It might be yes, judging from the docstring, the C implementation of
all-completions, and the evidence we're looking at.

>>                            - all-completions                        5   0%
>>                             - #<lambda 0x7018618bcc7fc8f>                  4   0%
>>                              - and                                  4   0%
>>                                 funcall                             4   0%
>>                         + #<lambda 0xf61e0170f96321f>                177  11%
>>                         + #<lambda 0xf61d91a2f96321f>                172  11%
>>                         + #<lambda -0x15745e458069cde1>                165  10%
>>                      + completion-pcm--hilit-commonality                 21   1%
>
> What are all those lambda-functions?

No idea.  I'd suppose they are the functions who are checking the
predicate against the symbols in obarray, since we want only the symbols
that verify commandp and aren't marked as obsolete.

I've expanded the first two lambdas of the 5 we have in total.  I think
that's the amount of types I pressed 'M-x C-g' waiting for the
completions to show up between the M-x and the C-g.

None of this was with an Emacs -Q, though.  If I use an Emacs -Q I just
have one of those mysterious lambdas, even if I do do the 'M-x
C-g' multiple times.

   - completion--some                         111  42%
    - #<compiled -0x1ed1f57d...                111  42%
     - completion-flex-all-c...                111  42%
      - completion-substring...                104  39%
       - completion-pcm--all...                103  39%
        - all-completions                     102  38%
         - #<lambda -0x1d5fc...                102  38%
          - if                                102  38%
           - let                              102  38%
            - complete-with-action            102  38%
             - all-completions                  8   3%
              - #<lambda 0x3...                  6   2%
               - and                            5   1%

Anyway, I'd wait for Stefan to now, or at least for a comparative study
with the other alternatives that reported instantaneous responses to
M-x, since with Emacs -Q we can also get such a response.

Thanks for your help,
João

RE: Add user customization fido-completion-styles

[Drew Adams]

> > As for performance, for `M-x' there certainly is
> > no problem.  `M-x' with no input pattern to match
> > shows the 6000-some candidates immediately.
> 
> Certainly it doesn't _show_ 6000 candidates, right?
> You must mean it calculates them quickly and shows
> the top-sorting set quickly.  That's great, I guess,
> and I join in asking if Emacs shouldn't do that.

No, it can show them all.  Sure, because of Emacs's
display code, it shows only as many as fit in the
`*Completions* window.  But I'm using a separate
frame for `*Completions*', and I can shrink its
font size to show zillions of them.  Yeah, at a
certain point the font size becomes too small (no
such font).  But there's no problem calculating
and showing lots of candidates.

However, with Icicles flx sorting, the score for
each candidate, when there's no input pattern,
i.e. (flx-score CANDIDATE ""), is nil.

And as I mentioned, when either of the flx scores
is nil the candidates are compared alphabetically.
`flx-score' (from `flx.el') is meaningless for
empty input.  (See my previous mail about this.)

Things are slower when an input pattern is used.
E.g. with `M-x b' I see a pause of a couple sec,
for 864 candidates.  So yeah, flx sorting isn't
super rapid.  But that's OK.  Icicles users can
change sort orders anytime, on the fly.

> I didnt' dig down very intensively, but my naive
> profiling shows the bulk of the work to be in
> `all-completions' or thereabouts to be the
> main sink of cpu time:

I was reporting about Icicles, not icomplete.  But
yes, Icicles uses `completion-all-completions' too
(in this context).

> As you know, all-completions is a C function.
> The 1% is suspicious, I don't know if I should
> trust it.
> 
> Anyway, I'm curious how Icycles evades this.

See above.  I was speaking of the empty-input
case, where `flx-score' returns nil, in which
case the sort predicate falls back to comparing
alphabetically.

Also, Icicles sorts (and displays candidates)
outside the standard completion code.  It sorts
the current set of completions at the end.

> Anyway2, I wasn't trying this in an Emacs -Q.
> When I do, and I set icomplete-compute-delay to 0,
> then M-x is practically instantaneous.
> 
> Maybe the default 0.3 value of that variable is
> excessively conservative.

(FWIW, Icicles's delay for the number of sec to
wait before updating *Completions* incrementally
is 0.7 sec, by default.  But there's no wait if
the number of candidates is =< the value of option
`icicle-incremental-completion-threshold', whose
default value is 1000.)

> So, to summarize: I do believe could be some
> shortcuts for the notable case of no input pattern
> to give a speed boost, maybe showing them them
> their relevant minibuffer history.

Yes, for that notable case.  I'd say no, to using
input-history order.  (And what do you do if two
candidates to compare can't be compared by flx and
one or both is not a previous input?)

In Icicles, one of the available sort orders is
sorting "by last use as input".  A user can
switch to that anytime during completion.  In
general (most cases), that's not a good default
sort order.  It all depends on the user, and on
what kind of candidates s?he's sorting, and why.

That last-use-as-input sort order is defined as:

 Non-nil means S1 was used as input more recently than S2.
 Also:
  S1 < S2 if S1 was used as input previously but S2 was not.
  S1 < S2 if neither was used as input previously
   and S1 `icicle-case-string-less-p' S2.

> I don't believe there's some kind
> of magical speed pickup to be had.

Agreed.  Sorry if I gave a false impression in
that regard.

FWIW, I'm a firm believer in giving users control
over this kind of thing, as opposed to trying to
bake in some optimal combination that an Emacs
developer thinks is a good idea in general, but
without giving users an easy way to override that.

Yes, a given command (given kind of completion
candidates) can call for a given kind of sorting
by default - that default sorting behavior should
be up to the designer of that command.

But past that, a user should be able to choose
the kind of (default) sorting to use, including
for a specific predefined command, and in general.

And beyond that, a user should be able to change
the current sort order on the fly, while completing.
(Icicles allows all of that.)

> Even the C rewrite of parts of
> completion-pcm--all-completions now seems dubious.

I can't speak to that.

> > This way of combining yes-no-maybe predicates is
> > described here:
> > https://www.emacswiki.org/emacs/ApplesAndOranges
> 
> Have you tried `fido-mode` and/or flex?  Can you
> think of specific ways to improve it?  If so,
> patches are very welcome.

No, sorry.  And I'm no expert on flex matching or
fuzzy matching (of any kind).  Icicles lets users
use different kinds of matching, some of which are
provided by other 3rd-party libraries, if you load
them.

> It's a bit harder, I admit, to try to learn Icicles
> and decide which features to migrate.  But if you
> can point some laser-targeted improvement to the
> flex algorithm (in efficiency of functionality)
> are very welcome.

I don't have anything to help wrt flex.  I just
enable Icicles to use library `flx.el' which has
been around for years.

I do the same for other kinds of matching.  See
this doc page:

https://www.emacswiki.org/emacs/Icicles_-_Completion_Methods_and_Styles

These are the matchings that require other
libraries, for use by Icicles:

* Swank fuzzy-symbol matching requires
  `el-swank-fuzzy.el'.

* Fuzzy matching requires `fuzzy-match.el'.

* Jaro-Winkler matching requires `fuzzy.el'
  (in package `autocomplete').

* Levenshtein matching requires `levenshtein.el'.
  (There are 2 kinds of Levenshtein matching.)

The only "fuzzy" matchings I define in Icicles
itself are these:

* Scatter matching (sometimes called "flex"
  elsewhere).  This is just regexp matching
  with `a.*b.*c’

* SPC-scatter matching (some other packages use
  this).  This matches input parts that are
  separated by SPC chars, matching arbitrary
  text at the separations between those parts.
  IOW, it's as if regexp `.*' were inserted in
  place of each substring of SPC chars.

HTH.

Re: Add user customization fido-completion-styles

[Tassilo Horn]

João Távora <joaotavora@gmail.com> writes:

Hi João,

>> Users of fido-mode (like me) may want to customize the completion
>> style it uses, so add a variable to allow them to do so.
>>
>> For instance, I really like using the “initials” style so “vlm”
>> quickly completes visual-line-mode, and then falling back to
>> flex. There are also neat packages like orderless
>> (https://github.com/oantolin/orderless) which add a new
>> completion-style, which folks may want to plug-in and use. Anyway, it
>> seems like it ought to be customizable.
>
> Actually, it is, just not with custom.  These two lines should do the trick:
>
>   (add-hook 'icomplete-minibuffer-setup-hook 'my-flex-styles)
>   (defun my-flex-styles () (setq-local completion-styles '(initials flex)))
>
> Or just use a lambda, which is even shorter.  I'd like to avoid a
> duplicate version of completion-styles customization vars.  

I guess fido-mode sets `completion-styles' locally in order to achieve
ido-like defaults, right?  But as a user with customized
`completion-styles' and `completion-category-overrides', I wish there
was a simple way to just use them with fido, too.  Especially, I can't
see how I could figure out the right category override.  Is the
completion metadata accessible in icomplete-minibuffer-setup-hook?

I would like to use fido (more than icomplete) because the keybindings
suite my intuition but I'd like to keep my completion styles and
overrides as I have them.

Bye,
Tassilo

Re: Add user customization fido-completion-styles

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2379 bytes --]

On Tue, Jun 2, 2020 at 4:40 PM Tassilo Horn <tsdh@gnu.org> wrote:

> >   (add-hook 'icomplete-minibuffer-setup-hook 'my-flex-styles)
> >   (defun my-flex-styles () (setq-local completion-styles '(initials
flex)))
> >
> > Or just use a lambda, which is even shorter.  I'd like to avoid a
> > duplicate version of completion-styles customization vars.
>
> I guess fido-mode sets `completion-styles' locally in order to achieve
> ido-like defaults, right?

Yes, exactly. fido-mode is broadly "choose these nice icomplete-mode
settings that make it fake ido-mode".

> But as a user with customized
> `completion-styles' and `completion-category-overrides', I wish there
> was a simple way to just use them with fido, too.

You can try icomplete-mode directly if you don't like fido-mode's.
fido-mode is a small customization layer on top of it that makes
some choices for you.

> Especially, I can't
> see how I could figure out the right category override.  Is the
> completion metadata accessible in icomplete-minibuffer-setup-hook?

I don't know what you mean.  Exactly what are you trying do to?
Anyway, I'm reasonably confident you can set
completion-category-overrides there.

> I would like to use fido (more than icomplete) because the keybindings
> suite my intuition but I'd like to keep my completion styles and
> overrides as I have them.

If (some of the keybindings) are the only thing that interest you,
you can bind them to icomplete-minibuffer-map.  The only thing
you can't do this way is this bit of behaviour hidden in
`icomplete--sorted-completions'

((and fido-mode
                             (not minibuffer-default)
                             (eq (icomplete--category) 'file))
                        ;; `fido-mode' has some extra file-sorting
                        ;; semantics even if there isn't a default,
                        ;; which is to bubble "./" to the top if it
                        ;; exists.  This makes M-x dired RET RET go to
                        ;; the directory of current file, which is
                        ;; what vanilla Emacs and `ido-mode' both do.
                        `(,(lambda (comp)
                             (string= "./" comp))))

But _that_ could be controlled by an icomplete variable or some
other category technique.

Also, you can argue for some of the defaults in icomplete to be
changed, slowly.

[-- Attachment #2: Type: text/html, Size: 3044 bytes --]

Re: Add user customization fido-completion-styles

[Tassilo Horn]

João Távora <joaotavora@gmail.com> writes:

> On Tue, Jun 2, 2020 at 4:40 PM Tassilo Horn <tsdh@gnu.org> wrote:
>
>> >   (add-hook 'icomplete-minibuffer-setup-hook 'my-flex-styles)
>> >   (defun my-flex-styles () (setq-local completion-styles '(initials
> flex)))
>> >
>> > Or just use a lambda, which is even shorter.  I'd like to avoid a
>> > duplicate version of completion-styles customization vars.
>>
>> I guess fido-mode sets `completion-styles' locally in order to achieve
>> ido-like defaults, right?
>
> Yes, exactly. fido-mode is broadly "choose these nice icomplete-mode
> settings that make it fake ido-mode".
>
>> But as a user with customized `completion-styles' and
>> `completion-category-overrides', I wish there was a simple way to
>> just use them with fido, too.
>
> You can try icomplete-mode directly if you don't like fido-mode's.
> fido-mode is a small customization layer on top of it that makes some
> choices for you.

Yes, I know.  And except for the change in completion-styles, I like it.
Not so much for plain icomplete-mode where the fido keybindings are
missing.

>> Especially, I can't see how I could figure out the right category
>> override.  Is the completion metadata accessible in
>> icomplete-minibuffer-setup-hook?
>
> I don't know what you mean.  Exactly what are you trying do to?
> Anyway, I'm reasonably confident you can set
> completion-category-overrides there.

If I understand the docs correctly, `completion-category-overrides'
override the normal `completion-styles' on a category basis.  I have
these customizations:

--8<---------------cut here---------------start------------->8---
(setq completion-styles '(basic partial-completion substring flex)
      completion-category-overrides
      '((ecomplete
         (styles basic partial-completion substring))
        (buffer
         (styles basic partial-completion substring))
        (unicode-name
         (styles basic partial-completion flex))
        (info-menu
         (styles basic partial-completion flex))
        (project-file
         (styles basic partial-completion substring initials))
        (file
         (styles basic partial-completion substring initials))
        ;; TODO: There's actually no command category (yet).
        (command (styles basic partial-completion substring initials))))
--8<---------------cut here---------------end--------------->8---

So when completing buffers, completion-styles will be

  (basic partial-completion substring)

whereas when completing files it'll be

  (basic partial-completion substring initials)

and when not completing something in the overrides, it'll go with the
normal customized completion-styles of

  (basic partial-completion substring flex).

I don't know when this happens, but it might be that it is before
`icomplete-minibuffer-setup-hook' is called and then setting
`completion-category-overrides' wouldn't have an effect.  I'll test...

Thanks,
Tassilo

Re: Add user customization fido-completion-styles

[João Távora]

Tassilo Horn <tsdh@gnu.org> writes:

>> You can try icomplete-mode directly if you don't like fido-mode's.
>> fido-mode is a small customization layer on top of it that makes some
>> choices for you.
>
> Yes, I know.  And except for the change in completion-styles, I like it.
> Not so much for plain icomplete-mode where the fido keybindings are
> missing.

You can grab all the bindings of fido-mode in one swoop like this, I
think you can put this:

  (use-local-map (make-composed-keymap icomplete-fido-mode-map
                                       (current-local-map)))

Somewhere in your icomplete-minibuffer-setup-hook.

> I don't know when this happens, but it might be that it is before
> `icomplete-minibuffer-setup-hook' is called and then setting
> `completion-category-overrides' wouldn't have an effect.  I'll test...

If it helps, here is a list of the variables that fido-mode sets for
you.

    (setq-local icomplete-tidy-shadowed-file-names t
                icomplete-show-matches-on-no-input t
                icomplete-hide-common-prefix nil
                completion-styles '(flex)
                completion-flex-nospace nil
                completion-category-defaults nil)

It seems to leave `completion-category-overrides' alone, so you should
be able to use to have the last word.  If completion-category-overrides
were to allow `t` entries, or some kind of catch-all, then it would be
the "ultimate" last word.

This would make sense to me, and fido-mode would commit to not touching
that variable explicitly.  I think this would make an(other) elegant
solution for this question, don't you think?

João

Re: Add user customization fido-completion-styles

[Tassilo Horn]

João Távora <joaotavora@gmail.com> writes:

>> Yes, I know.  And except for the change in completion-styles, I like
>> it.  Not so much for plain icomplete-mode where the fido keybindings
>> are missing.
>
> You can grab all the bindings of fido-mode in one swoop like this, I
> think you can put this:
>
>   (use-local-map (make-composed-keymap icomplete-fido-mode-map
>                                        (current-local-map)))
>
> Somewhere in your icomplete-minibuffer-setup-hook.

Indeed, that should also work.

Thanks,
Tassilo

Re: Add user customization fido-completion-styles

[Tassilo Horn]

Hi again,

> I don't know when this happens, but it might be that it is before
> `icomplete-minibuffer-setup-hook' is called and then setting
> `completion-category-overrides' wouldn't have an effect.  I'll test...

Ok, nothing to worry, it seems to work with those configs.

--8<---------------cut here---------------start------------->8---
(defvar th/completion-styles
  '(basic partial-completion substring flex))

(defvar th/completion-category-overrides
  '((ecomplete
     (styles basic partial-completion substring))
    (buffer
     (styles basic partial-completion substring))
    (unicode-name
     (styles basic partial-completion flex))
    (info-menu
     (styles basic partial-completion flex))
    (project-file
     (styles basic partial-completion substring initials))
    (file
     (styles basic partial-completion substring initials))
    ;; TODO: There's actually no command category (yet).
    (command (styles basic partial-completion substring initials))))

(setq read-buffer-completion-ignore-case t
      read-file-name-completion-ignore-case t
      completion-cycle-threshold 3
      completion-styles th/completion-styles
      completion-category-overrides th/completion-category-overrides)

(defun th/icomplete-minibuffer-setup-init ()
  (when fido-mode
    (setq-local completion-styles th/completion-styles)
    (setq-local completion-category-overrides
                th/completion-category-overrides)))

(add-hook 'icomplete-minibuffer-setup-hook
          #'th/icomplete-minibuffer-setup-init)

(fido-mode)
--8<---------------cut here---------------end--------------->8---

Re: Add user customization fido-completion-styles

[João Távora]

Tassilo Horn <tsdh@gnu.org> writes:

> Hi again,
>
>> I don't know when this happens, but it might be that it is before
>> `icomplete-minibuffer-setup-hook' is called and then setting
>> `completion-category-overrides' wouldn't have an effect.  I'll test...
>
> Ok, nothing to worry, it seems to work with those configs.

Just to note that it _should_ also work without special work on the
th/completion-category-overrides var.  Did you test that?

João