bug#47711: 27.1; Deferred highlighting support in `completion-all-completions', `vertico--all-completions`

bug#47711: 27.1; Deferred highlighting support in `completion-all-completions', `vertico--all-completions`

[Daniel Mendler]

Emacs is lacking a possibility to defer the completion highlighting when
computing completions via `completion-all-completions'. This feature is
important for the performance of completion UIs when the set of all
completions is much larger than the set of completions which are
displayed.

The Vertico package defers highlighting by modifying the
`completion*-hilit-*' function with advices.

(declare-function orderless-highlight-matches "ext:orderless")
(defun vertico--all-completions (&rest args)
   "Compute all completions for ARGS with deferred highlighting."
   (cl-letf* ((orig-pcm (symbol-function 
#'completion-pcm--hilit-commonality))
              (orig-flex (symbol-function 
#'completion-flex-all-completions))
              ((symbol-function #'completion-flex-all-completions)
               (lambda (&rest args)
                 ;; Unfortunately for flex we have to undo the deferred 
highlighting, since flex uses
                 ;; the completion-score for sorting, which is applied 
during highlighting.
                 (cl-letf (((symbol-function 
#'completion-pcm--hilit-commonality) orig-pcm))
                   (apply orig-flex args))))
              ;; Defer the following highlighting functions
              (hl #'identity)
              ((symbol-function #'completion-hilit-commonality)
               (lambda (cands prefix &optional base)
                 (setq hl (lambda (x) (nconc 
(completion-hilit-commonality x prefix base) nil)))
                 (and cands (nconc cands base))))
              ((symbol-function #'completion-pcm--hilit-commonality)
               (lambda (pattern cands)
                 (setq hl (lambda (x) (completion-pcm--hilit-commonality 
pattern x)))
                 cands))
              ((symbol-function #'orderless-highlight-matches)
               (lambda (pattern cands)
                 (setq hl (lambda (x) (orderless-highlight-matches 
pattern x)))
                 cands)))
     (cons (apply #'completion-all-completions args) hl)))

This function `vertico--all-completions` returns the list of completions 
and a highlighting
function which can then be used to highlight the completions on the fly.
It is a prototype of how some improved functionality in Emacs could look 
like.

(completion-all-completions STRING TABLE PRED POINT &optional METADATA 
DEFER-HL)

or

(completion-all-completions-defer-hl STRING TABLE PRED POINT &optional 
METADATA)

If DEFER-HL=t, then the function returns the completions and a
highlighting function. One may consider returning a triple of base,
completions and highlighting functions. Internally the completion styles
should be adapted such that they support the deferred highlighting.

It could be that this feature becomes less needed with the introduction
of gccemacs in the future.

bug#47711: Acknowledgement (27.1; Deferred highlighting support in `completion-all-completions', `vertico--all-completions`)

[Daniel Mendler]

Deferred highlighting is also useful for completion-at-point, see the 
ELPA Corfu package, `corfu--all-completions`, which uses the same method 
as Vertico.

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

I'm comparing

   ido-mode
   with ido-ubiquitous-mode (for support for arbitrary completion 
tables), available at 
https://github.com/DarwinAwardWinner/ido-completing-read-plus
   with (setq ido-enable-flex-matching t), of course

versus

  fido-mode
  with
    (setq icomplete-compute-delay 0)
    (setq icomplete-show-matches-on-no-input t)
    (setq icomplete-max-delay-chars 0)

The values chosen for behavior maximally close to ido.

Try something like:

  - Start a session with personal config and a number of loaded packages 
(so that there are a lot of functions defined in obarray)
  - Type 'C-h f'
  - Type 'a', then type 'b'.
  - Delete 'b', type it again, see how quickly you can make the 
completions update.

With ido, the updates seem instant (probably due to some magic in 
ido-completing-read-plus); with fido, there is some lag. Not huge, but 
easy enough to notice.

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> I'm comparing
>
>   ido-mode
>   with ido-ubiquitous-mode (for support for arbitrary completion
>   tables), available at
>   https://github.com/DarwinAwardWinner/ido-completing-read-plus
>   with (setq ido-enable-flex-matching t), of course
>
> versus
>
>  fido-mode
>  with
>    (setq icomplete-compute-delay 0)
>    (setq icomplete-show-matches-on-no-input t)
>    (setq icomplete-max-delay-chars 0)
>
> The values chosen for behavior maximally close to ido.
>
> Try something like:
>
>  - Start a session with personal config and a number of loaded
>    packages (so that there are a lot of functions defined in obarray)
>  - Type 'C-h f'
>  - Type 'a', then type 'b'.
>  - Delete 'b', type it again, see how quickly you can make the
>    completions update.
>
> With ido, the updates seem instant (probably due to some magic in
> ido-completing-read-plus); with fido, there is some lag. Not huge, but
> easy enough to notice.

Thanks for the report.  Before I try reproducing, can you try with
fido-vertical-mode and tell us it if that changes anything?  I think I
remember that skipping some suffix-calculation logic saved on a few
traversals of the big list of symbol completions.

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 05.06.2021 12:35, João Távora wrote:
> Thanks for the report.  Before I try reproducing, can you try with
> fido-vertical-mode and tell us it if that changes anything?  I think I
> remember that skipping some suffix-calculation logic saved on a few
> traversals of the big list of symbol completions.

Why, yes it is. fido-vertical-mode is definitely snappier with such 
settings.

Maybe still not on the level of ido-mode, but at least halfway there, 
compared to the "horizontal" version.

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 05.06.2021 12:35, João Távora wrote:
>> Thanks for the report.  Before I try reproducing, can you try with
>> fido-vertical-mode and tell us it if that changes anything?  I think I
>> remember that skipping some suffix-calculation logic saved on a few
>> traversals of the big list of symbol completions.
>
> Why, yes it is. fido-vertical-mode is definitely snappier with such
> settings.
>
> Maybe still not on the level of ido-mode, but at least halfway there,
> compared to the "horizontal" version.

Yes, that is also my assessment after trying your recipe.  fido-mode +
fido-vertical-mode is not quite as snappy as ido-ubiquitous-mode, but
decently close.

My bet is that the remaining lag is due to sorting.  In a dumb but
illustrative example, when given the pattern 'fmcro' flex-enabled
ido-mode pops 'flymake--backend-state-p--cmacro' to the top, while fido
mode selects the much more reasonable 'defmacro'.

Now, what I called here the "suffix-calculation logic" is what I also
called the "[mplete] dance" back in the emacs-devel thread.  Truth is,
it's always annoyed me in icomplete partially because I don't understand
what it does exactly and how it is supposed to help me.  I suppose
Stefan knows best here.  Regardless of its use, it seems to require
another try-completion call in all the filtered candidates (which might
be very big) so that's probably where the extra lag comes from.

So, in summary, to speed this up for whomever is _not_ using
fido-vertical-mode, either we manage to speed up that part of
icomplete.el, or we get rid of it completely (at least for fido-mode).
For reference, it lives in an "else" branch of one of the "if"s in
icomplete-completions.

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 06.06.2021 02:20, João Távora wrote:
> Now, what I called here the "suffix-calculation logic" is what I also
> called the "[mplete] dance" back in the emacs-devel thread.  Truth is,
> it's always annoyed me in icomplete partially because I don't understand
> what it does exactly and how it is supposed to help me.  I suppose
> Stefan knows best here.  Regardless of its use, it seems to require
> another try-completion call in all the filtered candidates (which might
> be very big) so that's probably where the extra lag comes from.

Shouldn't this logic (whether it's used) be governed by the variable 
icomplete-hide-common-prefix?

Which icomplete--fido-mode-setup sets to nil (appropriately, given than 
ido-mode does not have this behavior). And looking at its behavior, it 
only does the "[mplete] dance" when there is only one match remaining.

Whether to make icomplete-hide-common-prefix affect even the "only one 
match" case is a matter of taste: I could personally find an argument 
for either choice. But we're seeing performance degradation when there 
are many matches, not one.

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 06.06.2021 02:20, João Távora wrote:
> My bet is that the remaining lag is due to sorting.  In a dumb but
> illustrative example, when given the pattern 'fmcro' flex-enabled
> ido-mode pops 'flymake--backend-state-p--cmacro' to the top, while fido
> mode selects the much more reasonable 'defmacro'.

Perhaps not sorting exactly, but the scoring part? Lowering the 
implementation into C might help, we discussed something like that in 
the past.

And/or pick a different algorithm. E.g. Jaro-Winkler, which apparently 
is used in a lot of "fuzzy matching" implementations out there, it's 
pretty fast.

I took an example like

   (setq s (all-completions "" obarray))
   (setq ss (cl-delete-if-not (lambda (s) (string-match-p "a" s)) s))

then

   (benchmark 1 '(completion-all-completions "a" ss nil 1))

prints 0.180s here, whereas a "pure Ruby" implementation of Jaro-Winkler 
takes about 0.060s on the exact same set of strings. But perhaps Ruby is 
just faster than Elisp, I don't have a good comparison.

(The only J-W implementation in Elisp I have found yet -- 
https://github.com/rdiankov/emacs-config/blob/master/.emacs-lisp/auto-complete-1.3.1/fuzzy.el#L70 
-- is slower than the current scoring algo).

bug#48841: fido-mode is slower than ido-mode with similar settings

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Stefan knows best here.  Regardless of its use, it seems to require
> another try-completion call in all the filtered candidates (which might
> be very big) so that's probably where the extra lag comes from.

IIRC the `try-completion` call is performed on the list of possible
completions rather than on the original completion table, so it should
be quite fast.  I'd be surprised if it is a significant portion of the
overall time.


        Stefan

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 06.06.2021 02:20, João Távora wrote:
>> My bet is that the remaining lag is due to sorting.  In a dumb but
>> illustrative example, when given the pattern 'fmcro' flex-enabled
>> ido-mode pops 'flymake--backend-state-p--cmacro' to the top, while fido
>> mode selects the much more reasonable 'defmacro'.
>
> Perhaps not sorting exactly, but the scoring part? Lowering the
> implementation into C might help, we discussed something like that in
> the past.

Perhaps, all could be measured.  But I also remember explaining that
scoring is basically free.  The flex algorithm search algorithm is
greedy already, it doesn't backtrack.  Given a pattern 'foo' against
'fabrobazo', it takes 9 steps to see that it matches.  I can't see any
other way to improve that, short of a something like a tottaly different
string implementation.  The scoring's numeric calculations at each step
are trivial.

One way to verify this is to do the scoring, but simply disregard it for
sorting purposes.

> And/or pick a different algorithm. E.g. Jaro-Winkler, which apparently
> is used in a lot of "fuzzy matching" implementations out there, it's
> pretty fast.

That may be useful, but for other purposes.  If I understand correctly,
Jaro-Winkler is for finding the distante between two arbitrary strings,
where the first in not a subsequence of the second.  I bet google uses
stuff like that when you accitendally transpose characters.  Flex just
gives up.  Those other others algos still catch the match (and Google
than probably NL-scours your most intimate fears and checks with your
local dictator before showing you typing suggestions)

> I took an example like
>
>   (setq s (all-completions "" obarray))
>   (setq ss (cl-delete-if-not (lambda (s) (string-match-p "a" s)) s))
>
> then
>
>   (benchmark 1 '(completion-all-completions "a" ss nil 1))
>
> prints 0.180s here, whereas a "pure Ruby" implementation of
> Jaro-Winkler takes about 0.060s on the exact same set of strings. But
> perhaps Ruby is just faster than Elisp, I don't have a good
> comparison.

Go ahead and kill the scoring calculationg altogether in
completion-pcm--hilit-commonality.  I bet it won't make a difference.
If fact, for that experiment, try a simple substring search.  I bet
you're just seeing an inferior GC at work, or a string implementation
that's made optimized for other stuff that Ruby's can't, like
propertization.  Try making Ruby strings that mimic Elips if you've time
to spare...

> (The only J-W implementation in Elisp I have found yet --
> https://github.com/rdiankov/emacs-config/blob/master/.emacs-lisp/auto-complete-1.3.1/fuzzy.el#L70
> -- is slower than the current scoring algo).

There you have it.

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Stefan knows best here.  Regardless of its use, it seems to require
>> another try-completion call in all the filtered candidates (which might
>> be very big) so that's probably where the extra lag comes from.
>
> IIRC the `try-completion` call is performed on the list of possible
> completions rather than on the original completion table, so it should
> be quite fast.  I'd be surprised if it is a significant portion of the
> overall time.

Very true, but here's the suprise: In the flex style, there are a _lot_
of "possible completions" for the null or very short patterns.  So those
calculations -- which were more than certainly thought up for prefix-ish
styles -- are quite slow (and also quite useless for flex).  At least
that's my theory.

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 06.06.2021 09:59, João Távora wrote:
> Very true, but here's the suprise: In the flex style, there are a_lot_
> of "possible completions" for the null or very short patterns.  So those
> calculations -- which were more than certainly thought up for prefix-ish
> styles -- are quite slow (and also quite useless for flex).  At least
> that's my theory.

try-completion doesn't trigger any completion style machinery; only 
completion-try-completion does.

And are we talking about the 'try-completion' call which is guarded with 
(when icomplete-hide-common-prefix ...)?

icomplete--fido-mode-setup sets that variable to nil.

bug#48841: fido-mode is slower than ido-mode with similar settings

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Very true, but here's the suprise: In the flex style, there are a _lot_
> of "possible completions" for the null or very short patterns.  So those
> calculations -- which were more than certainly thought up for prefix-ish
> styles -- are quite slow (and also quite useless for flex).  At least
> that's my theory.

In the very worst possible case, `try-completion` will be just as slow
as the original computation of the set of possible completions.  So at
most it will double the total time (and this assumes we do basically
nothing else than a single call to `all-completions` to get the set of
candidates and then display them).
In practice I'd be surprised if it ever reaches the 20% mark of the time spent.


        Stefan

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

On Sun, Jun 6, 2021, 17:55 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 06.06.2021 09:59, João Távora wrote:
> > Very true, but here's the suprise: In the flex style, there are a_lot_
> > of "possible completions" for the null or very short patterns.  So those
> > calculations -- which were more than certainly thought up for prefix-ish
> > styles -- are quite slow (and also quite useless for flex).  At least
> > that's my theory.
>
> try-completion doesn't trigger any completion style machinery; only
> completion-try-completion does.
>

I have no idea if completion style stuff b is related. Just that else
branch is there to calculate some 'determ' thing and a cursory look
revealed try-completion calls being passed 'comps', or 'completions'.
Presumably lots of data given short flex style patterns. No idea what it
accomplishes, as I said.

Bottom line is that something (TM) happened to speed up the whole thing
when I skipped over that whole part. I had vertical mode basically visually
equivalent to vertical, but quite slower.  After skipping that part they
became practically equivalent. And you yourself witnessed this when
switching yo vertical mode, which is when the skip is made.

 I'll check later in the week, away from my computer now.

And are we talking about the 'try-completion' call which is guarded with
> (when icomplete-hide-common-prefix ...)?
>

No idea

>
> icomplete--fido-mode-setup sets that variable to nil.
>

May be.

João

>

[-- Attachment #2: Type: text/html, Size: 2650 bytes --]

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 446 bytes --]

On Sun, Jun 6, 2021, 18:55 Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> In practice I'd be surprised if it ever reaches the 20% mark of the time
> spent.


Personally, I'm usually suprised if less than 80% of my estimates aren't
totally off. :) Anyway, if not try-completion like I theorized, it should
be reasonably easy to pinpoint: something non-fido-essential in that else
branch is causing a real slowdown.

João

>
>

[-- Attachment #2: Type: text/html, Size: 973 bytes --]

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 06.06.2021 09:54, João Távora wrote:

>> Perhaps not sorting exactly, but the scoring part? Lowering the
>> implementation into C might help, we discussed something like that in
>> the past.
> 
> Perhaps, all could be measured.  But I also remember explaining that
> scoring is basically free.  The flex algorithm search algorithm is
> greedy already, it doesn't backtrack.  Given a pattern 'foo' against
> 'fabrobazo', it takes 9 steps to see that it matches.  I can't see any
> other way to improve that, short of a something like a tottaly different
> string implementation.  The scoring's numeric calculations at each step
> are trivial.

Thanks, if it's indeed that fast, I might have a use for it as well, in 
a different context. ;-)

>> And/or pick a different algorithm. E.g. Jaro-Winkler, which apparently
>> is used in a lot of "fuzzy matching" implementations out there, it's
>> pretty fast.
> 
> That may be useful, but for other purposes.  If I understand correctly,
> Jaro-Winkler is for finding the distante between two arbitrary strings,
> where the first in not a subsequence of the second.  I bet google uses
> stuff like that when you accitendally transpose characters.  Flex just
> gives up.  Those other others algos still catch the match (and Google
> than probably NL-scours your most intimate fears and checks with your
> local dictator before showing you typing suggestions)

I'm not crazy about shuffling completion, but some people did indeed ask 
for it. The filtering step has to be more complex, though (you can't 
just construct a straightforward regexp to filter with).

>> I took an example like
>>
>>    (setq s (all-completions "" obarray))
>>    (setq ss (cl-delete-if-not (lambda (s) (string-match-p "a" s)) s))
>>
>> then
>>
>>    (benchmark 1 '(completion-all-completions "a" ss nil 1))
>>
>> prints 0.180s here, whereas a "pure Ruby" implementation of
>> Jaro-Winkler takes about 0.060s on the exact same set of strings. But
>> perhaps Ruby is just faster than Elisp, I don't have a good
>> comparison.
> 
> Go ahead and kill the scoring calculationg altogether in
> completion-pcm--hilit-commonality.  I bet it won't make a difference.
> If fact, for that experiment, try a simple substring search.

Same result, indeed. We should note, though, that 
completion-pcm--hilit-commonality has some steps that were added 
together with the 'flex' style, for it to work.

> I bet
> you're just seeing an inferior GC at work, or a string implementation
> that's made optimized for other stuff that Ruby's can't, like
> propertization.  Try making Ruby strings that mimic Elips if you've time
> to spare...

I did some instrumenting, replacing (completion-pcm--hilit-commonality 
pattern all) inside completion-flex-all-completions with 
(benchmark-progn (completion-pcm--hilit-commonality pattern all)). 
Recompiled between each change (interpreted mode gives very different 
numbers).

Unmodified, the call takes ~85ms:

   Elapsed time: 0.085520s (0.068406s in 4 GCs)

If I comment everything inside its first lambda except the returned 
value (making the function the same as #'identity), the time goes down 
to <1ms.

Uncomment the 'copy-sequence' and 'string-match' calls (which one might 
suspect of garbage generation):

   Elapsed time: 0.006380s

Tried binding gc-cons-threshold to a high value, and even galling 
garbage-collect-maybe (or not): that speeds it up, but adds some 
unpredictable GC pauses later (though it would be nice to be able to 
consolidate the pauses into one collection pass).

Long story short, the patch I just installed, to reuse the match data, 
brings the runtime down to

   Elapsed time: 0.066388s (0.050087s in 3 GCs)

Tried other things like moving the update-score-and-face lambda out of 
the mapcar loop - that didn't move a needle. Someone more familiar with 
the code might get further. But perhaps it's just the cost of executing 
this logic in Lisp 12000 times, and doing some of that in C would be the 
next step.

And a weird part: replacing all repeated (length str) calls with a 
reference to an existing local binding makes it *slower* (back to the 
original performance). Check this out:

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index d5a0118b7c..d7102245a2 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3544,7 +3544,7 @@ completion-pcm--hilit-commonality
                      score-numerator   (+ score-numerator (- b a)))
                     (unless (or (= a last-b)
                                 (zerop last-b)
-                               (= a (length str)))
+                               (= a end))
                       (setq
                        score-denominator (+ score-denominator
                                             1
@@ -3562,12 +3562,12 @@ completion-pcm--hilit-commonality
             ;; for that extra bit of match (bug#42149).
             (unless (= from match-end)
               (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
+           (if (> end pos)
                 (add-face-text-property
                  pos (1+ pos)
                  'completions-first-difference
                  nil str))
-           (unless (zerop (length str))
+           (unless (zerop end)
               (put-text-property
                0 1 'completion-score
                (/ score-numerator (* end (1+ score-denominator)) 1.0) 
str)))
@@ -3980,7 +3980,7 @@ completion-flex-all-completions
                    string table pred point
                    #'completion-flex--make-flex-pattern)))
        (when all
-        (nconc (completion-pcm--hilit-commonality pattern all)
+        (nconc (benchmark-progn (completion-pcm--hilit-commonality 
pattern all))
                 (length prefix))))))

  ;; Initials completion

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 06.06.2021 21:37, João Távora wrote:
> Bottom line is that something (TM) happened to speed up the whole thing 
> when I skipped over that whole part. I had vertical mode basically 
> visually equivalent to vertical, but quite slower.  After skipping that 
> part they became practically equivalent. And you yourself witnessed this 
> when switching yo vertical mode, which is when the skip is made.

Yep.

>   I'll check later in the week, away from my computer now.

Looking forward to it.

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

On Sun, Jun 6, 2021, 23:21 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 06.06.2021 21:37, João Távora wrote:
> > Bottom line is that something (TM) happened to speed up the whole thing
> > when I skipped over that whole part. I had vertical mode basically
> > visually equivalent to vertical, but quite slower.  After skipping that
> > part they became practically equivalent. And you yourself witnessed this
> > when switching yo vertical mode, which is when the skip is made.
>
> Yep.
>

By the way, earlier I meant to write:

"I had vertical mode basically visually equivalent to vertico.el, but quite
slower.  After skipping that part they became practically equivalent. "

 autocorrect on my phone had other ideas...

João

[-- Attachment #2: Type: text/html, Size: 1637 bytes --]

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3743 bytes --]

On Sun, Jun 6, 2021, 23:20 Dmitry Gutov <dgutov@yandex.ru> wrote:

>
> >> And/or pick a different algorithm. E.g. Jaro-Winkler, which apparently
> >> is used in a lot of "fuzzy matching" implementations out there, it's
> >> pretty fast.
> >
> > That may be useful, but for other purposes.  If I understand correctly,
> > Jaro-Winkler is for finding the distante between two arbitrary strings,
> > where the first in not a subsequence of the second.  I bet google uses
> > stuff like that when you accitendally transpose characters.  Flex just
> > gives up.  Those other others algos still catch the match (and Google
> > than probably NL-scours your most intimate fears and checks with your

> local dictator before showing you typing suggestions)
>

Meant to write ML as in machine learning, not NL.

I'm not crazy about shuffling completion, but some people did indeed ask
> for it. The filtering step has to be more complex, though (you can't
> just construct a straightforward regexp to filter with).
>

I think you calculate the distance using one of these fancy multi-surname
algorithms , then do cutoff somewhere.

Same result, indeed. We should note, though, that
> completion-pcm--hilit-commonality has some steps that were added
> together with the 'flex' style, for it to work.
>

But nothing algorithmically aberrant, I think.

I did some instrumenting, replacing (completion-pcm--hilit-commonality
> pattern all) inside completion-flex-all-completions with
> (benchmark-progn (completion-pcm--hilit-commonality pattern all)).
> Recompiled between each change (interpreted mode gives very different
> numbers).
>
> Unmodified, the call takes ~85ms:
>
>    Elapsed time: 0.085520s (0.068406s in 4 GCs)
>


By the way, I think you should be running benchmarks multiple times to get
times in the seconds range, and reduce noise. Multiple levels of CPU cache
and other factors like temp thottling may skew results when running just
one time.

If I comment everything inside its first lambda except the returned
> value (making the function the same as #'identity), the time goes down
> to <1ms.
>
> Uncomment the 'copy-sequence' and 'string-match' calls (which one might
> suspect of garbage generation):
>
>    Elapsed time: 0.006380s
>
> Tried binding gc-cons-threshold to a high value, and even galling
> garbage-collect-maybe (or not): that speeds it up, but adds some
> unpredictable GC pauses later (though it would be nice to be able to
> consolidate the pauses into one collection pass).
>

Maybe in Elisp that's a good idea, in other lisps and other languages,
second-guessing the GC is a bad idea. I hear ours is so basic that indeed
it might be reasonable.

Long story short, the patch I just installed, to reuse the match data,
> brings the runtime down to
>
>    Elapsed time: 0.066388s (0.050087s in 3 GCs)
>

That's nice! but are you sure you're not seeing noise, too?

Tried other things like moving the update-score-and-face lambda out of
> the mapcar loop - that didn't move a needle.


If a lambda is non capturing of stuff inside the loop, only one copy of it
is ever made, I think. So it doesn't suprise me.

And a weird part: replacing all repeated (length str) calls with a
> reference to an existing local binding makes it *slower* (back to the
> original performance).


Might be noise, or you might be thrashing of CPU caches, who knows? If the
string length is on the same cache line as the contents of the string
you're reading, then evicting that to go read the value of a boxed integer
somewhere radically different is slow. Just speculation of course. Might
just be noise or something else entirely.

João

[-- Attachment #2: Type: text/html, Size: 6158 bytes --]

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 07.06.2021 02:49, João Távora wrote:

>     Same result, indeed. We should note, though, that
>     completion-pcm--hilit-commonality has some steps that were added
>     together with the 'flex' style, for it to work.
> 
> 
> But nothing algorithmically aberrant, I think.

Just some stuff adding work to GC, I think.

> By the way, I think you should be running benchmarks multiple times to 
> get times in the seconds range, and reduce noise. Multiple levels of CPU 
> cache and other factors like temp thottling may skew results when 
> running just one time.

Yeah, I repeat the action with each version for like a few dozen times, 
until I see the numbers stabilize, or just take the average.

>     Tried binding gc-cons-threshold to a high value, and even galling
>     garbage-collect-maybe (or not): that speeds it up, but adds some
>     unpredictable GC pauses later (though it would be nice to be able to
>     consolidate the pauses into one collection pass).
> 
> 
> Maybe in Elisp that's a good idea, in other lisps and other languages, 
> second-guessing the GC is a bad idea. I hear ours is so basic that 
> indeed it might be reasonable.

I never get good results with that.

>     Long story short, the patch I just installed, to reuse the match data,
>     brings the runtime down to
> 
>         Elapsed time: 0.066388s (0.050087s in 3 GCs)
> 
> 
> That's nice! but are you sure you're not seeing noise, too?

Pretty sure.

>     Tried other things like moving the update-score-and-face lambda out of
>     the mapcar loop - that didn't move a needle.
> 
> 
> If a lambda is non capturing of stuff inside the loop, only one copy of 
> it is ever made, I think. So it doesn't suprise me.

update-score-and-face references both variables in its closest binding 
form (score-numerator, score-denominator) and the parameter of its 
containing lambda (str).

Maybe moving all of them to parameters and return values (making it a 
static function and having the caller manage state) would help, I 
haven't tried that exactly.

>     And a weird part: replacing all repeated (length str) calls with a
>     reference to an existing local binding makes it *slower* (back to the
>     original performance).
> 
> 
> Might be noise, or you might be thrashing of CPU caches, who knows? If 
> the string length is on the same cache line as the contents of the 
> string you're reading, then evicting that to go read the value of a 
> boxed integer somewhere radically different is slow.

But the string value is boxed as well, right? So the code needs to 
follow one indirection either way. Perhaps there's also overhead in 
looking up the lexical scope.

I also tried using the new-and-shiny length> and length=. This simply 
made no measurable difference.

> Just speculation of 
> course. Might just be noise or something else entirely.

This is highly reproducible. On my machine, at least.

Given how weird it is, I wouldn't just write about it without 
recompiling, restarting Emacs and measuring the scenario several times, 
with different versions of code.

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1908 bytes --]

On Mon, Jun 7, 2021, 01:11 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 07.06.2021 02:49, João Távora wrote:
>
> >     Same result, indeed. We should note, though, that
> >     completion-pcm--hilit-commonality has some steps that were added
> >     together with the 'flex' style, for it to work.
> >
> >
> > But nothing algorithmically aberrant, I think.
>
> Just some stuff adding work to GC, I think.
>

O(n) stuff being a property and a number on each string, small in
comparison to the string.

Maybe moving all of them to parameters and return values (making it a
> static function and having the caller manage state) would help, I
> haven't tried that exactly.
>

Normally, in those adventures you end up with the same allocations
somewhere else, and uglier code. But you can try.

> Might be noise, or you might be thrashing of CPU caches, who knows? If
> > the string length is on the same cache line as the contents of the
> > string you're reading, then evicting that to go read the value of a
> > boxed integer somewhere radically different is slow.
>
> But the string value is boxed as well, right?


The key is locality. If the string length and data happen to live nearby in
memory (in the same box, so to speak), there's a decent chance that reading
one brings the other into the cache, and you get a nice hit depending on
your subsequent operation.

Here I'm just speculating, as I said. In managed languages such as Lisps,
it's somewhat unpredictable. It's also always hardware dependent. Though
given C/C++, a known processor and the right application, this will make a
world of a difference, and will yield truly "weird" results (which arent
weird at all after you understand the logic). Like, for example a vector
being much better at sorted insertion than a linked list. (!) Look it up.
Bjarne Stroustrup has one of those talks.

João

[-- Attachment #2: Type: text/html, Size: 2915 bytes --]

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 3655 bytes --]

On 07.06.2021 11:52, João Távora wrote:

>     Maybe moving all of them to parameters and return values (making it a
>     static function and having the caller manage state) would help, I
>     haven't tried that exactly.
> 
> 
> Normally, in those adventures you end up with the same allocations 
> somewhere else, and uglier code. But you can try.

I have it a try, with little success (patch attached, for posterity). 
Since there's no multiple value returns in Lisp, I had to define a 
container for the three values.

The performance is basically the same, which seems to indicate that 
either Elisp has to allocate very little for a compiled lambda code, or 
it's optimized out (which would also make sense: the only thing 
necessary for it is a new container for the current scope).

>      > Might be noise, or you might be thrashing of CPU caches, who
>     knows? If
>      > the string length is on the same cache line as the contents of the
>      > string you're reading, then evicting that to go read the value of a
>      > boxed integer somewhere radically different is slow.
> 
>     But the string value is boxed as well, right? 
> 
> 
> The key is locality. If the string length and data happen to live nearby 
> in memory (in the same box, so to speak), there's a decent chance that 
> reading one brings the other into the cache, and you get a nice hit 
> depending on your subsequent operation.
> 
> Here I'm just speculating, as I said. In managed languages such as 
> Lisps, it's somewhat unpredictable. It's also always hardware dependent.
> Though given C/C++, a known processor and the right application, this 
> will make a world of a difference, and will yield truly "weird" results 
> (which arent weird at all after you understand the logic). Like, for 
> example a vector being much better at sorted insertion than a linked 
> list. (!) Look it up. Bjarne Stroustrup has one of those talks.

When you have to do some work, better memory locality can indeed change 
a lot. But in this case we have an already computed value vs. something 
the code still needs to compute, however fast that is.

Accessing function arguments must be currently much faster than looking 
up the current scope defined with 'let'.

Anyway, looking at what else could be removed, now that the extra 
allocation in 'match-data' is gone, what really speeds it up 2x-11x 
(depending on whether GC kicks in, but it more often doesn't), is 
commenting out the line:

   (setq str (copy-sequence str))

So if it were possible to rearrange completion-pcm--hilit-commonality 
not to have to modify the strings (probably removing the function 
altogether?), that would improve the potential performance of c-a-p-f 
quite a bit, for fido-mode and other frontends (depending on how much 
overhead the other layers add).

Ultimately, the scoring information doesn't have to live in the text 
properties. For sorting, the frontend could allocate a hash table, then 
ask the [backends? styles?] for completion scores on each item and sort 
based on that. Since faces are needed only for the completions that are 
currently displayed, even having to repeat the regexp matching stuff for 
each of them later would be no big deal performance-wise, compared to 
the current approach.

Anyway, these are musing for the much-discussed future iteration of the 
API. With the current version, and tied by backward compatibility, it 
might be possible to wring 10ms of improvement by consolidating text 
property changes somehow, but likely no more than that.

Looking forward for your analysis of fido-vertical-mode's performance 
improvement over the "normal" one.

[-- Attachment #2: completion-pcm-score-struct.diff --]
[-- Type: text/x-patch, Size: 4525 bytes --]

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index d5a0118b7c..0cb247fc19 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3485,6 +3485,7 @@ completion-pcm--hilit-commonality
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
            (point-idx (completion-pcm--pattern-point-idx pattern))
            (case-fold-search completion-ignore-case)
+           (score (make-vector 3 0))
            last-md)
       (mapcar
        (lambda (str)
@@ -3531,37 +3532,17 @@ completion-pcm--hilit-commonality
                 ;;    (SUM_across_i(hole_i_contrib) + 1) * len
                 ;;
                 ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
+                )
+           (fillarray score 0)
            (while md
-             (funcall update-score-and-face from (pop md))
+             (completion-pcm--update-score-and-face str from (pop md) score)
              (setq from (pop md)))
            ;; If `pattern' doesn't have an explicit trailing any, the
            ;; regex `re' won't produce match data representing the
            ;; region after the match.  We need to account to account
            ;; for that extra bit of match (bug#42149).
            (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
+             (completion-pcm--update-score-and-face str from match-end score))
            (if (> (length str) pos)
                (add-face-text-property
                 pos (1+ pos)
@@ -3570,10 +3551,35 @@ completion-pcm--hilit-commonality
            (unless (zerop (length str))
              (put-text-property
               0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
+              (/ (completion-pcm--score-numerator score)
+                 (* end (1+ (completion-pcm--score-denominator score)))
+                 1.0)
+              str)))
          str)
        completions))))
 
+(cl-defstruct (completion-pcm--score (:type vector))
+  (numerator 0) (denominator 0) (last-b 0))
+
+(defun completion-pcm--update-score-and-face (str a b score)
+  "Update score and face in STR given match range (A B)."
+  (add-face-text-property a b
+                          'completions-common-part
+                          nil str)
+  (let ((last-b (completion-pcm--score-last-b score)))
+    (setf (completion-pcm--score-numerator score)
+          (+ (completion-pcm--score-numerator score) (- b a)))
+    (unless (or (= a last-b)
+                (zerop last-b)
+                (= a (length str)))
+      (setf (completion-pcm--score-denominator score)
+            (+ (completion-pcm--score-denominator score)
+               1
+               (expt (- a last-b 1)
+                     (/ 1.0
+                        flex-score-match-tightness)))))
+    (setf (completion-pcm--score-last-b score) b)))
+
 (defun completion-pcm--find-all-completions (string table pred point
                                                     &optional filter)
   "Find all completions for STRING at POINT in TABLE, satisfying PRED.
@@ -3980,7 +3986,7 @@ completion-flex-all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
       (when all
-        (nconc (completion-pcm--hilit-commonality pattern all)
+        (nconc (benchmark-progn (completion-pcm--hilit-commonality pattern all))
                (length prefix))))))
 
 ;; Initials completion

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 07.06.2021 11:52, João Távora wrote:
>
>>     Maybe moving all of them to parameters and return values (making it a
>>     static function and having the caller manage state) would help, I
>>     haven't tried that exactly.
>> Normally, in those adventures you end up with the same allocations
>> somewhere else, and uglier code. But you can try.
>
> I have it a try, with little success (patch attached, for
> posterity). Since there's no multiple value returns in Lisp, I had to
> define a container for the three values.

And if there were multiple value, you can bet the container for them
wouldn't be free ;-)

> The performance is basically the same, which seems to indicate that
> either Elisp has to allocate very little for a compiled lambda code,
> or it's optimized out (which would also make sense: the only thing
> necessary for it is a new container for the current scope).

Which lambda are we talking about?  Is it ` update-score-and-face`?  If
so, I would guess that the capture of `score-denominator` is what takes
space, and that space is no bigger than another variable in that let
scope.

>> Though given C/C++, a known processor and the right application,
>> this will make a world of a difference, and will yield truly "weird"
>> results (which arent weird at all after you understand the
>> logic). Like, for example a vector being much better at sorted
>> insertion than a linked list. (!) Look it up. Bjarne Stroustrup has
>> one of those talks.
> When you have to do some work, better memory locality can indeed
> change a lot. But in this case we have an already computed value
> vs. something the code still needs to compute, however fast that is.

But `length` of a string, in any sane string implementation, _is_
accessing "an already computed value".  Which likely lives just besides
the data.  In Emacs, it seems to be two pointers (8 bytes) apart from
the data.  In a system with 64bytes of L1/2/3 cache it still
theoretically makes up to 52 bytes come in "for free" after you read the
length.  But to be honest I tried a bit and haven't come up with
benchmarks to help confirm -- or help dispel -- this theory.  Maybe you
can distill your "weird" experiment down to a code snippet?

> Accessing function arguments must be currently much faster than
> looking up the current scope defined with 'let'.

In a compiled CL system, I would expect the former to use the stack, and
the to use the heap, but it wouldn't make any difference in reading the
variable's value, I think.  But Elisp is byte-compiled, not natively
compiled (except for that thing now, haven't tried it), and i don't
understand how the byte-compiler chooses byte-codes so all bets are off.

> Anyway, looking at what else could be removed, now that the extra
> allocation in 'match-data' is gone, what really speeds it up 2x-11x
> (depending on whether GC kicks in, but it more often doesn't), is
> commenting out the line:
>
>   (setq str (copy-sequence str))
>
> So if it were possible to rearrange completion-pcm--hilit-commonality
> not to have to modify the strings (probably removing the function
> altogether?), that would improve the potential performance of c-a-p-f
> quite a bit, for fido-mode and other frontends (depending on how much
> overhead the other layers add).

Very interesting.  I don't know what the matter is with modifying the
string itself.  Is it because we want to protect its 'face' property?
Maybe, but what's the harm in chaning it?  Maybe Stefan knows.  Stefan,
are you reading this far?

If we do want to protect the shared 'face' property -- and only 'face'
-- then we could very add some other property about face that the
frontend could read "just in time" before it itself makes a copy of the
string to display to the user.  

This technique appears to be slightly simpler than using the hash-table
indirection you propose (we would need something like that if, for some
reason, we absolutely could not touch the string's property list.)

> Anyway, these are musing for the much-discussed future iteration of
> the API. With the current version, and tied by backward compatibility,

Maybe I'm missing something, but I don't see why my above idea requires
changing _that_ much of the API (a bit would change yes).  It's a matter
of letting frontends opt-out of the current readily-available
face-propertized completions and opt-into a display-time facility that
does this propertization. 

But if the speedup is big, I'd revisit the rationale for requiring those
copies to be performed in the first place.  In my (very brief) testing
it doesn't hurt a bit to remove it.

> Looking forward for your analysis of fido-vertical-mode's performance
> improvement over the "normal" one.

Will take a look now.

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 11.06.2021 20:09, João Távora wrote:

>> The performance is basically the same, which seems to indicate that
>> either Elisp has to allocate very little for a compiled lambda code,
>> or it's optimized out (which would also make sense: the only thing
>> necessary for it is a new container for the current scope).
> 
> Which lambda are we talking about?  Is it ` update-score-and-face`?  If
> so, I would guess that the capture of `score-denominator` is what takes
> space, and that space is no bigger than another variable in that let
> scope.

Yes, that one.

> But `length` of a string, in any sane string implementation, _is_
> accessing "an already computed value".  Which likely lives just besides
> the data.  In Emacs, it seems to be two pointers (8 bytes) apart from
> the data.  In a system with 64bytes of L1/2/3 cache it still
> theoretically makes up to 52 bytes come in "for free" after you read the
> length.  But to be honest I tried a bit and haven't come up with
> benchmarks to help confirm -- or help dispel -- this theory.  Maybe you
> can distill your "weird" experiment down to a code snippet?

I've tried reproducing the effect with a small snippet, and failed. :-(

>> Anyway, looking at what else could be removed, now that the extra
>> allocation in 'match-data' is gone, what really speeds it up 2x-11x
>> (depending on whether GC kicks in, but it more often doesn't), is
>> commenting out the line:
>>
>>    (setq str (copy-sequence str))
>>
>> So if it were possible to rearrange completion-pcm--hilit-commonality
>> not to have to modify the strings (probably removing the function
>> altogether?), that would improve the potential performance of c-a-p-f
>> quite a bit, for fido-mode and other frontends (depending on how much
>> overhead the other layers add).
> 
> Very interesting.  I don't know what the matter is with modifying the
> string itself.  Is it because we want to protect its 'face' property?
> Maybe, but what's the harm in chaning it?

I imagine it's just a "correctness" thing. Text properties are part of 
the string's identity. We add text properties, so we make a copy because 
we don't own the original list (it might be saved to some constant and 
also used for, I don't know, IMenu items?)

> If we do want to protect the shared 'face' property -- and only 'face'
> -- then we could very add some other property about face that the
> frontend could read "just in time" before it itself makes a copy of the
> string to display to the user.

Yes, it's an option (though a less elegant one): apply some namespaced 
text properties with the necessary data. And then we'd also be able to 
fontify "just in time".

Do we have any "frozen strings" in Emacs, which absolutely could not be 
modified? Do we plan to?

> This technique appears to be slightly simpler than using the hash-table
> indirection you propose (we would need something like that if, for some
> reason, we absolutely could not touch the string's property list.)

I disagree it's a simpler technique, but it would indeed be a simpler 
change, based on the current implementation.

>> Anyway, these are musing for the much-discussed future iteration of
>> the API. With the current version, and tied by backward compatibility,
> 
> Maybe I'm missing something, but I don't see why my above idea requires
> changing _that_ much of the API (a bit would change yes).  It's a matter
> of letting frontends opt-out of the current readily-available
> face-propertized completions and opt-into a display-time facility that
> does this propertization.

Even your version is a breaking enough change to be a pain, but possibly 
not beneficial enough to bother all consumers with, until we also add 
some more awaited features, I guess.

But I don't mind it myself, and happy to update Company. Either way it's 
a step forward.

> But if the speedup is big, I'd revisit the rationale for requiring those
> copies to be performed in the first place.

With fido-vertical-mode, and with that particular input, it's

   Elapsed time: 0.130773s (0.031547s in 1 GCs)

without copy-sequence, and

   Elapsed time: 0.169842s (0.069740s in 4 GCs)

with it. Not game changing, but definitely measurable.

> In my (very brief) testing
> it doesn't hurt a bit to remove it.

Same. But it likely depends on where the strings came from. In the most 
usual case, of course, they are created at runtime.

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 12.06.2021 01:34, Dmitry Gutov wrote:
> With fido-vertical-mode, and with that particular input, it's
> 
>    Elapsed time: 0.130773s (0.031547s in 1 GCs)
> 
> without copy-sequence, and
> 
>    Elapsed time: 0.169842s (0.069740s in 4 GCs)
> 
> with it. Not game changing, but definitely measurable.

it = icomplete-completions' runtime

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Looking forward for your analysis of fido-vertical-mode's performance
> improvement over the "normal" one.

So, I benchmarked before and after this patch to icomplete.el:

    diff --git a/lisp/icomplete.el b/lisp/icomplete.el
    index 08b4ef2030..3561ebfa04 100644
    --- a/lisp/icomplete.el
    +++ b/lisp/icomplete.el
    @@ -858,16 +858,8 @@ icomplete-completions
                    ;; removing making `comps' a proper list.
                    (base-size (prog1 (cdr last)
                                 (if last (setcdr last nil))))
    -               (most-try
    -                (if (and base-size (> base-size 0))
    -                    (completion-try-completion
    -                     name candidates predicate (length name) md)
    -                  ;; If the `comps' are 0-based, the result should be
    -                  ;; the same with `comps'.
    -                  (completion-try-completion
    -                   name comps nil (length name) md)))
    -               (most (if (consp most-try) (car most-try)
    -                       (if most-try (car comps) "")))
    +               (most-try nil)
    +               (most "")
                    ;; Compare name and most, so we can determine if name is
                    ;; a prefix of most, or something else.
                    (compare (compare-strings name nil nil

The patch itself nullifies the calculation of the 'determ' thing that I
and presumably some other users don't value that much.  It doesn't
affect fido-mode's basic funcionality.

How did I benchmark?  Well, to measure the delay the user experiences
until all completions are presented I had to take out the
`while-no-input` in icomplete-exhibit so that this test would work:

    ;; After the form, type C-u C-x C-e C-m in quick succession
    (benchmark-run (completing-read "bla" obarray))

If I don't remove this `while-no-input`, icomplete will not lose time
showing all the completions and will instead select just the first one.
That's a very nice feature for actual use, but for this benchmark that
is not what I want: I want to measure the time to show all the
completions.

Then, the times presented by benchmark-run are the same that the user
sees if he waits to see the completions.  Now the values:

Before my patch:

    (1.802209488 5 1.3678843490000077)
    (1.609066281 4 1.1170432569999775)
    (1.878972079 5 1.3725165670000479)
    (1.901952581 5 1.3979494059999524)
    (1.820800064 5 1.3283940110000003)

After the patch:

    (0.552051921 1 0.3079724459999511)
    (0.58396499 1 0.3038616050000087)
    (0.861106587 2 0.6046198220000178)
    (0.611551175 1 0.30275532399997473)
    (0.62500199 1 0.3160454470000218)

Without the patch but with icomplete-vertical-mode:

    (0.645366711 1 0.3412892389999911)
    (0.6256968110000001 1 0.3234302760000105)
    (0.9716317630000001 2 0.6676939319999633)
    (0.6414442749999999 1 0.3325084230000357)
    (0.627684562 1 0.32241421699995954)

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 12.06.2021 02:24, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Looking forward for your analysis of fido-vertical-mode's performance
>> improvement over the "normal" one.
> 
> So, I benchmarked before and after this patch to icomplete.el:
> 
>      diff --git a/lisp/icomplete.el b/lisp/icomplete.el
>      index 08b4ef2030..3561ebfa04 100644
>      --- a/lisp/icomplete.el
>      +++ b/lisp/icomplete.el
>      @@ -858,16 +858,8 @@ icomplete-completions
>                      ;; removing making `comps' a proper list.
>                      (base-size (prog1 (cdr last)
>                                   (if last (setcdr last nil))))
>      -               (most-try
>      -                (if (and base-size (> base-size 0))
>      -                    (completion-try-completion
>      -                     name candidates predicate (length name) md)
>      -                  ;; If the `comps' are 0-based, the result should be
>      -                  ;; the same with `comps'.
>      -                  (completion-try-completion
>      -                   name comps nil (length name) md)))
>      -               (most (if (consp most-try) (car most-try)
>      -                       (if most-try (car comps) "")))
>      +               (most-try nil)
>      +               (most "")
>                      ;; Compare name and most, so we can determine if name is
>                      ;; a prefix of most, or something else.
>                      (compare (compare-strings name nil nil

All right, so this is not about try-completion, it's about 
completion-try-completion. That makes sense.

> The patch itself nullifies the calculation of the 'determ' thing that I
> and presumably some other users don't value that much.  It doesn't
> affect fido-mode's basic funcionality.
> 
> How did I benchmark?  Well, to measure the delay the user experiences
> until all completions are presented I had to take out the
> `while-no-input` in icomplete-exhibit so that this test would work:
> 
>      ;; After the form, type C-u C-x C-e C-m in quick succession
>      (benchmark-run (completing-read "bla" obarray))
> 
> If I don't remove this `while-no-input`, icomplete will not lose time
> showing all the completions and will instead select just the first one.
> That's a very nice feature for actual use, but for this benchmark that
> is not what I want: I want to measure the time to show all the
> completions.

Did the same, can repro.

> Then, the times presented by benchmark-run are the same that the user
> sees if he waits to see the completions.  Now the values:
> 
> Before my patch:
> 
>      (1.802209488 5 1.3678843490000077)
>      (1.609066281 4 1.1170432569999775)
>      (1.878972079 5 1.3725165670000479)
>      (1.901952581 5 1.3979494059999524)
>      (1.820800064 5 1.3283940110000003)
> 
> After the patch:
> 
>      (0.552051921 1 0.3079724459999511)
>      (0.58396499 1 0.3038616050000087)
>      (0.861106587 2 0.6046198220000178)
>      (0.611551175 1 0.30275532399997473)
>      (0.62500199 1 0.3160454470000218)

I get

(0.377195885 10 0.24448539800000013)

before and

(0.245218061 6 0.1390041310000001)

after. A solid improvement.

BTW, if I just stick benchmark-progn around icomplete-completions like

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index 08b4ef2030..b9fe3e1836 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -678,12 +678,13 @@ icomplete-exhibit
                   ;; seems to trigger it fairly often!
                   (while-no-input-ignore-events '(selection-request))
                   (text (while-no-input
-                         (icomplete-completions
-                          field-string
-                          (icomplete--completion-table)
-                          (icomplete--completion-predicate)
-                          (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                         (benchmark-progn
+                           (icomplete-completions
+                            field-string
+                            (icomplete--completion-table)
+                            (icomplete--completion-predicate)
+                            (if (window-minibuffer-p)
+                                (eq minibuffer--require-match t))))))
                   (buffer-undo-list t)
                   deactivate-mark)
              ;; Do nothing if while-no-input was aborted.

...it reports

Elapsed time: 0.329006s (0.246073s in 10 GCs)

vs

Elapsed time: 0.169200s (0.113762s in 5 GCs)

I suppose the 40-70ms difference is due to delay in typing.

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 12.06.2021 02:24, João Távora wrote:
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>>> Looking forward for your analysis of fido-vertical-mode's performance
>>> improvement over the "normal" one.
>> So, I benchmarked before and after this patch to icomplete.el:
>>      diff --git a/lisp/icomplete.el b/lisp/icomplete.el
>>      index 08b4ef2030..3561ebfa04 100644
>>      --- a/lisp/icomplete.el
>>      +++ b/lisp/icomplete.el
>>      @@ -858,16 +858,8 @@ icomplete-completions
>>                      ;; removing making `comps' a proper list.
>>                      (base-size (prog1 (cdr last)
>>                                   (if last (setcdr last nil))))
>>      -               (most-try
>>      -                (if (and base-size (> base-size 0))
>>      -                    (completion-try-completion
>>      -                     name candidates predicate (length name) md)
>>      -                  ;; If the `comps' are 0-based, the result should be
>>      -                  ;; the same with `comps'.
>>      -                  (completion-try-completion
>>      -                   name comps nil (length name) md)))
>>      -               (most (if (consp most-try) (car most-try)
>>      -                       (if most-try (car comps) "")))
>>      +               (most-try nil)
>>      +               (most "")
>>                      ;; Compare name and most, so we can determine if name is
>>                      ;; a prefix of most, or something else.
>>                      (compare (compare-strings name nil nil
>
> All right, so this is not about try-completion, it's about
> completion-try-completion. That makes sense.

Yeah, to be honest, once I'm done actually using these functions I
immediately evict the differences between try-completion,
completion-try-completion, try-try-completion-completion, or any of
these yoda-speak variations from my mental cache.  

Here I meant is that there was something apparently useless and slow (to
fido-mode at least) going on in that else branch.

> Elapsed time: 0.329006s (0.246073s in 10 GCs)
>
> vs
>
> Elapsed time: 0.169200s (0.113762s in 5 GCs)
>
> I suppose the 40-70ms difference is due to delay in typing.

No idea.  In my (slower?) system, I typed C-u C-x C-e C-m pretty fast.
Presumably the C-m goes in before pp-eval-last-sexp has a chance to read
more input so I wouldn't think it's a delay in typing.  I could
investigate, but since your measurements confirm the same tendency
anyway, I think this simple patch is what's needed to close this issue.

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index 08b4ef2030..5d37f47e7d 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -859,13 +859,14 @@ icomplete-completions
                (base-size (prog1 (cdr last)
                             (if last (setcdr last nil))))
                (most-try
-                (if (and base-size (> base-size 0))
-                    (completion-try-completion
-                     name candidates predicate (length name) md)
-                  ;; If the `comps' are 0-based, the result should be
-                  ;; the same with `comps'.
-                  (completion-try-completion
-                   name comps nil (length name) md)))
+                (and (not fido-mode) ; Fido avoids these expensive calculations.
+                     (if (and base-size (> base-size 0))
+                         (completion-try-completion
+                          name candidates predicate (length name) md)
+                       ;; If the `comps' are 0-based, the result should be
+                       ;; the same with `comps'.
+                       (completion-try-completion
+                        name comps nil (length name) md))))
                (most (if (consp most-try) (car most-try)
                        (if most-try (car comps) "")))
                ;; Compare name and most, so we can determine if name is

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> Very interesting.  I don't know what the matter is with modifying
>> the
>> string itself.  Is it because we want to protect its 'face' property?
>> Maybe, but what's the harm in chaning it?
>
> I imagine it's just a "correctness" thing. Text properties are part of
> the string's identity. We add text properties, so we make a copy
> because we don't own the original list (it might be saved to some
> constant and also used for, I don't know, IMenu items?)

I just confirmed it's not for correctness.  If one foregoes the copy, in
fido-mode a C-h f followed by some input followed by C-g and an M-x and
no input (empty pattern) will show interesting results, i.e. a list of
propertized strings when nothing should be propertized.

But maybe that's a question of removing the propertization when the
pattern is empty?  I'll try later.

>> If we do want to protect the shared 'face' property -- and only 'face'
>> -- then we could very add some other property about face that the
>> frontend could read "just in time" before it itself makes a copy of the
>> string to display to the user.
>
> Yes, it's an option (though a less elegant one): apply some namespaced
> text properties with the necessary data. And then we'd also be able to
> fontify "just in time".
>
> Do we have any "frozen strings" in Emacs, which absolutely could not
> be modified? Do we plan to?

Immutable strings? And error when one tries to?  Or just ignore the
modification?  And how would that help here?

> I disagree it's a simpler technique, but it would indeed be a simpler
> change, based on the current implementation.

simpler means simpler in my book :-)

> But I don't mind it myself, and happy to update Company. Either way
> it's a step forward.

If Company and fido-mode and a couple more outside the core/Elpa are all
that's needed, it's probably warranted.  But there are so many frontends
right now, I don't know...  We'd need some "opt into the optimization",
I think."

>> But if the speedup is big, I'd revisit the rationale for requiring those
>> copies to be performed in the first place.
>
> With fido-vertical-mode, and with that particular input, it's
>
>   Elapsed time: 0.130773s (0.031547s in 1 GCs)
>
> without copy-sequence, and
>
>   Elapsed time: 0.169842s (0.069740s in 4 GCs)
>
> with it. Not game changing, but definitely measurable.

I don't have these results.  I tried the previous experiment:

   ;; C-u C-x C-e C-m in quick succession
   (benchmark-run (completing-read "bla" obarray))

And turned icomplete.el's while-no-input into a progn.

In an Emacs -Q where (length (all-completions "" obarray)) is about
20000.

   ;; with copy
   (0.39647753 6 0.22811240199999983)
   (0.431950471 7 0.263988651)
   (0.451116177 6 0.2249686070000001)
    
   ;; without copy, small but measurable speedup
   (0.29890632 2 0.08419541699999966)
   (0.293501099 2 0.08622194699999985)
   (0.306566633 3 0.0853211100000002)

In a loaded Emacs where (length (all-completions "" obarray)) is 64554

   ;; with copy
   (2.869362171 6 2.3882547280000495)
   (2.909661303 6 2.4209153659999743)
   (2.845522439 6 2.3638140250000106)
    
   ;; without copy.  Huge speedup.
   (0.79817337 1 0.4526993239999797)
   (0.8231736510000001 1 0.4752496449999626)
   (0.719004478 1 0.4016016420000028)

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 13.06.2021 17:29, João Távora wrote:

> Yeah, to be honest, once I'm done actually using these functions I
> immediately evict the differences between try-completion,
> completion-try-completion, try-try-completion-completion, or any of
> these yoda-speak variations from my mental cache.
> 
> Here I meant is that there was something apparently useless and slow (to
> fido-mode at least) going on in that else branch.

And there it was!

>> Elapsed time: 0.329006s (0.246073s in 10 GCs)
>>
>> vs
>>
>> Elapsed time: 0.169200s (0.113762s in 5 GCs)
>>
>> I suppose the 40-70ms difference is due to delay in typing.
> 
> No idea.  In my (slower?) system, I typed C-u C-x C-e C-m pretty fast.
> Presumably the C-m goes in before pp-eval-last-sexp has a chance to read
> more input so I wouldn't think it's a delay in typing.

Some input latency must be there, of course it depends on how fast Emacs 
is handling the previous events, how fast the machine is, etc.

Anyway, I was just describing an easier way to benchmark the same effect 
(without having to hit the key sequence very quickly). Hope you or 
someone else find it useful.

> I could
> investigate, but since your measurements confirm the same tendency
> anyway, I think this simple patch is what's needed to close this issue.

Haven't tested it myself, but it looks like it will almost certainly work.

> diff --git a/lisp/icomplete.el b/lisp/icomplete.el
> index 08b4ef2030..5d37f47e7d 100644
> --- a/lisp/icomplete.el
> +++ b/lisp/icomplete.el
> @@ -859,13 +859,14 @@ icomplete-completions
>                  (base-size (prog1 (cdr last)
>                               (if last (setcdr last nil))))
>                  (most-try
> -                (if (and base-size (> base-size 0))
> -                    (completion-try-completion
> -                     name candidates predicate (length name) md)
> -                  ;; If the `comps' are 0-based, the result should be
> -                  ;; the same with `comps'.
> -                  (completion-try-completion
> -                   name comps nil (length name) md)))
> +                (and (not fido-mode) ; Fido avoids these expensive calculations.

Perhaps predicate it on the value of icomplete-hide-common-prefix instead?

fido-mode sets it to nil, and this way we retain a better level of 
abstraction, and better backward compatibility for vanilla 
icomplete-mode users.

Next step might be switching this var's default value to nil.

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

On Mon, Jun 14, 2021 at 1:08 AM Dmitry Gutov <dgutov@yandex.ru> wrote:

> Anyway, I was just describing an easier way to benchmark the same effect
> (without having to hit the key sequence very quickly). Hope you or
> someone else find it useful.

Yes.  It is useful.  I didn't know about benchmark-progn.

> > I could
> > investigate, but since your measurements confirm the same tendency
> > anyway, I think this simple patch is what's needed to close this issue.
>
> Haven't tested it myself, but it looks like it will almost certainly work.
>
> > diff --git a/lisp/icomplete.el b/lisp/icomplete.el
> > index 08b4ef2030..5d37f47e7d 100644
> > --- a/lisp/icomplete.el
> > +++ b/lisp/icomplete.el
> > @@ -859,13 +859,14 @@ icomplete-completions
> >                  (base-size (prog1 (cdr last)
> >                               (if last (setcdr last nil))))
> >                  (most-try
> > -                (if (and base-size (> base-size 0))
> > -                    (completion-try-completion
> > -                     name candidates predicate (length name) md)
> > -                  ;; If the `comps' are 0-based, the result should be
> > -                  ;; the same with `comps'.
> > -                  (completion-try-completion
> > -                   name comps nil (length name) md)))
> > +                (and (not fido-mode) ; Fido avoids these expensive calculations.
>
> Perhaps predicate it on the value of icomplete-hide-common-prefix instead?
>
> fido-mode sets it to nil, and this way we retain a better level of
> abstraction, and better backward compatibility for vanilla
> icomplete-mode users.

This is a good idea, the level of abstraction.  But what is this
"common prefix" anyway?  Is it the the same as the "determ"
thing,  or the "[mplete...] dance" as I called it earlier.  Shouldn't
fido-mode then _hide_ it?

I'm confused, but if you're not, go ahead and make that more
abstract change instead of relying on fido-mode.

> Next step might be switching this var's default value to nil.

Also confused, but no objections.

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 14.06.2021 03:16, João Távora wrote:
>> Perhaps predicate it on the value of icomplete-hide-common-prefix instead?
>>
>> fido-mode sets it to nil, and this way we retain a better level of
>> abstraction, and better backward compatibility for vanilla
>> icomplete-mode users.
> This is a good idea, the level of abstraction.  But what is this
> "common prefix" anyway?  Is it the the same as the "determ"
> thing,  or the "[mplete...] dance" as I called it earlier.  Shouldn't
> fido-mode then_hide_  it?
> 
> I'm confused, but if you're not, go ahead and make that more
> abstract change instead of relying on fido-mode.

So... it's a bit more complex than that. The 'most' value computes the 
biggest "fuzzy" match (taking into account completion styles) and bases 
the resulting display of the "single match" on that.

Before your patch the output could look like:

   starfshe|(...t-file-process-shell-command) [Matched]

with the patch it's much less informative:

   starfshe| [Matched]

...so it has value, whether the variable I mentioned above is t or nil.

It seems there are two ways to proceed from here:

- Just alter the printing logic in the "single match" case to print the 
match text in full is it's not equal to the input string. I haven't 
puzzled out the logic doing that yet.

- Try to keep the current behavior while avoiding the duplicate work.

About the latter option: the result of that most-try stuff is only 
useful when there is only one match, right? But that work is performed 
unconditionally.

Unless I'm missing something and the value does see some use in the 
multiple-matches situations, the patch below both keeps the current 
behavior and gives the same performance improvement:

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index 08b4ef2030..fc88e2a3e0 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -859,13 +859,14 @@ icomplete-completions
                 (base-size (prog1 (cdr last)
                              (if last (setcdr last nil))))
                 (most-try
-                (if (and base-size (> base-size 0))
+                (unless (cdr comps)
+                  (if (and base-size (> base-size 0))
+                      (completion-try-completion
+                       name candidates predicate (length name) md)
+                    ;; If the `comps' are 0-based, the result should be
+                    ;; the same with `comps'.
                      (completion-try-completion
-                     name candidates predicate (length name) md)
-                  ;; If the `comps' are 0-based, the result should be
-                  ;; the same with `comps'.
-                  (completion-try-completion
-                   name comps nil (length name) md)))
+                     name comps nil (length name) md))))
                 (most (if (consp most-try) (car most-try)
                         (if most-try (car comps) "")))
                 ;; Compare name and most, so we can determine if name is

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 13.06.2021 17:55, João Távora wrote:

> I just confirmed it's not for correctness.  If one foregoes the copy, in
> fido-mode a C-h f followed by some input followed by C-g and an M-x and
> no input (empty pattern) will show interesting results, i.e. a list of
> propertized strings when nothing should be propertized.

That's also an example of correctness problem, just the more obvious 
kind. It probably happens in a number of situations, e.g. all (?) uses 
of completion-table-with-cache.

> But maybe that's a question of removing the propertization when the
> pattern is empty?  I'll try later.

That must add some performance penalty as well, for extra mutation. And 
wouldn't cover some potential other uses of the same set of strings. In 
IMenu, maybe?

The latter is hypothetical, of course.

>> Do we have any "frozen strings" in Emacs, which absolutely could not
>> be modified? Do we plan to?
> 
> Immutable strings? And error when one tries to?  Or just ignore the
> modification?  And how would that help here?

It wouldn't help. It would do the opposite: basically forbid the 
approach you suggest, mutation without copying.

>> I disagree it's a simpler technique, but it would indeed be a simpler
>> change, based on the current implementation.
> 
> simpler means simpler in my book :-)

One is simpler diff, another is simpler resulting code. Both have their 
upsides.

>> But I don't mind it myself, and happy to update Company. Either way
>> it's a step forward.
> 
> If Company and fido-mode and a couple more outside the core/Elpa are all
> that's needed, it's probably warranted.  But there are so many frontends
> right now, I don't know...  We'd need some "opt into the optimization",
> I think."

Since all other users are third-party (and thus have short release 
cycles), it shouldn't be too much of a problem. Some highlighting code 
would start to fail, but probably without disastrous results. And then 
people will issue updates to look for some new property when the old 
expected ones are all missing.

>>> But if the speedup is big, I'd revisit the rationale for requiring those
>>> copies to be performed in the first place.
>>
>> With fido-vertical-mode, and with that particular input, it's
>>
>>    Elapsed time: 0.130773s (0.031547s in 1 GCs)
>>
>> without copy-sequence, and
>>
>>    Elapsed time: 0.169842s (0.069740s in 4 GCs)
>>
>> with it. Not game changing, but definitely measurable.
> 
> I don't have these results.  I tried the previous experiment:
> 
>     ;; C-u C-x C-e C-m in quick succession
>     (benchmark-run (completing-read "bla" obarray))
> 
> And turned icomplete.el's while-no-input into a progn.
> 
> In an Emacs -Q where (length (all-completions "" obarray)) is about
> 20000.
> 
>     ;; with copy
>     (0.39647753 6 0.22811240199999983)
>     (0.431950471 7 0.263988651)
>     (0.451116177 6 0.2249686070000001)
>      
>     ;; without copy, small but measurable speedup
>     (0.29890632 2 0.08419541699999966)
>     (0.293501099 2 0.08622194699999985)
>     (0.306566633 3 0.0853211100000002)
> 
> In a loaded Emacs where (length (all-completions "" obarray)) is 64554
> 
>     ;; with copy
>     (2.869362171 6 2.3882547280000495)
>     (2.909661303 6 2.4209153659999743)
>     (2.845522439 6 2.3638140250000106)
>      
>     ;; without copy.  Huge speedup.
>     (0.79817337 1 0.4526993239999797)
>     (0.8231736510000001 1 0.4752496449999626)
>     (0.719004478 1 0.4016016420000028)

Even better.

My current session has 37559 symbols, so it's somewhere in the middle.

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>>> I disagree it's a simpler technique, but it would indeed be a simpler
>>> change, based on the current implementation.
>> simpler means simpler in my book :-)
>
> One is simpler diff, another is simpler resulting code. Both have
> their upsides.

Oh, you meant the The Big Redesign?  I'm a fan of that too, not only
here but constantly, everywhere...  That indeed means simpler resulting
code in abstract.  Problem is that also means different resulting code
to different people.  But is definitely doable.

>>> But I don't mind it myself, and happy to update Company. Either way
>>> it's a step forward.
>> If Company and fido-mode and a couple more outside the core/Elpa are
>> all
>> that's needed, it's probably warranted.  But there are so many frontends
>> right now, I don't know...  We'd need some "opt into the optimization",
>> I think."
>
> Since all other users are third-party (and thus have short release
> cycles), it shouldn't be too much of a problem. Some highlighting code
> would start to fail, but probably without disastrous results. And then
> people will issue updates to look for some new property when the old
> expected ones are all missing.

OK.  I can live with that rationale.  So what are the places to touch
that "we" control?

- icomplete.el? for fido-mode & friends
- minibuffer.el, for the *Completions* buffer
- company.el
- Any notable others?

>>     ;; with copy
>>     (2.869362171 6 2.3882547280000495)
>>     (2.909661303 6 2.4209153659999743)
>>     (2.845522439 6 2.3638140250000106)
>>          ;; without copy.  Huge speedup.
>>     (0.79817337 1 0.4526993239999797)
>>     (0.8231736510000001 1 0.4752496449999626)
>>     (0.719004478 1 0.4016016420000028)
>
> Even better.
>
> My current session has 37559 symbols, so it's somewhere in the middle.

Yes, this is a big performance bottleneck.  But i wonder if tweaking GC
parameter would help here.  I know nothing of Emacs GC parameters.

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 14.06.2021 03:16, João Távora wrote:
>>> Perhaps predicate it on the value of icomplete-hide-common-prefix instead?
>>>
>>> fido-mode sets it to nil, and this way we retain a better level of
>>> abstraction, and better backward compatibility for vanilla
>>> icomplete-mode users.
>> This is a good idea, the level of abstraction.  But what is this
>> "common prefix" anyway?  Is it the the same as the "determ"
>> thing,  or the "[mplete...] dance" as I called it earlier.  Shouldn't
>> fido-mode then_hide_  it?
>> I'm confused, but if you're not, go ahead and make that more
>> abstract change instead of relying on fido-mode.
>
> So... it's a bit more complex than that.

Yes, my batch broke the things you mentioned.

> It seems there are two ways to proceed from here:
>
> - Just alter the printing logic in the "single match" case to print
>   the match text in full is it's not equal to the input string. I
>   haven't puzzled out the logic doing that yet.
>   
> - Try to keep the current behavior while avoiding the duplicate work.

Both sound absolutely fine to me.

> About the latter option: the result of that most-try stuff is only
> useful when there is only one match, right?

No idea, but may be.

> Unless I'm missing something and the value does see some use in the
> multiple-matches situations, the patch below both keeps the current
> behavior and gives the same performance improvement:

That'd be fantastic, but I doubt you'd be keeping the exact same
behaviour.  I never understood it -- that's the thing here -- but I
think that completion-try-completion is doing more stuff when multiple
candidates matched by a pattern happen to share the same prefix or
suffix or something like that.  I might be completely wrong, tho.

But really if you make this patch conditional to fido-mode or that other
var that you think is more abstract, I think it's fine and it's a very
clear win. I really doubt that the tiny number of fido-mode users care
about that behaviour anyway, but I'm sure they'll appreciate the
considerable speedup.

João

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 18.06.2021 00:29, João Távora wrote:

>> Unless I'm missing something and the value does see some use in the
>> multiple-matches situations, the patch below both keeps the current
>> behavior and gives the same performance improvement:
> 
> That'd be fantastic, but I doubt you'd be keeping the exact same
> behaviour.  I never understood it -- that's the thing here -- but I
> think that completion-try-completion is doing more stuff when multiple
> candidates matched by a pattern happen to share the same prefix or
> suffix or something like that.  I might be completely wrong, tho.

Turns out that indeed the logic is used in the "multiple matches" case: 
when icomplete-hide-common-prefix is non-nil.

Meaning, with icomplete-mode but not with fido-mode.

> But really if you make this patch conditional to fido-mode or that other
> var that you think is more abstract, I think it's fine and it's a very
> clear win. I really doubt that the tiny number of fido-mode users care
> about that behaviour anyway, but I'm sure they'll appreciate the
> considerable speedup.

So I have done the above. There is no change in observable behavior 
either (AFAICS), so it's win-win.

bug#48841: fido-mode is slower than ido-mode with similar settings

[Dmitry Gutov]

On 18.06.2021 00:21, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>>>> I disagree it's a simpler technique, but it would indeed be a simpler
>>>> change, based on the current implementation.
>>> simpler means simpler in my book :-)
>>
>> One is simpler diff, another is simpler resulting code. Both have
>> their upsides.
> 
> Oh, you meant the The Big Redesign?  I'm a fan of that too, not only
> here but constantly, everywhere...  That indeed means simpler resulting
> code in abstract.  Problem is that also means different resulting code
> to different people.  But is definitely doable.

I meant a particular change (not modifying the strings at all in 
advance) which would be bigger and indeed might better fit The Big Redesign.

>>>> But I don't mind it myself, and happy to update Company. Either way
>>>> it's a step forward.
>>> If Company and fido-mode and a couple more outside the core/Elpa are
>>> all
>>> that's needed, it's probably warranted.  But there are so many frontends
>>> right now, I don't know...  We'd need some "opt into the optimization",
>>> I think."
>>
>> Since all other users are third-party (and thus have short release
>> cycles), it shouldn't be too much of a problem. Some highlighting code
>> would start to fail, but probably without disastrous results. And then
>> people will issue updates to look for some new property when the old
>> expected ones are all missing.
> 
> OK.  I can live with that rationale.  So what are the places to touch
> that "we" control?
> 
> - icomplete.el? for fido-mode & friends
> - minibuffer.el, for the *Completions* buffer
> - company.el
> - Any notable others?

corfu, consult, etc? Probably Ivy too. All of these are in GNU ELPA.

BTW, I think Daniel had some ideas about applying the face property 
lazily as well. I can't find the particular discussion now, but perhaps 
he can add to this discussion as well.

>>>      ;; with copy
>>>      (2.869362171 6 2.3882547280000495)
>>>      (2.909661303 6 2.4209153659999743)
>>>      (2.845522439 6 2.3638140250000106)
>>>           ;; without copy.  Huge speedup.
>>>      (0.79817337 1 0.4526993239999797)
>>>      (0.8231736510000001 1 0.4752496449999626)
>>>      (0.719004478 1 0.4016016420000028)
>>
>> Even better.
>>
>> My current session has 37559 symbols, so it's somewhere in the middle.
> 
> Yes, this is a big performance bottleneck.  But i wonder if tweaking GC
> parameter would help here.  I know nothing of Emacs GC parameters.

I think the current understanding that by raising gc-cons-threshold we 
exchange the number of GC pauses for larger latencies. I suppose one 
could tune that value to a particular workload such that the 4 GCs in a 
row that I had will turn into just one (and thus save on re-scanning the 
heap 3 times), but data sets change.

bug#47711: bug#48841: fido-mode is slower than ido-mode with similar settings

[Daniel Mendler]

On 7/4/21 3:53 AM, Dmitry Gutov wrote:
>> - icomplete.el? for fido-mode & friends
>> - minibuffer.el, for the *Completions* buffer
>> - company.el
>> - Any notable others?
> 
> corfu, consult, etc? Probably Ivy too. All of these are in GNU ELPA.
> 
> BTW, I think Daniel had some ideas about applying the face property 
> lazily as well. I can't find the particular discussion now, but perhaps 
> he can add to this discussion as well.

Yes, Vertico and Corfu apply highlighting lazily. This leads to
significant performance wins. See `vertico--all-completions` in
https://github.com/minad/vertico/blob/main/vertico.el#L243-L279 and
bug#47711.

The technique I am using in Vertico and Corfu retains backward
compatibility, such that the strings are returned unmodified by the
completion style. Highlighting is applied lazily by copying the
candidate strings and mutating the copies. For now I am relying on advices.

One could add an optional argument (or dynamically bound variable) to
completion styles which tell the completion style to opt out of copying
the candidates and the highlighting.

Daniel

bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

I prepared a patch which provides the API
`completion-filter-completions`. This function supports deferred
highlighting and returns additional data with the list of matching
completion candidates. The API supersedes the existing function
`completion-all-completions`.

The main goal of the new API is to avoid expensive string allocations
and highlighting during completion. This is particularly relevant for
continuously updating completion UIs like Icomplete or Vertico.
Furthermore the end position of the completion boundaries is returned
with the completion results. This information is not provided by the
existing `completion-all-completions` API.

See also the relevant bugs bug#47711 and bug#48841. I am looking forward
to your feedback. Thank you!

Daniel Mendler

[-- Attachment #2: 0001-Add-new-completion-filter-completions-API-and-deferr.patch --]
[-- Type: text/x-diff, Size: 50976 bytes --]

From e7f26abc520ac36cc154a92bfb4744837d9f7e5e Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Mon, 12 Jul 2021 21:40:32 +0200
Subject: [PATCH] Add new `completion-filter-completions` API and deferred
 highlighting

Fix bug#47711.

Add a new `completion-filter-completions` API, which supersedes
`completion-all-completions`.  The new API returns the matching
completion candidates and additional data.  The return value is an
alist, with the keys `completions`, `base`, `end` and `highlight`.
The API can be extended in a backward compatible way later on thanks
to the use of an alist as return value.

The `completions` value is the list of completion strings *without*
applied highlighting.  The completion strings are returned unmodified,
which avoids allocations and results in performance gains for
continuously updating completion UIs, like Icomplete or Vertico (GNU
ELPA).  The value `base` is the base position of the completion.
Correspondingly the value `end` specifies the end position of the
completion counted from the beginning of the input strng.  In
comparison, the old function `completion-all-completions` only
returned the base position in the last cdr of the returned completions
list, which complicated usage.  The `end` position was not provided by
`completion-all-completions`.  Given the new API the
`completion-base-position` can be set accurately.  Finally the
`highlight` value is a function taking a list of completion strings
and returns a new list of new strings with highlighting applied.  A
continously updating UI can use the highlighting function to apply
highlighting only to the visible completions.

* lisp/minibuffer.el:
(completion-pcm--hilit-commonality): Remove scoring computation.
(completion--adjust-metadata): Rename to `completion--style-metadata`
due to change of calling convention.
(completion--nth-completion): Call renamed metadata adjustment
function.  Ignore the old property `completion--adjust-metadata`.
(completion--flex-adjust-metadata): Rename function.
(completion--twq-all): Attach `completion--unquoted` text property to
quoted completion strings.
(completion--flex-score): New optimized scoring function.  Use
`completion--unquoted` text property.
(completion--flex-style-metadata): Use it.
(completion--pattern-compiler): New function.
(completion-substring--all-completions,
completion--flex-score): Use it.
(completion--hilit-commonality): New function.
(completion-hilit-commonality): Use it.
(completion--deferred-hilit): New function.
(completion-basic-all-completions,
completion-emacs21-all-completions,
completion-emacs22-all-completions): Use it.
(completion--pcm-deferred-hilit): New function.
(completion-pcm-all-completions,
completion-flex-all-completions,
completion-initials-all-completions,
completion-substring-all-completions): Use it.
(completion--filter-completions): New variable to conditionally enable
the new alist completions result format.  This variable is for
internal use to preserve the existing calling convention of the
completion style `all` functions.
(completion-filter-completions): New API which returns the completion
strings and additional data as an an alist.  Transparently convert
old-fashioned completion style results to the new format.
(completion-all-completions): Transparently downgrade the
new-fashioned completion style result to the old list format.
(minibuffer-completion-help): Use the new API, set
`completion-base-position` correctly.
(completion-try-completion,
completion-all-completions): Update doc string.
(completion--replace): Remove unnecessary property removal.

* test/lisp/minibuffer-tests.el:
(completion--pcm-score): Remove obsolete function.
(completion-*-test-*): Remove obsolete functions, rename.
(completion-flex-score-test-*): Add new scoring test functions.
(completion--test-style): New test helper function.
(completion-*-style-test): Add new API tests for each built-in
completion style.
(completion--test-boundaries): New test helper function.
(completion-*-boundaries-test): New boundary tests for each built-in
completion style.
(completion-filter-completions-highlight-test): New API test.
(completion-emacs22orig-all-completions): New function.
(completion-upgrade-return-type-test): New test of transparent
completion style return value upgrade.
---
 lisp/minibuffer.el            | 453 +++++++++++++++++++++++-----------
 test/lisp/minibuffer-tests.el | 257 +++++++++++++++----
 2 files changed, 516 insertions(+), 194 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 9f327df28f..ba8855c4ea 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -692,6 +692,10 @@ completion--twq-all
                                              'completions-common-part)
                                qprefix))))
                         (qcompletion (concat qprefix qnew)))
+                   ;; Attach unquoted completion string, which is needed
+                   ;; to score the completion in `completion--flex-score'.
+                   (put-text-property 0 1 'completion--unquoted
+                                      completion qcompletion)
 		   ;; FIXME: Similarly here, Cygwin's mapping trips this
 		   ;; assertion.
                    ;;(cl-assert
@@ -1035,6 +1039,17 @@ completion--styles
         (delete-dups (append (cdr over) (copy-sequence completion-styles)))
        completion-styles)))
 
+(defvar completion--filter-completions nil
+  "Enable the new completions return value format.
+If this variable is non-nil the `all-completions' function of a
+completion style should return the results in the new alist format of
+`completion-filter-completions'.  This variable is purely needed to
+for backward compatibility of the existing builtin completion style
+functions.  New completion style functions may always return their
+results in the new alist format, since `completion-all-completions'
+transparently converts back to the old improper list of completions
+with base size in the last cdr.")
+
 (defun completion--nth-completion (n string table pred point metadata)
   "Call the Nth method of completion styles."
   ;; We provide special support for quoting/unquoting here because it cannot
@@ -1061,6 +1076,15 @@ completion--nth-completion
                  ;; the original table, in that case!
                  (functionp table))
             (let ((new (funcall table string point 'completion--unquote)))
+              ;; FIXME For now do not attempt deferred highlighting if
+              ;; quoting is used.  Not doing deferred highlighting is
+              ;; not too severe in this case, since
+              ;; `completion--twq-all' is already an expensive
+              ;; function, which allocates all completion strings.  In
+              ;; contrast to plain completion tables, the savings of
+              ;; deferred highlighting would be minimal in the case of
+              ;; quoted completion tables.
+              (setq completion--filter-completions nil)
               (setq string (pop new))
               (setq table (pop new))
               (setq point (pop new))
@@ -1074,9 +1098,10 @@ completion--nth-completion
                                    string table pred point)))
                (and probe (cons probe style))))
            (completion--styles md)))
-         (adjust-fn (get (cdr result-and-style) 'completion--adjust-metadata)))
-    (when (and adjust-fn metadata)
-      (setcdr metadata (cdr (funcall adjust-fn metadata))))
+         (style-md (get (cdr result-and-style) 'completion--style-metadata)))
+    (when (and style-md metadata)
+      (setcdr metadata (cdr (funcall style-md
+                                     string table pred point metadata))))
     (if requote
         (funcall requote (car result-and-style) n)
       (car result-and-style))))
@@ -1084,22 +1109,64 @@ completion--nth-completion
 (defun completion-try-completion (string table pred point &optional metadata)
   "Try to complete STRING using completion table TABLE.
 Only the elements of table that satisfy predicate PRED are considered.
-POINT is the position of point within STRING.
-The return value can be either nil to indicate that there is no completion,
-t to indicate that STRING is the only possible completion,
-or a pair (NEWSTRING . NEWPOINT) of the completed result string together with
-a new position for point."
+POINT is the position of point within STRING.  The return value can be
+either nil to indicate that there is no completion, t to indicate that
+STRING is the only possible completion, or a pair (NEWSTRING . NEWPOINT)
+of the completed result string together with a new position for point.
+The METADATA may be modified by the completion style."
   (completion--nth-completion 1 string table pred point metadata))
 
 (defun completion-all-completions (string table pred point &optional metadata)
   "List the possible completions of STRING in completion table TABLE.
 Only the elements of table that satisfy predicate PRED are considered.
-POINT is the position of point within STRING.
-The return value is a list of completions and may contain the base-size
-in the last `cdr'."
-  ;; FIXME: We need to additionally return the info needed for the
-  ;; second part of completion-base-position.
-  (completion--nth-completion 2 string table pred point metadata))
+POINT is the position of point within STRING.  The return value is a
+list of completions and may contain the base-size in the last `cdr'.
+The METADATA may be modified by the completion style.  This function
+has been superseded by `completion-filter-completions', which returns
+richer information and supports deferred candidate highlighting."
+  (let ((completion--filter-completions nil)
+        (result (completion--nth-completion 2 string table
+                                            pred point metadata)))
+    (if (and result (consp (car result)))
+        ;; Give the completion styles some freedom!
+        ;; If they are targeting Emacs 28 upwards only, they
+        ;; may always return a result with deferred
+        ;; highlighting.  We convert back to the old format
+        ;; here by applying the highlighting eagerly.
+        (nconc (funcall (cdr (assq 'highlight result))
+                        (cdr (assq 'completions result)))
+               (cdr (assq 'base result)))
+      result)))
+
+(defun completion-filter-completions (string table pred point metadata)
+  "Filter the possible completions of STRING in completion table TABLE.
+Only the elements of table that satisfy predicate PRED are considered.
+POINT is the position of point within STRING.  The METADATA may be
+modified by the completion style.  The return value is a alist with
+the keys:
+
+- base: Base position of the completion (from the start of STRING)
+- end: End position of the completion (from the start of STRING)
+- highlight: Highlighting function taking a list of completions and
+  returning a new list of new strings with applied highlighting.
+- completions: The list of completions.
+
+This function supersedes the function `completion-all-completions'."
+  (let* ((completion--filter-completions t)
+         (result (completion--nth-completion 2 string table
+                                             pred point metadata)))
+    (if (and result (not (consp (car result))))
+        ;; Deferred highlighting has been requested, but the completion
+        ;; style returned a non-deferred result. Convert the result to the
+        ;; new alist format.
+        (let* ((last (last result))
+               (base (or (cdr last) 0)))
+          (setcdr last nil)
+          `((base . ,base)
+            (end . ,(length string))
+            (highlight . identity)
+            (completions . ,result)))
+      result)))
 
 (defun minibuffer--bitset (modified completions exact)
   (logior (if modified    4 0)
@@ -1114,9 +1181,8 @@ completion--replace
   ;; include upon insertion.
   (if minibuffer-allow-text-properties
       ;; If we're preserving properties, then just remove the faces
-      ;; and other properties added by the completion machinery.
-      (remove-text-properties 0 (length newtext) '(face completion-score)
-                              newtext)
+      ;; added by the completion machinery.
+      (remove-text-properties 0 (length newtext) '(face nil) newtext)
     ;; Remove all text properties.
     (set-text-properties 0 (length newtext) nil newtext))
   ;; Maybe this should be in subr.el.
@@ -2021,34 +2087,48 @@ completion-hilit-commonality
 It returns a list with font-lock properties applied to each element,
 and with BASE-SIZE appended as the last element."
   (when completions
-    (let ((com-str-len (- prefix-len (or base-size 0))))
-      (nconc
-       (mapcar
-        (lambda (elem)
-          (let ((str
-                 ;; Don't modify the string itself, but a copy, since the
-                 ;; the string may be read-only or used for other purposes.
-                 ;; Furthermore, since `completions' may come from
-                 ;; display-completion-list, `elem' may be a list.
-                 (if (consp elem)
-                     (car (setq elem (cons (copy-sequence (car elem))
-                                           (cdr elem))))
-                   (setq elem (copy-sequence elem)))))
-            (font-lock-prepend-text-property
-             0
-             ;; If completion-boundaries returns incorrect
-             ;; values, all-completions may return strings
-             ;; that don't contain the prefix.
-             (min com-str-len (length str))
-             'face 'completions-common-part str)
-            (if (> (length str) com-str-len)
-                (font-lock-prepend-text-property com-str-len (1+ com-str-len)
-                                                 'face
-                                                 'completions-first-difference
-                                                 str)))
-          elem)
-        completions)
-       base-size))))
+    (nconc
+     (completion--hilit-commonality (- prefix-len (or base-size 0)) completions)
+     base-size)))
+
+(defun completion--hilit-commonality (com-size completions)
+  (mapcar
+   (lambda (elem)
+     (let ((str
+            ;; Don't modify the string itself, but a copy, since the
+            ;; the string may be read-only or used for other purposes.
+            ;; Furthermore, since `completions' may come from
+            ;; display-completion-list, `elem' may be a list.
+            (if (consp elem)
+                (car (setq elem (cons (copy-sequence (car elem))
+                                      (cdr elem))))
+              (setq elem (copy-sequence elem)))))
+       (font-lock-prepend-text-property
+        0
+        ;; If completion-boundaries returns incorrect
+        ;; values, all-completions may return strings
+        ;; that don't contain the prefix.
+        (min com-size (length str))
+        'face 'completions-common-part str)
+       (if (> (length str) com-size)
+           (font-lock-prepend-text-property com-size (1+ com-size)
+                                            'face
+                                            'completions-first-difference
+                                            str)))
+     elem)
+   completions))
+
+(defun completion--deferred-hilit (completions prefix-len base end)
+  "Return completions in old format or new alist format.
+If `completion--filter-completions' is non-nil use the new format."
+  (if completion--filter-completions
+      (when completions
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially #'completion--hilit-commonality
+                                         (- prefix-len base)))
+          (completions . ,completions)))
+    (completion-hilit-commonality completions prefix-len base)))
 
 (defun display-completion-list (completions &optional common-substring group-fun)
   "Display the list of completions, COMPLETIONS, using `standard-output'.
@@ -2163,15 +2243,16 @@ minibuffer-completion-help
          (end (or end (point-max)))
          (string (buffer-substring start end))
          (md (completion--field-metadata start))
-         (completions (completion-all-completions
-                       string
-                       minibuffer-completion-table
-                       minibuffer-completion-predicate
-                       (- (point) start)
-                       md)))
+         (filtered-completions (completion-filter-completions
+                                string
+                                minibuffer-completion-table
+                                minibuffer-completion-predicate
+                                (- (point) start)
+                                md))
+         (completions (alist-get 'completions filtered-completions)))
     (message nil)
     (if (or (null completions)
-            (and (not (consp (cdr completions)))
+            (and (not (cdr completions))
                  (equal (car completions) string)))
         (progn
           ;; If there are no completions, or if the current input is already
@@ -2181,8 +2262,7 @@ minibuffer-completion-help
           (completion--message
            (if completions "Sole completion" "No completions")))
 
-      (let* ((last (last completions))
-             (base-size (or (cdr last) 0))
+      (let* ((base-size (alist-get 'base filtered-completions))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
              (all-md (completion--metadata (buffer-substring-no-properties
                                             start (point))
@@ -2226,9 +2306,10 @@ minibuffer-completion-help
             (body-function
              . ,#'(lambda (_window)
                     (with-current-buffer mainbuf
-                      ;; Remove the base-size tail because `sort' requires a properly
-                      ;; nil-terminated list.
-                      (when last (setcdr last nil))
+                      ;; Apply highlighting
+                      (setq completions
+                            (funcall (alist-get 'highlight filtered-completions)
+                                     completions))
 
                       ;; Sort first using the `display-sort-function'.
                       ;; FIXME: This function is for the output of
@@ -2267,13 +2348,10 @@ minibuffer-completion-help
                                       completions))))
 
                       (with-current-buffer standard-output
-                        (setq-local completion-base-position
-                             (list (+ start base-size)
-                                   ;; FIXME: We should pay attention to completion
-                                   ;; boundaries here, but currently
-                                   ;; completion-all-completions does not give us the
-                                   ;; necessary information.
-                                   end))
+                        (setq-local
+                         completion-base-position
+                         (list (+ start base-size)
+                               (+ start (alist-get 'end filtered-completions))))
                         (setq-local completion-list-insert-choice-function
                              (let ((ctable minibuffer-completion-table)
                                    (cpred minibuffer-completion-predicate)
@@ -3223,10 +3301,11 @@ completion-emacs21-try-completion
       completion)))
 
 (defun completion-emacs21-all-completions (string table pred _point)
-  (completion-hilit-commonality
+  (completion--deferred-hilit
    (all-completions string table pred)
    (length string)
-   (car (completion-boundaries string table pred ""))))
+   (car (completion-boundaries string table pred ""))
+   (length string)))
 
 (defun completion-emacs22-try-completion (string table pred point)
   (let ((suffix (substring string point))
@@ -3249,11 +3328,12 @@ completion-emacs22-try-completion
       (cons (concat completion suffix) (length completion)))))
 
 (defun completion-emacs22-all-completions (string table pred point)
-  (let ((beforepoint (substring string 0 point)))
-    (completion-hilit-commonality
+  (let* ((beforepoint (substring string 0 point))
+         (afterpoint (substring string point))
+         (bounds (completion-boundaries beforepoint table pred afterpoint)))
+    (completion--deferred-hilit
      (all-completions beforepoint table pred)
-     point
-     (car (completion-boundaries beforepoint table pred "")))))
+     point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Basic completion.
 
@@ -3312,7 +3392,7 @@ completion-basic-all-completions
                             'point
                             (substring afterpoint 0 (cdr bounds)))))
          (all (completion-pcm--all-completions prefix pattern table pred)))
-    (completion-hilit-commonality all point (car bounds))))
+    (completion--deferred-hilit all point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Partial-completion-mode style completion.
 
@@ -3504,13 +3584,25 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defun completion-pcm--deferred-hilit (pattern completions base end)
+  "Return completions in old format or new alist format.
+If `completion--filter-completions' is non-nil use the new format."
+  (when completions
+    (if completion--filter-completions
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially
+                         #'completion-pcm--hilit-commonality
+                         pattern))
+          (completions . ,completions))
+      (nconc (completion-pcm--hilit-commonality pattern completions) base))))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
 string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
+each string is propertized with faces `completions-common-part',
 `completions-first-difference' in the relevant segments."
   (when completions
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
@@ -3525,6 +3617,54 @@ completion-pcm--hilit-commonality
            (error "Internal error: %s does not match %s" re str))
          (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
                 (match-end (match-end 0))
+                (md (cddr (setq last-md (match-data t last-md))))
+                (from 0))
+           (while md
+             (add-face-text-property from (pop md)
+                                     'completions-common-part
+                                     nil str)
+             (setq from (pop md)))
+           ;; If `pattern' doesn't have an explicit trailing any, the
+           ;; regex `re' won't produce match data representing the
+           ;; region after the match.  We need to account to account
+           ;; for that extra bit of match (bug#42149).
+           (unless (= from match-end)
+             (add-face-text-property from match-end
+                                     'completions-common-part
+                                     nil str))
+           (if (> (length str) pos)
+               (add-face-text-property
+                pos (1+ pos)
+                'completions-first-difference
+                nil str)))
+         str)
+       completions))))
+
+(defun completion--flex-score (pattern completions)
+  "Compute how well PATTERN matches COMPLETIONS.
+PATTERN, a list of strings is assumed to match every string in
+COMPLETIONS.  Return a copy of COMPLETIONS where each element is
+a pair of a score and the completion string.  The score lies in
+the range between -1 and 0, where -1 corresponds to the full
+match."
+  (when completions
+    (let* ((re (completion-pcm--pattern->regex pattern 'group))
+           (case-fold-search completion-ignore-case)
+           last-md)
+      (mapcar
+       (lambda (str)
+         ;; The flex completion style requires the completion to match
+         ;; the pattern to compute the scoring.  For quoted completion
+         ;; tables the completions are matched against the *unquoted
+         ;; input string*.  However `completion-all-completions' and
+         ;; `completion-filter-completions' return a list of *quoted
+         ;; completions*, which is subsequently sorted.  Therefore we
+         ;; obtain the unquoted completion string which is stored in
+         ;; the text property `completion--unquoted'.
+         (setq str (or (get-text-property 0 'completion--unquoted str) str))
+         (unless (string-match re str)
+           (error "Internal error: %s does not match %s" re str))
+         (let* ((match-end (match-end 0))
                 (md (cddr (setq last-md (match-data t last-md))))
                 (from 0)
                 (end (length str))
@@ -3564,13 +3704,10 @@ completion-pcm--hilit-commonality
                 ;; , where "len" is the string's length.
                 (score-numerator 0)
                 (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
+                (last-b 0))
+           (while md
+             (let ((a from)
+                   (b (pop md)))
                    (setq
                     score-numerator   (+ score-numerator (- b a)))
                    (unless (or (= a last-b)
@@ -3583,26 +3720,29 @@ completion-pcm--hilit-commonality
                                                  (/ 1.0
                                                     flex-score-match-tightness)))))
                    (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
+                    last-b              b))
              (setq from (pop md)))
            ;; If `pattern' doesn't have an explicit trailing any, the
            ;; regex `re' won't produce match data representing the
            ;; region after the match.  We need to account to account
            ;; for that extra bit of match (bug#42149).
            (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
+             (let ((a from)
+                   (b match-end))
+                   (setq
+                    score-numerator   (+ score-numerator (- b a)))
+                   (unless (or (= a last-b)
+                               (zerop last-b)
+                               (= a (length str)))
+                     (setq
+                      score-denominator (+ score-denominator
+                                           1
+                                           (expt (- a last-b 1)
+                                                 (/ 1.0
+                                                    flex-score-match-tightness)))))
+                   (setq
+                    last-b              b)))
+           (cons (- (/ score-numerator (* end (1+ score-denominator)) 1.0)) str)))
        completions))))
 
 (defun completion-pcm--find-all-completions (string table pred point
@@ -3700,11 +3840,11 @@ completion-pcm--find-all-completions
         (list pattern all prefix suffix)))))
 
 (defun completion-pcm-all-completions (string table pred point)
-  (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+  (pcase-let ((`(,pattern ,all ,prefix ,suffix)
                (completion-pcm--find-all-completions string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 (defun completion--common-suffix (strs)
   "Return the common suffix of the strings STRS."
@@ -3885,8 +4025,8 @@ completion-pcm-try-completion
 ;;; Substring completion
 ;; Mostly derived from the code of `basic' completion.
 
-(defun completion-substring--all-completions
-    (string table pred point &optional transform-pattern-fn)
+(defun completion--pattern-compiler
+    (string table pred point transform-pattern-fn)
   "Match the presumed substring STRING to the entries in TABLE.
 Respect PRED and POINT.  The pattern used is a PCM-style
 substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
@@ -3904,12 +4044,23 @@ completion-substring--all-completions
          (pattern (completion-pcm--optimize-pattern
                    (if transform-pattern-fn
                        (funcall transform-pattern-fn pattern)
-                     pattern)))
-         (all (completion-pcm--all-completions prefix pattern table pred)))
-    (list all pattern prefix suffix (car bounds))))
+                     pattern))))
+    (list pattern prefix suffix)))
+
+(defun completion-substring--all-completions
+    (string table pred point &optional transform-pattern-fn)
+  "Match the presumed substring STRING to the entries in TABLE.
+Respect PRED and POINT.  The pattern used is a PCM-style
+substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
+that is non-nil."
+  (pcase-let (((and result `(,pattern ,prefix ,_suffix))
+               (completion--pattern-compiler string table pred point
+                                             transform-pattern-fn)))
+    (cons (completion-pcm--all-completions prefix pattern table pred)
+          result)))
 
 (defun completion-substring-try-completion (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
     (if minibuffer-completing-file-name
@@ -3917,12 +4068,12 @@ completion-substring-try-completion
     (completion-pcm--merge-try pattern all prefix suffix)))
 
 (defun completion-substring-all-completions (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 ;;; "flex" completion, also known as flx/fuzzy/scatter completion
 ;; Completes "foo" to "frodo" and "farfromsober"
@@ -3932,42 +4083,40 @@ completion-flex-nospace
   :version "27.1"
   :type 'boolean)
 
-(put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata)
-
-(defun completion--flex-adjust-metadata (metadata)
-  (cl-flet
-      ((compose-flex-sort-fn
-        (existing-sort-fn) ; wish `cl-flet' had proper indentation...
-        (lambda (completions)
-          (let ((pre-sorted
-                 (if existing-sort-fn
-                     (funcall existing-sort-fn completions)
-                   completions)))
-            (cond
-             ((or (not (window-minibuffer-p))
-                  ;; JT@2019-12-23: FIXME: this is still wrong.  What
-                  ;; we need to test here is "some input that actually
-                  ;; leads to flex filtering", not "something after
-                  ;; the minibuffer prompt".  Among other
-                  ;; inconsistencies, the latter is always true for
-                  ;; file searches, meaning the next clauses will be
-                  ;; ignored.
-                  (> (point-max) (minibuffer-prompt-end)))
-              (sort
-               pre-sorted
-               (lambda (c1 c2)
-                 (let ((s1 (get-text-property 0 'completion-score c1))
-                       (s2 (get-text-property 0 'completion-score c2)))
-                   (> (or s1 0) (or s2 0))))))
-             (t pre-sorted))))))
-    `(metadata
-      (display-sort-function
-       . ,(compose-flex-sort-fn
-           (completion-metadata-get metadata 'display-sort-function)))
-      (cycle-sort-function
-       . ,(compose-flex-sort-fn
-           (completion-metadata-get metadata 'cycle-sort-function)))
-      ,@(cdr metadata))))
+(put 'flex 'completion--style-metadata 'completion--flex-style-metadata)
+
+(defun completion--flex-style-metadata (string table pred point metadata)
+  ;; Use the modified flex sorting function only for non-empty input.
+  ;; In an older version of `completion--flex-adjust-metadata', the
+  ;; check (> (point-max) (minibuffer-prompt-end))) was used instead.
+  (unless (eq string "")
+    (let ((pattern (car (completion--pattern-compiler
+                         string table pred point
+                         #'completion-flex--make-flex-pattern))))
+      (cl-flet
+          ((compose-flex-sort-fn
+            (existing-sort-fn) ; wish `cl-flet' had proper indentation...
+            (lambda (completions)
+              (let* ((sorted (sort (completion--flex-score
+                                    pattern
+                                    (if existing-sort-fn
+                                        (funcall existing-sort-fn completions)
+                                      completions))
+                                   #'car-less-than-car))
+                     (cell sorted))
+                ;; Remove score decorations, reuse the list to avoid allocations.
+                (while cell
+                  (setcar cell (cdar cell))
+                  (pop cell))
+                sorted))))
+        `(metadata
+          (display-sort-function
+           . ,(compose-flex-sort-fn
+               (completion-metadata-get metadata 'display-sort-function)))
+          (cycle-sort-function
+           . ,(compose-flex-sort-fn
+               (completion-metadata-get metadata 'cycle-sort-function)))
+          ,@(cdr metadata))))))
 
 (defun completion-flex--make-flex-pattern (pattern)
   "Convert PCM-style PATTERN into PCM-style flex pattern.
@@ -3989,7 +4138,7 @@ completion-flex--make-flex-pattern
 (defun completion-flex-try-completion (string table pred point)
   "Try to flex-complete STRING in TABLE given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
@@ -4006,13 +4155,13 @@ completion-flex-try-completion
 (defun completion-flex-all-completions (string table pred point)
   "Get flex-completions of STRING in TABLE, given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
-      (when all
-        (nconc (completion-pcm--hilit-commonality pattern all)
-               (length prefix))))))
+      (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix))))))
 
 ;; Initials completion
 ;; Complete /ums to /usr/monnier/src or lch to list-command-history.
@@ -4049,7 +4198,11 @@ completion-initials-expand
 (defun completion-initials-all-completions (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
     (when newstr
-      (completion-pcm-all-completions newstr table pred (length newstr)))))
+      (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+                   (completion-pcm--find-all-completions newstr table
+                                                         pred (length newstr))))
+        (completion-pcm--deferred-hilit pattern all
+                                        (length prefix) (length string))))))
 
 (defun completion-initials-try-completion (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
diff --git a/test/lisp/minibuffer-tests.el b/test/lisp/minibuffer-tests.el
index c3ba8f9a92..4ebf27fd1d 100644
--- a/test/lisp/minibuffer-tests.el
+++ b/test/lisp/minibuffer-tests.el
@@ -188,10 +188,6 @@ completion-all-sorted-completions
                   '("some/alpha" "base/epsilon" "base/delta"))
                  `("epsilon" "delta" "beta" "alpha" "gamma"  . 5))))
 
-(defun completion--pcm-score (comp)
-  "Get `completion-score' from COMP."
-  (get-text-property 0 'completion-score comp))
-
 (defun completion--pcm-first-difference-pos (comp)
   "Get `completions-first-difference' from COMP."
   (cl-loop for pos = (next-single-property-change 0 'face comp)
@@ -215,25 +211,12 @@ completion-pcm-test-2
            "barfoobar")))
 
 (ert-deftest completion-pcm-test-3 ()
-  ;; Full match!
-  (should (eql
-           (completion--pcm-score
-            (car (completion-pcm-all-completions
-                  "R" '("R" "hello") nil 1)))
-           1.0)))
-
-(ert-deftest completion-pcm-test-4 ()
-  ;; One fourth of a match and no match due to point being at the end
-  (should (eql
-           (completion--pcm-score
-            (car (completion-pcm-all-completions
-                  "RO" '("RaOb") nil 1)))
-           (/ 1.0 4.0)))
+  ;; No match due to point being at the end
   (should (null
            (completion-pcm-all-completions
             "RO" '("RaOb") nil 2))))
 
-(ert-deftest completion-pcm-test-5 ()
+(ert-deftest completion-pcm-test-4 ()
   ;; Since point is at the beginning, there is nothing that can really
   ;; be typed anymore
   (should (null
@@ -241,7 +224,7 @@ completion-pcm-test-5
             (car (completion-pcm-all-completions
                   "f" '("few" "many") nil 0))))))
 
-(ert-deftest completion-pcm-test-6 ()
+(ert-deftest completion-pcm-test-5 ()
   ;; Wildcards and delimiters work
   (should (equal
            (car (completion-pcm-all-completions
@@ -252,26 +235,12 @@ completion-pcm-test-6
                  "li-pac*" '("do-not-list-packages") nil 7)))))
 
 (ert-deftest completion-substring-test-1 ()
-  ;; One third of a match!
   (should (equal
            (car (completion-substring-all-completions
                  "foo" '("hello" "world" "barfoobar") nil 3))
-           "barfoobar"))
-  (should (eql
-           (completion--pcm-score
-            (car (completion-substring-all-completions
-                  "foo" '("hello" "world" "barfoobar") nil 3)))
-           (/ 1.0 3.0))))
+           "barfoobar")))
 
 (ert-deftest completion-substring-test-2 ()
-  ;; Full match!
-  (should (eql
-           (completion--pcm-score
-            (car (completion-substring-all-completions
-                  "R" '("R" "hello") nil 1)))
-           1.0)))
-
-(ert-deftest completion-substring-test-3 ()
   ;; Substring match
   (should (equal
            (car (completion-substring-all-completions
@@ -281,7 +250,7 @@ completion-substring-test-3
            (car (completion-substring-all-completions
                  "custgroup" '("customize-group") nil 5)))))
 
-(ert-deftest completion-substring-test-4 ()
+(ert-deftest completion-substring-test-3 ()
   ;; `completions-first-difference' should be at the right place
   (should (eql
            (completion--pcm-first-difference-pos
@@ -306,14 +275,6 @@ completion-flex-test-1
            "fabrobazo")))
 
 (ert-deftest completion-flex-test-2 ()
-  ;; Full match!
-  (should (eql
-           (completion--pcm-score
-            (car (completion-flex-all-completions
-                  "R" '("R" "hello") nil 1)))
-           1.0)))
-
-(ert-deftest completion-flex-test-3 ()
   ;; Another fuzzy match, but more of a "substring" one
   (should (equal
            (car (completion-flex-all-completions
@@ -331,5 +292,213 @@ completion-flex-test-3
                   "custgroup" '("customize-group-other-window") nil 9)))
            15)))
 
+(ert-deftest completion-flex-score-test-1 ()
+  ;; Full match!
+  (should (equal
+           (completion--flex-score '(prefix "R") '("R"))
+           (list (cons -1.0 "R")))))
+
+(ert-deftest completion-flex-score-test-2 ()
+  ;; One third and half of a match!
+  (should (equal
+           (completion--flex-score '(prefix "foo")
+                                   '("barfoobar" "fooboo"))
+           (list (cons (/ -1.0 3.0) "barfoobar")
+                 (cons (/ -1.0 2.0) "fooboo")))))
+
+(ert-deftest completion-flex-score-test-3 ()
+  ;; One fourth of a match
+  (should (eql
+           (caar (completion--flex-score '(prefix "R" point "O")
+                                         '("RaOb")))
+           (/ -1.0 4.0))))
+
+(ert-deftest completion-flex-score-test-4 ()
+  ;; For quoted completion tables, score the unquoted completion string.
+  (should (equal
+           (completion--flex-score
+            '(prefix "R")
+            (list (propertize "X" 'completion--unquoted "R")))
+           (list (cons -1.0 "R")))))
+
+(defun completion--test-style (style string point table filtered)
+  (let* ((completion-styles (list style))
+         (pred (lambda (x) (not (string-search "!" x))))
+         (result (completion-filter-completions
+                  string table pred point nil)))
+    (should (equal (alist-get 'base result) 0))
+    (should (equal (alist-get 'end result) (length string)))
+    (should (equal (alist-get 'completions result) filtered))
+    (should (not (memq (alist-get 'highlight result) '(nil identity))))
+    (should (equal (completion-all-completions string table pred point)
+                   (append filtered 0)))))
+
+(ert-deftest completion-basic-style-test-1 ()
+  ;; point at the beginning |foo
+  (completion--test-style 'basic "foo" 0
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-basic-style-test-2 ()
+  ;; point foo
+  (completion--test-style 'basic "foo" 2
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-substring-style-test ()
+  (completion--test-style 'substring "foo" 1
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-emacs21-style-test ()
+  (completion--test-style 'emacs21 "foo" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-emacs22-style-test ()
+  (completion--test-style 'emacs22 "fo0" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar" "fobar"))) ;; suffix ignored completely
+
+(ert-deftest completion-flex-style-test ()
+  (completion--test-style 'flex "abc" 1
+                          '("abc" "abc!" "xaybzc" "xaybz")
+                          '("abc" "xaybzc")))
+
+(ert-deftest completion-initials-style-test ()
+  (completion--test-style 'initials "abc" 1
+                          '("a-b-c" "a-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("a-b-c" "ax-by-cz")))
+
+(ert-deftest completion-pcm-style-test ()
+  (completion--test-style 'partial-completion "ax-b-c" 1
+                          '("ax-b-c" "ax-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("ax-b-c" "ax-by-cz")))
+
+(ert-deftest completion-filter-completions-highlight-test ()
+  ;; point at the beginning |foo
+  (let* ((completion-styles '(basic))
+         (result (completion-filter-completions
+                  "foo" '("foobar" "fbarfoo" "fxfooy" "bar")
+                  nil 1 nil)))
+    (should (equal
+             (format "%S" (alist-get 'completions result))
+             (format "%S" '("foobar" "fbarfoo" "fxfooy"))))
+    (should (equal
+             (format "%S" (funcall (alist-get 'highlight result)
+                                   (alist-get 'completions result)))
+             (format "%S"
+                     '(#("foobar" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fbarfoo" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fxfooy" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))))))))
+
+(defun completion--test-boundaries (style string table result)
+  (let ((table
+         (lambda (str pred action)
+           (pcase action
+             (`(boundaries . ,suffix) `(boundaries
+                                        ,(1+ (string-match-p "<\\|/" str))
+                                        . ,(or (string-search ">" suffix) (length suffix))))
+             (_ (complete-with-action action table
+                                      (replace-regexp-in-string ".*[</]" "" str)
+                                      pred)))))
+        (point (string-search "|" string))
+        (string (string-replace "|" "" string))
+        (completion-styles (list style)))
+    (should (equal
+             (assq-delete-all
+              (if (assq 'highlight result) '-does-not-exist 'highlight)
+              (completion-filter-completions
+               string table nil point nil))
+             result))
+    (should (equal
+             (completion-all-completions
+              string table nil point)
+             (append (alist-get 'completions result)
+                     (alist-get 'base result))))))
+
+(ert-deftest completion-emacs21-boundaries-test ()
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("ainput>after" "input>after" "inpux>after"
+                                 "inxputy>after" "input>after2")
+                               '((base . 7)
+                                 (end . 18)
+                                 (completions "input>after" "input>after2"))))
+
+(ert-deftest completion-emacs22-boundaries-test ()
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "inyy" "inzzz"))))
+
+(ert-deftest completion-basic-boundaries-test ()
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("ainput" "input" "inpux" "inxputy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "input" "inxputy"))))
+
+(ert-deftest completion-substring-boundaries-test ()
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("other") nil)
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("ainputs" "inputs" "inpux" "inxputsy")
+                               '((base . 7)
+                                 (end . 13)
+                                 (completions "ainputs" "inputs" "inxputsy"))))
+
+(ert-deftest completion-pcm-boundaries-test ()
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("other") nil)
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("ain-pu-ts" "in-pts" "in-pu-ts" "in-px" "inx-ptsy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "in-pts" "in-pu-ts" "inx-ptsy"))))
+
+(ert-deftest completion-initials-boundaries-test ()
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("other") nil)
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("ain/pu/ts" "in/pts" "in/pu/ts" "a/in/pu/ts"
+                                 "in/pu/ts/foo" "in/px" "inx/ptsy")
+                               '((base . 1)
+                                 (end . 4)
+                                 (completions "in/pu/ts" "in/pu/ts/foo"))))
+
+(defun completion-emacs22orig-all-completions (string table pred point)
+  (let ((beforepoint (substring string 0 point)))
+    (completion-hilit-commonality
+      (all-completions beforepoint table pred)
+     point
+     (car (completion-boundaries beforepoint table pred "")))))
+
+(ert-deftest completion-upgrade-return-type-test ()
+  ;; Test transparent upgrade of old completion style return value
+  ;; to new return value format.
+  (let ((completion-styles-alist
+         '((emacs22orig completion-emacs22-try-completion
+                        completion-emacs22orig-all-completions nil))))
+  (completion--test-boundaries 'emacs22orig "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 ;; 18 is incorrect, should be 12!
+                                 ;; But the information is not available
+                                 ;; due to the completion-style upgrade.
+                                 (end . 18)
+                                 ;; Identity highlighting function.
+                                 (highlight . identity)
+                                 (completions "inyy" "inzzz")))))
+
 (provide 'minibuffer-tests)
 ;;; minibuffer-tests.el ends here
-- 
2.20.1

bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/11/21 4:16 PM, Daniel Mendler wrote:
> I prepared a patch which provides the API
> `completion-filter-completions`. This function supports deferred
> highlighting and returns additional data with the list of matching
> completion candidates. The API supersedes the existing function
> `completion-all-completions`.
> 
> The main goal of the new API is to avoid expensive string allocations
> and highlighting during completion. This is particularly relevant for
> continuously updating completion UIs like Icomplete or Vertico.
> Furthermore the end position of the completion boundaries is returned
> with the completion results. This information is not provided by the
> existing `completion-all-completions` API.
> 
> See also the relevant bugs bug#47711 and bug#48841. I am looking forward
> to your feedback. Thank you!

There are currently two issues with the patch with regards to backward
compatibility. Fortunately they are fixable with a little effort.

1. I would like to deprecate `completion-score' or remove it altogether,
   but unfortunately `completion-score' is used in the wild. In order to
   preserve `completion-score', bind `completion--filter-completions' in
   the highlighting functions. Add `completion-score' in
   `completion-pcm--hilit-commonality' when
   `completion--filter-completions' is nil.

2. In `completion--nth-completion' set `completion--filter-completions'
   to nil, unless `(memq style '(emacs21 emacs22 basic
   partial-completion initials flex))' such that custom completion
   styles which wrap the completion functions don't see the new return
   value format, except if the custom style opts in explicitly by
   binding `completion--filter-completions'. An alternative criterion is
   `(memq fun '(completion-emacs22-all-completions) ...)'. Unfortunately
   this approach will still not work if the user has advised a
   `completion-x-all-completions' function. The only 100% safe approach
   seems to transparently redirect calls to
   `completion-x-all-completions' to `completion--x-filter-completions',
   which returns the results in the new format.

With these changes the patch should be 100% backward compatible.

Daniel

bug#47711: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Perhaps you should first provide a patch with these 2 "little effort" changes,
(that are presumably also backward compatible and don't affect the API) by
themselves.  Reading about these complex ideas isn't as clear as seeing
them in actual code.

Then it'll be easier to evaluate the merits of the patch you proposed in
your first email.

João

On Wed, Aug 11, 2021 at 5:11 PM Daniel Mendler <mail@daniel-mendler.de> wrote:
>
> On 8/11/21 4:16 PM, Daniel Mendler wrote:
> > I prepared a patch which provides the API
> > `completion-filter-completions`. This function supports deferred
> > highlighting and returns additional data with the list of matching
> > completion candidates. The API supersedes the existing function
> > `completion-all-completions`.
> >
> > The main goal of the new API is to avoid expensive string allocations
> > and highlighting during completion. This is particularly relevant for
> > continuously updating completion UIs like Icomplete or Vertico.
> > Furthermore the end position of the completion boundaries is returned
> > with the completion results. This information is not provided by the
> > existing `completion-all-completions` API.
> >
> > See also the relevant bugs bug#47711 and bug#48841. I am looking forward
> > to your feedback. Thank you!
>
> There are currently two issues with the patch with regards to backward
> compatibility. Fortunately they are fixable with a little effort.
>
> 1. I would like to deprecate `completion-score' or remove it altogether,
>    but unfortunately `completion-score' is used in the wild. In order to
>    preserve `completion-score', bind `completion--filter-completions' in
>    the highlighting functions. Add `completion-score' in
>    `completion-pcm--hilit-commonality' when
>    `completion--filter-completions' is nil.
>
> 2. In `completion--nth-completion' set `completion--filter-completions'
>    to nil, unless `(memq style '(emacs21 emacs22 basic
>    partial-completion initials flex))' such that custom completion
>    styles which wrap the completion functions don't see the new return
>    value format, except if the custom style opts in explicitly by
>    binding `completion--filter-completions'. An alternative criterion is
>    `(memq fun '(completion-emacs22-all-completions) ...)'. Unfortunately
>    this approach will still not work if the user has advised a
>    `completion-x-all-completions' function. The only 100% safe approach
>    seems to transparently redirect calls to
>    `completion-x-all-completions' to `completion--x-filter-completions',
>    which returns the results in the new format.
>
> With these changes the patch should be 100% backward compatible.
>
> Daniel



-- 
João Távora

bug#47711: bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

[I removed emacs-devel from the CC list, please don't cross-post to
both emacs-devel and the bug tracker.]

> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Wed, 11 Aug 2021 16:16:57 +0200
> Cc: 48841@debbugs.gnu.org, Dmitry Gutov <dgutov@yandex.ru>,
>  João Távora <joaotavora@gmail.com>,
>  Stefan Monnier <monnier@iro.umontreal.ca>, 47711@debbugs.gnu.org
> 
> I prepared a patch which provides the API
> `completion-filter-completions`. This function supports deferred
> highlighting and returns additional data with the list of matching
> completion candidates. The API supersedes the existing function
> `completion-all-completions`.

Thanks.  The discussion of this is still going on, and I don't
consider myself an expert in this area of Emacs, so below please find
only comments for minor formatting and documentation aspects.

> Add a new `completion-filter-completions` API, which supersedes
> `completion-all-completions`.  The new API returns the matching
> completion candidates and additional data.  The return value is an
> alist, with the keys `completions`, `base`, `end` and `highlight`.
> The API can be extended in a backward compatible way later on thanks
> to the use of an alist as return value.

Please don't use Markdown-style quoting `like this` in our comments
and log messages.  We quote 'like this' in these places.

> The `completions` value is the list of completion strings *without*
> applied highlighting.  The completion strings are returned unmodified,
> which avoids allocations and results in performance gains for

This is unclear: how can you return a list of strings which you
produce without allocating the strings?

>         The value `base` is the base position of the completion.

"Base position" where, or relative to what object?

> Correspondingly the value `end` specifies the end position of the
> completion counted from the beginning of the input strng.

So the base position is also relative to the beginning of the input
string?  If so, please say so explicitly.

>                                                    Finally the
> `highlight` value is a function taking a list of completion strings
> and returns a new list of new strings with highlighting applied.

If you say "taking a list...", then for consistent style please also
say "...and returning a new list...".

> A continously updating UI can use the highlighting function to apply
> highlighting only to the visible completions.

Not, "visible", but "displayed", right?  So I'd rephrase

  ...to apply highlighting only to the completions that are actually
  displayed.

> (completion-basic-all-completions,
> completion-emacs21-all-completions,
> completion-emacs22-all-completions): Use it.

That's incorrect format, I guess you did it by hand rather than
letting change-log-mode do it for you?  The correct format looks like
this:

  (completion-basic-all-completions)
  (completion-emacs21-all-completions)
  (completion-emacs22-all-completions): use it.

IOW, each line ends with a closing parentheses, not a comma.

> +(defvar completion--filter-completions nil
> +  "Enable the new completions return value format.

Using genitive construction should be limited to just 2 words.  Here
you have 4, which produces an awkward, hard to interpret phrase.
Suggest to reword:

  If non-nil, return completions in `completion-filter-completions' format.

Note that I also dropped the "new" part (which generally doesn't stand
the test of time), and instead gave a hint as to what that format is.

Btw, why is this an internal variable?  Shouldn't all completion
styles ideally support it?  If so, it should be a public variable,
documented in the ELisp manual.  And the name should also end with -p,
since it's a boolean.  How about completion-filter-completions-format-p?

> +            New completion style functions may always return their
> +results in the new alist format, since `completion-all-completions'
> +transparently converts back to the old improper list of completions
> +with base size in the last cdr.")

"may" and "always" are a kind of contradiction.

Also, I'd drop the "improper" part, as it sounds like a derogatory
adjective.

>  (defun completion-try-completion (string table pred point &optional metadata)
>    "Try to complete STRING using completion table TABLE.
>  Only the elements of table that satisfy predicate PRED are considered.
> -POINT is the position of point within STRING.
> -The return value can be either nil to indicate that there is no completion,
> -t to indicate that STRING is the only possible completion,
> -or a pair (NEWSTRING . NEWPOINT) of the completed result string together with
> -a new position for point."
> +POINT is the position of point within STRING.  The return value can be
> +either nil to indicate that there is no completion, t to indicate that
> +STRING is the only possible completion, or a pair (NEWSTRING . NEWPOINT)
> +of the completed result string together with a new position for point.
> +The METADATA may be modified by the completion style."

Here you changed whitespace by filling, and that ruined the
intentionally formatted doc string, which made it easy to find the
various forms of the return value and the important parts of the doc
string.  Please keep the original formatting.

>  (defun completion-all-completions (string table pred point &optional metadata)
>    "List the possible completions of STRING in completion table TABLE.
>  Only the elements of table that satisfy predicate PRED are considered.
> -POINT is the position of point within STRING.
> -The return value is a list of completions and may contain the base-size
> -in the last `cdr'."
> -  ;; FIXME: We need to additionally return the info needed for the
> -  ;; second part of completion-base-position.
> -  (completion--nth-completion 2 string table pred point metadata))
> +POINT is the position of point within STRING.  The return value is a
> +list of completions and may contain the base-size in the last `cdr'.
> +The METADATA may be modified by the completion style.  This function
> +has been superseded by `completion-filter-completions', which returns
> +richer information and supports deferred candidate highlighting."

Likewise here.

Also, the "This function has been superseded..." part should be a new
paragraph, so that it stands out.  (And I'm not yet sure we indeed
want to say "superseded" here, but that's part of the on-going
discussion.  maybe use a more neutral language here, like "See also".)

> +    (if (and result (consp (car result)))
> +        ;; Give the completion styles some freedom!
> +        ;; If they are targeting Emacs 28 upwards only, they
> +        ;; may always return a result with deferred
> +        ;; highlighting.  We convert back to the old format
> +        ;; here by applying the highlighting eagerly.

"May always" again.  How about "can always" instead?

> +        (nconc (funcall (cdr (assq 'highlight result))
> +                        (cdr (assq 'completions result)))
> +               (cdr (assq 'base result)))
> +      result)))
> +
> +(defun completion-filter-completions (string table pred point metadata)
> +  "Filter the possible completions of STRING in completion table TABLE.

Is "filter" really the right word here (in the doc string)?  "Filer"
means you take a sequence and produce another sequence with some
members removed.  That's not what this API does, is it?  Suggest to
use a different name, like completion-completions-alist or
completion-all-completions-as-alist.

> +Only the elements of table that satisfy predicate PRED are considered.
> +POINT is the position of point within STRING.  The METADATA may be
> +modified by the completion style.  The return value is a alist with
> +the keys:
> +
> +- base: Base position of the completion (from the start of STRING)

"Base" here means the beginning?  If so, why not call it "beg" or
somesuch?

> +This function supersedes the function `completion-all-completions'."

Again, "supersedes" is too strong, IMO.  I would say "is a variant of"
instead, and explain why this variant could be better suited to some
use cases.  IOW, explain the upsides (and downsides, if any), and let
the programmers decide whether they want this, instead of more-or-less
forcing them to use it.

> +        ;; Deferred highlighting has been requested, but the completion
> +        ;; style returned a non-deferred result. Convert the result to the
                                                  ^^
two spaces between sentences, please.

> +        ;; new alist format.

"New" is not a good word here.

> +      ;; added by the completion machinery.

Please start comments with a capital letter.

> +(defun completion--deferred-hilit (completions prefix-len base end)
> +  "Return completions in old format or new alist format.
> +If `completion--filter-completions' is non-nil use the new format."

Again, please don't use "old" and "new" here, but instead describe
explicitly the differences between them, or provide a hyperlink to
where that is described.

> +                      ;; Apply highlighting

Please end each sentence in a comment with a period.

> +(defun completion-pcm--deferred-hilit (pattern completions base end)
> +  "Return completions in old format or new alist format.
> +If `completion--filter-completions' is non-nil use the new format."

"Old" and "new" again.

>  (defun completion-pcm--hilit-commonality (pattern completions)
>    "Show where and how well PATTERN matches COMPLETIONS.
>  PATTERN, a list of symbols and strings as seen
>  `completion-pcm--merge-completions', is assumed to match every
>  string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
> -each string is propertized with `completion-score', a number
> -between 0 and 1, and with faces `completions-common-part',
> +each string is propertized with faces `completions-common-part',
>  `completions-first-difference' in the relevant segments."

Are we really losing the completion-score property here?  If so, why?

> +           ;; If `pattern' doesn't have an explicit trailing any,

This is confusing: what do you mean by "explicit trailing any" in the
context of patterns?

> +(defun completion--flex-score (pattern completions)
> +  "Compute how well PATTERN matches COMPLETIONS.
> +PATTERN, a list of strings is assumed to match every string in
> +COMPLETIONS.

Is PATTERN really a list?  It would be strange for a list to be called
PATTERN, and how can a list "match every string in COMPLETIONS"?

>               Return a copy of COMPLETIONS where each element is
> +a pair of a score and the completion string.

What is "the completion string" in this case? is it the same string
from COMPLETIONS, or is it something else?  The doc string leaves that
unclear.

>                                                The score lies in
> +the range between -1 and 0, where -1 corresponds to the full
> +match."

What score could a partial match have, and what is the meaning of the
numerical value for a partial match?

bug#47711: bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

Eli, thank you for your feedback and for pointing me to the mode which
helps with the formatting.  I will address the documentation and
formatting issues as soon as the discussion has concluded.

In the following I answer to a few of your questions about technical
details.

>> The `completions` value is the list of completion strings *without*
>> applied highlighting.  The completion strings are returned unmodified,
>> which avoids allocations and results in performance gains for
> 
> This is unclear: how can you return a list of strings which you
> produce without allocating the strings?

The function 'completion-filter-completions' receives a completion table
as argument.  The strings produced by this table are returned
unmodified, but of course the completion table has to produce them.  For
a static completion table (e.g., in the simplest case a list of strings)
the completion table itself will not allocate strings.  In this scenario
'completion-filter-completions' will not perform any string allocations,
only the list will be allocated.  This is what leads to major
performance gains.

>> +(defvar completion--filter-completions nil
>> +  "Enable the new completions return value format.
>
> Btw, why is this an internal variable?  Shouldn't all completion
> styles ideally support it?  If so, it should be a public variable,
> documented in the ELisp manual.  And the name should also end with -p,
> since it's a boolean.  How about completion-filter-completions-format-p?

(As I understood the style guide '-p' is not a good idea for boolean
variables, since a value is not a predicate in a strict sense.)

To address your technical comment - this variable is precisely what one
of the technical difficulties mentioned in my other mail is about.  The
question is how we can retain backward compatibility of the completion
style 'all' functions, e.g., 'completion-basic-all-completions', while
still allowing the function to return the newly introduced alist format
with more data, which enables 'completion-filter-completions' to perform
the efficient deferred highlighting.

> Also, the "This function has been superseded..." part should be a new
> paragraph, so that it stands out.  (And I'm not yet sure we indeed
> want to say "superseded" here, but that's part of the on-going
> discussion.  maybe use a more neutral language here, like "See also".)

The new API 'completion-filter-completions' will substitute the existing
API 'completion-all-completions'.  I only didn't go as far as
deprecating the 'completion-all-completions' API right away, but we
could also do this.

> Is "filter" really the right word here (in the doc string)?  "Filer"
> means you take a sequence and produce another sequence with some
> members removed.  That's not what this API does, is it?  Suggest to
> use a different name, like completion-completions-alist or
> completion-all-completions-as-alist.

"Filter" seems like exactly the right word to me.  The function takes a
list of strings (or a completion table) and returns a subset of matching
completion strings without further modifications to the strings. See
above what I wrote about allocations.

>> +Only the elements of table that satisfy predicate PRED are considered.
>> +POINT is the position of point within STRING.  The METADATA may be
>> +modified by the completion style.  The return value is a alist with
>> +the keys:
>> +
>> +- base: Base position of the completion (from the start of STRING)
> 
> "Base" here means the beginning?  If so, why not call it "beg" or
> somesuch?

Base position is a fixed term which is already used in minibuffer.el for
completions.  See also 'completion-base-position' for example.

> Are we really losing the completion-score property here?  If so, why?

Yes, the property is removed in the current patch.  It is not actually
used for anything in the new implementation.  But it is possible to
restore the property such that 'completion-all-completions' always
returns scored candidates as it does now.  See my other mail regarding
the caveats of the current patch.

Daniel

bug#47711: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

[-- Attachment #1: Type: text/plain, Size: 1747 bytes --]

On 8/11/21 6:11 PM, Daniel Mendler wrote:
> 2. In `completion--nth-completion' set `completion--filter-completions'
>    to nil, unless `(memq style '(emacs21 emacs22 basic
>    partial-completion initials flex))' such that custom completion
>    styles which wrap the completion functions don't see the new return
>    value format, except if the custom style opts in explicitly by
>    binding `completion--filter-completions'. An alternative criterion is
>    `(memq fun '(completion-emacs22-all-completions) ...)'. Unfortunately
>    this approach will still not work if the user has advised a
>    `completion-x-all-completions' function. The only 100% safe approach
>    seems to transparently redirect calls to
>    `completion-x-all-completions' to `completion--x-filter-completions',
>    which returns the results in the new format.

I attached two patch variants which can be placed on top of my previous
patch to improve the backward compatibility of the internal API.

Variant 1: Set 'completion--return-alist-flag' only for the existing
completion styles, such that they transparently upgrade to the alist
return format.  If the variable is not set, the completion styles return
the result as plain list retaining backward compatibility.  The variable
is purely for internal use, new completion styles should return their
results as an alist on Emacs 28 and newer.

Variant 2: Add an optional argument FILTER to each of the completion
styles 'all' functions, e.g., 'completion-basic-all-completions'.  In
'completion--nth-completion' try to call the function with the
additional FILTER argument to upgrade to the alist return format.  If
this fails with a 'wrong-number-of-arguments' error, retry again without
the argument.

Daniel

[-- Attachment #2: variant1-restrict.el --]
[-- Type: text/plain, Size: 7196 bytes --]

From 5539eea7df1570b6b5045be2b383d2bfbd44789e Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Thu, 12 Aug 2021 10:20:10 +0200
Subject: [PATCH] Set 'completion--return-alist-flag' only for the existing
 styles

This avoids problems if a package wraps an existing completion
style function in a custom completion style.
---
 lisp/minibuffer.el | 66 +++++++++++++++++++++++++---------------------
 1 file changed, 36 insertions(+), 30 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index ba8855c4ea..3257349a1a 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1039,16 +1039,16 @@ completion--styles
         (delete-dups (append (cdr over) (copy-sequence completion-styles)))
        completion-styles)))
 
-(defvar completion--filter-completions nil
+(defvar completion--return-alist-flag nil
   "Enable the new completions return value format.
 If this variable is non-nil the `all-completions' function of a
 completion style should return the results in the new alist format of
 `completion-filter-completions'.  This variable is purely needed to
 for backward compatibility of the existing builtin completion style
-functions.  New completion style functions may always return their
-results in the new alist format, since `completion-all-completions'
-transparently converts back to the old improper list of completions
-with base size in the last cdr.")
+functions as of Emacs 28.  New completion style functions should
+always return their results in the new alist format, since
+`completion-all-completions' transparently converts back to the old
+improper list of completions with base size in the last cdr.")
 
 (defun completion--nth-completion (n string table pred point metadata)
   "Call the Nth method of completion styles."
@@ -1084,7 +1084,7 @@ completion--nth-completion
               ;; contrast to plain completion tables, the savings of
               ;; deferred highlighting would be minimal in the case of
               ;; quoted completion tables.
-              (setq completion--filter-completions nil)
+              (setq completion--return-alist-flag nil)
               (setq string (pop new))
               (setq table (pop new))
               (setq point (pop new))
@@ -1093,18 +1093,35 @@ completion--nth-completion
          (result-and-style
           (completion--some
            (lambda (style)
-             (let ((probe (funcall (nth n (assq style
-                                                completion-styles-alist))
-                                   string table pred point)))
+             (let* ((fun (nth n (assq style completion-styles-alist)))
+                    ;; Transparently upgrade the return value for
+                    ;; existing built-in styles as of Emacs 28.  No
+                    ;; new styles should be added here. New completion
+                    ;; styles should directly return the new
+                    ;; completion format.el
+                    (completion--return-alist-flag
+                     (and completion--return-alist-flag
+                          (memq style '(emacs21 emacs22 basic substring
+                                        partial-completion initials flex))))
+                    (probe (funcall fun string table pred point)))
                (and probe (cons probe style))))
            (completion--styles md)))
-         (style-md (get (cdr result-and-style) 'completion--style-metadata)))
+         (style-md (get (cdr result-and-style) 'completion--style-metadata))
+         (result (car result-and-style)))
     (when (and style-md metadata)
       (setcdr metadata (cdr (funcall style-md
                                      string table pred point metadata))))
+    (when (and (not completion--return-alist-flag) (= n 2) (consp (car result)))
+      ;; Give the completion styles some freedom!  If they are
+      ;; targeting Emacs 28 upwards only, they may return a result
+      ;; with deferred highlighting.  We convert back to the old
+      ;; format here by applying the highlighting eagerly.
+      (setq result (nconc (funcall (cdr (assq 'highlight result))
+                                   (cdr (assq 'completions result)))
+                          (cdr (assq 'base result)))))
     (if requote
-        (funcall requote (car result-and-style) n)
-      (car result-and-style))))
+        (funcall requote result n)
+      result)))
 
 (defun completion-try-completion (string table pred point &optional metadata)
   "Try to complete STRING using completion table TABLE.
@@ -1124,19 +1141,8 @@ completion-all-completions
 The METADATA may be modified by the completion style.  This function
 has been superseded by `completion-filter-completions', which returns
 richer information and supports deferred candidate highlighting."
-  (let ((completion--filter-completions nil)
-        (result (completion--nth-completion 2 string table
-                                            pred point metadata)))
-    (if (and result (consp (car result)))
-        ;; Give the completion styles some freedom!
-        ;; If they are targeting Emacs 28 upwards only, they
-        ;; may always return a result with deferred
-        ;; highlighting.  We convert back to the old format
-        ;; here by applying the highlighting eagerly.
-        (nconc (funcall (cdr (assq 'highlight result))
-                        (cdr (assq 'completions result)))
-               (cdr (assq 'base result)))
-      result)))
+  (let ((completion--return-alist-flag nil))
+    (completion--nth-completion 2 string table pred point metadata)))
 
 (defun completion-filter-completions (string table pred point metadata)
   "Filter the possible completions of STRING in completion table TABLE.
@@ -1152,7 +1158,7 @@ completion-filter-completions
 - completions: The list of completions.
 
 This function supersedes the function `completion-all-completions'."
-  (let* ((completion--filter-completions t)
+  (let* ((completion--return-alist-flag t)
          (result (completion--nth-completion 2 string table
                                              pred point metadata)))
     (if (and result (not (consp (car result))))
@@ -2120,8 +2126,8 @@ completion--hilit-commonality
 
 (defun completion--deferred-hilit (completions prefix-len base end)
   "Return completions in old format or new alist format.
-If `completion--filter-completions' is non-nil use the new format."
-  (if completion--filter-completions
+If `completion--return-alist-flag' is non-nil use the new format."
+  (if completion--return-alist-flag
       (when completions
         `((base . ,base)
           (end . ,end)
@@ -3586,9 +3592,9 @@ flex-score-match-tightness
 
 (defun completion-pcm--deferred-hilit (pattern completions base end)
   "Return completions in old format or new alist format.
-If `completion--filter-completions' is non-nil use the new format."
+If `completion--return-alist-flag' is non-nil use the new format."
   (when completions
-    (if completion--filter-completions
+    (if completion--return-alist-flag
         `((base . ,base)
           (end . ,end)
           (highlight . ,(apply-partially
-- 
2.20.1


[-- Attachment #3: variant2-argument.el --]
[-- Type: text/plain, Size: 13882 bytes --]

From f0da1f1d92f20df7d92d9762295ac51390fcee83 Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Thu, 12 Aug 2021 10:10:02 +0200
Subject: [PATCH] Use FILTER argument instead of global
 `completion--filter-completions`

---
 lisp/minibuffer.el | 119 ++++++++++++++++++++++++---------------------
 1 file changed, 64 insertions(+), 55 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index ba8855c4ea..0f8b85ddd4 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1039,18 +1039,7 @@ completion--styles
         (delete-dups (append (cdr over) (copy-sequence completion-styles)))
        completion-styles)))
 
-(defvar completion--filter-completions nil
-  "Enable the new completions return value format.
-If this variable is non-nil the `all-completions' function of a
-completion style should return the results in the new alist format of
-`completion-filter-completions'.  This variable is purely needed to
-for backward compatibility of the existing builtin completion style
-functions.  New completion style functions may always return their
-results in the new alist format, since `completion-all-completions'
-transparently converts back to the old improper list of completions
-with base size in the last cdr.")
-
-(defun completion--nth-completion (n string table pred point metadata)
+(defun completion--nth-completion (n string table pred point metadata &optional filter)
   "Call the Nth method of completion styles."
   ;; We provide special support for quoting/unquoting here because it cannot
   ;; reliably be done within the normal completion-table routines: Completion
@@ -1084,7 +1073,7 @@ completion--nth-completion
               ;; contrast to plain completion tables, the savings of
               ;; deferred highlighting would be minimal in the case of
               ;; quoted completion tables.
-              (setq completion--filter-completions nil)
+              (setq filter nil)
               (setq string (pop new))
               (setq table (pop new))
               (setq point (pop new))
@@ -1093,18 +1082,37 @@ completion--nth-completion
          (result-and-style
           (completion--some
            (lambda (style)
-             (let ((probe (funcall (nth n (assq style
-                                                completion-styles-alist))
-                                   string table pred point)))
+             (let* ((fun (nth n (assq style completion-styles-alist)))
+                    (probe
+                     (if filter
+                         ;; In order to retain backward compatibility
+                         ;; of the calling convention of the exisiting
+                         ;; completion styles, add a fourth FILTER
+                         ;; argument to opt-in to the new return value
+                         ;; format.
+                         (condition-case nil
+                             (funcall fun string table pred point filter)
+                           (wrong-number-of-arguments
+                            (funcall fun string table pred point)))
+                       (funcall fun string table pred point))))
                (and probe (cons probe style))))
            (completion--styles md)))
-         (style-md (get (cdr result-and-style) 'completion--style-metadata)))
+         (style-md (get (cdr result-and-style) 'completion--style-metadata))
+         (result (car result-and-style)))
     (when (and style-md metadata)
       (setcdr metadata (cdr (funcall style-md
                                      string table pred point metadata))))
+    (when (and (not filter) (= n 2) (consp (car result)))
+      ;; Give the completion styles some freedom!  If they are
+      ;; targeting Emacs 28 upwards only, they may return a result
+      ;; with deferred highlighting.  We convert back to the old
+      ;; format here by applying the highlighting eagerly.
+      (setq result (nconc (funcall (cdr (assq 'highlight result))
+                                   (cdr (assq 'completions result)))
+                          (cdr (assq 'base result)))))
     (if requote
-        (funcall requote (car result-and-style) n)
-      (car result-and-style))))
+        (funcall requote result n)
+      result)))
 
 (defun completion-try-completion (string table pred point &optional metadata)
   "Try to complete STRING using completion table TABLE.
@@ -1124,19 +1132,7 @@ completion-all-completions
 The METADATA may be modified by the completion style.  This function
 has been superseded by `completion-filter-completions', which returns
 richer information and supports deferred candidate highlighting."
-  (let ((completion--filter-completions nil)
-        (result (completion--nth-completion 2 string table
-                                            pred point metadata)))
-    (if (and result (consp (car result)))
-        ;; Give the completion styles some freedom!
-        ;; If they are targeting Emacs 28 upwards only, they
-        ;; may always return a result with deferred
-        ;; highlighting.  We convert back to the old format
-        ;; here by applying the highlighting eagerly.
-        (nconc (funcall (cdr (assq 'highlight result))
-                        (cdr (assq 'completions result)))
-               (cdr (assq 'base result)))
-      result)))
+  (completion--nth-completion 2 string table pred point metadata))
 
 (defun completion-filter-completions (string table pred point metadata)
   "Filter the possible completions of STRING in completion table TABLE.
@@ -1152,9 +1148,8 @@ completion-filter-completions
 - completions: The list of completions.
 
 This function supersedes the function `completion-all-completions'."
-  (let* ((completion--filter-completions t)
-         (result (completion--nth-completion 2 string table
-                                             pred point metadata)))
+  (let* ((result (completion--nth-completion 2 string table
+                                             pred point metadata 'filter)))
     (if (and result (not (consp (car result))))
         ;; Deferred highlighting has been requested, but the completion
         ;; style returned a non-deferred result. Convert the result to the
@@ -2118,10 +2113,10 @@ completion--hilit-commonality
      elem)
    completions))
 
-(defun completion--deferred-hilit (completions prefix-len base end)
+(defun completion--deferred-hilit (completions prefix-len base end filter)
   "Return completions in old format or new alist format.
-If `completion--filter-completions' is non-nil use the new format."
-  (if completion--filter-completions
+If FILTER is non-nil use the new format."
+  (if filter
       (when completions
         `((base . ,base)
           (end . ,end)
@@ -3300,12 +3295,14 @@ completion-emacs21-try-completion
         (cons completion (length completion))
       completion)))
 
-(defun completion-emacs21-all-completions (string table pred _point)
+(defun completion-emacs21-all-completions (string table pred _point
+                                                  &optional filter)
   (completion--deferred-hilit
    (all-completions string table pred)
    (length string)
    (car (completion-boundaries string table pred ""))
-   (length string)))
+   (length string)
+   filter))
 
 (defun completion-emacs22-try-completion (string table pred point)
   (let ((suffix (substring string point))
@@ -3327,13 +3324,14 @@ completion-emacs22-try-completion
           (setq suffix (substring suffix 1)))
       (cons (concat completion suffix) (length completion)))))
 
-(defun completion-emacs22-all-completions (string table pred point)
+(defun completion-emacs22-all-completions (string table pred point
+                                                  &optional filter)
   (let* ((beforepoint (substring string 0 point))
          (afterpoint (substring string point))
          (bounds (completion-boundaries beforepoint table pred afterpoint)))
     (completion--deferred-hilit
      (all-completions beforepoint table pred)
-     point (car bounds) (+ point (cdr bounds)))))
+     point (car bounds) (+ point (cdr bounds)) filter)))
 
 ;;; Basic completion.
 
@@ -3381,7 +3379,8 @@ completion-basic-try-completion
             (setq all (completion-pcm--filename-try-filter all)))
         (completion-pcm--merge-try pattern all prefix suffix)))))
 
-(defun completion-basic-all-completions (string table pred point)
+(defun completion-basic-all-completions (string table pred point
+                                                &optional filter)
   (let* ((beforepoint (substring string 0 point))
          (afterpoint (substring string point))
          (bounds (completion-boundaries beforepoint table pred afterpoint))
@@ -3392,7 +3391,9 @@ completion-basic-all-completions
                             'point
                             (substring afterpoint 0 (cdr bounds)))))
          (all (completion-pcm--all-completions prefix pattern table pred)))
-    (completion--deferred-hilit all point (car bounds) (+ point (cdr bounds)))))
+    (completion--deferred-hilit all point
+                                (car bounds) (+ point (cdr bounds))
+                                filter)))
 
 ;;; Partial-completion-mode style completion.
 
@@ -3584,11 +3585,11 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
-(defun completion-pcm--deferred-hilit (pattern completions base end)
+(defun completion-pcm--deferred-hilit (pattern completions base end filter)
   "Return completions in old format or new alist format.
-If `completion--filter-completions' is non-nil use the new format."
+If FILTER is non-nil use the new format."
   (when completions
-    (if completion--filter-completions
+    (if filter
         `((base . ,base)
           (end . ,end)
           (highlight . ,(apply-partially
@@ -3839,12 +3840,14 @@ completion-pcm--find-all-completions
           (signal (car firsterror) (cdr firsterror))
         (list pattern all prefix suffix)))))
 
-(defun completion-pcm-all-completions (string table pred point)
+(defun completion-pcm-all-completions (string table pred point
+                                              &optional filter)
   (pcase-let ((`(,pattern ,all ,prefix ,suffix)
                (completion-pcm--find-all-completions string table pred point)))
     (completion-pcm--deferred-hilit pattern all
                                     (length prefix)
-                                    (- (length string) (length suffix)))))
+                                    (- (length string) (length suffix))
+                                    filter)))
 
 (defun completion--common-suffix (strs)
   "Return the common suffix of the strings STRS."
@@ -4067,13 +4070,15 @@ completion-substring-try-completion
         (setq all (completion-pcm--filename-try-filter all)))
     (completion-pcm--merge-try pattern all prefix suffix)))
 
-(defun completion-substring-all-completions (string table pred point)
+(defun completion-substring-all-completions (string table pred point
+                                                    &optional filter)
   (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
     (completion-pcm--deferred-hilit pattern all
                                     (length prefix)
-                                    (- (length string) (length suffix)))))
+                                    (- (length string) (length suffix))
+                                    filter)))
 
 ;;; "flex" completion, also known as flx/fuzzy/scatter completion
 ;; Completes "foo" to "frodo" and "farfromsober"
@@ -4152,7 +4157,8 @@ completion-flex-try-completion
       ;; "farfromsober".
       (completion-pcm--merge-try pattern all prefix suffix))))
 
-(defun completion-flex-all-completions (string table pred point)
+(defun completion-flex-all-completions (string table pred point
+                                               &optional filter)
   "Get flex-completions of STRING in TABLE, given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
     (pcase-let ((`(,all ,pattern ,prefix ,suffix)
@@ -4161,7 +4167,8 @@ completion-flex-all-completions
                   #'completion-flex--make-flex-pattern)))
       (completion-pcm--deferred-hilit pattern all
                                     (length prefix)
-                                    (- (length string) (length suffix))))))
+                                    (- (length string) (length suffix))
+                                    filter))))
 
 ;; Initials completion
 ;; Complete /ums to /usr/monnier/src or lch to list-command-history.
@@ -4195,14 +4202,16 @@ completion-initials-expand
             (concat (substring str 0 (car bounds))
                     (mapconcat 'string (substring str (car bounds)) sep))))))))
 
-(defun completion-initials-all-completions (string table pred _point)
+(defun completion-initials-all-completions (string table pred _point
+                                                   &optional filter)
   (let ((newstr (completion-initials-expand string table pred)))
     (when newstr
       (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
                    (completion-pcm--find-all-completions newstr table
                                                          pred (length newstr))))
         (completion-pcm--deferred-hilit pattern all
-                                        (length prefix) (length string))))))
+                                        (length prefix) (length string)
+                                        filter)))))
 
 (defun completion-initials-try-completion (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
-- 
2.20.1

bug#48841: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

[-- Attachment #1: Type: text/plain, Size: 290 bytes --]

I attached the overhauled patch, which addresses most of the comments by
Eli.  In comparison to my last patch, the patch is fully backward
compatible and preserves all existing tests.  As before, there are tests
which check the new functionality for each existing completion style.

Daniel

[-- Attachment #2: 0001-Add-new-completion-filter-completions-API-and-deferr.patch --]
[-- Type: text/x-diff, Size: 53501 bytes --]

From 30ca42b49d5b5316abb3ebad38b0e9629eb52920 Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Mon, 12 Jul 2021 21:40:32 +0200
Subject: [PATCH] Add new 'completion-filter-completions' API and deferred
 highlighting

Fix bug#47711.

Add a new 'completion-filter-completions' API, which supersedes
'completion-all-completions'.  The new API returns the matching
completion candidates and additional data.  The return value is an
alist, with the keys 'completions', 'base', 'end' and 'highlight'.
The API can be extended in a backward compatible way later on thanks
to the use of an alist as return value.

The 'completions' value is the list of completion strings *without*
applied highlighting.  The completion strings are returned unmodified,
which avoids allocations and results in performance gains for
continuously updating completion UIs, like Icomplete or Vertico (GNU
ELPA).  The value 'base' is the base position of the completion
relative to the beginning of the input string.  Correspondingly the
value 'end' specifies the end position of the completion relative to
the beginning of the input string.  In comparison, the old function
'completion-all-completions' only returned the base position in the
last cdr of the returned completions list, which complicated usage.
The 'end' position was not provided by 'completion-all-completions'.
Given the new API the 'completion-base-position' can be set
accurately.  Finally the 'highlight' value is a function taking a list
of completion strings and returns a new list of new strings with
highlighting applied.  A continously updating UI can use the
highlighting function to apply highlighting only to the visible
completions.

* lisp/minibuffer.el:
(completion--adjust-metadata): Rename to 'completion--style-metadata'
due to change of calling convention.
(completion--nth-completion): Call renamed metadata adjustment
function.  Ignore the old property 'completion--adjust-metadata'.
(completion--flex-adjust-metadata): Rename function.
(completion--twq-all): Attach 'completion--unquoted' text property to
quoted completion strings.
(completion--flex-score-1): Extract new function from
'completion-pcm--hilit-commonality'.
(completion-pcm--hilit-commonality): Use it.  Add SCORE argument.
(completion--flex-score): Use 'completion--flex-score-1'.  Use
'completion--unquoted' text property.
(completion--flex-style-metadata): Use it.
(completion--pattern-compiler): New function.
(completion-substring--all-completions)
(completion--flex-score): Use it.
(completion--hilit-commonality): New function.
(completion-hilit-commonality): Use it.
(completion--deferred-hilit): New function.
(completion-basic-all-completions)
(completion-emacs21-all-completions)
(completion-emacs22-all-completions): Use it.
(completion--pcm-deferred-hilit): New function.
(completion-pcm-all-completions)
(completion-flex-all-completions)
(completion-initials-all-completions)
(completion-substring-all-completions): Use it.
(completion--return-alist-flag): New variable to conditionally enable
the new alist completions result format.  This variable is for
internal use to preserve the existing calling convention of the
completion style 'all' functions.
(completion-filter-completions): New API which returns the completion
strings and additional data as an an alist.  Transparently convert
old list completion style results to the new alist format.
(completion-all-completions): Transparently convert the new
alist completion style result to the old list format.
(minibuffer-completion-help): Use the new API, set
'completion-base-position' correctly.
(completion-try-completion)
(completion-all-completions): Update doc string.
(completion--replace): Fix property removal.

* test/lisp/minibuffer-tests.el:
(completion--test-style)
(completion--test-boundaries): New test helper function.
(completion-emacs22orig-all-completions): New function.
(completion-flex-score-test-*): Add new scoring test functions.
(completion-*-style-test): Add new API tests for each built-in
completion style.
(completion-*-boundaries-test): New boundary tests for each built-in
completion style.
(completion-filter-completions-highlight-test): New API test.
(completion-upgrade-return-type-test): New test of transparent
completion style return value upgrade.
---
 lisp/minibuffer.el            | 580 +++++++++++++++++++++++-----------
 test/lisp/minibuffer-tests.el | 217 ++++++++++++-
 2 files changed, 603 insertions(+), 194 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 9f327df28f..66ac6b3763 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -692,6 +692,10 @@ completion--twq-all
                                              'completions-common-part)
                                qprefix))))
                         (qcompletion (concat qprefix qnew)))
+                   ;; Attach unquoted completion string, which is needed
+                   ;; to score the completion in `completion--flex-score'.
+                   (put-text-property 0 1 'completion--unquoted
+                                      completion qcompletion)
 		   ;; FIXME: Similarly here, Cygwin's mapping trips this
 		   ;; assertion.
                    ;;(cl-assert
@@ -1035,6 +1039,17 @@ completion--styles
         (delete-dups (append (cdr over) (copy-sequence completion-styles)))
        completion-styles)))
 
+(defvar completion--return-alist-flag nil
+  "Non-nil means to return completions in alist format.
+If this variable is non-nil the `all-completions' function of a
+completion style should return the results in the alist format of
+`completion-filter-completions'.  This variable is purely needed to
+for backward compatibility of the existing builtin completion style
+functions as of Emacs 28.  Newer completion style functions should
+always return their results in the alist format, since
+`completion-all-completions' transparently converts back to a list of
+completions with base size in the last cdr.")
+
 (defun completion--nth-completion (n string table pred point metadata)
   "Call the Nth method of completion styles."
   ;; We provide special support for quoting/unquoting here because it cannot
@@ -1061,6 +1076,15 @@ completion--nth-completion
                  ;; the original table, in that case!
                  (functionp table))
             (let ((new (funcall table string point 'completion--unquote)))
+              ;; FIXME For now do not attempt deferred highlighting if
+              ;; quoting is used.  Not doing deferred highlighting is
+              ;; not too severe in this case, since
+              ;; `completion--twq-all' is already an expensive
+              ;; function, which allocates all completion strings.  In
+              ;; contrast to plain completion tables, the savings of
+              ;; deferred highlighting would be minimal in the case of
+              ;; quoted completion tables.
+              (setq completion--return-alist-flag nil)
               (setq string (pop new))
               (setq table (pop new))
               (setq point (pop new))
@@ -1069,17 +1093,35 @@ completion--nth-completion
          (result-and-style
           (completion--some
            (lambda (style)
-             (let ((probe (funcall (nth n (assq style
-                                                completion-styles-alist))
-                                   string table pred point)))
+             (let* ((fun (nth n (assq style completion-styles-alist)))
+                    ;; Transparently upgrade the return value for
+                    ;; existing built-in styles as of Emacs 28.  No
+                    ;; new styles should be added here. New completion
+                    ;; styles should directly return the new
+                    ;; completion format.el
+                    (completion--return-alist-flag
+                     (and completion--return-alist-flag
+                          (memq style '(emacs21 emacs22 basic substring
+                                        partial-completion initials flex))))
+                    (probe (funcall fun string table pred point)))
                (and probe (cons probe style))))
            (completion--styles md)))
-         (adjust-fn (get (cdr result-and-style) 'completion--adjust-metadata)))
-    (when (and adjust-fn metadata)
-      (setcdr metadata (cdr (funcall adjust-fn metadata))))
+         (style-md (get (cdr result-and-style) 'completion--style-metadata))
+         (result (car result-and-style)))
+    (when (and style-md metadata)
+      (setcdr metadata (cdr (funcall style-md
+                                     string table pred point metadata))))
+    (when (and (not completion--return-alist-flag) (= n 2) (consp (car result)))
+      ;; Give the completion styles some freedom!  If they are
+      ;; targeting Emacs 28 upwards only, they may return a result
+      ;; with deferred highlighting.  We convert back to the old
+      ;; format here by applying the highlighting eagerly.
+      (setq result (nconc (funcall (cdr (assq 'highlight result))
+                                   (cdr (assq 'completions result)))
+                          (cdr (assq 'base result)))))
     (if requote
-        (funcall requote (car result-and-style) n)
-      (car result-and-style))))
+        (funcall requote result n)
+      result)))
 
 (defun completion-try-completion (string table pred point &optional metadata)
   "Try to complete STRING using completion table TABLE.
@@ -1088,7 +1130,8 @@ completion-try-completion
 The return value can be either nil to indicate that there is no completion,
 t to indicate that STRING is the only possible completion,
 or a pair (NEWSTRING . NEWPOINT) of the completed result string together with
-a new position for point."
+a new position for point.
+The METADATA may be modified by the completion style."
   (completion--nth-completion 1 string table pred point metadata))
 
 (defun completion-all-completions (string table pred point &optional metadata)
@@ -1096,10 +1139,47 @@ completion-all-completions
 Only the elements of table that satisfy predicate PRED are considered.
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
-in the last `cdr'."
-  ;; FIXME: We need to additionally return the info needed for the
-  ;; second part of completion-base-position.
-  (completion--nth-completion 2 string table pred point metadata))
+in the last `cdr'.
+The METADATA may be modified by the completion style.
+
+This function has been superseded by `completion-filter-completions',
+which returns richer information and supports deferred candidate
+highlighting."
+  (let ((completion--return-alist-flag nil))
+    (completion--nth-completion 2 string table pred point metadata)))
+
+(defun completion-filter-completions (string table pred point metadata)
+  "Filter the possible completions of STRING in completion table TABLE.
+Only the elements of table that satisfy predicate PRED are considered.
+POINT is the position of point within STRING.
+The METADATA may be modified by the completion style.
+The return value is a alist with the keys:
+
+- base: Base position of the completion (from the start of STRING)
+- end: End position of the completion (from the start of STRING)
+- highlight: Highlighting function taking a list of completions and
+  returning a new list of new strings with applied highlighting.
+- completions: The list of completions.
+
+This function supersedes the function `completion-all-completions',
+which does not provide the `end' position of the completion and does
+not support deferred highlighting."
+  (let* ((completion--return-alist-flag t)
+         (result (completion--nth-completion 2 string table
+                                             pred point metadata)))
+    (if (and result (not (consp (car result))))
+        ;; Deferred highlighting has been requested, but the
+        ;; completion style returned a non-deferred result.  Convert
+        ;; the result to the alist format of
+        ;; `completion-filter-completions'.
+        (let* ((last (last result))
+               (base (or (cdr last) 0)))
+          (setcdr last nil)
+          `((base . ,base)
+            (end . ,(length string))
+            (highlight . identity)
+            (completions . ,result)))
+      result)))
 
 (defun minibuffer--bitset (modified completions exact)
   (logior (if modified    4 0)
@@ -1115,7 +1195,8 @@ completion--replace
   (if minibuffer-allow-text-properties
       ;; If we're preserving properties, then just remove the faces
       ;; and other properties added by the completion machinery.
-      (remove-text-properties 0 (length newtext) '(face completion-score)
+      (remove-text-properties 0 (length newtext)
+                              '(face nil completion-score nil)
                               newtext)
     ;; Remove all text properties.
     (set-text-properties 0 (length newtext) nil newtext))
@@ -2021,34 +2102,49 @@ completion-hilit-commonality
 It returns a list with font-lock properties applied to each element,
 and with BASE-SIZE appended as the last element."
   (when completions
-    (let ((com-str-len (- prefix-len (or base-size 0))))
-      (nconc
-       (mapcar
-        (lambda (elem)
-          (let ((str
-                 ;; Don't modify the string itself, but a copy, since the
-                 ;; the string may be read-only or used for other purposes.
-                 ;; Furthermore, since `completions' may come from
-                 ;; display-completion-list, `elem' may be a list.
-                 (if (consp elem)
-                     (car (setq elem (cons (copy-sequence (car elem))
-                                           (cdr elem))))
-                   (setq elem (copy-sequence elem)))))
-            (font-lock-prepend-text-property
-             0
-             ;; If completion-boundaries returns incorrect
-             ;; values, all-completions may return strings
-             ;; that don't contain the prefix.
-             (min com-str-len (length str))
-             'face 'completions-common-part str)
-            (if (> (length str) com-str-len)
-                (font-lock-prepend-text-property com-str-len (1+ com-str-len)
-                                                 'face
-                                                 'completions-first-difference
-                                                 str)))
-          elem)
-        completions)
-       base-size))))
+    (nconc
+     (completion--hilit-commonality (- prefix-len (or base-size 0)) completions)
+     base-size)))
+
+(defun completion--hilit-commonality (com-size completions)
+  (mapcar
+   (lambda (elem)
+     (let ((str
+            ;; Don't modify the string itself, but a copy, since the
+            ;; the string may be read-only or used for other purposes.
+            ;; Furthermore, since `completions' may come from
+            ;; display-completion-list, `elem' may be a list.
+            (if (consp elem)
+                (car (setq elem (cons (copy-sequence (car elem))
+                                      (cdr elem))))
+              (setq elem (copy-sequence elem)))))
+       (font-lock-prepend-text-property
+        0
+        ;; If completion-boundaries returns incorrect
+        ;; values, all-completions may return strings
+        ;; that don't contain the prefix.
+        (min com-size (length str))
+        'face 'completions-common-part str)
+       (if (> (length str) com-size)
+           (font-lock-prepend-text-property com-size (1+ com-size)
+                                            'face
+                                            'completions-first-difference
+                                            str)))
+     elem)
+   completions))
+
+(defun completion--deferred-hilit (completions prefix-len base end)
+  "Return completions as a list or as an alist.
+If `completion--return-alist-flag' is non-nil use the alist format of
+`completion-filter-completions'."
+  (if completion--return-alist-flag
+      (when completions
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially #'completion--hilit-commonality
+                                         (- prefix-len base)))
+          (completions . ,completions)))
+    (completion-hilit-commonality completions prefix-len base)))
 
 (defun display-completion-list (completions &optional common-substring group-fun)
   "Display the list of completions, COMPLETIONS, using `standard-output'.
@@ -2163,15 +2259,16 @@ minibuffer-completion-help
          (end (or end (point-max)))
          (string (buffer-substring start end))
          (md (completion--field-metadata start))
-         (completions (completion-all-completions
-                       string
-                       minibuffer-completion-table
-                       minibuffer-completion-predicate
-                       (- (point) start)
-                       md)))
+         (filtered-completions (completion-filter-completions
+                                string
+                                minibuffer-completion-table
+                                minibuffer-completion-predicate
+                                (- (point) start)
+                                md))
+         (completions (alist-get 'completions filtered-completions)))
     (message nil)
     (if (or (null completions)
-            (and (not (consp (cdr completions)))
+            (and (not (cdr completions))
                  (equal (car completions) string)))
         (progn
           ;; If there are no completions, or if the current input is already
@@ -2181,8 +2278,7 @@ minibuffer-completion-help
           (completion--message
            (if completions "Sole completion" "No completions")))
 
-      (let* ((last (last completions))
-             (base-size (or (cdr last) 0))
+      (let* ((base-size (alist-get 'base filtered-completions))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
              (all-md (completion--metadata (buffer-substring-no-properties
                                             start (point))
@@ -2226,9 +2322,12 @@ minibuffer-completion-help
             (body-function
              . ,#'(lambda (_window)
                     (with-current-buffer mainbuf
-                      ;; Remove the base-size tail because `sort' requires a properly
-                      ;; nil-terminated list.
-                      (when last (setcdr last nil))
+                      ;; Apply highlighting using the deferred
+                      ;; highlighting function provided by
+                      ;; `completion-format-completions'.
+                      (setq completions
+                            (funcall (alist-get 'highlight filtered-completions)
+                                     completions))
 
                       ;; Sort first using the `display-sort-function'.
                       ;; FIXME: This function is for the output of
@@ -2267,13 +2366,10 @@ minibuffer-completion-help
                                       completions))))
 
                       (with-current-buffer standard-output
-                        (setq-local completion-base-position
-                             (list (+ start base-size)
-                                   ;; FIXME: We should pay attention to completion
-                                   ;; boundaries here, but currently
-                                   ;; completion-all-completions does not give us the
-                                   ;; necessary information.
-                                   end))
+                        (setq-local
+                         completion-base-position
+                         (list (+ start base-size)
+                               (+ start (alist-get 'end filtered-completions))))
                         (setq-local completion-list-insert-choice-function
                              (let ((ctable minibuffer-completion-table)
                                    (cpred minibuffer-completion-predicate)
@@ -3223,10 +3319,11 @@ completion-emacs21-try-completion
       completion)))
 
 (defun completion-emacs21-all-completions (string table pred _point)
-  (completion-hilit-commonality
+  (completion--deferred-hilit
    (all-completions string table pred)
    (length string)
-   (car (completion-boundaries string table pred ""))))
+   (car (completion-boundaries string table pred ""))
+   (length string)))
 
 (defun completion-emacs22-try-completion (string table pred point)
   (let ((suffix (substring string point))
@@ -3249,11 +3346,12 @@ completion-emacs22-try-completion
       (cons (concat completion suffix) (length completion)))))
 
 (defun completion-emacs22-all-completions (string table pred point)
-  (let ((beforepoint (substring string 0 point)))
-    (completion-hilit-commonality
+  (let* ((beforepoint (substring string 0 point))
+         (afterpoint (substring string point))
+         (bounds (completion-boundaries beforepoint table pred afterpoint)))
+    (completion--deferred-hilit
      (all-completions beforepoint table pred)
-     point
-     (car (completion-boundaries beforepoint table pred "")))))
+     point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Basic completion.
 
@@ -3312,7 +3410,7 @@ completion-basic-all-completions
                             'point
                             (substring afterpoint 0 (cdr bounds)))))
          (all (completion-pcm--all-completions prefix pattern table pred)))
-    (completion-hilit-commonality all point (car bounds))))
+    (completion--deferred-hilit all point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Partial-completion-mode style completion.
 
@@ -3504,13 +3602,26 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
-(defun completion-pcm--hilit-commonality (pattern completions)
+(defun completion-pcm--deferred-hilit (pattern completions base end)
+  "Return completions as a list or as an alist.
+If `completion--return-alist-flag' is non-nil use the alist format of
+`completion-filter-completions'."
+  (when completions
+    (if completion--return-alist-flag
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially
+                         #'completion-pcm--hilit-commonality
+                         pattern))
+          (completions . ,completions))
+      (nconc (completion-pcm--hilit-commonality pattern completions 'score) base))))
+
+(defun completion-pcm--hilit-commonality (pattern completions &optional score)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
 string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
+each string is propertized with faces `completions-common-part',
 `completions-first-difference' in the relevant segments."
   (when completions
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
@@ -3527,84 +3638,143 @@ completion-pcm--hilit-commonality
                 (match-end (match-end 0))
                 (md (cddr (setq last-md (match-data t last-md))))
                 (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by ]0..1]: the higher the better and only a
-                ;; perfect match (pattern equals string) will have
-                ;; score 1.  The formula takes the form of a quotient.
-                ;; For the numerator, we use the number of +, i.e. the
-                ;; length of the pattern.  For the denominator, it
-                ;; first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
+                (len (length str)))
+           (when (and score (/= 0 len))
+             (put-text-property
+              0 1 'completion-score (- (completion--flex-score-1 md match-end len)) str))
            (while md
-             (funcall update-score-and-face from (pop md))
+             (add-face-text-property from (pop md)
+                                     'completions-common-part
+                                     nil str)
              (setq from (pop md)))
            ;; If `pattern' doesn't have an explicit trailing any, the
            ;; regex `re' won't produce match data representing the
            ;; region after the match.  We need to account to account
            ;; for that extra bit of match (bug#42149).
            (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
+             (add-face-text-property from match-end
+                                     'completions-common-part
+                                     nil str))
+           (if (> len pos)
                (add-face-text-property
                 pos (1+ pos)
                 'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
+                nil str)))
          str)
        completions))))
 
+(defun completion--flex-score-1 (md match-end len)
+  "Compute matching score of completion.
+The score lies in the range between-1 and 0, where -1 corresponds to
+the full match.
+MD is the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by ]0..1]: the higher the better and only a
+         ;; perfect match (pattern equals string) will have
+         ;; score 1.  The formula takes the form of a quotient.
+         ;; For the numerator, we use the number of +, i.e. the
+         ;; length of the pattern.  For the denominator, it
+         ;; first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while md
+      (let ((a from)
+            (b (pop md)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (- (/ score-numerator (* len (1+ score-denominator)) 1.0))))
+
+(defun completion--flex-score (pattern completions)
+  "Compute how well PATTERN matches COMPLETIONS.
+PATTERN, a pcm pattern is assumed to match every string in the
+COMPLETIONS list.  Return a copy of COMPLETIONS where each element is
+a pair of a score and the string.  The score lies in the range between
+-1 and 0, where -1 corresponds to the full match."
+  (when completions
+    (let* ((re (completion-pcm--pattern->regex pattern 'group))
+           (case-fold-search completion-ignore-case)
+           last-md)
+      (mapcar
+       (lambda (str)
+         ;; The flex completion style requires the completion to match
+         ;; the pattern to compute the scoring.  For quoted completion
+         ;; tables the completions are matched against the *unquoted
+         ;; input string*.  However `completion-all-completions' and
+         ;; `completion-filter-completions' return a list of *quoted
+         ;; completions*, which is subsequently sorted.  Therefore we
+         ;; obtain the unquoted completion string which is stored in
+         ;; the text property `completion--unquoted'.
+         (let ((unquoted (or (get-text-property 0 'completion--unquoted str) str)))
+           (unless (string-match re unquoted)
+             (error "Internal error: %s does not match %s" re unquoted))
+           (cons (completion--flex-score-1 (cddr (setq last-md (match-data t last-md)))
+                                           (match-end 0) (length unquoted))
+                 str)))
+       completions))))
+
 (defun completion-pcm--find-all-completions (string table pred point
                                                     &optional filter)
   "Find all completions for STRING at POINT in TABLE, satisfying PRED.
@@ -3700,11 +3870,11 @@ completion-pcm--find-all-completions
         (list pattern all prefix suffix)))))
 
 (defun completion-pcm-all-completions (string table pred point)
-  (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+  (pcase-let ((`(,pattern ,all ,prefix ,suffix)
                (completion-pcm--find-all-completions string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 (defun completion--common-suffix (strs)
   "Return the common suffix of the strings STRS."
@@ -3885,8 +4055,8 @@ completion-pcm-try-completion
 ;;; Substring completion
 ;; Mostly derived from the code of `basic' completion.
 
-(defun completion-substring--all-completions
-    (string table pred point &optional transform-pattern-fn)
+(defun completion--pattern-compiler
+    (string table pred point transform-pattern-fn)
   "Match the presumed substring STRING to the entries in TABLE.
 Respect PRED and POINT.  The pattern used is a PCM-style
 substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
@@ -3904,12 +4074,23 @@ completion-substring--all-completions
          (pattern (completion-pcm--optimize-pattern
                    (if transform-pattern-fn
                        (funcall transform-pattern-fn pattern)
-                     pattern)))
-         (all (completion-pcm--all-completions prefix pattern table pred)))
-    (list all pattern prefix suffix (car bounds))))
+                     pattern))))
+    (list pattern prefix suffix)))
+
+(defun completion-substring--all-completions
+    (string table pred point &optional transform-pattern-fn)
+  "Match the presumed substring STRING to the entries in TABLE.
+Respect PRED and POINT.  The pattern used is a PCM-style
+substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
+that is non-nil."
+  (pcase-let (((and result `(,pattern ,prefix ,_suffix))
+               (completion--pattern-compiler string table pred point
+                                             transform-pattern-fn)))
+    (cons (completion-pcm--all-completions prefix pattern table pred)
+          result)))
 
 (defun completion-substring-try-completion (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
     (if minibuffer-completing-file-name
@@ -3917,12 +4098,12 @@ completion-substring-try-completion
     (completion-pcm--merge-try pattern all prefix suffix)))
 
 (defun completion-substring-all-completions (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 ;;; "flex" completion, also known as flx/fuzzy/scatter completion
 ;; Completes "foo" to "frodo" and "farfromsober"
@@ -3932,42 +4113,53 @@ completion-flex-nospace
   :version "27.1"
   :type 'boolean)
 
-(put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata)
-
-(defun completion--flex-adjust-metadata (metadata)
-  (cl-flet
-      ((compose-flex-sort-fn
-        (existing-sort-fn) ; wish `cl-flet' had proper indentation...
-        (lambda (completions)
-          (let ((pre-sorted
-                 (if existing-sort-fn
-                     (funcall existing-sort-fn completions)
-                   completions)))
-            (cond
-             ((or (not (window-minibuffer-p))
-                  ;; JT@2019-12-23: FIXME: this is still wrong.  What
-                  ;; we need to test here is "some input that actually
-                  ;; leads to flex filtering", not "something after
-                  ;; the minibuffer prompt".  Among other
-                  ;; inconsistencies, the latter is always true for
-                  ;; file searches, meaning the next clauses will be
-                  ;; ignored.
-                  (> (point-max) (minibuffer-prompt-end)))
-              (sort
-               pre-sorted
-               (lambda (c1 c2)
-                 (let ((s1 (get-text-property 0 'completion-score c1))
-                       (s2 (get-text-property 0 'completion-score c2)))
-                   (> (or s1 0) (or s2 0))))))
-             (t pre-sorted))))))
-    `(metadata
-      (display-sort-function
-       . ,(compose-flex-sort-fn
-           (completion-metadata-get metadata 'display-sort-function)))
-      (cycle-sort-function
-       . ,(compose-flex-sort-fn
-           (completion-metadata-get metadata 'cycle-sort-function)))
-      ,@(cdr metadata))))
+(put 'flex 'completion--style-metadata 'completion--flex-style-metadata)
+
+(defun completion--flex-style-metadata (string table pred point metadata)
+  ;; Use the modified flex sorting function only for non-empty input.
+  ;; In an older version of `completion--flex-adjust-metadata', the
+  ;; check (> (point-max) (minibuffer-prompt-end))) was used instead.
+  (unless (eq string "")
+    (let ((pattern (car (completion--pattern-compiler
+                         string table pred point
+                         #'completion-flex--make-flex-pattern))))
+      (cl-flet
+          ((compose-flex-sort-fn
+            (existing-sort-fn) ; wish `cl-flet' had proper indentation...
+            (lambda (completions)
+              (let ((pre-sorted (if existing-sort-fn
+                                    (funcall existing-sort-fn completions)
+                                  completions)))
+                ;; If `completion-scores' are already present use
+                ;; those instead of recomputing the scores with
+                ;; `completion--flex-score'.  The scores are already
+                ;; present, when the candidates have been computed by
+                ;; `completion-all-completions'.  In contrast, the
+                ;; score is not yet present, when the candidates have
+                ;; been computed by `completion-filter-completions'.
+                (if (and (car pre-sorted)
+                         (get-text-property 0 'completion-score (car pre-sorted)))
+                    (sort
+                     pre-sorted
+                     (lambda (c1 c2)
+                       (> (or (get-text-property 0 'completion-score c1) 0)
+                          (or (get-text-property 0 'completion-score c2) 0))))
+                  (let* ((sorted (sort (completion--flex-score pattern pre-sorted)
+                                       #'car-less-than-car))
+                         (cell sorted))
+                    ;; Remove score decorations, reuse the list to avoid allocations.
+                    (while cell
+                      (setcar cell (cdar cell))
+                      (pop cell))
+                    sorted))))))
+        `(metadata
+          (display-sort-function
+           . ,(compose-flex-sort-fn
+               (completion-metadata-get metadata 'display-sort-function)))
+          (cycle-sort-function
+           . ,(compose-flex-sort-fn
+               (completion-metadata-get metadata 'cycle-sort-function)))
+          ,@(cdr metadata))))))
 
 (defun completion-flex--make-flex-pattern (pattern)
   "Convert PCM-style PATTERN into PCM-style flex pattern.
@@ -3989,7 +4181,7 @@ completion-flex--make-flex-pattern
 (defun completion-flex-try-completion (string table pred point)
   "Try to flex-complete STRING in TABLE given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
@@ -4006,13 +4198,13 @@ completion-flex-try-completion
 (defun completion-flex-all-completions (string table pred point)
   "Get flex-completions of STRING in TABLE, given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
-      (when all
-        (nconc (completion-pcm--hilit-commonality pattern all)
-               (length prefix))))))
+      (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix))))))
 
 ;; Initials completion
 ;; Complete /ums to /usr/monnier/src or lch to list-command-history.
@@ -4049,7 +4241,11 @@ completion-initials-expand
 (defun completion-initials-all-completions (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
     (when newstr
-      (completion-pcm-all-completions newstr table pred (length newstr)))))
+      (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+                   (completion-pcm--find-all-completions newstr table
+                                                         pred (length newstr))))
+        (completion-pcm--deferred-hilit pattern all
+                                        (length prefix) (length string))))))
 
 (defun completion-initials-try-completion (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
diff --git a/test/lisp/minibuffer-tests.el b/test/lisp/minibuffer-tests.el
index c3ba8f9a92..22de3c9ff5 100644
--- a/test/lisp/minibuffer-tests.el
+++ b/test/lisp/minibuffer-tests.el
@@ -28,8 +28,7 @@
 
 (require 'ert)
 (require 'ert-x)
-
-(eval-when-compile (require 'cl-lib))
+(require 'cl-lib)
 
 (ert-deftest completion-test1 ()
   (with-temp-buffer
@@ -331,5 +330,219 @@ completion-flex-test-3
                   "custgroup" '("customize-group-other-window") nil 9)))
            15)))
 
+(ert-deftest completion-flex-score-test-1 ()
+  ;; Full match!
+  (should (equal
+           (completion--flex-score '(prefix "R") '("R"))
+           (list (cons -1.0 "R")))))
+
+(ert-deftest completion-flex-score-test-2 ()
+  ;; One third and half of a match!
+  (should (equal
+           (completion--flex-score '(prefix "foo")
+                                   '("barfoobar" "fooboo"))
+           (list (cons (/ -1.0 3.0) "barfoobar")
+                 (cons (/ -1.0 2.0) "fooboo")))))
+
+(ert-deftest completion-flex-score-test-3 ()
+  ;; One fourth of a match
+  (should (eql
+           (caar (completion--flex-score '(prefix "R" point "O")
+                                         '("RaOb")))
+           (/ -1.0 4.0))))
+
+(ert-deftest completion-flex-score-test-4 ()
+  ;; For quoted completion tables, score the unquoted completion string.
+  (should (equal
+           (completion--flex-score
+            '(prefix "R")
+            (list (propertize "X" 'completion--unquoted "R")))
+           (list (cons -1.0 "X")))))
+
+(defun completion--test-style (style string point table filtered)
+  (let* ((completion-styles (list style))
+         (pred (lambda (x) (not (string-search "!" x))))
+         (result (completion-filter-completions
+                  string table pred point nil)))
+    (should (equal (alist-get 'base result) 0))
+    (should (equal (alist-get 'end result) (length string)))
+    (should (equal (alist-get 'completions result) filtered))
+    ;; The highlighting function should be present.
+    (should (not (memq (alist-get 'highlight result) '(nil identity))))
+    ;; Equal results as `completion-all-completions'.
+    (should (equal (completion-all-completions string table pred point)
+                   (append filtered 0)))
+    ;; The returned strings should be identical to the original strings.
+    ;; The `completion-filter-completions' function avoids allocations!
+    (should (cl-intersection (alist-get 'completions result)
+                             table :test #'eq))))
+
+(ert-deftest completion-basic-style-test-1 ()
+  ;; point at the beginning |foo
+  (completion--test-style 'basic "foo" 0
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-basic-style-test-2 ()
+  ;; point foo
+  (completion--test-style 'basic "foo" 2
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-substring-style-test ()
+  (completion--test-style 'substring "foo" 1
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-emacs21-style-test ()
+  (completion--test-style 'emacs21 "foo" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-emacs22-style-test ()
+  (completion--test-style 'emacs22 "fo0" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar" "fobar"))) ;; suffix ignored completely
+
+(ert-deftest completion-flex-style-test ()
+  (completion--test-style 'flex "abc" 1
+                          '("abc" "abc!" "xaybzc" "xaybz")
+                          '("abc" "xaybzc")))
+
+(ert-deftest completion-initials-style-test ()
+  (completion--test-style 'initials "abc" 1
+                          '("a-b-c" "a-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("a-b-c" "ax-by-cz")))
+
+(ert-deftest completion-pcm-style-test ()
+  (completion--test-style 'partial-completion "ax-b-c" 1
+                          '("ax-b-c" "ax-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("ax-b-c" "ax-by-cz")))
+
+(ert-deftest completion-filter-completions-highlight-test ()
+  ;; point at the beginning |foo
+  (let* ((completion-styles '(basic))
+         (result (completion-filter-completions
+                  "foo" '("foobar" "fbarfoo" "fxfooy" "bar")
+                  nil 1 nil)))
+    (should (equal
+             (format "%S" (alist-get 'completions result))
+             (format "%S" '("foobar" "fbarfoo" "fxfooy"))))
+    (should (equal
+             (format "%S" (funcall (alist-get 'highlight result)
+                                   (alist-get 'completions result)))
+             (format "%S"
+                     '(#("foobar" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fbarfoo" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fxfooy" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))))))))
+
+(defun completion--test-boundaries (style string table result)
+  (let ((table
+         (lambda (str pred action)
+           (pcase action
+             (`(boundaries . ,suffix) `(boundaries
+                                        ,(1+ (string-match-p "<\\|/" str))
+                                        . ,(or (string-search ">" suffix) (length suffix))))
+             (_ (complete-with-action action table
+                                      (replace-regexp-in-string ".*[</]" "" str)
+                                      pred)))))
+        (point (string-search "|" string))
+        (string (string-replace "|" "" string))
+        (completion-styles (list style)))
+    (should (equal
+             (assq-delete-all
+              (if (assq 'highlight result) '-does-not-exist 'highlight)
+              (completion-filter-completions
+               string table nil point nil))
+             result))
+    (should (equal
+             (completion-all-completions
+              string table nil point)
+             (append (alist-get 'completions result)
+                     (alist-get 'base result))))))
+
+(ert-deftest completion-emacs21-boundaries-test ()
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("ainput>after" "input>after" "inpux>after"
+                                 "inxputy>after" "input>after2")
+                               '((base . 7)
+                                 (end . 18)
+                                 (completions "input>after" "input>after2"))))
+
+(ert-deftest completion-emacs22-boundaries-test ()
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "inyy" "inzzz"))))
+
+(ert-deftest completion-basic-boundaries-test ()
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("ainput" "input" "inpux" "inxputy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "input" "inxputy"))))
+
+(ert-deftest completion-substring-boundaries-test ()
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("other") nil)
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("ainputs" "inputs" "inpux" "inxputsy")
+                               '((base . 7)
+                                 (end . 13)
+                                 (completions "ainputs" "inputs" "inxputsy"))))
+
+(ert-deftest completion-pcm-boundaries-test ()
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("other") nil)
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("ain-pu-ts" "in-pts" "in-pu-ts" "in-px" "inx-ptsy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "in-pts" "in-pu-ts" "inx-ptsy"))))
+
+(ert-deftest completion-initials-boundaries-test ()
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("other") nil)
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("ain/pu/ts" "in/pts" "in/pu/ts" "a/in/pu/ts"
+                                 "in/pu/ts/foo" "in/px" "inx/ptsy")
+                               '((base . 1)
+                                 (end . 4)
+                                 (completions "in/pu/ts" "in/pu/ts/foo"))))
+
+(defun completion-emacs22orig-all-completions (string table pred point)
+  (let ((beforepoint (substring string 0 point)))
+    (completion-hilit-commonality
+      (all-completions beforepoint table pred)
+     point
+     (car (completion-boundaries beforepoint table pred "")))))
+
+(ert-deftest completion-upgrade-return-type-test ()
+  ;; Test transparent upgrade of list completion style return value
+  ;; to the alist return value format of `completion-format-completions'.
+  (let ((completion-styles-alist
+         '((emacs22orig completion-emacs22-try-completion
+                        completion-emacs22orig-all-completions nil))))
+  (completion--test-boundaries 'emacs22orig "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 ;; 18 is incorrect, should be 12!
+                                 ;; But the information is not available
+                                 ;; due to the completion-style upgrade.
+                                 (end . 18)
+                                 ;; Identity highlighting function.
+                                 (highlight . identity)
+                                 (completions "inyy" "inzzz")))))
+
 (provide 'minibuffer-tests)
 ;;; minibuffer-tests.el ends here
-- 
2.20.1

bug#48841: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

> In comparison to my last patch, the patch is fully backward
> compatible and preserves all existing tests.

This a very good thing (the fact that the patch is fully backward compatible,
I mean).

It is quite a large patch that touches many completion internals.  I'd like
some time to look it over.

I've read the discussion and am indeed aware of some non-neglibile
performance problems in the flex and pcm completion styles since
they need to copy strings around.  Other -- completely different --
performance problems affect fido-mode specifically (but not
fido-vertical-mode, curiously).

In some conversation with Dmitry

  bug#48841: fido-mode is slower than ido-mode with similar settings

We discussed this.

There was also talk of removing the string copying with minimal (but not null)
backward compatibility breakage.  I recall Dmitry saying it was easy
to fix on the
completion frontend side.  Many such frontends live in Emacs or GNU Elpa.
On the other hand, the patch that we (or at least I) envisioned in
that discussion
was almost certainly much, much simpler than the one being presented here,
and thus much easier to reason about and discuss.

But to avoid comparing apples to oranges, I would you to summarize exactly,
perhaps in the forms of code snippets, and/or benchmarks exactly what problems
your large patch solves. State the problem(s) first, then the solution
(to each).
If there are multiple problems, then there's a good chance that multiple patches
that address each of these are preferred.

Thank you very much.
João

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/13/21 12:56 PM, João Távora wrote:
> I've read the discussion and am indeed aware of some non-neglibile
> performance problems in the flex and pcm completion styles since
> they need to copy strings around.  Other -- completely different --
> performance problems affect fido-mode specifically (but not
> fido-vertical-mode, curiously).
> 
> In some conversation with Dmitry
> 
>   bug#48841: fido-mode is slower than ido-mode with similar settings
> 
> We discussed this.

I've read the discussion.  You are probably aware of my efforts to in
Vertico to implement deferred highlighting.  The patch I implemented
here implements the deferred highlighting in a clean way.

> There was also talk of removing the string copying with minimal (but not null)
> backward compatibility breakage.  I recall Dmitry saying it was easy
> to fix on the
> completion frontend side.  Many such frontends live in Emacs or GNU Elpa.
> On the other hand, the patch that we (or at least I) envisioned in
> that discussion
> was almost certainly much, much simpler than the one being presented here,
> and thus much easier to reason about and discuss.

No, this is not the case. There is no simple fix of the allocation issue
on the frontend side.  The existing API `completion-all-completions`
necessarily has to allocate all the strings in order to attach
highlighting and scoring.  The new API solves this in a clean way by
both deferring highlighting and scoring.

I claim that my patch is easy to reason about and refactors the existing
code to address the exact problem we are having. Please take some time
in reviewing it.

> But to avoid comparing apples to oranges, I would you to summarize exactly,
> perhaps in the forms of code snippets, and/or benchmarks exactly what problems
> your large patch solves. State the problem(s) first, then the solution
> (to each).

The main problem is that `completion-all-completions` allocates all the
strings every time the completions are filtered.  This is the same
performance issue you encountered in fido-mode/icomplete-mode.

The second problem addressed by the new API
`completion-filter-completions` is that `completion-all-completions` is
limited in what it can return.  For example it cannot return the end
position of the completion.  This is also solved by the new API.  The
new API is a clean extensible way forward.

Daniel

bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Fri, Aug 13, 2021 at 12:21 PM Daniel Mendler <mail@daniel-mendler.de> wrote:

> No, this is not the case. There is no simple fix of the allocation issue
> on the frontend side.

I didn't claim that. At all. I claimed that the frontends that would be
affected by the (small)  backend patch are easy to adapt.  I think
you completely read past my idea.

> The existing API `completion-all-completions`
> necessarily has to allocate all the strings in order to attach
> highlighting and scoring.  The new API solves this in a clean way by
> both deferring highlighting and scoring.

I'm not sure you understand my alternative idea.  As far as I
understand (and have actually measured) the lines:

   ;; Don't modify the string itself.
   (setq str (copy-sequence str))

in minibuffer.el, in the function completion-pcm--hilit-commonality

Are the cause of the problem that _I am talking about_ and that
I have actually measured.  Again you may be referring to a
_different_ problem that I am unaware of.

If one removes these lines, the process becomes much faster, but there is a
problem with highlighting.  My idea is indeed to defer highlighting by not
setting the 'face property directly on that shared string, but some
other property
that is read later from the shared string by compliant frontents.

If you have understood this idea, can you comment on it?
(Preferably in terms of less adjectification regarding "cleanliness", but in
terms of actual drawbacks/advantages?)

The drawback that I can see in it is that frontends directly relying
on 'face are
broken by that patch. But according to Dmitry (and I tend to agree), it's
quite easy to address those frontends.  Most of them live in Emacs core or
GNU Elpa.

The advantage that I see is that those adaptations apart, it is a small
localized and effective change.

> I claim that my patch is easy to reason about and refactors the existing
> code to address the exact problem we are having. Please take some time
> in reviewing it.

I am already taking some time. I need your assistance in explaining the
problems first. I take into account your claims of cleanliness and elegance,
but in terms of their power of persuasion, they are much more limited
than hard material evidence.

> The main problem is that `completion-all-completions` allocates all the
> strings every time the completions are filtered.  This is the same
> performance issue you encountered in fido-mode/icomplete-mode.

OK. I encountered at least two different performance problems there, with
quite different causes. So let's stick to the string-allocation problem.  Post
a code snippet that demonstrates the problem the way you see it/experience it?

Some benchmark code would be very welcome.  You can probably grab my
benchmarking code from that other bug.

Then it becomes easy to study multiple solutions to that problem and
choose the best one!

> The second problem addressed by the new API
> `completion-filter-completions` is that `completion-all-completions` is
> limited in what it can return.  For example it cannot return the end
> position of the completion.

And why is this a problem? Can you post an example of something you'd
like to do, but can't?  Regardless, it does seem indeed like a "second" problem
(as you state) so perhaps something that can be addressed separately.

Is your particular solution to this second problem instrumental in solving
the "main problem"

> This is also solved by the new API.  The new API is a clean extensible way forward.

I understand you've put time and effort into producing this work. We are
all indebted and I promise to read it. But every API writer in history of
programming has claimed those things and reality often shows otherwise.
So it's not that your work can't be those things you claim, maybe it is, but
generally the larger and broader the work the harder it is to reason about.

João

bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/13/21 2:05 PM, João Távora wrote:
>> The existing API `completion-all-completions`
>> necessarily has to allocate all the strings in order to attach
>> highlighting and scoring.  The new API solves this in a clean way by
>> both deferring highlighting and scoring.
> 
> I'm not sure you understand my alternative idea.  As far as I
> understand (and have actually measured) the lines:
> 
>    ;; Don't modify the string itself.
>    (setq str (copy-sequence str))
> 
> in minibuffer.el, in the function completion-pcm--hilit-commonality
> 
> Are the cause of the problem that _I am talking about_ and that
> I have actually measured.  Again you may be referring to a
> _different_ problem that I am unaware of.

You are right that the call to `copy-sequence` is a major bottleneck in
the filtering.  However you are wrong that this line can simply be
removed/disabled and the candidates can be modified.  The API guarantees
and has always guaranteed that the candidate strings are not mutated.
It is important to keep this property since this will preclude many bugs
due to string mutation.  By separating the filtering and mutation
(highlighting, scoring) my patch addresses the problem at hand in the
proper way.

Note that the UI also has no possibility to opt-out of the mutation.
The UI is actually not the one being concerned about the mutation here,
it is the backends (completion tables), which produce the strings.  If
one starts mutating these strings you will see bugs cropping up
throughout Emacs where shared strings suddenly have spurious additional
properties due to the completion filtering.

Mutation would be a reasonable choice here if the problem could not be
solved in a proper way.  But in fact it can be solved in a proper way
without mutating the strings at all as my patch shows.

> If one removes these lines, the process becomes much faster, but there is a
> problem with highlighting.  My idea is indeed to defer highlighting by not
> setting the 'face property directly on that shared string, but some
> other property
> that is read later from the shared string by compliant frontents.

This solution is much more ad-hoc and you still mutate the string which
is not allowed.

> The advantage that I see is that those adaptations apart, it is a small
> localized and effective change.

Note that your idea also does not address the other issues which are
addressed by my patch.  The new API `completion-filter-completions`
returns data which hasn't been available before, e.g., the end position,
which cannot be fixed given the existing API.

>> The main problem is that `completion-all-completions` allocates all the
>> strings every time the completions are filtered.  This is the same
>> performance issue you encountered in fido-mode/icomplete-mode.
> 
> OK. I encountered at least two different performance problems there, with
> quite different causes. So let's stick to the string-allocation problem.  Post
> a code snippet that demonstrates the problem the way you see it/experience it?

You can try my Vertico completion UI, which is available on GNU ELPA.
It implements deferred highlighting and there the performance difference
is perceivable.  Currently Vertico uses an advice-based hack to avoid
the over-eager string-allocations and the highlighting.

>> The second problem addressed by the new API
>> `completion-filter-completions` is that `completion-all-completions` is
>> limited in what it can return.  For example it cannot return the end
>> position of the completion.
> 
> And why is this a problem? Can you post an example of something you'd
> like to do, but can't?  Regardless, it does seem indeed like a "second" problem
> (as you state) so perhaps something that can be addressed separately.

Please look at the FIXMEs in minibuffer.el which address this.
Currently only the beginning position of the completion boundary is
returned, which is only half of the information.

> I understand you've put time and effort into producing this work. We are
> all indebted and I promise to read it. But every API writer in history of
> programming has claimed those things and reality often shows otherwise.
> So it's not that your work can't be those things you claim, maybe it is, but
> generally the larger and broader the work the harder it is to reason about.

I stand by my claim and I also stand by the claim that
removing/disabling `copy-sequence` is not a proper way to address the
issues at hand and will introduce many bugs in the long run.  Please
take your time to look at the patch in earnest.  I would also like to
see others chime in here with their opinion.

Daniel

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@daniel-mendler.de> wrote:

> It is important to keep this property since this will preclude many bugs
> due to string mutation.

I am aware of this, of course.  Can you give examples of these "many bugs"?
Perhaps other than the one I already described and addressed?

> By separating the filtering and mutation
> (highlighting, scoring) my patch addresses the problem at hand in the
> proper way.
>[ ... ]
> Mutation would be a reasonable choice here if the problem could not be
> solved in a proper way.  But in fact it can be solved in a proper way
> without mutating the strings at all as my patch shows.

"proper" is just an reasonably empty adjective.  There are different ways to
go about this, of course.  What's "proper" and isn't is hard to debate
objectively.

> This solution is much more ad-hoc and you still mutate the string which
> is not allowed.

It's also difficult to debate "ad-hoc" or not.  If you've studied the
problem, what
makes you say that mutating the string (in this case, adding a
'completion--style-face' property to it) is not allowed? What negative things
would derive from it.

> > The advantage that I see is that those adaptations apart, it is a small
> > localized and effective change.
>
> Note that your idea also does not address the other issues which are
> addressed by my patch.

That's for sure.  My patch idea addresses only that single problem.
I think this is a good property of patches: to solve one thing, not many.

We can make more patches to solve other problems, once we
identify them clearly.

> The new API `completion-filter-completions`
> returns data which hasn't been available before, e.g., the end position,
> which cannot be fixed given the existing API.
>
> >> The main problem is that `completion-all-completions` allocates all the
> >> strings every time the completions are filtered.  This is the same
> >> performance issue you encountered in fido-mode/icomplete-mode.
> >
> > OK. I encountered at least two different performance problems there, with
> > quite different causes. So let's stick to the string-allocation problem.  Post
> > a code snippet that demonstrates the problem the way you see it/experience it?

Look, one needs to evaluate things quantitively. Your patch is not
to Vertico, it's to Emacs. I'm concerned with changes to Emacs and their
effect on all completion frontends.  So trying Vertico isn't very useful.

If you're solving a performance problem (and it seems that you are, among
other things) we really need benchmarks, a description of an experiment whose
results can be reproduced independently. It's the normal scientific method.

Something like:

"before my patch, this code takes 123 seconds to run, after my patch it
takes 12."

> >> The second problem addressed by the new API
> >> `completion-filter-completions` is that `completion-all-completions` is
> >> limited in what it can return.  For example it cannot return the end
> >> position of the completion.
> >
> > And why is this a problem? Can you post an example of something you'd
> > like to do, but can't?  Regardless, it does seem indeed like a "second" problem
> > (as you state) so perhaps something that can be addressed separately.
>
> Please look at the FIXMEs in minibuffer.el which address this.
> Currently only the beginning position of the completion boundary is
> returned, which is only half of the information.

OK. It does seem like a separate problem, so maybe open a new bug for it?

João Távora

bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/13/21 2:37 PM, João Távora wrote:
> On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@daniel-mendler.de> wrote:
> 
>> It is important to keep this property since this will preclude many bugs
>> due to string mutation.
> 
> I am aware of this, of course.  Can you give examples of these "many bugs"?
> Perhaps other than the one I already described and addressed?

No, João, this is not how it goes.  I don't have to prove to you that
your idea introduces bugs.  You have to show that mutation of the
completion table strings (which are not supposed to be mutated) will not
lead to bugs, which are hard to find.

In contrast with the new API `completion-filter-completions` this entire
class of bugs is avoided by construction of the API.  Furthermore the
`completion-filter-completions` API is easy to use in comparison to your
idea, where "compliant" backends have to apply string manipulations to
apply the highlighting and revert the strings back to their old pristine
state.  The only thing the API user has to do is to call the `highlight`
function returned in the alist by `completion-filter-completions`.

>> By separating the filtering and mutation
>> (highlighting, scoring) my patch addresses the problem at hand in the
>> proper way.
>> [ ... ]
>> Mutation would be a reasonable choice here if the problem could not be
>> solved in a proper way.  But in fact it can be solved in a proper way
>> without mutating the strings at all as my patch shows.
> 
> "proper" is just an reasonably empty adjective.  There are different ways to
> go about this, of course.  What's "proper" and isn't is hard to debate
> objectively.

You are contradicting yourself here. You agree that string mutation is
better be avoid. If we define "proper" as avoids string mutation if this
is easily possible, then my patch implements a proper solution to the
problem.

>>> The advantage that I see is that those adaptations apart, it is a small
>>> localized and effective change.
>>
>> Note that your idea also does not address the other issues which are
>> addressed by my patch.
> 
> That's for sure.  My patch idea addresses only that single problem.
> I think this is a good property of patches: to solve one thing, not many.

No, this is not necessarily true.  This is only good if the problem is
solved in a way which is future proof.  The idea of mutating the strings
is a hack and not a solution. In contrast, I am presenting a
future-proof new API as solution which addresses multiple problems.  If
you look at the patch, only 196 new lines are added to minibuffer.el.
Furthermore the patch adds 213 lines of new tests.

> Look, one needs to evaluate things quantitively. Your patch is not
> to Vertico, it's to Emacs. I'm concerned with changes to Emacs and their
> effect on all completion frontends.  So trying Vertico isn't very useful.
> 
> If you're solving a performance problem (and it seems that you are, among
> other things) we really need benchmarks, a description of an experiment whose
> results can be reproduced independently. It's the normal scientific method.

João, you don't have to lecture me on these things.  Of course I can
provide such numbers.  You cannot reasonably make the claim that
`copy-sequence` is the problem and at the same time claim that my patch
does not solve the performance issues, when in fact my patch avoids this
exact string copying.

>>>> The second problem addressed by the new API
>>>> `completion-filter-completions` is that `completion-all-completions` is
>>>> limited in what it can return.  For example it cannot return the end
>>>> position of the completion.
>>>
>>> And why is this a problem? Can you post an example of something you'd
>>> like to do, but can't?  Regardless, it does seem indeed like a "second" problem
>>> (as you state) so perhaps something that can be addressed separately.
>>
>> Please look at the FIXMEs in minibuffer.el which address this.
>> Currently only the beginning position of the completion boundary is
>> returned, which is only half of the information.
> 
> OK. It does seem like a separate problem, so maybe open a new bug for it?

There is already a FIXME in minibuffer.el, so I assume Stefan Monnier is
well aware of these issues.  It is an additional win of the new API that
such open problems can be fixed too.  As I see it, a new API is the way
to go here.

Daniel

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

didnOn Fri, Aug 13, 2021 at 1:56 PM Daniel Mendler
<mail@daniel-mendler.de> wrote:
>
> On 8/13/21 2:37 PM, João Távora wrote:
> > On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@daniel-mendler.de> wrote:
> >
> >> It is important to keep this property since this will preclude many bugs
> >> due to string mutation.
> >
> > I am aware of this, of course.  Can you give examples of these "many bugs"?
> > Perhaps other than the one I already described and addressed?
>
> No, João, this is not how it goes.  I don't have to prove to you that
> your idea introduces bugs.

So you just say it and I have to believe it?  Then I could say the same to
you, right?  I won't of course, that would be silly.

You have to show that mutation of the
> completion table strings (which are not supposed to be mutated) will not
> lead to bugs, which are hard to find.
>
> In contrast with the new API `completion-filter-completions` this entire
> class of bugs is avoided by construction of the API.  Furthermore the
> `completion-filter-completions` API is easy to use in comparison to your
> idea, where "compliant" backends have to apply string manipulations to
> apply the highlighting and revert the strings back to their old pristine
> state.  The only thing the API user has to do is to call the `highlight`
> function returned in the alist by `completion-filter-completions`.
>
> >> By separating the filtering and mutation
> >> (highlighting, scoring) my patch addresses the problem at hand in the
> >> proper way.
> >> [ ... ]
> >> Mutation would be a reasonable choice here if the problem could not be
> >> solved in a proper way.  But in fact it can be solved in a proper way
> >> without mutating the strings at all as my patch shows.
> >
> > "proper" is just an reasonably empty adjective.  There are different ways to
> > go about this, of course.  What's "proper" and isn't is hard to debate
> > objectively.
>
> You are contradicting yourself here. You agree that string mutation is
> better be avoid. If we define "proper" as avoids string mutation if this
> is easily possible, then my patch implements a proper solution to the> problem.

I didn't say it's better avoided, though of course I will avoid _any_ change if
I can. I said I have identified one drawback with doing it.  Then I
have addressed
that drawback. So that's what I said.

I am unaware of _other_ drawbacks.  They might exist, but I am unaware of
them.  Perhaps you are, and indeed you state they exist, but you refuse to
let me know about them.  Or perhaps others know of them and will let me know.
In my long-running discussion with Dmitry they were not presented (again,
except for the one I identified).

> > That's for sure.  My patch idea addresses only that single problem.
> > I think this is a good property of patches: to solve one thing, not many.
> No, this is not necessarily true.  This is only good if the problem is
> solved in a way which is future proof.

OK, but what thing of the future, real or academic, do you envision that
would bring back the problem, or create other problems?

> The idea of mutating the strings is a hack and not a solution.

Without facts to back it up, I have to take this as gratuitous disparagement.
Nicht so gut.

> In contrast, I am presenting a
> future-proof new API as solution which addresses multiple problems.

That's the issue.  The completion system is very complex and there are many
good ideas, different, floated by many people. But if you make a patch to
address "multiple" fuzzily-described problems, it's hard to judge how good
your ideas even are! Maybe they are indeed very good,  I never said
they weren't.  No need to get worked up about it!

Again, my proposal is to first focus on the performance problems caused by
string allocation.  _That_ problem is well understood, at least by me (but it
would help to settle on convenient benchmarks understood by others, too).
Then we can go from there.

> you look at the patch, only 196 new lines are added to minibuffer.el.
> Furthermore the patch adds 213 lines of new tests.

It's a large patch, over 1000 lines.  One does not review a patch
merely by looking at
lines added, when one needs to read much more, to understand implications, etc.
It needs documentation, for one, much more than just docstrings, on
how to use the
new API.

> João, you don't have to lecture me on these things.  Of course I can
> provide such numbers.

Then please do! Not meaning to lecture you, just that your suggestion that
I try Vertico UI as a substitution for these numbers seemed completely
misguided.  So if you have them (or "can provide them") let's see them.
All I'm asking,  preferably from Emacs -Q recipe.

> You cannot reasonably make the claim that
> `copy-sequence` is the problem and at the same time claim that my patch
> does not solve the performance issues, when in fact my patch avoids this
> exact string copying.

I didn't say it didn't solve them! Now, where did I say that? I would
like to see a
benchmark so that I can witness it _and_ study alternative solutions. With
that, there's a better chance that I will be persuaded there are none
as elegant,
clean, proper, pure, etc as yours!

Maybe others review patches on other aspects that's fine.   Maybe
others will. Eli reviewed on minor formatting and documentation aspects.
I review them on substance, using numbers and conducting my own
experiments and tests. This takes time and help from the scientist on the
other end.

Simple and in summary, let's hope your next reply has some benchmarks
so we can make progress.

Thanks,
João

bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/13/21 3:36 PM, João Távora wrote:
>> You are contradicting yourself here. You agree that string mutation is
>> better be avoid. If we define "proper" as avoids string mutation if this
>> is easily possible, then my patch implements a proper solution to the> problem.
> 
> I didn't say it's better avoided, though of course I will avoid _any_ change if
> I can. I said I have identified one drawback with doing it.  Then I
> have addressed
> that drawback. So that's what I said.
> 
> I am unaware of _other_ drawbacks.  They might exist, but I am unaware of
> them.  Perhaps you are, and indeed you state they exist, but you refuse to
> let me know about them.  Or perhaps others know of them and will let me know.
> In my long-running discussion with Dmitry they were not presented (again,
> except for the one I identified).

In the discussion with Dmitry, I already pointed out that there is an
alternative principled approach implemented by my Vertico UI, which is
in fact the same approach as implemented in this patch.

If there are other useful conclusions from the discussion I will adopt
them here for this patch.

>>> That's for sure.  My patch idea addresses only that single problem.
>>> I think this is a good property of patches: to solve one thing, not many.
>> No, this is not necessarily true.  This is only good if the problem is
>> solved in a way which is future proof.
> 
> OK, but what thing of the future, real or academic, do you envision that
> would bring back the problem, or create other problems?
> 
>> The idea of mutating the strings is a hack and not a solution.
> 
> Without facts to back it up, I have to take this as gratuitous disparagement.
> Nicht so gut.

João, your whole answers are "nicht so gut" or not useful.  What is your
point?  Please give constructive technical feedback instead of such
empty phrases.

>> In contrast, I am presenting a
>> future-proof new API as solution which addresses multiple problems.
> 
> That's the issue.  The completion system is very complex and there are many
> good ideas, different, floated by many people. But if you make a patch to
> address "multiple" fuzzily-described problems, it's hard to judge how good
> your ideas even are! Maybe they are indeed very good,  I never said
> they weren't.  No need to get worked up about it!
> 
> Again, my proposal is to first focus on the performance problems caused by
> string allocation.  _That_ problem is well understood, at least by me (but it
> would help to settle on convenient benchmarks understood by others, too).
> Then we can go from there.

No, it is not the correct approach to fix larger issues by applying
localized patches.  We both have identified the string allocations and
highlighting as problem.  My patch resolves the problem, by exposing
just the right pieces of the already existing completion machinery. More
about this below.

>> you look at the patch, only 196 new lines are added to minibuffer.el.
>> Furthermore the patch adds 213 lines of new tests.
>
> It's a large patch, over 1000 lines.  One does not review a patch
> merely by looking at
> lines added, when one needs to read much more, to understand implications, etc.
> It needs documentation, for one, much more than just docstrings, on
> how to use the
> new API.

I suggest you take a step back here and try to understand the high-level
idea first.  It seems that you are misjudging the complexity of the
patch.  The minibuffer completion machinery is already constructed such
that filtering and highlighting are separate.

If you look at `completion-basic-all-completions` for example, there is
first a filtering step and then the highlighting is applied in a second
step by the function `completion-hilit-commonality`.  This separation
exists for all completion styles.

My patch does nothing else than separating these two processing steps.
The new API `completion-filter-completions` returns the filtered list
and a function to apply highlighting afterwards only to the actually
displayed candidates where highlighting is needed.

In contrast your idea totally misses this.

> Maybe others review patches on other aspects that's fine.   Maybe
> others will. Eli reviewed on minor formatting and documentation aspects.

I am looking forward to more reviews by other people.

Your desire for benchmarks is understandable, but I doubt that it will
lead to progress in the discussion here and I doubt that it will
convince you.

The outcome of the benchmark is the following - my patch only filters
and does not mutate the strings, so it will be slightly faster than your
idea where the strings are mutated first and afterwards the mutation has
to be undone again.  However the mutations are of course not expensive,
so the differences will be small.  The discussion we should be having
here is about technical details and internals and not about the numbers
which won't give any guidance in this case regarding the correct API design.

You seem to always come back to the "scientific method". Note that there
is not only statistics, there is only "scientific reasoning" and
mathematics, which allows to reason about transformations and drawing
conclusions from that.  If you don't do this, you are only doing half of
the science.

Daniel

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Fri, Aug 13, 2021 at 3:03 PM Daniel Mendler <mail@daniel-mendler.de> wrote:

> > Without facts to back it up, I have to take this as gratuitous disparagement.
> > Nicht so gut.
>
> João, your whole answers are "nicht so gut" or not useful.  What is your
> point?  Please give constructive technical feedback instead of such
> empty phrases.

Look, you disparaged an idea of mine without absolutely any facts. I don't think
that's good. "Nicht so gut" was a lighthearted way of pointing it out.
Lighten up.
Post the benchmarks you say you have and stop the pompous handwaving.

Bye,
João

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/13/21 4:11 PM, João Távora wrote:
> On Fri, Aug 13, 2021 at 3:03 PM Daniel Mendler <mail@daniel-mendler.de> wrote:
> 
>>> Without facts to back it up, I have to take this as gratuitous disparagement.
>>> Nicht so gut.
>>
>> João, your whole answers are "nicht so gut" or not useful.  What is your
>> point?  Please give constructive technical feedback instead of such
>> empty phrases.
> 
> Look, you disparaged an idea of mine without absolutely any facts. I don't think
> that's good. "Nicht so gut" was a lighthearted way of pointing it out.
> Lighten up.
> Post the benchmarks you say you have and stop the pompous handwaving.

João, the way you argue is not in any way "lighthearted".  It also
depends on what the other party receives as the message.  And here you
just repeat this style by calling my reasoning "pompous handwaving".
This is not a fair way to discuss.  In contrast my arguments were
generally of a technical nature.  I propose we both calm down a bit and
let others chime in here.

Daniel

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

Hi folks,

Sorry I'm late to this party.

On 13.08.2021 16:36, João Távora wrote:
>>>> By separating the filtering and mutation
>>>> (highlighting, scoring) my patch addresses the problem at hand in the
>>>> proper way.
>>>> [ ... ]
>>>> Mutation would be a reasonable choice here if the problem could not be
>>>> solved in a proper way.  But in fact it can be solved in a proper way
>>>> without mutating the strings at all as my patch shows.
>>> "proper" is just an reasonably empty adjective.  There are different ways to
>>> go about this, of course.  What's "proper" and isn't is hard to debate
>>> objectively.
>> You are contradicting yourself here. You agree that string mutation is
>> better be avoid. If we define "proper" as avoids string mutation if this
>> is easily possible, then my patch implements a proper solution to the> problem.
> I didn't say it's better avoided, though of course I will avoid_any_  change if
> I can. I said I have identified one drawback with doing it.  Then I
> have addressed
> that drawback. So that's what I said.
> 
> I am unaware of_other_  drawbacks.  They might exist, but I am unaware of
> them.  Perhaps you are, and indeed you state they exist, but you refuse to
> let me know about them.  Or perhaps others know of them and will let me know.
> In my long-running discussion with Dmitry they were not presented (again,
> except for the one I identified).

I thought I explained the problem with this previously.

It's basically this: we cannot mutate what we don't own. Across all of 
completion functions out there, there will be such that return "shared" 
strings (meaning, not copied or newly allocated) from their completion 
tables. And modifying them is bad, with consequences which can present 
themselves in unexpected, often subtle ways.

Since up until now completion-pcm--hilit-commonality copied all strings 
before modifying, completion tables such as described (with "shared" 
strings) have all been "legal". Suddenly deciding to stop supporting 
them would be a major API breakage with consequences that are hard to 
predict. And while I perhaps agree that it's an inconvenience, I don't 
think it's a choice we can simply make as this stage in c-a-p-f's 
development.

To give an example of a subtle consequence:

   1. (setq s (symbol-name 'car))

   2. (put-text-property 1 3 'face 'error s)

   3. Switch to a buffer in fundamental mode

   4. (insert (symbol-name 'car)) --> see the error face in the buffer

Now imagine that some completion table collects symbol names by passing 
obarray through #'symbol-name rather than #'all-completions, and voila, 
if the completion machinery adds properties (any properties, not just 
face) to those strings, you have just modified a bunch of global values. 
That's not good.

And in the example above, the values are those that the 
lispref/objects.texi says we should not change (though it gives 
(symbol-name 'cons) as example). "Not mutable", in its parlance. IIRC 
the related discussions mentioned that modifying such values could lead 
to a segfault in some previous Emacs versions. Maybe not anymore, but 
it's still not a good idea.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

Aside from the mutability/ownership issue,

On 13.08.2021 15:05, João Távora wrote:
> If one removes these lines, the process becomes much faster, but there is a
> problem with highlighting.  My idea is indeed to defer highlighting by not
> setting the 'face property directly on that shared string, but some
> other property
> that is read later from the shared string by compliant frontents.

I haven't done any direct benchmarking, but I'm pretty sure that this 
approach cannot, by definition, be as fast as the non-mutating one.

Because you go through the whole list and mutate all of its elements, 
you have to perform a certain bit of work W x N times, where N is the 
size of the whole list.

Whereas the deferred-mutation approach will only have to do its bit 
(which is likely more work, like, WWW) only 20 times, or 100 times, or 
however many completions are displayed. And this is usually negligible.

However big the difference is going to be, I can't say in advance, of 
course, or whether it's going to be shadowed by some other performance 
costs. But the non-mutating approach should have the best optimization 
potential when the list is long.

bug#47711: bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

Hi Daniel,

I haven't yet read the patch in detail, but it sounds like a move in the 
right direction (even if it doesn't include the long-overdue overhaul of 
the whole API).

A few notes on the new stuff:

 > Finally the
`highlight` value is a function taking a list of completion strings
and returns a new list of new strings with highlighting applied.

First of all, I'd really like it to be a function that applies to 
individual completion strings, not the whole collection. That would make 
it much easier to use in company-capf without having to rewrite a lot of 
code in the presentation layer.

Second, perhaps instead of modifying the strings themselves it could 
return some data (like a list of faces-intervals tuples) that could be 
used to do so?

Again, in company-capf's we end up parsing the face properties in the 
string, so those modifications are just extra work for CPU which we 
could eliminate.

This is less critical, though.

On 11.08.2021 19:11, Daniel Mendler wrote:
> There are currently two issues with the patch with regards to backward
> compatibility. Fortunately they are fixable with a little effort.
> 
> 1. I would like to deprecate `completion-score' or remove it altogether,
>     but unfortunately `completion-score' is used in the wild. In order to
>     preserve `completion-score', bind `completion--filter-completions' in
>     the highlighting functions. Add `completion-score' in
>     `completion-pcm--hilit-commonality' when
>     `completion--filter-completions' is nil.

And third: I think completion-score could ultimately use the same 
treatment as 'highlight'. Meaning, being returned up the stack together 
with completions, so other bits of code could look up those values.

I don't have a clear picture of this yet, but see the recently filed 
bug#49888. If we want to be able to combine matching scores with recency 
scores, simply sorting the completions after matching is not going to 
cut it.

Not sure if this is something we can tackle now, but keeping this 
possible evolution in mind could help us make good choices in the 
current migration.

bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> Cc: 48841@debbugs.gnu.org, dgutov@yandex.ru, joaotavora@gmail.com,
>  monnier@iro.umontreal.ca, 47711@debbugs.gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Thu, 12 Aug 2021 10:47:17 +0200
> 
> >> The `completions` value is the list of completion strings *without*
> >> applied highlighting.  The completion strings are returned unmodified,
> >> which avoids allocations and results in performance gains for
> > 
> > This is unclear: how can you return a list of strings which you
> > produce without allocating the strings?
> 
> The function 'completion-filter-completions' receives a completion table
> as argument.  The strings produced by this table are returned
> unmodified, but of course the completion table has to produce them.  For
> a static completion table (e.g., in the simplest case a list of strings)
> the completion table itself will not allocate strings.  In this scenario
> 'completion-filter-completions' will not perform any string allocations,
> only the list will be allocated.  This is what leads to major
> performance gains.

My point was that at least some of this should be in the description,
otherwise it will leave the reader wondering.

> >> +(defvar completion--filter-completions nil
> >> +  "Enable the new completions return value format.
> >
> > Btw, why is this an internal variable?  Shouldn't all completion
> > styles ideally support it?  If so, it should be a public variable,
> > documented in the ELisp manual.  And the name should also end with -p,
> > since it's a boolean.  How about completion-filter-completions-format-p?
> 
> (As I understood the style guide '-p' is not a good idea for boolean
> variables, since a value is not a predicate in a strict sense.)
> 
> To address your technical comment - this variable is precisely what one
> of the technical difficulties mentioned in my other mail is about.  The
> question is how we can retain backward compatibility of the completion
> style 'all' functions, e.g., 'completion-basic-all-completions', while
> still allowing the function to return the newly introduced alist format
> with more data, which enables 'completion-filter-completions' to perform
> the efficient deferred highlighting.

I understand, but given that we provide this for other packages, it
shouldn't be an internal variable.

> > Also, the "This function has been superseded..." part should be a new
> > paragraph, so that it stands out.  (And I'm not yet sure we indeed
> > want to say "superseded" here, but that's part of the on-going
> > discussion.  maybe use a more neutral language here, like "See also".)
> 
> The new API 'completion-filter-completions' will substitute the existing
> API 'completion-all-completions'.

That's your hope, and I understand.  But we as a project didn't yet
decide to deprecate the original APIs, so talking about superseding is
premature.

> > Is "filter" really the right word here (in the doc string)?  "Filer"
> > means you take a sequence and produce another sequence with some
> > members removed.  That's not what this API does, is it?  Suggest to
> > use a different name, like completion-completions-alist or
> > completion-all-completions-as-alist.
> 
> "Filter" seems like exactly the right word to me.  The function takes a
> list of strings (or a completion table) and returns a subset of matching
> completion strings without further modifications to the strings. See
> above what I wrote about allocations.

But the name says "filter completions".  Which would mean you take a
list of completions and filter out some of them.  A completion table
is much more general object than a list of strings.  Thus, I think
using "filter" here is sub-optimal.

> >> +Only the elements of table that satisfy predicate PRED are considered.
> >> +POINT is the position of point within STRING.  The METADATA may be
> >> +modified by the completion style.  The return value is a alist with
> >> +the keys:
> >> +
> >> +- base: Base position of the completion (from the start of STRING)
> > 
> > "Base" here means the beginning?  If so, why not call it "beg" or
> > somesuch?
> 
> Base position is a fixed term which is already used in minibuffer.el for
> completions.  See also 'completion-base-position' for example.

Well, we don't have to keep bad habits indefinitely.  It's okay to
lose them and use better terminology.  Or at least to explain that
terminology in parentheses the first time it is used in some context.

> > Are we really losing the completion-score property here?  If so, why?
> 
> Yes, the property is removed in the current patch.  It is not actually
> used for anything in the new implementation.  But it is possible to
> restore the property such that 'completion-all-completions' always
> returns scored candidates as it does now.  See my other mail regarding
> the caveats of the current patch.

I'd prefer not to lose existing features, because that'd potentially
make the changes backward-incompatible.

Thanks.

bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: Daniel Mendler <mail@daniel-mendler.de>
> Cc: Eli Zaretskii <eliz@gnu.org>, João Távora
>  <joaotavora@gmail.com>, Stefan Monnier <monnier@iro.umontreal.ca>,
>  Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 13 Aug 2021 12:38:28 +0200
> 
> I attached the overhauled patch, which addresses most of the comments by
> Eli.  In comparison to my last patch, the patch is fully backward
> compatible and preserves all existing tests.  As before, there are tests
> which check the new functionality for each existing completion style.

Thanks.  You were faster than me, so I sent a few more comments to the
old patch today.

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Fri, 13 Aug 2021 14:56:38 +0200
> Cc: 47711@debbugs.gnu.org, Stefan Monnier <monnier@iro.umontreal.ca>,
>  48841@debbugs.gnu.org, Dmitry Gutov <dgutov@yandex.ru>
> 
> On 8/13/21 2:37 PM, João Távora wrote:
> > On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@daniel-mendler.de> wrote:
> > 
> >> It is important to keep this property since this will preclude many bugs
> >> due to string mutation.
> > 
> > I am aware of this, of course.  Can you give examples of these "many bugs"?
> > Perhaps other than the one I already described and addressed?
> 
> No, João, this is not how it goes.  I don't have to prove to you that
> your idea introduces bugs.  You have to show that mutation of the
> completion table strings (which are not supposed to be mutated) will not
> lead to bugs, which are hard to find.

Please calm down, both of you.  No one has to prove anything to anyone
here, that's not how Emacs development works.  We need to see which
idea is better, and if none is significantly better, we will probably
have both of them living side by side.

And while asking for an example of potential bugs is reasonable,
asking for a proof that a change will NOT lead to bugs isn't.  So how
about a couple of examples where having original strings unchanged is
important, which could then be discussed?

> >> Note that your idea also does not address the other issues which are
> >> addressed by my patch.
> > 
> > That's for sure.  My patch idea addresses only that single problem.
> > I think this is a good property of patches: to solve one thing, not many.
> 
> No, this is not necessarily true.  This is only good if the problem is
> solved in a way which is future proof.  The idea of mutating the strings
> is a hack and not a solution.

Just to make sure we are on the same page: adding a text property to a
string doesn't mutate a string.  Lisp programs that process these
strings will not necessarily see any difference, and displaying those
strings will also not show any difference if the property is not
related to display.  So the assumption that seems to be made here,
that adding a property is the same as mutating a string, is IMO
inaccurate if not incorrect.

And once again: please tone down your responses, both of you.  TIA.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 14 Aug 2021 05:47:43 +0300
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 48841@debbugs.gnu.org,
>  47711@debbugs.gnu.org
> 
> I thought I explained the problem with this previously.
> 
> It's basically this: we cannot mutate what we don't own. Across all of 
> completion functions out there, there will be such that return "shared" 
> strings (meaning, not copied or newly allocated) from their completion 
> tables. And modifying them is bad, with consequences which can present 
> themselves in unexpected, often subtle ways.
> 
> Since up until now completion-pcm--hilit-commonality copied all strings 
> before modifying, completion tables such as described (with "shared" 
> strings) have all been "legal". Suddenly deciding to stop supporting 
> them would be a major API breakage with consequences that are hard to 
> predict. And while I perhaps agree that it's an inconvenience, I don't 
> think it's a choice we can simply make as this stage in c-a-p-f's 
> development.

This sounds like an argument against Daniel's approach as well, no?
Because if a completion API returns strings it "doesn't own", there
will be restrictions on Lisp programs that use those strings, because
those Lisp programs previously could do anything they liked with those
strings, and now they cannot.  Or am I missing something?

>    1. (setq s (symbol-name 'car))
> 
>    2. (put-text-property 1 3 'face 'error s)
> 
>    3. Switch to a buffer in fundamental mode
> 
>    4. (insert (symbol-name 'car)) --> see the error face in the buffer
> 
> Now imagine that some completion table collects symbol names by passing 
> obarray through #'symbol-name rather than #'all-completions, and voila, 
> if the completion machinery adds properties (any properties, not just 
> face) to those strings, you have just modified a bunch of global values. 
> That's not good.

How is this different from Daniel's proposal of returning the original
strings?  AFAIU, he just shifts the responsibility from the completion
code to the caller of the completion code, but basically leaves the
problem still very much real and pretty much into our face.

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 14 Aug 2021 05:55:17 +0300
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 48841@debbugs.gnu.org,
>  47711@debbugs.gnu.org
> 
> On 13.08.2021 15:05, João Távora wrote:
> > If one removes these lines, the process becomes much faster, but there is a
> > problem with highlighting.  My idea is indeed to defer highlighting by not
> > setting the 'face property directly on that shared string, but some
> > other property
> > that is read later from the shared string by compliant frontents.
> 
> I haven't done any direct benchmarking, but I'm pretty sure that this 
> approach cannot, by definition, be as fast as the non-mutating one.

Daniel seems to think otherwise, AFAIU.

> Because you go through the whole list and mutate all of its elements, 
> you have to perform a certain bit of work W x N times, where N is the 
> size of the whole list.
> 
> Whereas the deferred-mutation approach will only have to do its bit 
> (which is likely more work, like, WWW) only 20 times, or 100 times, or 
> however many completions are displayed. And this is usually negligible.
> 
> However big the difference is going to be, I can't say in advance, of 
> course, or whether it's going to be shadowed by some other performance 
> costs. But the non-mutating approach should have the best optimization 
> potential when the list is long.

So I guess the suggestion to have a benchmark is still useful, because
the estimations of which approach has better performance vary between
you three.  So maybe producing such benchmarks would be a good step?

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Aside from the mutability/ownership issue,
>
> On 13.08.2021 15:05, João Távora wrote:
>> If one removes these lines, the process becomes much faster, but there is a
>> problem with highlighting.  My idea is indeed to defer highlighting by not
>> setting the 'face property directly on that shared string, but some
>> other property
>> that is read later from the shared string by compliant frontents.
>
> I haven't done any direct benchmarking, but I'm pretty sure that this
> approach cannot, by definition, be as fast as the non-mutating one.
>
> Because you go through the whole list and mutate all of its elements,
> you have to perform a certain bit of work W x N times, where N is the
> size of the whole list.

Let's call W the work that you perform N times in this istuation.  In
the non-mutation, let's call it Z.  So

W <= Z, because Z not only propertizes the string with a calculation of
faces but _also copies its character contents_.

Also I think it's better to start about copying rather than mutating.
As Eli points out, putting a text property in a string (my idea) is not
equivalent with "mutating" it.

> Whereas the deferred-mutation approach will only have to do its bit
> (which is likely more work, like, WWW) only 20 times, or 100 times, or
> however many completions are displayed. And this is usually
> negligible.

I think you're going in the same fallacy you went briefly in the other
bug report.  The flex and pcm styles (meaning
completion-pcm--hilit-commonality) has to go through all the completions
when deciding the score to atribute to each completion that we already
know matches the pattern.  That's because this scoring is essential to
sorting.  That's a given in both scenarios, copying and non-copying.

Then, it's true that only a very small set of those eventually have to
be displayed to the user, depending on where wants she wants her
scrolling page to be.  So that's when you have to apply 'face' to, say
20 strings, and that can indeed be pretty fast.  But where does the
information come from?

- Currently, it comes from the string's 'face' itself, which was copied
  entirely.

- In the non-copying approach, it must come from somewhere else.  One
  idea is that it comes from a new "private" property 'lazy-face', also
  in the string itselv, but which was _not_ copied.  Another idea is
  just to remember the pattern and re-match it to those 20 strings.

I think the second alternative is always faster.

> However big the difference is going to be, I can't say in advance, of
> course, or whether it's going to be shadowed by some other performance
> costs. But the non-mutating approach should have the best optimization
> potential when the list is long.

Don't think so.  I'm doing benchmarks, will post soon.

João

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 5666 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> > On Fri, Aug 13, 2021 at 1:22 PM Daniel Mendler <mail@daniel-mendler.de> wrote:
>> > 
>> >> It is important to keep this property since this will preclude many bugs
>> >> due to string mutation.
>> > I am aware of this, of course.  Can you give examples of these "many bugs"?
>> > Perhaps other than the one I already described and addressed?
>> No, João, this is not how it goes.  I don't have to prove to you that
>> your idea introduces bugs.  You have to show that mutation of the
>> completion table strings (which are not supposed to be mutated) will not
>> lead to bugs, which are hard to find.
> Please calm down, both of you.  No one has to prove anything to anyone
> here, that's not how Emacs development works.  We need to see which
> idea is better, and if none is significantly better, we will probably
> have both of them living side by side.
>
> And while asking for an example of potential bugs is reasonable,
> asking for a proof that a change will NOT lead to bugs isn't.

As far as I remember, I have done the first. I found bugs and addressed
them. I cannot _prove_ that my change will not leads to bugs indeed:
no-one can with any change.  I've just stated repeteadly that I'm not
aware of any such bugs.  I can understand intuition" for bugs to a
certain extent (everyone has intuition), but this intuition must always
resolve into actual reality to be useful in the end.

> So how
> about a couple of examples where having original strings unchanged is
> important, which could then be discussed?

Good idea, so in the absence of any controlled benchmarks I did some of
my own, using the most controlled environment I could devise.  I start
Emacs like so:

   ~/Source/Emacs/emacs/src/emacs -Q -f fido-mode -f fido-vertical-mode -l ~/tmp/benchmark.el ~/tmp/benchmark.el

I prime the obarry with lots of symblos to make completion purposedly
slow:

   (require 'cl-lib)
   (cl-loop repeat 300000 do (intern (symbol-name (gensym))))

I attach the file. Then I try a run of 10 invocations of

  ;; Press C-u C-x C-e C-m quickly to produce a sample.  
  (benchmark-run (completing-read "" obarray))

This, I think, is a good representation of the responsiveness of the
completion system.  It always prints well after I finish typing, so I
don't think I'm inducing any artificial slowdows while it waits for my
input.  When not measuring quantitatively, I also feel the difference in
responsiveness between different approaches.

Summarized results with an assortment of Emacs builds.

   - the current master (254dc6ab4ca8e6a549a795f9eaf45378ce51b61f).

     20.25 seconds total
   
   - Applying Daniel's patch over 254dc6.

     23.41 seconds total
   
   - The theoretical best situation where we don't highlight in
     completion-pcm--hilit-commonality (like 254dc6, but just removed
     the copy-sequence)

     10.70 seconds total

   - Experimental patch published in
     scratch/icomplete-lazy-highlight-attempt-2 (not finished, still
     needs a way for frontends to opt into the optimization).

     10.80 seconds total

I invite you all to reproduce these results.

In conclusion, I don't think Daniel's patch is going in the right
direction, *performance-wise*, for the kind of responsiveness scenarios
that I am concerned with, and which were discussed with Dmitry in
bug#48841.  It seems to slow down the process by about 10%.

Note 1: there may be *other* performance scenarios that I am not aware
of, where Daniel's patch excels.  I've requested these benchmarks,
regrettably without any success.

Note 2: doesn't mean that there aren't *other* merits to Daniel's patch,
but I have not understood these yet.  That is due to the stated fact
that the patch is very long, and seems to comprise performance
improvements, refactorings, and API redesign.  It has no documentation
in manual and/or examples on how to use the new API.

>> >> Note that your idea also does not address the other issues which are
>> >> addressed by my patch.
>> > 
>> > That's for sure.  My patch idea addresses only that single problem.
>> > I think this is a good property of patches: to solve one thing, not many.
>> 
>> No, this is not necessarily true.  This is only good if the problem is
>> solved in a way which is future proof.  The idea of mutating the strings
>> is a hack and not a solution.
>
> Just to make sure we are on the same page: adding a text property to a
> string doesn't mutate a string.  Lisp programs that process these
> strings will not necessarily see any difference, and displaying those
> strings will also not show any difference if the property is not
> related to display.  So the assumption that seems to be made here,
> that adding a property is the same as mutating a string, is IMO
> inaccurate if not incorrect.

Yes, in Lisp it is very common to attach a "private" property to an
object.  If no-one else knows about the existence of that property, then
attaching it is not harmful. Generally, of course: there are situations
where adding a private property brings side-effects to other parts of
the code.  But IMO that other code is in the wrong, not the one that
adds properties.

Also, to be clear, attaching a different property (as in, not 'face') to
the completion string is only _one_ of the ways of the ways to bypass
copying.  According to my measurements, performance doesn't seem to be
decided by property attachments, but by copying or not copying of the
character data of said strings in completion-pcm--hilit-commonality.

João


[-- Attachment #2: benchmark.el --]
[-- Type: application/emacs-lisp, Size: 2973 bytes --]

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> It's basically this: we cannot mutate what we don't own. Across all of
> completion functions out there, there will be such that return
> "shared" strings (meaning, not copied or newly allocated) from their
> completion tables. And modifying them is bad, with consequences which
> can present themselves in unexpected, often subtle ways.

I agree with this premise.  But would you call putting a uniquely named
text property in them a modification or mutation of said strings?  I
don't.

> Since up until now completion-pcm--hilit-commonality copied all
> strings before modifying, completion tables such as described (with
> "shared" strings) have all been "legal".

Again, if I take one of this shared strings, in whichever environment
running row now, and I secretly attach a privatte "joaot/blargh"
property to it, there is very very low likelyhood that that will hurt
anybody.

You seem to be worrying about re-setting the 'face' property (a public
property by excellence) and that's the very same bug I experienced and
have described early.  It's not even a hard bug to see.  Just remove the
copy-sequence in `completion-pcm--hilit-commonality' and see strange
stuff happening.

But if you set some other property, _that_ bug _doesn't_ occur.  Do some
other bugs occur?  I don't know, I don't think we'll ever know, for
_any_ change.

Furthermore, there are other ways to forego the copying in
`completion-pcm--hilit-commonality and not even touch _ANY_ string
property.

> Suddenly deciding to stop supporting them would be a major API
> breakage with consequences that are hard to predict. And while I
> perhaps agree that it's an inconvenience, I don't think it's a choice
> we can simply make as this stage in c-a-p-f's development.
>
> To give an example of a subtle consequence:
>
>   1. (setq s (symbol-name 'car))
>
>   2. (put-text-property 1 3 'face 'error s)
>
>   3. Switch to a buffer in fundamental mode
>
>   4. (insert (symbol-name 'car)) --> see the error face in the buffer

It's not even subtle :-) Yes this is why have seen from the beginning in
bug#48841.  I think it was even I who reported it to you.

The principle to follow can be summarized as this: "Don't touch values
of properties you don't own in objects you don't own."

So just don't touch the 'face' property in things you don't own!  But
feel free to touch the "dmitry/blargh" property even in objects you
don't own.

So 'c-p--h-l' doesn't "own" face.  So it must either create an object
that it owns or set something that it does own.  'completion-score' is
"owned" by 'c-p--h-l'.  Only it can write it (though others can read
it).

> Now imagine that some completion table collects symbol names by
> passing obarray through #'symbol-name rather than #'all-completions,
> and voila, if the completion machinery adds properties (any
> properties, not just face) to those strings, you have just modified a
> bunch of global values. That's not good.

Why?  Maybe I'm missing something.  Why is adding properties -- that
no-one but the completion machinery knows about -- to those shared
strings "not good"?  What bad thing can happen if I do?

> And in the example above, the values are those that the
> lispref/objects.texi says we should not change (though it gives
> (symbol-name 'cons) as example). "Not mutable", in its parlance. IIRC
> the related discussions mentioned that modifying such values could
> lead to a segfault in some previous Emacs versions. Maybe not anymore,
> but it's still not a good idea.

You're extrapolating "change" to also include attaching properties to
symbols.  I think that document just means that you can't do stuff like

    (aset "cons" 0 ?z)

or

    (aset (symbol-name 'cons) 0 ?z)

I don't think it means you can't

    (put-text-property 0 1 'joaot/muahahah 42 (symbol-name 'cons))

But maybe Eli or someone else more knowledgeable in the deep internals
of Emacs can correct me.

If indeed I'm wrong, there are other ways to forego the copying in
`c-p---hilit-commonality` and still don't incurr in any such "mutation".
We must keep our eyes on the prize: copying -- not property-attaching --
is the real bummer here.

scratch/icomplete-lazy-highlight-attempt-2, although still incomplete,
is one such approach, though it still sets `completion-score` on the
"shared" string, used later for sorting.  But also that could be
prevented (again, only if it turns out to be actually problematic IMO).

João

PS: Maybe I've not stated it clearly enough: I *don't* object to -- or
endorse -- Daniel's patch.  My point was solely that it mixes too many
things for me to be intellectually able to review its functional merits,
and that those things should be separated into multiple problems and
patches to make this evaluation easier.  Maybe someone with superior
intellecutal capacity can review -- on substance -- as it stands.

See my other reply containing benchmarks.  Daniel's patch doesn't
perform well there, but for all I know, it can co-exist with my
non-copying approach, and we can all have our cake.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 14.08.2021 10:12, Eli Zaretskii wrote:
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sat, 14 Aug 2021 05:47:43 +0300
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, 48841@debbugs.gnu.org,
>>   47711@debbugs.gnu.org
>>
>> I thought I explained the problem with this previously.
>>
>> It's basically this: we cannot mutate what we don't own. Across all of
>> completion functions out there, there will be such that return "shared"
>> strings (meaning, not copied or newly allocated) from their completion
>> tables. And modifying them is bad, with consequences which can present
>> themselves in unexpected, often subtle ways.
>>
>> Since up until now completion-pcm--hilit-commonality copied all strings
>> before modifying, completion tables such as described (with "shared"
>> strings) have all been "legal". Suddenly deciding to stop supporting
>> them would be a major API breakage with consequences that are hard to
>> predict. And while I perhaps agree that it's an inconvenience, I don't
>> think it's a choice we can simply make as this stage in c-a-p-f's
>> development.
> 
> This sounds like an argument against Daniel's approach as well, no?
> Because if a completion API returns strings it "doesn't own", there
> will be restrictions on Lisp programs that use those strings, because
> those Lisp programs previously could do anything they liked with those
> strings, and now they cannot.  Or am I missing something?

Good question. It is not.

The completion tables described above have all been doing "legal" 
things, in our general understanding.

But any callers of completion-all-completions were never really allowed 
to modify the returned strings (those still were strings that code 
"doesn't own").

Of course, some of those callers (I don't know any, though) might have 
taken advantage of being able to modify the strings with impunity 
because of completion-all-completions' implementation detail, but 
they'll have a chance to clean up their act when switching to 
completion-filter-completions.

>>     1. (setq s (symbol-name 'car))
>>
>>     2. (put-text-property 1 3 'face 'error s)
>>
>>     3. Switch to a buffer in fundamental mode
>>
>>     4. (insert (symbol-name 'car)) --> see the error face in the buffer
>>
>> Now imagine that some completion table collects symbol names by passing
>> obarray through #'symbol-name rather than #'all-completions, and voila,
>> if the completion machinery adds properties (any properties, not just
>> face) to those strings, you have just modified a bunch of global values.
>> That's not good.
> 
> How is this different from Daniel's proposal of returning the original
> strings?  AFAIU, he just shifts the responsibility from the completion
> code to the caller of the completion code, but basically leaves the
> problem still very much real and pretty much into our face.

This is a shift of responsibility in the right direction. The callers 
might as well do the string copying when needed, but the fact of the 
matter is, they usually only need to "copy" 20-100 strings (or however 
many is displayed), if they need to modify them at all. That's where we 
win: copying 100 strings is better than copying 10000.

Gotta run now, will reply to other email later.

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 14 Aug 2021 11:36:32 +0100
> Cc: Daniel Mendler <mail@daniel-mendler.de>,
>  Stefan Monnier <monnier@iro.umontreal.ca>, 48841@debbugs.gnu.org,
>  47711@debbugs.gnu.org
> 
> > And in the example above, the values are those that the
> > lispref/objects.texi says we should not change (though it gives
> > (symbol-name 'cons) as example). "Not mutable", in its parlance. IIRC
> > the related discussions mentioned that modifying such values could
> > lead to a segfault in some previous Emacs versions. Maybe not anymore,
> > but it's still not a good idea.
> 
> You're extrapolating "change" to also include attaching properties to
> symbols.  I think that document just means that you can't do stuff like
> 
>     (aset "cons" 0 ?z)
> 
> or
> 
>     (aset (symbol-name 'cons) 0 ?z)
> 
> I don't think it means you can't
> 
>     (put-text-property 0 1 'joaot/muahahah 42 (symbol-name 'cons))
> 
> But maybe Eli or someone else more knowledgeable in the deep internals
> of Emacs can correct me.

Text properties are stored separately from the string, so I don't
think adding properties can in general be referred to as "change".

Whether in some particular situation that could count as a "change"
depends on that situation and on the particular property, of course.
I'm not sure in the context of completion there's any reason to count
as "change" adding properties that don't affect display.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Text properties are stored separately from the string, so I don't
> think adding properties can in general be referred to as "change".
>
> Whether in some particular situation that could count as a "change"
> depends on that situation and on the particular property, of course.
> I'm not sure in the context of completion there's any reason to count
> as "change" adding properties that don't affect display.

It is a destructive change, but we may just declare that completion
functions are allowed to destructively change the inputs in certain very
prescribed ways.  I'd rather avoid that, though, if at all possible,
because it may lead to subtle bugs all over the place.

Stefan just reminded me (in a different bug report) that we've long
meant to extend the text property machinery with a "namespace" or
"plane" concept.  The impetus for this is really the font locking
machinery which wants to manage some text properties that other packages
also want to manage.

The idea is that the display machinery would combine all the planes
before displaying, but each package would just manage its own "plane".
If we had something like this, then using this mechanism in the
completion context would make sense -- we could then say that completion
isn't allowed to alter anything except text properties in its private
plane.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: João Távora <joaotavora@gmail.com>,
>   mail@daniel-mendler.de,
>   dgutov@yandex.ru,  monnier@iro.umontreal.ca,  48841@debbugs.gnu.org,
>   47711@debbugs.gnu.org
> Date: Sat, 14 Aug 2021 14:12:31 +0200
> 
> Stefan just reminded me (in a different bug report) that we've long
> meant to extend the text property machinery with a "namespace" or
> "plane" concept.  The impetus for this is really the font locking
> machinery which wants to manage some text properties that other packages
> also want to manage.
> 
> The idea is that the display machinery would combine all the planes
> before displaying, but each package would just manage its own "plane".
> If we had something like this, then using this mechanism in the
> completion context would make sense -- we could then say that completion
> isn't allowed to alter anything except text properties in its private
> plane.

How can that work if at display time all the "planes" are combined?
It would mean that the code which produced the original strings will
get them displayed differently after completion.

Anyway, I'm not sure why you are talking about display here: the
properties which are supposed to store the information about
completion aren't supposed to affect display.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> How can that work if at display time all the "planes" are combined?
> It would mean that the code which produced the original strings will
> get them displayed differently after completion.

That's in the font-lock context, where font-lock will do faces on its
"plane" while other packages can do faces on their own "planes", and
they'll be combined on display.

It not relevant in the context of this bug report, but I thought I'd
just mention the general design.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> in the absence of any controlled benchmarks I did some of
> my own, using the most controlled environment I could devise.  I start
> Emacs like so:
>
>    ~/Source/Emacs/emacs/src/emacs -Q -f fido-mode -f fido-vertical-mode -l ~/tmp/benchmark.el ~/tmp/benchmark.el

I have know tweaked the benchmark slightly to make it easier to evaluate
speed qualitatively.  Here's what I've been using.

   (require 'cl-lib)
    
   (fido-mode 1)
   (fido-vertical-mode 1)
    
   ;; Introduce 150 000 new functions to really slow things down.
   ;; Probably more than most non-Spacemancs people will have :-)
   (defmacro lotsoffunctions ()
     `(progn
        ,@(cl-loop repeat 150000
                  collect `(defun ,(intern (symbol-name (gensym "heyhey"))) () 42))))
    
   (lotsoffunctions)
    
   (when nil
     ;; Press C-u C-x C-e C-m quickly to produce a quantitative sample
     (benchmark-run (completing-read "" obarray))
    
     ;; Or just press C-h f to experience how fast/slow completion is.
     )

The results are the same as the ones I reported in the previous email.

I've also cleaned up my previous patch of the
scratch/icomplete-lazy-highlight-attempt-2 branch slightly.  It is now
fully opt-in for frontends and completion-styles, so the backward
compatibility problems which I speculated seem to have been exaggerated.

I'm still studying it for flaws, but anyone can have a look.  And, of
course, there are many different ways to realize the "opt-in" for
frontends/styles.  I just chose the one that seemed the simplest given
the current completion framework.

The performance is still very good, it reduces the usual waiting time in
long lists of completions to about half of what it currently is.

João

bug#48841: [PATCH] Make fido-mode about as fast as ido-mode even with many completions

[João Távora]

[I've removed bug#47711 from the list, since I haven't read the bug.
This is only directly concerned with this report bug#48841 about speed
differences between fido-mode and ido-mode.]

João Távora <joaotavora@gmail.com> writes:

> scratch/icomplete-lazy-highlight-attempt-2, although still incomplete,
> is one such approach, though it still sets `completion-score` on the
> "shared" string, used later for sorting.  But also that could be
> prevented (again, only if it turns out to be actually problematic
> IMO).

I have tested the patch more thoroughly now, and have not found any
problems.  It's now opt-in for both frontends and completion styles.
Used four combinations:

- For adhering styles (substring, pcm and flex) and adhering frontends
  (icomplete-mode and fido-mode).

- For adherint styles with non-adhering but style-respecting frontends
  (company and "vanilla" completion).

- Non-adhering styles with adhering frontends, also no problems.

- Non-adhering styles and non-adhering frontends, also no problems.

As for performance, I'm using the usual simple benchmark from Emacs -Q

   (require 'cl-lib)
   (fido-mode 1)
   (fido-vertical-mode 1)

   (defmacro lotsoffunctions ()
     `(progn
        ,@(cl-loop repeat 300000
                  collect `(defun ,(intern (symbol-name (gensym "heyhey"))) () 42))))

   (lotsoffunctions)

I then press C-u C-x C-e C-m

   (benchmark-run (completing-read "" obarray))

To get a quantitative benchmark or just C-h f to get a qualitative
one. Without the optimization this takes about 5s to evaluate, with the
optimization is usually around 2.6s.

I also tested ido-mode on this, with

   (ido-mode)
   (ido-ubiquitous-mode)
   (setq ido-enable-flex-matching t)

But it seems with ido-ubiquitous-mode, C-h f gives up around 20000
functions.  So I tested around that mark and found:

* ido-mode to be minutely faster still in displaying the first set
  though it isn't as well sorted by recency, size and alphanumericity.
  However, I don't now if I'm seeing correctly ifo-mode 

* ido-mode to be slower in responding to quick input (C-h f a), for
  example.  There's some flickering.  It's also problematic when
  quitting with C-g (almost hanging sometimes).

All in all I'm satisfied with the speed increase imparted to fido-mode
and fido-vertical mode.  In particular, the sluggishness reported for
short inputs (which makes the flex style consider a great deal of
matches) seems to be completely gone.

I'll let people try this out and review the patch, which is after my
sig.

If there's one thing I'm not crazy about, it's the opt-in interface for
frontends which requires them to somehow explain to the completion
machinery what constitutes a completion "session".

That, of course, is essential to allow any caches to be invalidated.  I
don't know if the current completion framework has any better mechanism
than the one explained in the docstring of `completion-lazy-hilit'.
Maybe Stefan can speak to that, maybe the table "metadata" is a good
place, but that object seems complicated to access and manipuate.

Other than that detail, the fact that the opt-in is just a variable and
a function call seems simple enough, in my opinion.

Another note: the actual cache implementation is done with "private",
non-display-related string properties on non-copied strings.  That's
somewhat of a bad practice to some us, but not to others.  I haven't
seen any evidence of mischief, real or academic, but if it ever comes
forward, the implementation can use some off-string caching mechanism
(likely just a hash-table).

João

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index cd1979d04a..d69cb7568d 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -494,6 +494,7 @@ icomplete-minibuffer-setup
     (setq-local icomplete--initial-input (icomplete--field-string))
     (setq-local completion-show-inline-help nil)
     (setq icomplete--scrolled-completions nil)
+    (setq completion-lazy-hilit (cl-gensym))
     (use-local-map (make-composed-keymap icomplete-minibuffer-map
     					 (current-local-map)))
     (add-hook 'post-command-hook #'icomplete-post-command-hook nil t)
@@ -800,7 +801,9 @@ icomplete--render-vertical
                (cl-return-from icomplete--render-vertical
                  (concat
                   " \n"
-                  (mapconcat #'identity torender icomplete-separator))))
+                  (mapconcat #'identity
+                             (mapcar #'completion-lazy-hilit torender)
+                             icomplete-separator))))
    for (comp prefix) in triplets
    maximizing (length prefix) into max-prefix-len
    maximizing (length comp) into max-comp-len
@@ -812,7 +815,7 @@ icomplete--render-vertical
     (cl-loop for (comp prefix suffix) in triplets
              concat prefix
              concat (make-string (- max-prefix-len (length prefix)) ? )
-             concat comp
+             concat (completion-lazy-hilit comp)
              concat (make-string (- max-comp-len (length comp)) ? )
              concat suffix
              concat icomplete-separator))))
@@ -962,7 +965,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index f335a9e13b..c21f234053 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3512,6 +3512,54 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request lazy highlighting of completion matches.
+
+Completion-presenting frontends may opt to bind this variable to
+a unique non-nil value in the context of completion-producing
+calls (such as `completion-all-sorted-completions').  This hints
+the intervening completion styles that they do not need to
+propertize completion strings with the `face' property.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible for `face'-propertizing the completion matches meant
+to be displayed to the user, frequently a small subset of all
+completion matches.  This can be done by calling the function
+`completion-lazy-hilit' which returns a `face'-propertized
+string.
+
+The value stored in this variable by the completion frontend must
+be unique to each completion attempt/session.  For instance,
+frontends which utilize the minibuffer as the locus of completion
+may set it to a buffer-local value returned by `gensym'.  For
+frontends operating within a recursive command loop, let-binding
+it to `gensym' is appropriate.
+
+Note that the optimization enabled by variable is only actually
+performed some completions styles.  To others, it is a harmless
+and useless hint.  To author a completion style that takes
+advantage of this, look in the source of
+`completion-pcm--hilit-commonality' for ideas.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (let* ((str (copy-sequence str))
+         (data (get-text-property 0 'completion-lazy-hilit-data str))
+         (re (and
+              completion-lazy-hilit
+              (eq completion-lazy-hilit (car data)) (cdr data)))
+         (md (and re (string-match re str) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil str)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil str))
+    str))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
@@ -3527,8 +3575,10 @@ completion-pcm--hilit-commonality
            last-md)
       (mapcar
        (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
+         (unless completion-lazy-hilit
+           ;; Make a copy of `str' since in this case we're about to
+           ;; `face'-propertize it.
+           (setq str (copy-sequence str)))
          (unless (string-match re str)
            (error "Internal error: %s does not match %s" re str))
          (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
@@ -3576,9 +3626,10 @@ completion-pcm--hilit-commonality
                 (update-score-and-face
                  (lambda (a b)
                    "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
+                   (unless completion-lazy-hilit
+                     (add-face-text-property a b
+                                             'completions-common-part
+                                             nil str))
                    (setq
                     score-numerator   (+ score-numerator (- b a)))
                    (unless (or (= a last-b)
@@ -3601,7 +3652,10 @@ completion-pcm--hilit-commonality
            ;; for that extra bit of match (bug#42149).
            (unless (= from match-end)
              (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
+           (put-text-property 0 1 'completion-lazy-hilit-data
+                              (cons completion-lazy-hilit re) str)
+           (if (and (> (length str) pos)
+                    (not completion-lazy-hilit))
                (add-face-text-property
                 pos (1+ pos)
                 'completions-first-difference

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 14.08.2021 14:29, Eli Zaretskii wrote:
> Text properties are stored separately from the string, so I don't
> think adding properties can in general be referred to as "change".

Are you thinking of C strings?

Lisp strings carry text properties in addition to the array of 
characters. It doesn't really matter where in the memory the properties 
and the characters reside.

> Whether in some particular situation that could count as a "change"
> depends on that situation and on the particular property, of course.

I was talking in the general sense: modifying a value.

One can talk about whether a certain modification matters in certain 
situations, but that's not the way to discount a general principle.

> I'm not sure in the context of completion there's any reason to count
> as "change" adding properties that don't affect display.

For the context in question, whether the properties affect display is 
not particularly important. Properties affecting display just make it 
easier to notice that something's wrong. Bug involving other properties 
should be more difficult to investigate.

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 14.08.2021 15:12, Lars Ingebrigtsen wrote:
> It is a destructive change, but we may just declare that completion
> functions are allowed to destructively change the inputs in certain very
> prescribed ways.  I'd rather avoid that, though, if at all possible,
> because it may lead to subtle bugs all over the place.

That would be a breaking change, but it's a possibility, of course.

If we couldn't find a better way to implement this.

> Stefan just reminded me (in a different bug report) that we've long
> meant to extend the text property machinery with a "namespace" or
> "plane" concept.  The impetus for this is really the font locking
> machinery which wants to manage some text properties that other packages
> also want to manage.

"Planes" for text properties are just prefixed properties, I guess? 
That's different from the situation with font-lock.

> The idea is that the display machinery would combine all the planes
> before displaying, but each package would just manage its own "plane".
> If we had something like this, then using this mechanism in the
> completion context would make sense -- we could then say that completion
> isn't allowed to alter anything except text properties in its private
> plane.

Yes, if the code makes sure to only use prefixed properties, that would 
limit the damage. It could still affect repeated (parallel?) uses of the 
same values in the same piece of code.

And even if the effects are usually not serious, are we really okay with 
evaluating (symbol-name 'car) someday and seeing lots of properties 
attached to it?

bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 14.08.2021 10:01, Eli Zaretskii wrote:

> Just to make sure we are on the same page: adding a text property to a
> string doesn't mutate a string.  Lisp programs that process these
> strings will not necessarily see any difference, and displaying those
> strings will also not show any difference if the property is not
> related to display.  So the assumption that seems to be made here,
> that adding a property is the same as mutating a string, is IMO
> inaccurate if not incorrect.

This is nonsense.

A program won't necessarily see a difference in *any* changed value, as 
long as some part of it stays the same.

I can zero out the tail of a string, and have a program that only looks 
at its first few characters. It wouldn't mean that a string hasn't changed.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 4:21 AM Dmitry Gutov <dgutov@yandex.ru> wrote:

> And even if the effects are usually not serious, are we really okay with
> evaluating (symbol-name 'car) someday and seeing lots of properties
> attached to it?

I wouldn't mind that at all.  For me, it's quite the same as evaluating
(symbol-plist 'car) and seeing (is-vehicle t number-of-wheels 4) along with
all the other byte-compilation stuff already there.

João Távora

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 06:27, João Távora wrote:
> I wouldn't mind that at all.  For me, it's quite the same as evaluating
> (symbol-plist 'car) and seeing (is-vehicle t number-of-wheels 4) along with
> all the other byte-compilation stuff already there.

Those serve a real purpose, not just work as an accidental cache for 
some earlier computation.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 14.08.2021 11:23, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Aside from the mutability/ownership issue,
>>
>> On 13.08.2021 15:05, João Távora wrote:
>>> If one removes these lines, the process becomes much faster, but there is a
>>> problem with highlighting.  My idea is indeed to defer highlighting by not
>>> setting the 'face property directly on that shared string, but some
>>> other property
>>> that is read later from the shared string by compliant frontents.
>>
>> I haven't done any direct benchmarking, but I'm pretty sure that this
>> approach cannot, by definition, be as fast as the non-mutating one.
>>
>> Because you go through the whole list and mutate all of its elements,
>> you have to perform a certain bit of work W x N times, where N is the
>> size of the whole list.
> 
> Let's call W the work that you perform N times in this istuation.  In
> the non-mutation, let's call it Z.  So
> 
> W <= Z, because Z not only propertizes the string with a calculation of
> faces but _also copies its character contents_.

As I pointed out later in the email you're replying to, copying won't 
happen N times.

> Also I think it's better to start about copying rather than mutating.
> As Eli points out, putting a text property in a string (my idea) is not
> equivalent with "mutating" it.

In common industry terms, that's mutation. Lisp strings are not C 
strings, they are aggregate objects.

>> Whereas the deferred-mutation approach will only have to do its bit
>> (which is likely more work, like, WWW) only 20 times, or 100 times, or
>> however many completions are displayed. And this is usually
>> negligible.
> 
> I think you're going in the same fallacy you went briefly in the other
> bug report.  The flex and pcm styles (meaning
> completion-pcm--hilit-commonality) has to go through all the completions
> when deciding the score to atribute to each completion that we already
> know matches the pattern.  That's because this scoring is essential to
> sorting.  That's a given in both scenarios, copying and non-copying.

First of all, note that scoring is only essential to the 'flex' style. 
Whereas the improvements we're discussing should benefit all, and can be 
more pronounced if the scoring don't need to be performed.

But ok, let's talk about flex in particular.

> Then, it's true that only a very small set of those eventually have to
> be displayed to the user, depending on where wants she wants her
> scrolling page to be.  So that's when you have to apply 'face' to, say
> 20 strings, and that can indeed be pretty fast.  But where does the
> information come from?
> 
> - Currently, it comes from the string's 'face' itself, which was copied
>    entirely.
> 
> - In the non-copying approach, it must come from somewhere else.  One
>    idea is that it comes from a new "private" property 'lazy-face', also
>    in the string itselv, but which was _not_ copied.  Another idea is
>    just to remember the pattern and re-match it to those 20 strings.

Let's say that the cost to compute the score (on one completion) is S. 
And the cost to highlight it is H. The cost to copy a string is C (that 
would be amortized cost, including the subsequent GCs).

The current algorithm costs: N x (C + S + H)

C is unavoidable because of the existing API guarantees.

A non-mutating algorithm can cost:

   N x S (for sorting)

   +

   100 x (C + S + H) (let's say we didn't even cache the scoring info)

...where 100 is the number of displayed completions (the number is 
usually lower).

As we measured previously, copying is quite expensive. Even in the 
above, not-caching approach we win ((N - 100) x (C + H)), and, okay, 
lose 100 x S. For high values of N, it should be a clear win.

> I think the second alternative is always faster.
> 
>> However big the difference is going to be, I can't say in advance, of
>> course, or whether it's going to be shadowed by some other performance
>> costs. But the non-mutating approach should have the best optimization
>> potential when the list is long.
> 
> Don't think so.  I'm doing benchmarks, will post soon.

I'm guessing you just skip the C step in your benchmarks? Which is 
something that breaks our current contract.

Still, Daniel's patch should provide a comparable performance 
improvement. If you're saying it doesn't give numbers as good, I'll have 
to give it a more thorough read and testing tomorrow to comment on that.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 4:31 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 16.08.2021 06:27, João Távora wrote:
> > I wouldn't mind that at all.  For me, it's quite the same as evaluating
> > (symbol-plist 'car) and seeing (is-vehicle t number-of-wheels 4) along with
> > all the other byte-compilation stuff already there.
>
> Those serve a real purpose, not just work as an accidental cache for
> some earlier computation.

Caches also serve "a real purpose".  the gv-expander there
would be the "cache of an earlier computation". And I'm not
sure what "accidental" means, but if it means "implementation
detail for something I don't care about", I agree `completion-score`
is "accidental".  Should it be called
`completion-score-internal-cache-dont-look-here`?
Maybe.

Bottom line here is that an outside observer has no clue, and
shouldn't need or want to have a clue, on what "foreign" properties
attached to strings or symbols are meant.  This is why Eli says, and
I agree, that if the property isn't display related, it's all good.  No-one
but the setter and reader of that particular property mind.  The CL
systems I've worked with use package qualifiers to fine-grain this
even further, but they use the same principle. That Elisp allows a
string property list doesn't really make a difference IMO.

And none of this really really matters to the discussion.  If we absolutely
had to store these associations away from the string plist, (for
some aesthetic reason, I guess), we could just use hash-tables.

Then we could return the string unchanged (and uncopied) and largely
keep performance intact.
But why do it, since a string plist is a such a nice place to do these
associations and there's -- apart from your aesthetics considerations
-- 0 drawbacks identified?

João Távora

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 06:53, João Távora wrote:
> But why do it, since a string plist is a such a nice place to do these
> associations and there's -- apart from your aesthetics considerations
> -- 0 drawbacks identified?

You read all the explanations, and THAT'S your conclusion?

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> As I pointed out later in the email you're replying to, copying won't
> happen N times.

_Currently_, as in origin/master,  it happens N times.

In my patch, when a frontend adheres to the thing, it happens D times
where D is the amount of strings you need to display.  I guess if you do
the work to adapt a frontend to work with the new API proposed in
Daniel's patch it will be about the same (though likely in many more
lines of Elisp).

> First of all, note that scoring is only essential to the 'flex'
> style. Whereas the improvements we're discussing should benefit all,
> and can be more pronounced if the scoring don't need to be performed.

Yep, I agree.  But I don't see why the same principles I espouse --
which really amount to: let the style know it doesn't need to
face-propertize -- can't be applied to other styles that don't need
scoring.  Although those don't seem to suffer from any performance
problems, at least I haven't seens any complaints/reports/mesurements
like you did for bug#48841.

> But ok, let's talk about flex in particular.

Yes, I think that is important since it is the style known to be least
performant by its very lax nature.

> I'm guessing you just skip the C step in your benchmarks? Which is
> something that breaks our current contract.

Right.  Skipping the 'C = Copying step' is the whole point.  It breaks
our contract because the completion styles currently promise to
"face"-propertize the string.  So this is why I propose to let the
completion-style know that it doesn't need to.  When it is told of that,
it is relieved of the necessity of copying the string.  It is the
frontend that will do that copy just before face-propertizing and
displaying the string.  As you note, and reality shows, that's much
faster.  There is no disagreement here.

> Still, Daniel's patch should provide a comparable performance

Kinf of, from what I've read, it _should_ open the way for that to
happen.  From what I understand, you must then change the frontend (in a
big way?) to stop using completion-all-completions and start using the
new thing.  That work has not been done, as far as I know.  Whereas in
my proposal (which is now a patch posted to bug#48841) you change the
frontend in a very minor way, and that work _has_ been done.

Icomplete was very easy to adapt.  I can try adapting company soon.

In practice, we can't kill off completion-all-completions and start
everyone on completion-filter-completions (if that's what it's called).
So if the latter does turn out to be a step in the right direction (I'm
mostly waiting on Stefan to chime in), then I also don't see why we
couldn't have, as Eli suggested, both strategies for lazy highlighting
at some point in the future.

> improvement. If you're saying it doesn't give numbers as good, I'll
> have to give it a more thorough read and testing tomorrow to comment
> on that.

It's not me who is saying it, it's my Emacs :-) The real problem is that
with Daniel's patch, the frontends using the current API (as in
icomplete/fido) measurably become _slower_.  Though not by much (around
10%), it is still a shame.

Yes, do your testing and please, as always, try to report as
quantitatively as possible, so that we can really compare apples to
apples.

João

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 16.08.2021 06:53, João Távora wrote:
>> But why do it, since a string plist is a such a nice place to do these
>> associations and there's -- apart from your aesthetics considerations
>> -- 0 drawbacks identified?
>
> You read all the explanations, and THAT'S your conclusion?

Yes, I currently conclude that there are 0 drawbacks identified to
creating, _via_ string properties, an association of, say, property
'joaot/answer' with value '42' to _any_ string in my current and future
Emacs runtimes.

I conclude that because --- apart from your aesthetics considerations
("do we really want to see...") --- I have not seen identification of
such drawbacks.  Have I missed any?

João

bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/14/21 9:01 AM, Eli Zaretskii wrote:
> Just to make sure we are on the same page: adding a text property to a
> string doesn't mutate a string.  Lisp programs that process these
> strings will not necessarily see any difference, and displaying those
> strings will also not show any difference if the property is not
> related to display.  So the assumption that seems to be made here,
> that adding a property is the same as mutating a string, is IMO
> inaccurate if not incorrect.

While I agree that adding text properties is not mutation of the string
text itself it still mutates the string data structure. Dmitry made a
good point about this - if a completion table uses obarray as backend to
the completion table you suddenly attach text properties to symbol
names. This is not a good idea. I actually had such an issue once in a
package I developed, where I accidentally attached text properties (via
the mutating put-text-property API) to some strings I didn't own. This
lead to unexpected side effects at other places where I didn't expect
the strings to have properties attached.

Daniel

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/14/21 9:12 AM, Eli Zaretskii wrote:
>> Since up until now completion-pcm--hilit-commonality copied all strings 
>> before modifying, completion tables such as described (with "shared" 
>> strings) have all been "legal". Suddenly deciding to stop supporting 
>> them would be a major API breakage with consequences that are hard to 
>> predict. And while I perhaps agree that it's an inconvenience, I don't 
>> think it's a choice we can simply make as this stage in c-a-p-f's 
>> development.
> 
> This sounds like an argument against Daniel's approach as well, no?
> Because if a completion API returns strings it "doesn't own", there
> will be restrictions on Lisp programs that use those strings, because
> those Lisp programs previously could do anything they liked with those
> strings, and now they cannot.  Or am I missing something?

No, in my patch the displayed candidate strings are still copied before
the text properties are attached. The strings are kept intact as they
are now.

Daniel

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/16/21 6:20 AM, João Távora wrote:
> It's not me who is saying it, it's my Emacs :-) The real problem is that
> with Daniel's patch, the frontends using the current API (as in
> icomplete/fido) measurably become _slower_.  Though not by much (around
> 10%), it is still a shame.

I have to check this. I claim that 'completion-all-completions' should
not get slower with my patch. If it gets indeed slower as your benchmark
shows, this should be fixed and can be fixed since I am not doing
something else than decomposing the highlighting and filtering
processes, which are already present in the current machinery. The
amount of work stays the same. However in the case the new
'completion-filter-completions' API is used, the filtering will get much
faster since no highlighting and copying takes place.

Daniel

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/16/21 6:25 AM, João Távora wrote:
> Yes, I currently conclude that there are 0 drawbacks identified to
> creating, _via_ string properties, an association of, say, property
> 'joaot/answer' with value '42' to _any_ string in my current and future
> Emacs runtimes.
> 
> I conclude that because --- apart from your aesthetics considerations
> ("do we really want to see...") --- I have not seen identification of
> such drawbacks.  Have I missed any?

There are serious drawbacks of attaching "private" string properties to
arbitrary strings. For once it complicates debugging seriously if there
are suddenly string properties attached to symbol names. It also leads
to a potential memory leak. But even if we could chose this approach
with global side-effects I don't see a reason to do it given the
approach I am proposing in my patch, which avoids these problems entirely.

1. My approach decomposes the already existing completion machinery into
two steps: 1. filtering and 2. highlighting. It does not change the
fundamentals of the completion machinery. This decomposition of the
existing infrastructure is also what leads to the small number of added
lines to minibuffer.el, even if the patch itself is not as small as we
would like to have it.

2. Why take the chances of introducing potentially harmful global side
effects by attaching string properties to arbitrary strings if we can
avoid it easily?

3. The `completion-filter-completions` API is the fastest possible API
for filtering since it does not change the strings at all, it does not
attach string properties or modify the strings in any other way. By
construction, it is a pure function which only filters the list of
candidates. This makes the function easy to use and easy to reason
about. An API which adds private properties to the strings cannot be as
fast and non-intrusive.

4. If 'completion-all-completions' does indeed get slower thanks to my
patch, it is a performance regression of my patch. I will fix this. And
I thank you João for bringing this to my attention. However one should
also consider that in the end, 'fido-mode' and 'icomplete-mode' should
move to the new API 'completion-filter-completions' such that they
benefit from the fast filtering and only copy and highlight the actually
displayed strings. Given this a potential regression of
'completion-all-completions' would matter less for the incrementally
updating UIs. But of course I feel the same way as João - a performance
regression in the API 'completion-all-completions' is unacceptable. It
will be fixed.

Daniel

bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

Hello Eli,

here I respond to the comments you sent out after I've already sent the
overhauled patch.

On 8/14/21 8:27 AM, Eli Zaretskii wrote:
>> The function 'completion-filter-completions' receives a completion table
>> as argument.  The strings produced by this table are returned
>> unmodified, but of course the completion table has to produce them.  For
>> a static completion table (e.g., in the simplest case a list of strings)
>> the completion table itself will not allocate strings.  In this scenario
>> 'completion-filter-completions' will not perform any string allocations,
>> only the list will be allocated.  This is what leads to major
>> performance gains.
> 
> My point was that at least some of this should be in the description,
> otherwise it will leave the reader wondering.

I agree with this. The documentation should be improved.

>>>> +(defvar completion--filter-completions nil
>>>> +  "Enable the new completions return value format.
>>>
>>> Btw, why is this an internal variable?  Shouldn't all completion
>>> styles ideally support it?  If so, it should be a public variable,
>>> documented in the ELisp manual.  And the name should also end with -p,
>>> since it's a boolean.  How about completion-filter-completions-format-p?
>>
>> (As I understood the style guide '-p' is not a good idea for boolean
>> variables, since a value is not a predicate in a strict sense.)
>>
>> To address your technical comment - this variable is precisely what one
>> of the technical difficulties mentioned in my other mail is about.  The
>> question is how we can retain backward compatibility of the completion
>> style 'all' functions, e.g., 'completion-basic-all-completions', while
>> still allowing the function to return the newly introduced alist format
>> with more data, which enables 'completion-filter-completions' to perform
>> the efficient deferred highlighting.
> 
> I understand, but given that we provide this for other packages, it
> shouldn't be an internal variable.

No, we explicitly don't provide this variable for other packages. It is
explicitly only meant to be used for the existing completion styles
emacs21, emacs22, basic, substring, partial-completion, initials and
flex to opt-in in a backward-compatible/calling convention preserving
way to the alist return format. The idea is to keep the existing APIs
fully backward compatible.

Other packages should select the format returned from the completion
styles differently. They should return the alist format on Emacs 28 or
if the API 'completion-filter-completions' API is present. In the not so
near future external packages which support only Emacs 28 and upwards
will then only return the alist format and don't have to perform any
detection anymore.

>>> Also, the "This function has been superseded..." part should be a new
>>> paragraph, so that it stands out.  (And I'm not yet sure we indeed
>>> want to say "superseded" here, but that's part of the on-going
>>> discussion.  maybe use a more neutral language here, like "See also".)
>>
>> The new API 'completion-filter-completions' will substitute the existing
>> API 'completion-all-completions'.
> 
> That's your hope, and I understand.  But we as a project didn't yet
> decide to deprecate the original APIs, so talking about superseding is
> premature.

It is not the hope - it is the explicit goal. The API has been designed
to replace the existing API 'completion-all-completions'. We can of
course tone this down. However I, as a package author, would appreciate
if Emacs tells me when a newer API aims to replace another API and when
the documentation is explicit about it. Of course if you decide to have
the doc strings written in a different tone, I will adapt my patch
accordingly. Here I am just explaining why I chose the word "superseded"
instead of a more neutral word.

>>> Is "filter" really the right word here (in the doc string)?  "Filer"
>>> means you take a sequence and produce another sequence with some
>>> members removed.  That's not what this API does, is it?  Suggest to
>>> use a different name, like completion-completions-alist or
>>> completion-all-completions-as-alist.
>>
>> "Filter" seems like exactly the right word to me.  The function takes a
>> list of strings (or a completion table) and returns a subset of matching
>> completion strings without further modifications to the strings. See
>> above what I wrote about allocations.
> 
> But the name says "filter completions".  Which would mean you take a
> list of completions and filter out some of them.  A completion table
> is much more general object than a list of strings.  Thus, I think
> using "filter" here is sub-optimal.

Okay, you are right about this. But I think even if the name
'completion-filter-completions' is not 100% precise it still conveys
what the API is about. 'completion-completions-alist' or
'completion-all-completions-as-alist' are valid names of course, but I
dislike them for their verbosity. Already 'completion-all-completions'
is quite verbose. A strong argument to use this long name is that the
completion style functions are still called
'completion-basic-all-completions' etc. But if we accept that the new
API 'completion-filter-completions' will actually supersede the existing
API 'completion-all-completions' it makes sense to use a name which will
not hurt our eyes in the long run. However this is of course just a
personal preference of mine. I don't want to spent much time with name
bikeshedding discussions. If you decide on a name, I will adapt my patch
accordingly.

>>>> +Only the elements of table that satisfy predicate PRED are considered.
>>>> +POINT is the position of point within STRING.  The METADATA may be
>>>> +modified by the completion style.  The return value is a alist with
>>>> +the keys:
>>>> +
>>>> +- base: Base position of the completion (from the start of STRING)
>>>
>>> "Base" here means the beginning?  If so, why not call it "beg" or
>>> somesuch?
>>
>> Base position is a fixed term which is already used in minibuffer.el for
>> completions.  See also 'completion-base-position' for example.
> 
> Well, we don't have to keep bad habits indefinitely.  It's okay to
> lose them and use better terminology.  Or at least to explain that
> terminology in parentheses the first time it is used in some context.

Okay, I agree. However I tried to avoid including superfluous changes
with my patch set. We should add more context and documentation and then
rename the variables in another patch if we decide that we want to go
through with it.

>>> Are we really losing the completion-score property here?  If so, why?
>>
>> Yes, the property is removed in the current patch.  It is not actually
>> used for anything in the new implementation.  But it is possible to
>> restore the property such that 'completion-all-completions' always
>> returns scored candidates as it does now.  See my other mail regarding
>> the caveats of the current patch.
> 
> I'd prefer not to lose existing features, because that'd potentially
> make the changes backward-incompatible.

The overhauled patch (version 2) ensures that no features are lost. The
patch is fully backward compatible. There is a potential performance
regression in 'completion-all-completions' as identified by João. I have
yet to confirm this regression. In any case, it should be fixable since
the refactored 'completion-all-completions' API does precisely the same
amount of work as it does now.

Furthermore more documentation should be added to my patch. As of now
'completion-all-completions' is not mentioned in the info manual and
'completion-filter-completions' is also not added there. We may want to
improve the documentation of that. But for now I would like to limit my
documentation improvements to expanding the doc strings of the functions
involved in my patch.

Daniel

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 10:09 AM Daniel Mendler <mail@daniel-mendler.de> wrote:

> There are serious drawbacks of attaching "private" string properties to
> arbitrary strings. For once it complicates debugging seriously if there
> are suddenly string properties attached to symbol names. It also leads
> to a potential memory leak.

Please, in the name of the sanity of this discussion, justify these two
statements with examples or follow them with a clause like "because...".

> 3. The `completion-filter-completions` API is the fastest possible API

Again that's quite a statement that I cannot evaluate the veracity of
without hard proof.

What I _can_ evaluate is the material you have put forward, and using
your patch and your Vertico completion software, version 0.14, the very
latest one. I try

   emacs -Q -f package-initialize benchmark.el

where benchmark.el is:

   (setq completion-styles '(flex))
   (defmacro heyhey ()
     `(progn
        ,@(cl-loop repeat 300000
           collect `(defun ,(intern (symbol-name (gensym "heyhey"))) () 42))))
   (heyhey)

I then turn on vertico-mode and press C-h f.  It takes about 4-5 seconds.
It's *faster* than if I do the same with fido-vertical-mode and the current
master, but is noticeably *slower* than if I do the same with the patch
provided earlier and available at scratch/icomplete-lazy-highlight-attempt-2.

Unfortunately, I cannot measure quantitatively here, because I don't
know how to tell Vertico to wait until it gets the correct result.
In other words, take this form:

    (completing-read "bla" obarray)<cursor here>

if you type C-u C-x C-e C-m veeery s-l-o-w-l-y in Vertico, if prints
, correctly, the character "%".  But if you evaluate it quickly wrapped
in a benchmark-run, it returns immediately and prints "".

In fido-mode, it always waits blockinly until it prints the correct result
and the time it took it to achieve that result.  Not questioning if this is
a bug in Vertico, but it would help if it could do the same, or be
configured to do the same, so that we can measure.

Without that, we can't evaluate the speed of Vertico where,
presumably, the fastest API in the world is being used right now.

> 4. If 'completion-all-completions' does indeed get slower thanks to my
> patch, it is a performance regression of my patch. I will fix this. And
> I thank you João for bringing this to my attention. However one should
> also consider that in the end, 'fido-mode' and 'icomplete-mode' should
> move to the new API 'completion-filter-completions' such that they
> benefit from the fast filtering and only copy and highlight the actually
> displayed strings.

Maybe they will, maybe they will.  But it's still quite early to decide if
we're going to move all frontends to that API.  There's no manual
documentation for it. Conceivably, if you appreciate your API so, you
could demonstrate in practice us how easy it is to use by providing
a separate patch that adapts icomplete-mode and fido-mode to use it.

Then, I or other fido-mode users could test it for a while, evaluating
its speed and correctness.  If it performs well and the completion
API architects have a good outlook for it, I see no reason why it
shouldn't be merged and eventually supersede the new one.

In the meantime, there is a patch with a measured and documented
performance boost where fido-mode and icomplete-mode move
opt-in to a new `completion-lazy-hilit` feature in the "old" API with
a total or four 1-line changes.  That patch lives in the branch
scratch/icomplete-lazy-highlight-attempt-2.

I think we should move to that, solving the bug#48841 while we
do the evaluation of all aspects of your contributions.

João Távora

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/16/21 12:15 PM, João Távora wrote:
> On Mon, Aug 16, 2021 at 10:09 AM Daniel Mendler <mail@daniel-mendler.de> wrote:
> 
>> There are serious drawbacks of attaching "private" string properties to
>> arbitrary strings. For once it complicates debugging seriously if there
>> are suddenly string properties attached to symbol names. It also leads
>> to a potential memory leak.
> 
> Please, in the name of the sanity of this discussion, justify these two
> statements with examples or follow them with a clause like "because...".

João, I am giving hard examples here. What is not an example about
"memory leak" or making debugging output verbose thanks to the attached
string properties? You always repeat your demands but whatever I write
it is not satisfactory for you. Is it possible to convince you? Can you
try to interpret my arguments in a positive and constructive light
somehow and fill them with meaning such that it makes sense to you? My
goal is not to be right here just for the sake of being right. As Eli
said, nobody has to prove anything.

>> 3. The `completion-filter-completions` API is the fastest possible API
> 
> Again that's quite a statement that I cannot evaluate the veracity of
> without hard proof.

As I said, I will ensure that my API does not introduce performance
regressions. And since my new API performs strictly less work than your
proposal it will necessarily be faster if you consider only the
filtering, which is what matters for incrementally updating UIs.

>> 4. If 'completion-all-completions' does indeed get slower thanks to my
>> patch, it is a performance regression of my patch. I will fix this. And
>> I thank you João for bringing this to my attention. However one should
>> also consider that in the end, 'fido-mode' and 'icomplete-mode' should
>> move to the new API 'completion-filter-completions' such that they
>> benefit from the fast filtering and only copy and highlight the actually
>> displayed strings.
> 
> Maybe they will, maybe they will.  But it's still quite early to decide if
> we're going to move all frontends to that API.  There's no manual
> documentation for it. Conceivably, if you appreciate your API so, you
> could demonstrate in practice us how easy it is to use by providing
> a separate patch that adapts icomplete-mode and fido-mode to use it.

There is also no manual documentation of 'completion-all-completions'.
Of course you are right that it is early to make these decisions, but
the new API 'completion-filter-completions' is designed in a way to
allow adoption by 'icomplete-mode'. It is important to design the API
such that adoption is possible. I have a patch for Vertico which shows
this. I can provide patches for 'icomplete-mode' separately later on.

> In the meantime, there is a patch with a measured and documented
> performance boost where fido-mode and icomplete-mode move
> opt-in to a new `completion-lazy-hilit` feature in the "old" API with
> a total or four 1-line changes.  That patch lives in the branch
> scratch/icomplete-lazy-highlight-attempt-2.

As argued multiple times here now, the change you are proposing is
fragile. It will lead to problems later on. The goal is not to find the
smallest change which leads to a performance boost, all API violations
allowed. Attaching "private" string properties to arbitrary strings is
an API violation which we will regret later and which will make Emacs
harder to debug and harder to use.

> I think we should move to that, solving the bug#48841 while we
> do the evaluation of all aspects of your contributions.

No, we should not merge this problematic patch of yours. See the many
arguments against this proposal.

Daniel

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 11:53 AM Daniel Mendler <mail@daniel-mendler.de> wrote:

> >> There are serious drawbacks of attaching "private" string properties to
> >> arbitrary strings. For once it complicates debugging seriously if there
> >> are suddenly string properties attached to symbol names. It also leads
> >> to a potential memory leak.
> > Please, in the name of the sanity of this discussion, justify these two
> > statements with examples or follow them with a clause like "because...".
> João, I am giving hard examples here.

If I say to you:  "It's quite obvious your patch breaks all Git repositories
in Kathmandu, Nepal"  I am expected to demonstrate how a patch to
Emacs, leads to a obscure security flaw in the Linux operating system,
that tickles a butterfly at a certain angle that causes an earthquake
in the Kathmandu data center, literally breaking their Git repositories.

This is the the kind of statement you are making to me and the kind
of logical reasoning I'm looking for.

Alternatively, you can provide an actual experiment I can run in
my computer that demonstrates the bug.

I am not a native English speaker, and maybe you don't understand
my language.  Another way to explain what I am talking  is to talk about
"bug reproduction".  You say there's a bug in my patch, I am asking you
for a "bug reproduction recipe" as defined by most,  if not all, the results
you get by searching "bug reproduction recipe" in the Google search engine.

> goal is not to be right here just for the sake of being right. As Eli
> said, nobody has to prove anything.

This is clearly not what he said.

> >> 3. The `completion-filter-completions` API is the fastest possible API
> >
> > Again that's quite a statement that I cannot evaluate the veracity of
> > without hard proof.
>
> As I said, I will ensure that my API does not introduce performance
> regressions.

Not only that, to produce  veracity on that statement you would need
some much more demanding proof, which is somehow be able to
evaluate all possible APIs to compellingly demonstrate that yours
triumphs.

>  I have a patch for Vertico which shows
> this. I can provide patches for 'icomplete-mode' separately later on.

Yes, please do. The earlier, the better.

> No, we should not merge this problematic patch of yours. See the many
> arguments against this proposal.

I'm sorry to speak this child-like language, but a problem is a "bad thing".
An undesirable thing that happens when presumably safe and good
action(s) is taken by some user.  Can you explain how, given my patch,
a user would take a sequence of innocent actions that would lead to a
"bad thing" that would _not_ happen if the same sequence of innocent
actions were taken  in a version of Emacs without the patch applied?
That, to me, is what constitutes "a bug/problem in the patch".

Let me give you an example: if I make a patch that deletes `/lisp` in
the Emacs source tree, the innocent action "make"  would probably
not work.  That would be the problem/"bad thing"/bug in that patch.

We cannot proceed this discussion without these explanations.  Mind
you, I'm not stating, because it is impossible to prove, that my
patch cannot possibly generate problems, subtle or obvious (that, by
the way, is my interpretation of what Eli meant). But since you so
confidently state that it does, it's quite reasonable that I ask you
for examples that demonstrate it.

Once you do demonstrate these bugs, it's reasonable I will go about
fixing them.  Exactly as you say you are going to do.  I demonstrated
with code and numbers a regression in _your_ patch, and you say are
going to fix it.  That's great, and that's the way it should be.  But
you possibly wouldn't go about fixing it if I hadn't demonstrated the
regression compellingly, just as I can't go about fixing a "memory leak"
or "debugging difficulties" if you don't explain what these things mean
to you or how my patch causes them.

João

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca, 48841@debbugs.gnu.org,
>  47711@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 16 Aug 2021 06:17:32 +0300
> 
> On 14.08.2021 14:29, Eli Zaretskii wrote:
> > Text properties are stored separately from the string, so I don't
> > think adding properties can in general be referred to as "change".
> 
> Are you thinking of C strings?

No, about the implementation of Lisp strings in Emacs.

> Lisp strings carry text properties in addition to the array of 
> characters. It doesn't really matter where in the memory the properties 
> and the characters reside.

Well, it does, at least in some situations.  The string text is not
affected, and so the code which processes the string will not notice
that it has a property about which that code has no idea.  Only
properties that are known to the processing code can affect it;
non-standard properties private to some other code will generally pass
unnoticed.

> > Whether in some particular situation that could count as a "change"
> > depends on that situation and on the particular property, of course.
> 
> I was talking in the general sense: modifying a value.
> 
> One can talk about whether a certain modification matters in certain 
> situations, but that's not the way to discount a general principle.

I didn't want to start a general philosophical discussion about string
mutability.  I hoped to provide input of specific practical use in the
context of this discussion.  If what I said is not useful, just
disregard it.

> > I'm not sure in the context of completion there's any reason to count
> > as "change" adding properties that don't affect display.
> 
> For the context in question, whether the properties affect display is 
> not particularly important. Properties affecting display just make it 
> easier to notice that something's wrong. Bug involving other properties 
> should be more difficult to investigate.

Once again, if some code invents its private property, not used
anywhere else and not documented anywhere else, then putting such a
property on a string has very high chances of going unnoticed.  I hope
this consideration helps this discussion, because saying that
properties change a string blurs the distinction between actually
changing the string text or its properties that affect many parts in
Emacs, and adding some obscure property that is not known to anyone.

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> Cc: 47711@debbugs.gnu.org, monnier@iro.umontreal.ca, joaotavora@gmail.com,
>  48841@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 16 Aug 2021 06:26:58 +0300
> 
> On 14.08.2021 10:01, Eli Zaretskii wrote:
> 
> > Just to make sure we are on the same page: adding a text property to a
> > string doesn't mutate a string.  Lisp programs that process these
> > strings will not necessarily see any difference, and displaying those
> > strings will also not show any difference if the property is not
> > related to display.  So the assumption that seems to be made here,
> > that adding a property is the same as mutating a string, is IMO
> > inaccurate if not incorrect.
> 
> This is nonsense.
> 
> A program won't necessarily see a difference in *any* changed value, as 
> long as some part of it stays the same.
> 
> I can zero out the tail of a string, and have a program that only looks 
> at its first few characters. It wouldn't mean that a string hasn't changed.

You are not making any sense.

Anyway, if what I wrote doesn't help, feel free to disregard it.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> Cc: joaotavora@gmail.com, monnier@iro.umontreal.ca, 48841@debbugs.gnu.org,
>  47711@debbugs.gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Mon, 16 Aug 2021 10:48:19 +0200
> 
> On 8/14/21 9:12 AM, Eli Zaretskii wrote:
> >> Since up until now completion-pcm--hilit-commonality copied all strings 
> >> before modifying, completion tables such as described (with "shared" 
> >> strings) have all been "legal". Suddenly deciding to stop supporting 
> >> them would be a major API breakage with consequences that are hard to 
> >> predict. And while I perhaps agree that it's an inconvenience, I don't 
> >> think it's a choice we can simply make as this stage in c-a-p-f's 
> >> development.
> > 
> > This sounds like an argument against Daniel's approach as well, no?
> > Because if a completion API returns strings it "doesn't own", there
> > will be restrictions on Lisp programs that use those strings, because
> > those Lisp programs previously could do anything they liked with those
> > strings, and now they cannot.  Or am I missing something?
> 
> No, in my patch the displayed candidate strings are still copied before
> the text properties are attached. The strings are kept intact as they
> are now.

I was talking about the infrastructure that produces the completion
candidates, not about the application that uses them.  My point is
that your approach requires the applications using the candidates to
copy them, whereas previously they could use them without copying.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 12:57 PM Eli Zaretskii <eliz@gnu.org> wrote:

> I was talking about the infrastructure that produces the completion
> candidates, not about the application that uses them.  My point is
> that your approach requires the applications using the candidates to
> copy them, whereas previously they could use them without copying.

If it helps, I think that that is true of all alternatives presented
so far (though I haven't read the big patch fully yet). The difference is
that the consumers who copy the candidate strings will only copy a much
smaller number, typically only the ones that need to be displayed.
Whereas currently, all candidate strings are copied, displayed or not.

João Távora

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

João, the discussion is clearly not progressing. I propose that we both
take a step back and let the Emacs maintainers, who participated in this
discussion, decide on how to proceed. It seems to me that all arguments
and data has been presented and there is no need for further
reiterations in more and more colorful language. I would also like to
point out that implying that I don't understand your language is
borderline acceptable. I understand the discussion very well, but I
don't understand why you are using these unfair and invalid means of
discussion.

For example there could be these decision outcomes:

1. The information presented up to now does not allow the maintainers to
make a decision. For example the maintainers may ask for further
clarification from you, João, or they may ask for benchmarks from me or
a prove that my patch does not lead to performance regressions.

2. The maintainers decide that no patch should be merged.

2. The maintainers decide that your patch will be merged. I will accept
this decision.

3. The maintainers decide that my patch will be merged. You will accept
this decision.

4. The maintainers decide that both patches will be merged such that
both approaches will be supported. We both will accept this decision.

I want to summarize the situation in the following:

The patches in question address a performance issue in the current
completion machinery which is caused by over-eager copying of the
completion candidate strings and over-eager application of the
highlighting to all candidate strings. For incrementally updating UIs it
would be sufficient to only copy and highlight the strings which are
actually going to be displayed.

My patch takes the approach to expose the existing two-step completion
process, which consists of filtering and highlighting. By returning the
filtered completion strings and a highlighting function this two-step
process is decomposed and the caller of the API has the ability to call
the highlighting function on only the displayed subset of completion
candidates. I argue that exposing the filtering and highlighting
procession steps is the logical and natural conclusion of the existing
machinery.

My patch is fully backward compatible and aims to not introduce any
regressions (also no performance regressions) to the existing API.
Furthermore my patch adheres to the current guarantees given by the
existing 'completion-all-completions' API. The completion strings
provided by the completion backend are not mutated in any way, no string
properties are attached. Since the API 'completion-filter-completions'
proposed in my patch does the minimal amount of work necessary (only
filtering), if no highlighting is requested, I argue that the new API is
the most efficient possible API, given the current constraints.

Furthermore since I am introducing a new API, outstanding issues can be
solved which could not be solved until now given the constraints of the
existing 'completion-all-completions' API. In particular the new API
'completion-filter-completions' API returns additional data like the end
position of the completion boundaries. Until now the end position was
not made available and 'completion-base-position' just used the length
of the input to guess the end position. In a strict sense this guess is
incorrect and there is a FIXME in minibuffer.el, mentioning this issue.

The downside of my patch is that it is a large patch. While it adds only
196 lines of code, which is not much and expected given that it only
reshuffles the existing machinery, it is still a large patch in total.

On the other side, João's patch avoids the complication of adhering to
the existing guarantees of the APIs and takes the liberty of attaching
"private" string properties to the completion strings of the completion
table backend. I argue that attaching the string properties is a
violation of the guarantees of the existing API and violates the
expectations of the existing many completion tables. One very severe
scenario is when obarray is used as completion table, since then each
symbol name gets a private property attached. I argue that such global
side effects like attaching string properties to all completion
candidates should better be avoided. There is the issue that the
attached string properties are a potential memory leak. When dumping the
string representation of symbol names, the symbols will have additional
properties which will complicate the debugging experience. Furthermore
it will lead to confusion since the global side effect during completion
will suddenly have an influence of symbols which don't have to do
anything with completion. The big advantage of João's patch is that it
is very limited in scope and very simple. However I argue that this
simplicity is hard-won and we will regret this approach later due to the
global side effects.

Therefore I conclude that the two-step process proposed in my patch,
which does not introduce problematic global side effects is the better
approach forward. Furthermore a new API is needed such that more
completion data can be returned, e.g., the completion end position. One
could even return additional match data in the future given that the new
API 'completion-filter-completions' is extensible thanks to its alist
return value.

João, please feel free to also present your closing arguments here.

Daniel

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/16/21 1:57 PM, Eli Zaretskii wrote:
> I was talking about the infrastructure that produces the completion
> candidates, not about the application that uses them.  My point is
> that your approach requires the applications using the candidates to
> copy them, whereas previously they could use them without copying.

Okay, I understand. Yes, in my patch the strings returned by
'completion-filter-completions' must not be mutated by the API consumer
directly. This should be documented clearly, but it is not unexpected.
For example the API 'all-completions' which one can use to obtain the
strings from a completion table also requires the caller to not mutate
the returned strings.

Daniel

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

> João, please feel free to also present your closing arguments here.

Sorry, I don't "close" arguments like this.

I hope you can provide:

* the fixes to the regression identified
* the benchmarks you say you have
* the patches to icomplete.el that show how it uses your new API
* the demonstrations of the bugs you accuse my patch to suffer from

Thanks.

On Mon, Aug 16, 2021 at 1:05 PM Daniel Mendler <mail@daniel-mendler.de> wrote:
>
> João, the discussion is clearly not progressing. I propose that we both
> take a step back and let the Emacs maintainers, who participated in this
> discussion, decide on how to proceed. It seems to me that all arguments
> and data has been presented and there is no need for further
> reiterations in more and more colorful language. I would also like to
> point out that implying that I don't understand your language is
> borderline acceptable. I understand the discussion very well, but I
> don't understand why you are using these unfair and invalid means of
> discussion.
>
> For example there could be these decision outcomes:
>
> 1. The information presented up to now does not allow the maintainers to
> make a decision. For example the maintainers may ask for further
> clarification from you, João, or they may ask for benchmarks from me or
> a prove that my patch does not lead to performance regressions.
>
> 2. The maintainers decide that no patch should be merged.
>
> 2. The maintainers decide that your patch will be merged. I will accept
> this decision.
>
> 3. The maintainers decide that my patch will be merged. You will accept
> this decision.
>
> 4. The maintainers decide that both patches will be merged such that
> both approaches will be supported. We both will accept this decision.
>
> I want to summarize the situation in the following:
>
> The patches in question address a performance issue in the current
> completion machinery which is caused by over-eager copying of the
> completion candidate strings and over-eager application of the
> highlighting to all candidate strings. For incrementally updating UIs it
> would be sufficient to only copy and highlight the strings which are
> actually going to be displayed.
>
> My patch takes the approach to expose the existing two-step completion
> process, which consists of filtering and highlighting. By returning the
> filtered completion strings and a highlighting function this two-step
> process is decomposed and the caller of the API has the ability to call
> the highlighting function on only the displayed subset of completion
> candidates. I argue that exposing the filtering and highlighting
> procession steps is the logical and natural conclusion of the existing
> machinery.
>
> My patch is fully backward compatible and aims to not introduce any
> regressions (also no performance regressions) to the existing API.
> Furthermore my patch adheres to the current guarantees given by the
> existing 'completion-all-completions' API. The completion strings
> provided by the completion backend are not mutated in any way, no string
> properties are attached. Since the API 'completion-filter-completions'
> proposed in my patch does the minimal amount of work necessary (only
> filtering), if no highlighting is requested, I argue that the new API is
> the most efficient possible API, given the current constraints.
>
> Furthermore since I am introducing a new API, outstanding issues can be
> solved which could not be solved until now given the constraints of the
> existing 'completion-all-completions' API. In particular the new API
> 'completion-filter-completions' API returns additional data like the end
> position of the completion boundaries. Until now the end position was
> not made available and 'completion-base-position' just used the length
> of the input to guess the end position. In a strict sense this guess is
> incorrect and there is a FIXME in minibuffer.el, mentioning this issue.
>
> The downside of my patch is that it is a large patch. While it adds only
> 196 lines of code, which is not much and expected given that it only
> reshuffles the existing machinery, it is still a large patch in total.
>
> On the other side, João's patch avoids the complication of adhering to
> the existing guarantees of the APIs and takes the liberty of attaching
> "private" string properties to the completion strings of the completion
> table backend. I argue that attaching the string properties is a
> violation of the guarantees of the existing API and violates the
> expectations of the existing many completion tables. One very severe
> scenario is when obarray is used as completion table, since then each
> symbol name gets a private property attached. I argue that such global
> side effects like attaching string properties to all completion
> candidates should better be avoided. There is the issue that the
> attached string properties are a potential memory leak. When dumping the
> string representation of symbol names, the symbols will have additional
> properties which will complicate the debugging experience. Furthermore
> it will lead to confusion since the global side effect during completion
> will suddenly have an influence of symbols which don't have to do
> anything with completion. The big advantage of João's patch is that it
> is very limited in scope and very simple. However I argue that this
> simplicity is hard-won and we will regret this approach later due to the
> global side effects.
>
> Therefore I conclude that the two-step process proposed in my patch,
> which does not introduce problematic global side effects is the better
> approach forward. Furthermore a new API is needed such that more
> completion data can be returned, e.g., the completion end position. One
> could even return additional match data in the future given that the new
> API 'completion-filter-completions' is extensible thanks to its alist
> return value.
>
> João, please feel free to also present your closing arguments here.
>
> Daniel



-- 
João Távora

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Mon, 16 Aug 2021 13:02:04 +0100
> Cc: Daniel Mendler <mail@daniel-mendler.de>, Dmitry Gutov <dgutov@yandex.ru>, 
> 	Stefan Monnier <monnier@iro.umontreal.ca>, 48841@debbugs.gnu.org, 47711@debbugs.gnu.org
> 
> On Mon, Aug 16, 2021 at 12:57 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I was talking about the infrastructure that produces the completion
> > candidates, not about the application that uses them.  My point is
> > that your approach requires the applications using the candidates to
> > copy them, whereas previously they could use them without copying.
> 
> If it helps, I think that that is true of all alternatives presented
> so far

Yes, I know.  I was comparing the proposed alternatives to what we
have now, and specifically because Dmitry mentioned this aspect.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> Cc: Dmitry Gutov <dgutov@yandex.ru>, Lars Ingebrigtsen <larsi@gnus.org>,
>  47711@debbugs.gnu.org, 48841@debbugs.gnu.org,
>  Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Mon, 16 Aug 2021 12:52:58 +0200
> 
> On 8/16/21 12:15 PM, João Távora wrote:
> > On Mon, Aug 16, 2021 at 10:09 AM Daniel Mendler <mail@daniel-mendler.de> wrote:
> > 
> >> There are serious drawbacks of attaching "private" string properties to
> >> arbitrary strings. For once it complicates debugging seriously if there
> >> are suddenly string properties attached to symbol names. It also leads
> >> to a potential memory leak.
> > 
> > Please, in the name of the sanity of this discussion, justify these two
> > statements with examples or follow them with a clause like "because...".
> 
> João, I am giving hard examples here. What is not an example about
> "memory leak" or making debugging output verbose thanks to the attached
> string properties?

FWIW, I also don't understand how adding properties could cause a
memory leak.  When a string is GCed, its properties get GCed as well,
all of them.  Am I missing something?

As to more difficult debugging, I think adding a couple of properties
that have simple structure will not impair debugging too much.
Strings with many properties are not uncommon in Emacs, so we already
have to deal with that.

> As I said, I will ensure that my API does not introduce performance
> regressions. And since my new API performs strictly less work than your
> proposal it will necessarily be faster if you consider only the
> filtering, which is what matters for incrementally updating UIs.

I would indeed suggest both to make sure there's no performance
regressions, and would like to see some data similar to what João
presented, which backs up your assessments about your proposal being
faster.  Since performance is the main motivation for these changes, I
think it's important for us to be on the same page wrt facts related
to performance, before we make the decision how to proceed.

Thanks.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> Cc: Dmitry Gutov <dgutov@yandex.ru>, Lars Ingebrigtsen <larsi@gnus.org>,
>  47711@debbugs.gnu.org, 48841@debbugs.gnu.org,
>  Stefan Monnier <monnier@iro.umontreal.ca>, Eli Zaretskii <eliz@gnu.org>
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Mon, 16 Aug 2021 14:05:54 +0200
> 
> João, the discussion is clearly not progressing. I propose that we both
> take a step back and let the Emacs maintainers, who participated in this
> discussion, decide on how to proceed. It seems to me that all arguments
> and data has been presented and there is no need for further
> reiterations in more and more colorful language.

As I wrote elsewhere, I'd like to see the performance aspects of this
to be presented from both sides, and agreed upon.  I don't think we
can make the decision before we have performance data we all agree
about.  The other pros and cons are all of qualitative nature, and
involve intuition, personal experience, and personal preferences, so
each one will have their own balance.  But performance is both basic
and qualitative, and we should have the facts and agree on them.

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler]

On 8/16/21 2:39 PM, Eli Zaretskii wrote:
>> João, I am giving hard examples here. What is not an example about
>> "memory leak" or making debugging output verbose thanks to the attached
>> string properties?
> 
> FWIW, I also don't understand how adding properties could cause a
> memory leak.  When a string is GCed, its properties get GCed as well,
> all of them.  Am I missing something?

If you add string properties to all symbol names this memory will stay
alive for much longer than necessary. It is not a memory leak in the
strongest sense. The memory is still reachable, but there is still no
need to keep the string properties allocated. This is comparable to
memory leaks in other GCed languages where memory is also kept alive for
longer than necessary.

> As to more difficult debugging, I think adding a couple of properties
> that have simple structure will not impair debugging too much.
> Strings with many properties are not uncommon in Emacs, so we already
> have to deal with that.

I disagree with that. We are talking about adding string properties to
every symbol name. This is a global side effect and different from
adding string properties to a set of isolated string in a controlled
manner. I also don't understand why one would even want to take any
chances here given that the feature can be implemented in a way which
avoids this global side effect entirely as my patch shows.

>> As I said, I will ensure that my API does not introduce performance
>> regressions. And since my new API performs strictly less work than your
>> proposal it will necessarily be faster if you consider only the
>> filtering, which is what matters for incrementally updating UIs.
> 
> I would indeed suggest both to make sure there's no performance
> regressions, and would like to see some data similar to what João
> presented, which backs up your assessments about your proposal being
> faster.  Since performance is the main motivation for these changes, I
> think it's important for us to be on the same page wrt facts related
> to performance, before we make the decision how to proceed.

I will prepare some benchmarks.

Daniel

bug#47711: bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> Cc: joaotavora@gmail.com, dgutov@yandex.ru, monnier@iro.umontreal.ca,
>  48841@debbugs.gnu.org, 47711@debbugs.gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Mon, 16 Aug 2021 11:42:22 +0200
> 
> >> To address your technical comment - this variable is precisely what one
> >> of the technical difficulties mentioned in my other mail is about.  The
> >> question is how we can retain backward compatibility of the completion
> >> style 'all' functions, e.g., 'completion-basic-all-completions', while
> >> still allowing the function to return the newly introduced alist format
> >> with more data, which enables 'completion-filter-completions' to perform
> >> the efficient deferred highlighting.
> > 
> > I understand, but given that we provide this for other packages, it
> > shouldn't be an internal variable.
> 
> No, we explicitly don't provide this variable for other packages. It is
> explicitly only meant to be used for the existing completion styles
> emacs21, emacs22, basic, substring, partial-completion, initials and
> flex to opt-in in a backward-compatible/calling convention preserving
> way to the alist return format. The idea is to keep the existing APIs
> fully backward compatible.
> 
> Other packages should select the format returned from the completion
> styles differently. They should return the alist format on Emacs 28 or
> if the API 'completion-filter-completions' API is present. In the not so
> near future external packages which support only Emacs 28 and upwards
> will then only return the alist format and don't have to perform any
> detection anymore.

What if some package outside minibuffer.el will want to control the
format of the returned value, for some reason, like support for old
Emacs versions? are we going to disallow that?

> >> The new API 'completion-filter-completions' will substitute the existing
> >> API 'completion-all-completions'.
> > 
> > That's your hope, and I understand.  But we as a project didn't yet
> > decide to deprecate the original APIs, so talking about superseding is
> > premature.
> 
> It is not the hope - it is the explicit goal. The API has been designed
> to replace the existing API 'completion-all-completions'.

A goal is not a fact.  Until that goal is reached, we cannot in good
faith tell people an API is superseded.

> We can of
> course tone this down. However I, as a package author, would appreciate
> if Emacs tells me when a newer API aims to replace another API and when
> the documentation is explicit about it.

That's understood, and when we make that decision, we will of course
announce it.  But we didn't do so yet, and this discussion is not even
about that decision.  It could be, for example, that both APIs will
live side by side until we decide whether to deprecate the old one.

> Of course if you decide to have
> the doc strings written in a different tone, I will adapt my patch
> accordingly. Here I am just explaining why I chose the word "superseded"
> instead of a more neutral word.

I understand your motivation, I'm just saying that we cannot announce
deprecation before we actually decide to deprecate.

> > But the name says "filter completions".  Which would mean you take a
> > list of completions and filter out some of them.  A completion table
> > is much more general object than a list of strings.  Thus, I think
> > using "filter" here is sub-optimal.
> 
> Okay, you are right about this. But I think even if the name
> 'completion-filter-completions' is not 100% precise it still conveys
> what the API is about. 'completion-completions-alist' or
> 'completion-all-completions-as-alist' are valid names of course, but I
> dislike them for their verbosity. Already 'completion-all-completions'
> is quite verbose. A strong argument to use this long name is that the
> completion style functions are still called
> 'completion-basic-all-completions' etc. But if we accept that the new
> API 'completion-filter-completions' will actually supersede the existing
> API 'completion-all-completions' it makes sense to use a name which will
> not hurt our eyes in the long run. However this is of course just a
> personal preference of mine. I don't want to spent much time with name
> bikeshedding discussions. If you decide on a name, I will adapt my patch
> accordingly.

Emacs is frequently accused in having names that are hard to
discover.  The only time where we can improve that is when a symbol is
introduced, because later it's impossible for compatibility reasons.
So I'd like to come up with a good name before we install the changes.

That said, I'll let others chime in and agree or disagree with the
name you've chosen.

Thanks.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> Cc: joaotavora@gmail.com, 48841@debbugs.gnu.org, dgutov@yandex.ru,
>  larsi@gnus.org, monnier@iro.umontreal.ca, 47711@debbugs.gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Mon, 16 Aug 2021 14:49:45 +0200
> 
> On 8/16/21 2:39 PM, Eli Zaretskii wrote:
> >> João, I am giving hard examples here. What is not an example about
> >> "memory leak" or making debugging output verbose thanks to the attached
> >> string properties?
> > 
> > FWIW, I also don't understand how adding properties could cause a
> > memory leak.  When a string is GCed, its properties get GCed as well,
> > all of them.  Am I missing something?
> 
> If you add string properties to all symbol names this memory will stay
> alive for much longer than necessary.

That's a very extreme example, something that I wouldn't expect a Lisp
program to do, without removing the properties shortly thereafter.
And even that isn't a leak.

Note that we already put all kind of properties (although not text
properties) on symbols.

> > As to more difficult debugging, I think adding a couple of properties
> > that have simple structure will not impair debugging too much.
> > Strings with many properties are not uncommon in Emacs, so we already
> > have to deal with that.
> 
> I disagree with that. We are talking about adding string properties to
> every symbol name. This is a global side effect and different from
> adding string properties to a set of isolated string in a controlled
> manner. I also don't understand why one would even want to take any
> chances here given that the feature can be implemented in a way which
> avoids this global side effect entirely as my patch shows.

I understand your aversion from such global effects, but I was talking
specifically about the debugging difficulties.

> > I would indeed suggest both to make sure there's no performance
> > regressions, and would like to see some data similar to what João
> > presented, which backs up your assessments about your proposal being
> > faster.  Since performance is the main motivation for these changes, I
> > think it's important for us to be on the same page wrt facts related
> > to performance, before we make the decision how to proceed.
> 
> I will prepare some benchmarks.

Thank you.

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 14:46, Eli Zaretskii wrote:

>>> Text properties are stored separately from the string, so I don't
>>> think adding properties can in general be referred to as "change".
>>
>> Are you thinking of C strings?
> 
> No, about the implementation of Lisp strings in Emacs.

I was talking about their behavior.

>> Lisp strings carry text properties in addition to the array of
>> characters. It doesn't really matter where in the memory the properties
>> and the characters reside.
> 
> Well, it does, at least in some situations.  The string text is not
> affected, and so the code which processes the string will not notice
> that it has a property about which that code has no idea.  Only
> properties that are known to the processing code can affect it;
> non-standard properties private to some other code will generally pass
> unnoticed.

I don't think anybody was suggesting that changing text properties 
changes the character codes inside the "C string" part of the Lisp string.

>>> I'm not sure in the context of completion there's any reason to count
>>> as "change" adding properties that don't affect display.
>>
>> For the context in question, whether the properties affect display is
>> not particularly important. Properties affecting display just make it
>> easier to notice that something's wrong. Bug involving other properties
>> should be more difficult to investigate.
> 
> Once again, if some code invents its private property, not used
> anywhere else and not documented anywhere else, then putting such a
> property on a string has very high chances of going unnoticed.  I hope
> this consideration helps this discussion, because saying that
> properties change a string blurs the distinction between actually
> changing the string text or its properties that affect many parts in
> Emacs, and adding some obscure property that is not known to anyone.

What muddies the water is arguing against a solid engineering principle 
with statements like "those mutations are not mutations".

Yes, when the properties are prefixed, the damage is reduced. Even then, 
that increases the possibility of introducing bugs in the very code that 
sets those properties (like having different code paths where one branch 
sets them and another does not; forgetting to clear them in the other 
branch; having subsequent code use the property values set by some 
previous invocation of the code in question where it took another 
branch; not to mention the potential troubles with parallel execution, 
which is not a real concern these days, but we're designing for years 
ahead, and someday it can be). Memory leaks, too.

Our completion pipeline has multiple interchangeable/pluggable parts, so 
we have to be on the lookout even for problems which do not reproduce 
with stock Emacs, and that requires solid abstractions.

And speaking of "only private properties", the completion-score property 
can be used by downstream code, with all the associated potential for 
trouble.

bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 2:38 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> And speaking of "only private properties", the completion-score property
> can be used by downstream code, with all the associated potential for
> trouble.

That's true.  When I created it, I meant for it to be private, I think,
but indeed did forget to mark it as such.  It is not documented anywhere
but that hasn't stopped anyone in the past it, indeed.Can you point to
place(s) where it is indeed used other than the flex machinery of
`minibuffer.el`?  Thanks.

João Távora

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 16:21, Eli Zaretskii wrote:

>>> FWIW, I also don't understand how adding properties could cause a
>>> memory leak.  When a string is GCed, its properties get GCed as well,
>>> all of them.  Am I missing something?
>>
>> If you add string properties to all symbol names this memory will stay
>> alive for much longer than necessary.
> 
> That's a very extreme example, something that I wouldn't expect a Lisp
> program to do, without removing the properties shortly thereafter.

And that *will* happen with Joao's approach, as soon as some package 
implements a Lisp completion backend in a certain (legal) fashion.

Or using one of a few different fashions, actually.

> And even that isn't a leak.
> 
> Note that we already put all kind of properties (although not text
> properties) on symbols.

Those do not, generally, change over time.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 16:41, João Távora wrote:
> It is not documented anywhere
> but that hasn't stopped anyone in the past it, indeed.Can you point to
> place(s) where it is indeed used other than the flex machinery of
> `minibuffer.el`?  Thanks.

Try either of these:

https://github.com/rustify-emacs/fuz.el/blob/master/helm-fuz.el#L228
https://github.com/emacs-helm/helm/blob/master/helm-utils.el

And I'm considering using it in company-sort-by-occurrence, to make sure 
that flex sorting is at least semi-honored there (or create a variation 
of that transformer). For that to happen, the possible score values and 
their meanings will need to be documented, though.

The main scenario (and source of the completion-score property) I have 
in mind is not related to fido-mode or flex, but the users can always 
put flex into completion-styles by default, which affects company-capf.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 16.08.2021 16:21, Eli Zaretskii wrote:
>
>>>> FWIW, I also don't understand how adding properties could cause a
>>>> memory leak.  When a string is GCed, its properties get GCed as well,
>>>> all of them.  Am I missing something?
>>>
>>> If you add string properties to all symbol names this memory will stay
>>> alive for much longer than necessary.
>> That's a very extreme example, something that I wouldn't expect a
>> Lisp
>> program to do, without removing the properties shortly thereafter.
>
> And that *will* happen with Joao's approach, as soon as some package
> implements a Lisp completion backend in a certain (legal) fashion.

There is no leak, not in the strong or weak sense. There is a constant
usage memory footprint proportional to the size of obarray, yes, but the
factor isn't huge.  From the top of my head, I think it's about two
conses and a fp number for each sym. does anyone know how much that is
or can teach me how to measure?  Thanks.

Anyway the current situation is constant copies of strings that put
pressure on the GC, as the benchmarks show.

Anyhoo, in the interest of placating this string property bogeyman, I've
prepared a version of my patch that is based on weak-keyed hash tables.
Slightly slower, but not much. Here are the usual benchmarks:

   (defmacro heyhey ()
     `(progn
        ,@(cl-loop repeat 300000
                  collect `(defun ,(intern (symbol-name (gensym "heyhey"))) () 42))))
   (setq completion-styles '(flex))
   (heyhey)
   (when nil
     ;; Press C-u C-x C-e C-m quickly to produce a sample
     (benchmark-run (completing-read "" obarray))
    
     ;; my patch with private string properties
     (3.596972924 4 1.125298095999998)
     (3.584963294 4 1.1266740010000014)
     (3.4622216089999998 4 1.0924070069999985)
     (3.565632813 4 1.1066678320000012)
     (3.456130054 4 1.099950519)
     (3.49538751 4 1.1085273779999998)
     (3.4882531059999997 4 1.0928655200000001)
     (3.526581152 4 1.0909935229999999)
     (3.710919876 4 1.13883417)
     (3.576690379 4 1.09514685)
    
     ;; my patch with an no string properties (global weak hts)
     ;; Probably the extra gc sweeps are paranoid cleaning up of the
     ;; hash tables.
     (3.981110008 7 1.6466288340000013)
     (3.819598429 7 1.5200578379999996)
     (3.823931386 7 1.5175787589999992)
     (3.9161236720000003 7 1.6184865899999998)
     (3.835148066 7 1.5686207249999988)
     (3.791906221 7 1.5481051090000015)
     (3.798378812 7 1.5164137029999996)
     (4.049880173 7 1.7670989089999996)
     (3.716469474 6 1.3442434509999996)
     (3.422806969 6 1.3272816180000002)
    
     ;; current master
     (5.405534502 12 2.8778620629999994)
     (5.038353216999999 12 2.553688440000002)
     (4.94358915 12 2.4917956500000003)
     (4.950710861 12 2.4638737510000013)
     (5.0242796929999995 12 2.5226992029999984)
     (5.020964648 12 2.495171900999999)
     (4.914088866 12 2.4218276250000024)
     (5.003779622 12 2.502260272000001)
     (4.969347707 12 2.4814790469999988)
     (5.376038238 11 2.565757513000001)
    
     ;; didn't bother with daniel's patch as I've already shown it to be
     ;; about 10% slower than current master.
     
     )

The patch lives in the branch
scratch/icomplete-lazy-highlight-no-string-props.  It's a bit more
complicated to follow, but not much if you understand hash tables.  The
interface to icomplete.el is completely unchanged.

All in all, still a very good improvement over the current situation,
and I think I can make it faster.

(Though really do consider Eli's arguments the fastest approach)

>> And even that isn't a leak.
>> Note that we already put all kind of properties (although not text
>> properties) on symbols.
>
> Those do not, generally, change over time.

Neither does this one! At least in size, which is the thing that
matters.  So in terms of "negative" consequences it's exactly
equivalent. Read the patch it will be obvious, I think.

João

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 14:37, João Távora wrote:
> I am not a native English speaker, and maybe you don't understand
> my language.  Another way to explain what I am talking  is to talk about
> "bug reproduction".  You say there's a bug in my patch, I am asking you
> for a "bug reproduction recipe" as defined by most,  if not all, the results
> you get by searching "bug reproduction recipe" in the Google search engine.

I hope you, or at least other here, can someday see and understand that 
asking to prove standard engineering practices from the first 
principles, time and time again in various discussions, is not a way to 
encourage good atmosphere or promote project participation.

Are you really not imagine a buggy scenario coming from a combination of 
downstream uses of 'completion-score' property, different completion 
styles (some setting it, and some not), and a completion table that 
either uses global string values outright, or caches them for the 
duration of the current command?

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 3:26 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 16.08.2021 14:37, João Távora wrote:
> > I am not a native English speaker, and maybe you don't understand
> > my language.  Another way to explain what I am talking  is to talk about
> > "bug reproduction".  You say there's a bug in my patch, I am asking you
> > for a "bug reproduction recipe" as defined by most,  if not all, the results
> > you get by searching "bug reproduction recipe" in the Google search engine.
>
> I hope you, or at least other here, can someday see and understand that
> asking to prove standard engineering practices from the first
> principles, time and time again in various discussions, is not a way to
> encourage good atmosphere or promote project participation.
>
> Are you really not imagine a buggy scenario coming from a combination of
> downstream uses of 'completion-score' property, different completion
> styles (some setting it, and some not), and a completion table that
> either uses global string values outright, or caches them for the
> duration of the current command?

I don't. Please prime my imagination with some illustration based on
your fertile imagination of these things and the patches I have provided.
Oh and spare me the lectures.  Thanks.

João

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 17:20, João Távora wrote:

>>>>> FWIW, I also don't understand how adding properties could cause a
>>>>> memory leak.  When a string is GCed, its properties get GCed as well,
>>>>> all of them.  Am I missing something?
>>>>
>>>> If you add string properties to all symbol names this memory will stay
>>>> alive for much longer than necessary.
>>> That's a very extreme example, something that I wouldn't expect a
>>> Lisp
>>> program to do, without removing the properties shortly thereafter.
>>
>> And that *will* happen with Joao's approach, as soon as some package
>> implements a Lisp completion backend in a certain (legal) fashion.
> 
> There is no leak, not in the strong or weak sense.

Eli already said that, in a sentence that I also quoted. And still: "I 
wouldn't expect a Lisp program to do <so>".

> There is a constant
> usage memory footprint proportional to the size of obarray, yes, but the
> factor isn't huge.  From the top of my head, I think it's about two
> conses and a fp number for each sym. does anyone know how much that is
> or can teach me how to measure?  Thanks.

If we say that your approach is legal, those are only "two conses and a 
number" coming from minibuffer.el. But since other packages will also be 
allowed to do that, the factor will only be limited by the amount of 
installed packages.

> Anyway the current situation is constant copies of strings that put
> pressure on the GC, as the benchmarks show.
> 
> Anyhoo, in the interest of placating this string property bogeyman, I've
> prepared a version of my patch that is based on weak-keyed hash tables.
> Slightly slower, but not much. Here are the usual benchmarks:

Cool, I'll take a look, thanks.
>>> And even that isn't a leak.
>>> Note that we already put all kind of properties (although not text
>>> properties) on symbols.
>>
>> Those do not, generally, change over time.
> 
> Neither does this one! At least in size, which is the thing that
> matters.  So in terms of "negative" consequences it's exactly
> equivalent. Read the patch it will be obvious, I think.

I was talking about the values of the properties, not the size in memory.

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 3:33 PM Dmitry Gutov <dgutov@yandex.ru> wrote:

> I was talking about the values of the properties, not the size in memory.

Then what's the problem if the value of a property that is an implementation
detail changes? What do you (meaning the user of Emacs) care, ultimately?

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 17:36, João Távora wrote:
> On Mon, Aug 16, 2021 at 3:33 PM Dmitry Gutov<dgutov@yandex.ru>  wrote:
> 
>> I was talking about the values of the properties, not the size in memory.
> Then what's the problem if the value of a property that is an implementation
> detail changes? What do you (meaning the user of Emacs) care, ultimately?

You said "we already have global symbol properties". I pointed out the 
differences.

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Aug 16, 2021 at 3:47 PM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
> On 16.08.2021 17:36, João Távora wrote:
> > On Mon, Aug 16, 2021 at 3:33 PM Dmitry Gutov<dgutov@yandex.ru>  wrote:
> >
> >> I was talking about the values of the properties, not the size in memory.
> > Then what's the problem if the value of a property that is an implementation
> > detail changes? What do you (meaning the user of Emacs) care, ultimately?
>
> You said "we already have global symbol properties". I pointed out the
> differences.

Eli said that, I think. Anyway,  it doesn't present any kind of problem
whether their values of change or not, as long as the space they
occupy doesn't.

João

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> prepared a version of my patch that is based on weak-keyed hash tables.
>> Slightly slower, but not much. Here are the usual benchmarks:
>
> Cool, I'll take a look, thanks.

I've made it faster, now very close to the string-propertizing approach,
itself very close to the theoretical best (which is no copy, no
highlight).  See the tip of the
scratch/icomplete-lazy-highlight-no-string-props branch, which I had to
rewrite (some Git flub-up).  All benchmarks after sig.

João

(require 'cl-lib)

;; Introduce 300 000 new symbols to slow things down.  Probably more
;; than most non-Spacemancs people will have :-)

;; (setq ido-enable-flex-matching t)
;; (ido-mode)
;; (ignore-errors (ido-ubiquitous-mode))
;; (fido-mode)
;; (fido-vertical-mode)
;; (vertico-mode)

;; (hash-table-keys completion--get-lazy-highlight-cache)

(defmacro heyhey ()
  `(progn
     ,@(cl-loop repeat 300000
	       collect `(defun ,(intern (symbol-name (gensym "heyhey"))) () 42))))
;; (setq completion-styles '(substring))
(setq completion-styles '(flex))
(heyhey)
(setq icomplete-show-matches-on-no-input t)

(symbol-name 'mouse-kill)

(when nil
  ;; Press C-u C-x C-e C-m quickly to produce a sample. Always discard
  ;; the first sample in the session, it tends to have an extra GC and be
  ;; slower.
  (benchmark-run (completing-read "" obarray))

  ;; don't use string props
  (2.848873438 6 0.8307729419999994)
  (2.848416202 6 0.8370667889999996)
  (2.786944063 6 0.8230433460000004)
  (2.7815761840000004 6 0.819654023)
  (2.6929080819999998 5 0.7036257240000001)

  ;; string props
  (2.630354852 4 0.7071441910000011)
  (2.594761891 4 0.7082679669999994)
  (2.589480755 4 0.7112978109999997)
  (2.661196709 4 0.7130021060000011)
  (2.844372962 4 0.7378870879999999)

  ;; master
  (3.6339847759999997 12 1.601142523)
  (3.757091055 12 1.6231055449999996)
  (3.785980977 12 1.6333413839999995)
  (3.716144927 12 1.6100998260000008)
  (3.808275042 11 1.611891043)


  ;; these next two are not comparable to the above, because in
  ;; ab23fa4eb22f6557414724769958a63f1c59b49a I added sorting to flex
  ;; which changes results, and Daniel's patch no longer applies
  ;; cleanly.

  ;; daniel's patch
  (3.420418068 10 1.451012855)
  (3.603226896 10 1.672325507)
  (3.501318685 10 1.6150095739999992)
  (3.659821971 10 1.6580361760000004)
  (3.624743674 10 1.657498823)


  ;; master just before daniel's patch (254dc6ab4c)
  (2.611424665 10 1.5267066549999981)
  (2.48811409 10 1.486639387000002)
  (2.472587389 10 1.479865191)
  (2.543277273 10 1.510667634999999)
  (2.546243312 10 1.4986345790000009)
  
  )

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 16.08.2021 21:25, João Távora wrote:
> I've made it faster, now very close to the string-propertizing approach,
> itself very close to the theoretical best (which is no copy, no
> highlight).  See the tip of the
> scratch/icomplete-lazy-highlight-no-string-props branch, which I had to
> rewrite (some Git flub-up).  All benchmarks after sig.

Thanks. I've read it now.

This implementation style (quick exfiltration via a dynamic var with 
some special-cased logic) reminds me of the recent changes to eldoc, 
really not my cup of tea.

At the very least, though, you have done the work of proving that the 
no-string-propertization approach can be just as fast. Thank you for that.

A hash table with :test 'eq is a good choice. I'd be happy to try to 
tweak it further, but it also seems that at this point we can transition 
to the discussion about what kind of implementation style we want, since 
the performance is proven to be more or less on par.

Though of course that should start with an alternative patch which adds 
icomplete support as well (either Daniel does it, or I'll give it a try).

bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 16.08.2021 21:25, João Távora wrote:
>> I've made it faster, now very close to the string-propertizing approach,
>> itself very close to the theoretical best (which is no copy, no
>> highlight).  See the tip of the
>> scratch/icomplete-lazy-highlight-no-string-props branch, which I had to
>> rewrite (some Git flub-up).  All benchmarks after sig.
>
> Thanks. I've read it now.
>
> This implementation style (quick exfiltration via a dynamic var with
> some special-cased logic) reminds me of the recent changes to eldoc,
> really not my cup of tea.

I'm sorry, but I'm not drinking from your herbarium.  Googling for
"exfiltration" brings up "malware" and data security.  Why is mine
"quick" at that?  Is this some kind of metaphor?  And what does "some
special-cased logic" refer to exactly?  I see as much similarity to
Eldoc and I do to the Sistine chapel.

The only thing I understand, I think, is "dynamic var".  If you mean the
variable 'completion-lazy-hilit', notice it is not necessarily used as
dynamic var (in fact in icomplete.el it's just a buffer-local var).  As
I explained elsewhere, if the completion machinery had a realiable
abstraction for "session" I would use that.  

I don't think it does, does it?  Currently, it's the frontend who holds
that knowledge.  It will either have an object representing it (maybe a
fancy CLOS thing); or a stack frame with some kind of command loop; or, in
the case of icomplete, a minibuffer session delineated by
kill-all-local-variables.

So, for icomplete.el, setting that variable buffer-locally is the
appropriate thing.  For the command-loopy frontend, dynamically binding
it will be.  The the objecty frontend, the object itself it proabably a
good value for complation-lazy.hilit.

For completion-capf, if you cared to optimize it with this stuff, it
will likely be ... something something.

Anyway, the "implementation style" I went for is speed, brevity and a
decent docstring.

And it'd be a bit shorter if it used string properties...

> At the very least, though, you have done the work of proving that the
> no-string-propertization approach can be just as fast. Thank you for
> that.

You're welcome.  Not really just as fast, but in the big-O ballpark, of
course.

I had hoped to show also that the particular choice of global structure
for string/symbol/whatever association is irrelevant.

I'm still missing the imminent catastrophe (that is so clear to you) of
the put-text-property approach.  I'd like these slower and more complex
techniques to appease more than superstition.

> A hash table with :test 'eq is a good choice. I'd be happy to try to
> tweak it further, but it also seems that at this point we can
> transition to the discussion about what kind of implementation style
> we want, since the performance is proven to be more or less on par.
>
> Though of course that should start with an alternative patch which
> adds icomplete support as well (either Daniel does it, or I'll give it
> a try).

I'm curious to see those, yes.  But Eli pointed out on, two different
APIs will need to cohabitate since the new API won't kill off the old.

To be very clear, I'm interested in the performance of Daniel's patch,
not really in insufferable claims of its beauty and virginity.

minibuffer.el is a great big mess, I'll leave it to the Great Designers
of the Big Redesign, godspeed to them.

Currently, I just want changes to not assassinate, in speed or form,
icomplete.el or the flex completion style, two fundamental daily drivers
to my work, and other's work.  So if/when Daniel's patch doesn't do any
of that (it seems that it currently does), I'll be all for it.

João

bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 17 Aug 2021 09:59:25 +0100
> Cc: Daniel Mendler <mail@daniel-mendler.de>, larsi@gnus.org,
>  monnier@iro.umontreal.ca, 48841@debbugs.gnu.org, 47711@debbugs.gnu.org
> 
> I'm sorry, but I'm not drinking from your herbarium.

Once again, I'm asking everyone to please remove the emotional and
sarcastic parts from the exchange.  It is not helping to have
constructive discussions.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Tue, Aug 17, 2021 at 12:49 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Tue, 17 Aug 2021 09:59:25 +0100
> > Cc: Daniel Mendler <mail@daniel-mendler.de>, larsi@gnus.org,
> >  monnier@iro.umontreal.ca, 48841@debbugs.gnu.org, 47711@debbugs.gnu.org
> >
> > I'm sorry, but I'm not drinking from your herbarium.
>
> Once again, I'm asking everyone to please remove the emotional and
> sarcastic parts from the exchange.  It is not helping to have
> constructive discussions.

I thought it was rather appropriate for the "my cup of tea" line :-)  But
I get the message, and I apologize.

João

bug#48841: [PATCH] Make fido-mode about as fast as ido-mode even with many completions

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> [I've removed bug#47711 from the list, since I haven't read the bug.
> This is only directly concerned with this report bug#48841 about speed
> differences between fido-mode and ido-mode.]
>
> João Távora <joaotavora@gmail.com> writes:
>
>> scratch/icomplete-lazy-highlight-attempt-2, although still incomplete,
>> is one such approach, though it still sets `completion-score` on the
>> "shared" string, used later for sorting.  But also that could be
>> prevented (again, only if it turns out to be actually problematic
>> IMO).
>
> I have tested the patch more thoroughly now, and have not found any
> problems.

As I wait for genuine reports or explanations of the much dramatized
problems in the above patch, I've pushed a much simpler patch that has a
dramatic beneficial effect: simply don't do any copying, highlighting or
scoring if the pcm-style pattern (used by the styles 'flex', 'substring'
and others) is empty.

This more than halves the waiting time for the candidate display when
the pattern is empty.  As far as i can tell, `fido-mode` is now faster
than `ido-mode` and so I'm marking this bug closed.

Of course, when there is a pattern of a single character or more, the
icomplete waiting times using my earlier 'completion-lazy-highlight'
patch are still around 70% of the current master.  But those times are
always quite shorter than the empty-pattern case.  I'll wait a bit for
the alternatives presumably being worked on before pushing that or
something based on it.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 4181 bytes --]

Hi all!

Time flies, doesn't it?

On 14/08/2021 10:16, Eli Zaretskii wrote:
>>> If one removes these lines, the process becomes much faster, but there is a
>>> problem with highlighting.  My idea is indeed to defer highlighting by not
>>> setting the 'face property directly on that shared string, but some
>>> other property
>>> that is read later from the shared string by compliant frontents.
>>
>> I haven't done any direct benchmarking, but I'm pretty sure that this
>> approach cannot, by definition, be as fast as the non-mutating one.
> 
> Daniel seems to think otherwise, AFAIU.
> 
>> Because you go through the whole list and mutate all of its elements,
>> you have to perform a certain bit of work W x N times, where N is the
>> size of the whole list.
>>
>> Whereas the deferred-mutation approach will only have to do its bit
>> (which is likely more work, like, WWW) only 20 times, or 100 times, or
>> however many completions are displayed. And this is usually negligible.
>>
>> However big the difference is going to be, I can't say in advance, of
>> course, or whether it's going to be shadowed by some other performance
>> costs. But the non-mutating approach should have the best optimization
>> potential when the list is long.
> 
> So I guess the suggestion to have a benchmark is still useful, because
> the estimations of which approach has better performance vary between
> you three.  So maybe producing such benchmarks would be a good step?

To cross this out from my TODO, I spent most of the day rebasing both of 
the proposed patches (one of them longer than the other) -- one from an 
attachment here and another from a commit inside the 
scratch/icomplete-lazy-highlight-attempt-2 branch, porting icomplete to 
Daniel's new completion-filter-completions API (*), and benchmarking.

(*) Included in the attached patch: it needed changing just two lines 
inside icomplete, but also new variable completion-all-sorted-highlight 
and updates to completion--cache-all-sorted-completions and 
completion-all-sorted-completions.

Both rebased patches are attached to this email for your convenience.

AFAICT, the results confirmed my expectations quoted above.

Using Joao's benchmark, with setup:

   (defmacro lotsoffunctions ()
     `(progn
        ,@(cl-loop repeat 150000
                   collect `(defun ,(intern (symbol-name (gensym 
"heyhey"))) () 42))))

   (lotsoffunctions)

I ran the comparisons for empty and non-empty inputs.

With no characters typed:

   (benchmark-run 1
     (let ((completion-styles '(flex))
           (completion-lazy-hilit (cl-gensym)) ; might not be defined
           )
       ;; Uncomment one of the lines below, depending on the patch used.
       ;; (completion-all-completions "" obarray 'fboundp 0 nil)
       ;; (completion-filter-completions "" obarray 'fboundp 0 nil)
       ))

master => 0.066
lazy-hilit => 0.045
filter-and-defer => 0.041 (but more often ~0.110 including GC, somehow)

With one character typed:

   (benchmark-run 1
     (let ((completion-styles '(flex))
           (completion-lazy-hilit (cl-gensym)) ; might not be defined
           )
       ;; Uncomment one of the lines below, depending on the patch used.
       ;; (completion-all-completions "h" obarray 'fboundp 1 nil)
       ;; (completion-filter-completions "h" obarray 'fboundp 1 nil)
       ))

master => 0.824
lazy-hilit => 0.395
filter-and-defer => 0.125 (!)

This more or less translates into the improvement in speed of 
fido-vertical-mode, according to my benchmark-progn call inside 
icomplete-exhibit (included in both attached patches for convenience). 
For non-empty inputs (h or hh or hhe, to match the generated functions), 
filter-and-defer is about 1.5x faster than lazy-hilit, like 0.450ms vs 
0.640ms.

lazy-hilit is slightly faster than filter-and-defer with the empty input 
(like 380ms vs 420ms), and I'm not yet sure why, but it's the scenario 
with 0 highlighting (and so no flex scoring/sorting). Perhaps some 
short-circuiting can be added somewhere to reach parity, or it's the 
cost of extra branching somewhere for backward compatibility (which 
could be removed in the future). Worth additional study.

[-- Attachment #2: 0001-Add-new-completion-filter-completions-API-and-deferr-v3.diff --]
[-- Type: text/x-patch, Size: 54102 bytes --]

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..234113e607c 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -754,12 +754,13 @@ icomplete-exhibit
                            (overlay-end rfn-eshadow-overlay)))
           (let* ((field-string (icomplete--field-string))
                  (text (while-no-input
+                         (benchmark-progn
                          (icomplete-completions
                           field-string
                           (icomplete--completion-table)
                           (icomplete--completion-predicate)
                           (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                              (eq minibuffer--require-match t))))))
                  (buffer-undo-list t)
                  deactivate-mark)
             ;; Do nothing if while-no-input was aborted.
@@ -878,7 +879,7 @@ icomplete--render-vertical
   ;; Halfway there...
   (let* ((selected (propertize (car comps) 'icomplete-selected t))
          (chosen (append scroll-above (list selected) scroll-below))
-         (tuples (icomplete--augment md chosen))
+         (tuples (icomplete--augment md (funcall completion-all-sorted-highlight chosen)))
          max-prefix-len max-comp-len lines nsections)
     (add-face-text-property 0 (length selected)
                             'icomplete-selected-match 'append selected)
@@ -1067,7 +1068,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects (nreverse
+                                 (funcall completion-all-sorted-highlight prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..e8b2849f45a 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -677,6 +677,10 @@ completion--twq-all
                                              'completions-common-part)
                                qprefix))))
                         (qcompletion (concat qprefix qnew)))
+                   ;; Attach unquoted completion string, which is needed
+                   ;; to score the completion in `completion--flex-score'.
+                   (put-text-property 0 1 'completion--unquoted
+                                      completion qcompletion)
 		   ;; FIXME: Similarly here, Cygwin's mapping trips this
 		   ;; assertion.
                    ;;(cl-assert
@@ -1171,6 +1175,17 @@ completion--styles
         (delete-dups (append (cdr over) (copy-sequence completion-styles)))
        completion-styles)))
 
+(defvar completion--return-alist-flag nil
+  "Non-nil means to return completions in alist format.
+If this variable is non-nil the `all-completions' function of a
+completion style should return the results in the alist format of
+`completion-filter-completions'.  This variable is purely needed to
+for backward compatibility of the existing builtin completion style
+functions as of Emacs 28.  Newer completion style functions should
+always return their results in the alist format, since
+`completion-all-completions' transparently converts back to a list of
+completions with base size in the last cdr.")
+
 (defun completion--nth-completion (n string table pred point metadata)
   "Call the Nth method of completion styles."
   ;; We provide special support for quoting/unquoting here because it cannot
@@ -1197,6 +1212,15 @@ completion--nth-completion
                  ;; the original table, in that case!
                  (functionp table))
             (let ((new (funcall table string point 'completion--unquote)))
+              ;; FIXME For now do not attempt deferred highlighting if
+              ;; quoting is used.  Not doing deferred highlighting is
+              ;; not too severe in this case, since
+              ;; `completion--twq-all' is already an expensive
+              ;; function, which allocates all completion strings.  In
+              ;; contrast to plain completion tables, the savings of
+              ;; deferred highlighting would be minimal in the case of
+              ;; quoted completion tables.
+              (setq completion--return-alist-flag nil)
               (setq string (pop new))
               (setq table (pop new))
               (setq point (pop new))
@@ -1205,18 +1229,36 @@ completion--nth-completion
          (result-and-style
           (seq-some
            (lambda (style)
-             (let ((probe (funcall
-                           (or (nth n (assq style completion-styles-alist))
-                               (error "Invalid completion style %s" style))
-                           string table pred point)))
+             (let* ((fun (or (nth n (assq style completion-styles-alist))
+                               (error "Invalid completion style %s" style)))
+                    ;; Transparently upgrade the return value for
+                    ;; existing built-in styles as of Emacs 28.  No
+                    ;; new styles should be added here. New completion
+                    ;; styles should directly return the new
+                    ;; completion format.el
+                    (completion--return-alist-flag
+                     (and completion--return-alist-flag
+                          (memq style '(emacs21 emacs22 basic substring
+                                        partial-completion initials flex))))
+                    (probe (funcall fun string table pred point)))
                (and probe (cons probe style))))
            (completion--styles md)))
-         (adjust-fn (get (cdr result-and-style) 'completion--adjust-metadata)))
-    (when (and adjust-fn metadata)
-      (setcdr metadata (cdr (funcall adjust-fn metadata))))
+         (style-md (get (cdr result-and-style) 'completion--style-metadata))
+         (result (car result-and-style)))
+    (when (and style-md metadata)
+      (setcdr metadata (cdr (funcall style-md
+                                     string table pred point metadata))))
+    (when (and (not completion--return-alist-flag) (= n 2) (consp (car result)))
+      ;; Give the completion styles some freedom!  If they are
+      ;; targeting Emacs 28 upwards only, they may return a result
+      ;; with deferred highlighting.  We convert back to the old
+      ;; format here by applying the highlighting eagerly.
+      (setq result (nconc (funcall (cdr (assq 'highlight result))
+                                   (cdr (assq 'completions result)))
+                          (cdr (assq 'base result)))))
     (if requote
-        (funcall requote (car result-and-style) n)
-      (car result-and-style))))
+        (funcall requote result n)
+      result)))
 
 (defun completion-try-completion (string table pred point &optional metadata)
   "Try to complete STRING using completion table TABLE.
@@ -1225,7 +1267,8 @@ completion-try-completion
 The return value can be either nil to indicate that there is no completion,
 t to indicate that STRING is the only possible completion,
 or a pair (NEWSTRING . NEWPOINT) of the completed result string together with
-a new position for point."
+a new position for point.
+The METADATA may be modified by the completion style."
   (completion--nth-completion 1 string table pred point metadata))
 
 (defun completion-all-completions (string table pred point &optional metadata)
@@ -1233,10 +1276,47 @@ completion-all-completions
 Only the elements of table that satisfy predicate PRED are considered.
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
-in the last `cdr'."
-  ;; FIXME: We need to additionally return the info needed for the
-  ;; second part of completion-base-position.
-  (completion--nth-completion 2 string table pred point metadata))
+in the last `cdr'.
+The METADATA may be modified by the completion style.
+
+This function has been superseded by `completion-filter-completions',
+which returns richer information and supports deferred candidate
+highlighting."
+  (let ((completion--return-alist-flag nil))
+    (completion--nth-completion 2 string table pred point metadata)))
+
+(defun completion-filter-completions (string table pred point metadata)
+  "Filter the possible completions of STRING in completion table TABLE.
+Only the elements of table that satisfy predicate PRED are considered.
+POINT is the position of point within STRING.
+The METADATA may be modified by the completion style.
+The return value is a alist with the keys:
+
+- base: Base position of the completion (from the start of STRING)
+- end: End position of the completion (from the start of STRING)
+- highlight: Highlighting function taking a list of completions and
+  returning a new list of new strings with applied highlighting.
+- completions: The list of completions.
+
+This function supersedes the function `completion-all-completions',
+which does not provide the `end' position of the completion and does
+not support deferred highlighting."
+  (let* ((completion--return-alist-flag t)
+         (result (completion--nth-completion 2 string table
+                                             pred point metadata)))
+    (if (and result (not (consp (car result))))
+        ;; Deferred highlighting has been requested, but the
+        ;; completion style returned a non-deferred result.  Convert
+        ;; the result to the alist format of
+        ;; `completion-filter-completions'.
+        (let* ((last (last result))
+               (base (or (cdr last) 0)))
+          (setcdr last nil)
+          `((base . ,base)
+            (end . ,(length string))
+            (highlight . identity)
+            (completions . ,result)))
+      result)))
 
 (defun minibuffer--bitset (modified completions exact)
   (logior (if modified    4 0)
@@ -1253,7 +1333,8 @@ completion--replace
   (if minibuffer-allow-text-properties
       ;; If we're preserving properties, then just remove the faces
       ;; and other properties added by the completion machinery.
-      (remove-text-properties 0 (length newtext) '(face completion-score)
+      (remove-text-properties 0 (length newtext)
+                              '(face nil completion-score nil)
                               newtext)
     ;; Remove all text properties.
     (set-text-properties 0 (length newtext) nil newtext))
@@ -1356,6 +1437,7 @@ completion--cycle-threshold
 
 (defvar-local completion-all-sorted-completions nil)
 (defvar-local completion--all-sorted-completions-location nil)
+(defvar-local completion-all-sorted-highlight nil)
 (defvar completion-cycling nil)      ;Function that takes down the cycling map.
 (defvar completion-tab-width nil)
 
@@ -1562,12 +1644,15 @@ completion--in-region-1
           ;; When the completion list window was displayed, select it.
           (switch-to-completions))))))
 
-(defun completion--cache-all-sorted-completions (beg end comps)
+(defun completion--cache-all-sorted-completions (beg end comps &optional highlight)
   (add-hook 'after-change-functions
             #'completion--flush-all-sorted-completions nil t)
   (setq completion--all-sorted-completions-location
         (cons (copy-marker beg) (copy-marker end)))
-  (setq completion-all-sorted-completions comps))
+  (setq completion-all-sorted-completions comps)
+  (when highlight
+    (setq completion-all-sorted-highlight highlight))
+  comps)
 
 (defun completion--flush-all-sorted-completions (&optional start end _len)
   (unless (and start end
@@ -1578,7 +1663,8 @@ completion--flush-all-sorted-completions
     ;; Remove the transient map if applicable.
     (when completion-cycling
       (funcall (prog1 completion-cycling (setq completion-cycling nil))))
-    (setq completion-all-sorted-completions nil)))
+    (setq completion-all-sorted-completions nil
+          completion-all-sorted-highlight nil)))
 
 (defun completion--metadata (string base md-at-point table pred)
   ;; Like completion-metadata, but for the specific case of getting the
@@ -1656,14 +1742,15 @@ completion-all-sorted-completions
              (end (or end (point-max)))
              (string (buffer-substring start end))
              (md (completion--field-metadata start))
-             (all (completion-all-completions
-                   string
-                   minibuffer-completion-table
-                   minibuffer-completion-predicate
-                   (- (point) start)
-                   md))
-             (last (last all))
-             (base-size (or (cdr last) 0))
+             (alist (completion-filter-completions
+                     string
+                     minibuffer-completion-table
+                     minibuffer-completion-predicate
+                     (- (point) start)
+                     md))
+             (all (assoc-default 'completions alist))
+             (base-size (assoc-default 'base alist))
+             (highlight (assoc-default 'highlight alist))
              (all-md (completion--metadata (buffer-substring-no-properties
                                             start (point))
                                            base-size md
@@ -1671,14 +1758,11 @@ completion-all-sorted-completions
                                            minibuffer-completion-predicate))
              (sort-fun (completion-metadata-get all-md 'cycle-sort-function))
              (group-fun (completion-metadata-get all-md 'group-function)))
-        (when last
-          (setcdr last nil)
-
+        (when all
           ;; Delete duplicates: do it after setting last's cdr to nil (so
           ;; it's a proper list), and be careful to reset `last' since it
           ;; may be a different cons-cell.
           (setq all (delete-dups all))
-          (setq last (last all))
 
           (cond
            (sort-fun (setq all (funcall sort-fun all)))
@@ -1704,7 +1788,7 @@ completion-all-sorted-completions
           ;; repeated calls to minibuffer-force-complete can cycle through
           ;; all possibilities.
           (completion--cache-all-sorted-completions
-           start end (nconc all base-size))))))
+           start end (nconc all base-size) highlight)))))
 
 (defun minibuffer-force-complete-and-exit ()
   "Complete the minibuffer with first of the matches and exit."
@@ -2238,34 +2322,49 @@ completion-hilit-commonality
 It returns a list with font-lock properties applied to each element,
 and with BASE-SIZE appended as the last element."
   (when completions
-    (let ((com-str-len (- prefix-len (or base-size 0))))
-      (nconc
-       (mapcar
-        (lambda (elem)
-          (let ((str
-                 ;; Don't modify the string itself, but a copy, since the
-                 ;; string may be read-only or used for other purposes.
-                 ;; Furthermore, since `completions' may come from
-                 ;; display-completion-list, `elem' may be a list.
-                 (if (consp elem)
-                     (car (setq elem (cons (copy-sequence (car elem))
-                                           (cdr elem))))
-                   (setq elem (copy-sequence elem)))))
-            (font-lock-prepend-text-property
-             0
-             ;; If completion-boundaries returns incorrect
-             ;; values, all-completions may return strings
-             ;; that don't contain the prefix.
-             (min com-str-len (length str))
-             'face 'completions-common-part str)
-            (if (> (length str) com-str-len)
-                (font-lock-prepend-text-property com-str-len (1+ com-str-len)
-                                                 'face
-                                                 'completions-first-difference
-                                                 str)))
-          elem)
-        completions)
-       base-size))))
+    (nconc
+     (completion--hilit-commonality (- prefix-len (or base-size 0)) completions)
+     base-size)))
+
+(defun completion--hilit-commonality (com-size completions)
+  (mapcar
+   (lambda (elem)
+     (let ((str
+            ;; Don't modify the string itself, but a copy, since the
+            ;; string may be read-only or used for other purposes.
+            ;; Furthermore, since `completions' may come from
+            ;; display-completion-list, `elem' may be a list.
+            (if (consp elem)
+                (car (setq elem (cons (copy-sequence (car elem))
+                                      (cdr elem))))
+              (setq elem (copy-sequence elem)))))
+       (font-lock-prepend-text-property
+        0
+        ;; If completion-boundaries returns incorrect
+        ;; values, all-completions may return strings
+        ;; that don't contain the prefix.
+        (min com-size (length str))
+        'face 'completions-common-part str)
+       (if (> (length str) com-size)
+           (font-lock-prepend-text-property com-size (1+ com-size)
+                                            'face
+                                            'completions-first-difference
+                                            str)))
+     elem)
+   completions))
+
+(defun completion--deferred-hilit (completions prefix-len base end)
+  "Return completions as a list or as an alist.
+If `completion--return-alist-flag' is non-nil use the alist format of
+`completion-filter-completions'."
+  (if completion--return-alist-flag
+      (when completions
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially #'completion--hilit-commonality
+                                         (- prefix-len base)))
+          (completions . ,completions)))
+    (completion-hilit-commonality completions prefix-len base)))
 
 (defun display-completion-list (completions &optional common-substring group-fun)
   "Display the list of completions, COMPLETIONS, using `standard-output'.
@@ -2373,15 +2472,16 @@ minibuffer-completion-help
          (end (or end (point-max)))
          (string (buffer-substring start end))
          (md (completion--field-metadata start))
-         (completions (completion-all-completions
-                       string
-                       minibuffer-completion-table
-                       minibuffer-completion-predicate
-                       (- (point) start)
-                       md)))
+         (filtered-completions (completion-filter-completions
+                                string
+                                minibuffer-completion-table
+                                minibuffer-completion-predicate
+                                (- (point) start)
+                                md))
+         (completions (alist-get 'completions filtered-completions)))
     (message nil)
     (if (or (null completions)
-            (and (not (consp (cdr completions)))
+            (and (not (cdr completions))
                  (equal (car completions) string)))
         (progn
           ;; If there are no completions, or if the current input is already
@@ -2393,8 +2493,7 @@ minibuffer-completion-help
 	      (ding)
 	      (completion--message "No match"))))
 
-      (let* ((last (last completions))
-             (base-size (or (cdr last) 0))
+      (let* ((base-size (alist-get 'base filtered-completions))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
              (base-prefix (buffer-substring (minibuffer--completion-prompt-end)
                                             (+ start base-size)))
@@ -2442,9 +2541,12 @@ minibuffer-completion-help
             (body-function
              . ,#'(lambda (_window)
                     (with-current-buffer mainbuf
-                      ;; Remove the base-size tail because `sort' requires a properly
-                      ;; nil-terminated list.
-                      (when last (setcdr last nil))
+                      ;; Apply highlighting using the deferred
+                      ;; highlighting function provided by
+                      ;; `completion-format-completions'.
+                      (setq completions
+                            (funcall (alist-get 'highlight filtered-completions)
+                                     completions))
 
                       ;; Sort first using the `display-sort-function'.
                       ;; FIXME: This function is for the output of
@@ -2486,13 +2588,10 @@ minibuffer-completion-help
                                       completions))))
 
                       (with-current-buffer standard-output
-                        (setq-local completion-base-position
-                             (list (+ start base-size)
-                                   ;; FIXME: We should pay attention to completion
-                                   ;; boundaries here, but currently
-                                   ;; completion-all-completions does not give us the
-                                   ;; necessary information.
-                                   end))
+                        (setq-local
+                         completion-base-position
+                         (list (+ start base-size)
+                               (+ start (alist-get 'end filtered-completions))))
                         (setq-local completion-base-affixes
                                     (list base-prefix base-suffix))
                         (setq-local completion-list-insert-choice-function
@@ -3468,10 +3567,11 @@ completion-emacs21-try-completion
       completion)))
 
 (defun completion-emacs21-all-completions (string table pred _point)
-  (completion-hilit-commonality
+  (completion--deferred-hilit
    (all-completions string table pred)
    (length string)
-   (car (completion-boundaries string table pred ""))))
+   (car (completion-boundaries string table pred ""))
+   (length string)))
 
 (defun completion-emacs22-try-completion (string table pred point)
   (let ((suffix (substring string point))
@@ -3494,11 +3594,12 @@ completion-emacs22-try-completion
       (cons (concat completion suffix) (length completion)))))
 
 (defun completion-emacs22-all-completions (string table pred point)
-  (let ((beforepoint (substring string 0 point)))
-    (completion-hilit-commonality
+  (let* ((beforepoint (substring string 0 point))
+         (afterpoint (substring string point))
+         (bounds (completion-boundaries beforepoint table pred afterpoint)))
+    (completion--deferred-hilit
      (all-completions beforepoint table pred)
-     point
-     (car (completion-boundaries beforepoint table pred "")))))
+     point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Basic completion.
 
@@ -3557,7 +3658,7 @@ completion-basic-all-completions
                             'point
                             (substring afterpoint 0 (cdr bounds)))))
          (all (completion-pcm--all-completions prefix pattern table pred)))
-    (completion-hilit-commonality all point (car bounds))))
+    (completion--deferred-hilit all point (car bounds) (+ point (cdr bounds)))))
 
 ;;; Partial-completion-mode style completion.
 
@@ -3749,14 +3850,27 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
-(defun completion-pcm--hilit-commonality (pattern completions)
+(defun completion-pcm--deferred-hilit (pattern completions base end)
+  "Return completions as a list or as an alist.
+If `completion--return-alist-flag' is non-nil use the alist format of
+`completion-filter-completions'."
+  (when completions
+    (if completion--return-alist-flag
+        `((base . ,base)
+          (end . ,end)
+          (highlight . ,(apply-partially
+                         #'completion-pcm--hilit-commonality
+                         pattern))
+          (completions . ,completions))
+      (nconc (completion-pcm--hilit-commonality pattern completions 'score) base))))
+
+(defun completion-pcm--hilit-commonality (pattern completions &optional score)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
 string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
+each string is propertized with faces `completions-common-part',
 `completions-first-difference' in the relevant segments."
   (cond
    ((and completions (cl-loop for e in pattern thereis (stringp e)))
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
@@ -3773,86 +3888,145 @@ completion-pcm--hilit-commonality
                 (match-end (match-end 0))
                 (md (cddr (setq last-md (match-data t last-md))))
                 (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by (0..1] (i.e., larger than (but not equal
-                ;; to) zero, and smaller or equal to one): the higher
-                ;; the better and only a perfect match (pattern equals
-                ;; string) will have score 1.  The formula takes the
-                ;; form of a quotient.  For the numerator, we use the
-                ;; number of +, i.e. the length of the pattern.  For
-                ;; the denominator, it first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
+                (len (length str)))
+           (when (and score (/= 0 len))
+             (put-text-property
+              0 1 'completion-score (- (completion--flex-score-1 md match-end len)) str))
            (while md
-             (funcall update-score-and-face from (pop md))
+             (add-face-text-property from (pop md)
+                                     'completions-common-part
+                                     nil str)
              (setq from (pop md)))
            ;; If `pattern' doesn't have an explicit trailing any, the
            ;; regex `re' won't produce match data representing the
            ;; region after the match.  We need to account to account
            ;; for that extra bit of match (bug#42149).
            (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
+             (add-face-text-property from match-end
+                                     'completions-common-part
+                                     nil str))
+           (if (> len pos)
                (add-face-text-property
                 pos (1+ pos)
                 'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
+                nil str)))
          str)
        completions)))
    (t completions)))
 
+(defun completion--flex-score-1 (md match-end len)
+  "Compute matching score of completion.
+The score lies in the range between-1 and 0, where -1 corresponds to
+the full match.
+MD is the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by (0..1] (i.e., larger than (but not equal
+         ;; to) zero, and smaller or equal to one): the higher
+         ;; the better and only a perfect match (pattern equals
+         ;; string) will have score 1.  The formula takes the
+         ;; form of a quotient.  For the numerator, we use the
+         ;; number of +, i.e. the length of the pattern.  For
+         ;; the denominator, it first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while md
+      (let ((a from)
+            (b (pop md)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (- (/ score-numerator (* len (1+ score-denominator)) 1.0))))
+
+(defun completion--flex-score (pattern completions)
+  "Compute how well PATTERN matches COMPLETIONS.
+PATTERN, a pcm pattern is assumed to match every string in the
+COMPLETIONS list.  Return a copy of COMPLETIONS where each element is
+a pair of a score and the string.  The score lies in the range between
+-1 and 0, where -1 corresponds to the full match."
+  (when completions
+    (let* ((re (completion-pcm--pattern->regex pattern 'group))
+           (case-fold-search completion-ignore-case)
+           last-md)
+      (mapcar
+       (lambda (str)
+         ;; The flex completion style requires the completion to match
+         ;; the pattern to compute the scoring.  For quoted completion
+         ;; tables the completions are matched against the *unquoted
+         ;; input string*.  However `completion-all-completions' and
+         ;; `completion-filter-completions' return a list of *quoted
+         ;; completions*, which is subsequently sorted.  Therefore we
+         ;; obtain the unquoted completion string which is stored in
+         ;; the text property `completion--unquoted'.
+         (let ((unquoted (or (get-text-property 0 'completion--unquoted str) str)))
+           (unless (string-match re unquoted)
+             (error "Internal error: %s does not match %s" re unquoted))
+           (cons (completion--flex-score-1 (cddr (setq last-md (match-data t last-md)))
+                                           (match-end 0) (length unquoted))
+                 str)))
+       completions))))
+
 (defun completion-pcm--find-all-completions (string table pred point
                                                     &optional filter)
   "Find all completions for STRING at POINT in TABLE, satisfying PRED.
@@ -3948,11 +4122,11 @@ completion-pcm--find-all-completions
         (list pattern all prefix suffix)))))
 
 (defun completion-pcm-all-completions (string table pred point)
-  (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+  (pcase-let ((`(,pattern ,all ,prefix ,suffix)
                (completion-pcm--find-all-completions string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 (defun completion--common-suffix (strs)
   "Return the common suffix of the strings STRS."
@@ -4136,8 +4310,8 @@ completion-pcm-try-completion
 ;;; Substring completion
 ;; Mostly derived from the code of `basic' completion.
 
-(defun completion-substring--all-completions
-    (string table pred point &optional transform-pattern-fn)
+(defun completion--pattern-compiler
+    (string table pred point transform-pattern-fn)
   "Match the presumed substring STRING to the entries in TABLE.
 Respect PRED and POINT.  The pattern used is a PCM-style
 substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
@@ -4155,12 +4329,23 @@ completion-substring--all-completions
          (pattern (completion-pcm--optimize-pattern
                    (if transform-pattern-fn
                        (funcall transform-pattern-fn pattern)
-                     pattern)))
-         (all (completion-pcm--all-completions prefix pattern table pred)))
-    (list all pattern prefix suffix (car bounds))))
+                     pattern))))
+    (list pattern prefix suffix)))
+
+(defun completion-substring--all-completions
+    (string table pred point &optional transform-pattern-fn)
+  "Match the presumed substring STRING to the entries in TABLE.
+Respect PRED and POINT.  The pattern used is a PCM-style
+substring pattern, but it be massaged by TRANSFORM-PATTERN-FN, if
+that is non-nil."
+  (pcase-let (((and result `(,pattern ,prefix ,_suffix))
+               (completion--pattern-compiler string table pred point
+                                             transform-pattern-fn)))
+    (cons (completion-pcm--all-completions prefix pattern table pred)
+          result)))
 
 (defun completion-substring-try-completion (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
     (if minibuffer-completing-file-name
@@ -4168,12 +4353,12 @@ completion-substring-try-completion
     (completion-pcm--merge-try pattern all prefix suffix)))
 
 (defun completion-substring-all-completions (string table pred point)
-  (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+  (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                (completion-substring--all-completions
                 string table pred point)))
-    (when all
-      (nconc (completion-pcm--hilit-commonality pattern all)
-             (length prefix)))))
+    (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix)))))
 
 ;;; "flex" completion, also known as flx/fuzzy/scatter completion
 ;; Completes "foo" to "frodo" and "farfromsober"
@@ -4183,19 +4368,11 @@ completion-flex-nospace
   :version "27.1"
   :type 'boolean)
 
-(put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata)
+(put 'flex 'completion--style-metadata 'completion--flex-style-metadata)
 
-(defun completion--flex-adjust-metadata (metadata)
+(defun completion--flex-style-metadata (string table pred point metadata)
   "If `flex' is actually doing filtering, adjust sorting."
-  (let ((flex-is-filtering-p
-         ;; JT@2019-12-23: FIXME: this is kinda wrong.  What we need
-         ;; to test here is "some input that actually leads/led to
-         ;; flex filtering", not "something after the minibuffer
-         ;; prompt".  E.g. The latter is always true for file
-         ;; searches, meaning we'll be doing extra work when we
-         ;; needn't.
-         (or (not (window-minibuffer-p))
-             (> (point-max) (minibuffer-prompt-end))))
+  (let ((flex-is-filtering-p (not (equal string "")))
         (existing-dsf
          (completion-metadata-get metadata 'display-sort-function))
         (existing-csf
@@ -4204,12 +4381,32 @@ completion--flex-adjust-metadata
         ((compose-flex-sort-fn
           (existing-sort-fn) ; wish `cl-flet' had proper indentation...
           (lambda (completions)
-            (sort
-             (funcall existing-sort-fn completions)
-             (lambda (c1 c2)
-               (let ((s1 (get-text-property 0 'completion-score c1))
-                     (s2 (get-text-property 0 'completion-score c2)))
-                 (> (or s1 0) (or s2 0))))))))
+            (let ((pre-sorted (funcall existing-sort-fn completions))
+                  (pattern (car (completion--pattern-compiler
+                                 string table pred point
+                                 #'completion-flex--make-flex-pattern))))
+              ;; If `completion-scores' are already present use
+              ;; those instead of recomputing the scores with
+              ;; `completion--flex-score'.  The scores are already
+              ;; present, when the candidates have been computed by
+              ;; `completion-all-completions'.  In contrast, the
+              ;; score is not yet present, when the candidates have
+              ;; been computed by `completion-filter-completions'.
+              (if (and (car pre-sorted)
+                       (get-text-property 0 'completion-score (car pre-sorted)))
+                  (sort
+                   pre-sorted
+                   (lambda (c1 c2)
+                     (> (or (get-text-property 0 'completion-score c1) 0)
+                        (or (get-text-property 0 'completion-score c2) 0))))
+                (let* ((sorted (sort (completion--flex-score pattern pre-sorted)
+                                     #'car-less-than-car))
+                       (cell sorted))
+                  ;; Remove score decorations, reuse the list to avoid allocations.
+                  (while cell
+                    (setcar cell (cdar cell))
+                    (pop cell))
+                  sorted))))))
       `(metadata
         ,@(and flex-is-filtering-p
                `((display-sort-function
@@ -4239,7 +4436,7 @@ completion-flex--make-flex-pattern
 (defun completion-flex-try-completion (string table pred point)
   "Try to flex-complete STRING in TABLE given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
@@ -4256,13 +4453,13 @@ completion-flex-try-completion
 (defun completion-flex-all-completions (string table pred point)
   "Get flex-completions of STRING in TABLE, given PRED and POINT."
   (unless (and completion-flex-nospace (string-search " " string))
-    (pcase-let ((`(,all ,pattern ,prefix ,_suffix ,_carbounds)
+    (pcase-let ((`(,all ,pattern ,prefix ,suffix)
                  (completion-substring--all-completions
                   string table pred point
                   #'completion-flex--make-flex-pattern)))
-      (when all
-        (nconc (completion-pcm--hilit-commonality pattern all)
-               (length prefix))))))
+      (completion-pcm--deferred-hilit pattern all
+                                    (length prefix)
+                                    (- (length string) (length suffix))))))
 
 ;; Initials completion
 ;; Complete /ums to /usr/monnier/src or lch to list-command-history.
@@ -4299,7 +4496,11 @@ completion-initials-expand
 (defun completion-initials-all-completions (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
     (when newstr
-      (completion-pcm-all-completions newstr table pred (length newstr)))))
+      (pcase-let ((`(,pattern ,all ,prefix ,_suffix)
+                   (completion-pcm--find-all-completions newstr table
+                                                         pred (length newstr))))
+        (completion-pcm--deferred-hilit pattern all
+                                        (length prefix) (length string))))))
 
 (defun completion-initials-try-completion (string table pred _point)
   (let ((newstr (completion-initials-expand string table pred)))
diff --git a/test/lisp/minibuffer-tests.el b/test/lisp/minibuffer-tests.el
index 4f92d7f841c..1ab80cd1364 100644
--- a/test/lisp/minibuffer-tests.el
+++ b/test/lisp/minibuffer-tests.el
@@ -28,8 +28,7 @@
 
 (require 'ert)
 (require 'ert-x)
-
-(eval-when-compile (require 'cl-lib))
+(require 'cl-lib)
 
 (ert-deftest completion-test1 ()
   (with-temp-buffer
@@ -344,6 +343,221 @@ completion-flex-test-3
                   "custgroup" '("customize-group-other-window") nil 9)))
            15)))
 
+(ert-deftest completion-flex-score-test-1 ()
+  ;; Full match!
+  (should (equal
+           (completion--flex-score '(prefix "R") '("R"))
+           (list (cons -1.0 "R")))))
+
+(ert-deftest completion-flex-score-test-2 ()
+  ;; One third and half of a match!
+  (should (equal
+           (completion--flex-score '(prefix "foo")
+                                   '("barfoobar" "fooboo"))
+           (list (cons (/ -1.0 3.0) "barfoobar")
+                 (cons (/ -1.0 2.0) "fooboo")))))
+
+(ert-deftest completion-flex-score-test-3 ()
+  ;; One fourth of a match
+  (should (eql
+           (caar (completion--flex-score '(prefix "R" point "O")
+                                         '("RaOb")))
+           (/ -1.0 4.0))))
+
+(ert-deftest completion-flex-score-test-4 ()
+  ;; For quoted completion tables, score the unquoted completion string.
+  (should (equal
+           (completion--flex-score
+            '(prefix "R")
+            (list (propertize "X" 'completion--unquoted "R")))
+           (list (cons -1.0 "X")))))
+
+(defun completion--test-style (style string point table filtered)
+  (let* ((completion-styles (list style))
+         (pred (lambda (x) (not (string-search "!" x))))
+         (result (completion-filter-completions
+                  string table pred point nil)))
+    (should (equal (alist-get 'base result) 0))
+    (should (equal (alist-get 'end result) (length string)))
+    (should (equal (alist-get 'completions result) filtered))
+    ;; The highlighting function should be present.
+    (should (not (memq (alist-get 'highlight result) '(nil identity))))
+    ;; Equal results as `completion-all-completions'.
+    (should (equal (completion-all-completions string table pred point)
+                   (append filtered 0)))
+    ;; The returned strings should be identical to the original strings.
+    ;; The `completion-filter-completions' function avoids allocations!
+    (should (cl-intersection (alist-get 'completions result)
+                             table :test #'eq))))
+
+(ert-deftest completion-basic-style-test-1 ()
+  ;; point at the beginning |foo
+  (completion--test-style 'basic "foo" 0
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-basic-style-test-2 ()
+  ;; point foo
+  (completion--test-style 'basic "foo" 2
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-substring-style-test ()
+  (completion--test-style 'substring "foo" 1
+                          '("foobar" "foo!" "barfoo" "xfooy" "boobar")
+                          '("foobar" "barfoo" "xfooy")))
+
+(ert-deftest completion-emacs21-style-test ()
+  (completion--test-style 'emacs21 "foo" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar")))
+
+(ert-deftest completion-emacs22-style-test ()
+  (completion--test-style 'emacs22 "fo0" 1
+                          '("foobar" "foo!" "fobar" "barfoo" "xfooy" "boobar")
+                          '("foobar" "fobar"))) ;; suffix ignored completely
+
+(ert-deftest completion-flex-style-test ()
+  (completion--test-style 'flex "abc" 1
+                          '("abc" "abc!" "xaybzc" "xaybz")
+                          '("abc" "xaybzc")))
+
+(ert-deftest completion-initials-style-test ()
+  (completion--test-style 'initials "abc" 1
+                          '("a-b-c" "a-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("a-b-c" "ax-by-cz")))
+
+(ert-deftest completion-pcm-style-test ()
+  (completion--test-style 'partial-completion "ax-b-c" 1
+                          '("ax-b-c" "ax-b-c!" "ax-by-cz" "xax-by-cz")
+                          '("ax-b-c" "ax-by-cz")))
+
+(ert-deftest completion-filter-completions-highlight-test ()
+  ;; point at the beginning |foo
+  (let* ((completion-styles '(basic))
+         (result (completion-filter-completions
+                  "foo" '("foobar" "fbarfoo" "fxfooy" "bar")
+                  nil 1 nil)))
+    (should (equal
+             (format "%S" (alist-get 'completions result))
+             (format "%S" '("foobar" "fbarfoo" "fxfooy"))))
+    (should (equal
+             (format "%S" (funcall (alist-get 'highlight result)
+                                   (alist-get 'completions result)))
+             (format "%S"
+                     '(#("foobar" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fbarfoo" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))
+                       #("fxfooy" 0 1 (face (completions-common-part))
+                         1 2 (face (completions-first-difference)))))))))
+
+(defun completion--test-boundaries (style string table result)
+  (let ((table
+         (lambda (str pred action)
+           (pcase action
+             (`(boundaries . ,suffix) `(boundaries
+                                        ,(1+ (string-match-p "<\\|/" str))
+                                        . ,(or (string-search ">" suffix) (length suffix))))
+             (_ (complete-with-action action table
+                                      (replace-regexp-in-string ".*[</]" "" str)
+                                      pred)))))
+        (point (string-search "|" string))
+        (string (string-replace "|" "" string))
+        (completion-styles (list style)))
+    (should (equal
+             (assq-delete-all
+              (if (assq 'highlight result) '-does-not-exist 'highlight)
+              (completion-filter-completions
+               string table nil point nil))
+             result))
+    (should (equal
+             (completion-all-completions
+              string table nil point)
+             (append (alist-get 'completions result)
+                     (alist-get 'base result))))))
+
+(ert-deftest completion-emacs21-boundaries-test ()
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs21 "before<in|put>after"
+                               '("ainput>after" "input>after" "inpux>after"
+                                 "inxputy>after" "input>after2")
+                               '((base . 7)
+                                 (end . 18)
+                                 (completions "input>after" "input>after2"))))
+
+(ert-deftest completion-emacs22-boundaries-test ()
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'emacs22 "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "inyy" "inzzz"))))
+
+(ert-deftest completion-basic-boundaries-test ()
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("other") nil)
+  (completion--test-boundaries 'basic "before<in|put>after"
+                               '("ainput" "input" "inpux" "inxputy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "input" "inxputy"))))
+
+(ert-deftest completion-substring-boundaries-test ()
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("other") nil)
+  (completion--test-boundaries 'substring "before<in|puts>after"
+                               '("ainputs" "inputs" "inpux" "inxputsy")
+                               '((base . 7)
+                                 (end . 13)
+                                 (completions "ainputs" "inputs" "inxputsy"))))
+
+(ert-deftest completion-pcm-boundaries-test ()
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("other") nil)
+  (completion--test-boundaries 'partial-completion "before<in-p|t>after"
+                               '("ain-pu-ts" "in-pts" "in-pu-ts" "in-px" "inx-ptsy")
+                               '((base . 7)
+                                 (end . 12)
+                                 (completions "in-pts" "in-pu-ts" "inx-ptsy"))))
+
+(ert-deftest completion-initials-boundaries-test ()
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("other") nil)
+  (completion--test-boundaries 'initials "/ip|t"
+                               '("ain/pu/ts" "in/pts" "in/pu/ts" "a/in/pu/ts"
+                                 "in/pu/ts/foo" "in/px" "inx/ptsy")
+                               '((base . 1)
+                                 (end . 4)
+                                 (completions "in/pu/ts" "in/pu/ts/foo"))))
+
+(defun completion-emacs22orig-all-completions (string table pred point)
+  (let ((beforepoint (substring string 0 point)))
+    (completion-hilit-commonality
+      (all-completions beforepoint table pred)
+     point
+     (car (completion-boundaries beforepoint table pred "")))))
+
+(ert-deftest completion-upgrade-return-type-test ()
+  ;; Test transparent upgrade of list completion style return value
+  ;; to the alist return value format of `completion-format-completions'.
+  (let ((completion-styles-alist
+         '((emacs22orig completion-emacs22-try-completion
+                        completion-emacs22orig-all-completions nil))))
+  (completion--test-boundaries 'emacs22orig "before<in|put>after"
+                               '("ainxxx" "inyy" "inzzz")
+                               '((base . 7)
+                                 ;; 18 is incorrect, should be 12!
+                                 ;; But the information is not available
+                                 ;; due to the completion-style upgrade.
+                                 (end . 18)
+                                 ;; Identity highlighting function.
+                                 (highlight . identity)
+                                 (completions "inyy" "inzzz")))))
+
+
 \f
 (defmacro completing-read-with-minibuffer-setup (collection &rest body)
   (declare (indent 1) (debug (collection body)))

[-- Attachment #3: completion-lazy-hilit.patch --]
[-- Type: text/x-patch, Size: 6740 bytes --]

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..a9ac0b3f040 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -545,6 +545,7 @@ icomplete-minibuffer-setup
     (setq-local icomplete--initial-input (icomplete--field-string))
     (setq-local completion-show-inline-help nil)
     (setq icomplete--scrolled-completions nil)
+    (setq completion-lazy-hilit (cl-gensym))
     (use-local-map (make-composed-keymap icomplete-minibuffer-map
     					 (current-local-map)))
     (add-hook 'post-command-hook #'icomplete-post-command-hook nil t)
@@ -754,12 +755,13 @@ icomplete-exhibit
                            (overlay-end rfn-eshadow-overlay)))
           (let* ((field-string (icomplete--field-string))
                  (text (while-no-input
+                         (benchmark-progn
                          (icomplete-completions
                           field-string
                           (icomplete--completion-table)
                           (icomplete--completion-predicate)
                           (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                              (eq minibuffer--require-match t))))))
                  (buffer-undo-list t)
                  deactivate-mark)
             ;; Do nothing if while-no-input was aborted.
@@ -901,7 +903,7 @@ icomplete--render-vertical
                                 'icomplete-selected-match 'append comp)
      collect (concat prefix
                      (make-string (- max-prefix-len (length prefix)) ? )
-                     comp
+                     (completion-lazy-hilit comp)
                      (make-string (- max-comp-len (length comp)) ? )
                      suffix)
      into lines-aux
@@ -1067,7 +1069,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..c56ef64494a 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3749,6 +3749,54 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request completion lazy hilighting.
+
+Completion-presenting frontends may opt to bind this variable to
+a unique non-nil value in the context of completion-producing
+calls (such as `completion-all-sorted-completions').  This hints
+the intervening completion styles that they do not need to
+propertize completion strings with the `face' property.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible for `face'-propertizing only the completion strings
+that are meant to be displayed to the user.  This can be done by
+calling the function `completion-lazy-hilit' which returns a
+`face'-propertized string.
+
+The value stored in this variable by the completion frontend
+should be unique to each completion attempt or session that
+utilizes the same completion style in `completion-styles-alist'.
+For frontends using the minibuffer as the locus of completion
+calls and display, setting it to a buffer-local value given by
+`gensym' is appropriate.  For frontends operating entirely in a
+single command, let-binding it to `gensym' is appropriate.
+
+Note that the optimization enabled by variable is only actually
+performed some completions styles.  To others, it is a harmless
+and useless hint.  To author a completion style that takes
+advantage of this, look in the source of
+`completion-pcm--hilit-commonality'.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (let* ((str (copy-sequence str))
+         (data (get-text-property 0 'completion-lazy-hilit-data str))
+         (re (and
+              completion-lazy-hilit
+              (eq completion-lazy-hilit (car data)) (cdr data)))
+         (md (and re (string-match re str) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil str)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil str))
+    str))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
@@ -3765,8 +3813,9 @@ completion-pcm--hilit-commonality
            last-md)
       (mapcar
        (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
+         (unless completion-lazy-hilit
+           ;; Don't modify the string itself.
+           (setq str (copy-sequence str)))
          (unless (string-match re str)
            (error "Internal error: %s does not match %s" re str))
          (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
@@ -3815,9 +3864,10 @@ completion-pcm--hilit-commonality
                 (update-score-and-face
                  (lambda (a b)
                    "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
+                   (unless completion-lazy-hilit
+                     (add-face-text-property a b
+                                             'completions-common-part
+                                             nil str))
                    (setq
                     score-numerator   (+ score-numerator (- b a)))
                    (unless (or (= a last-b)
@@ -3840,7 +3890,10 @@ completion-pcm--hilit-commonality
            ;; for that extra bit of match (bug#42149).
            (unless (= from match-end)
              (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
+           (put-text-property 0 1 'completion-lazy-hilit-data
+                              (cons completion-lazy-hilit re) str)
+           (if (and (> (length str) pos)
+                    (not completion-lazy-hilit))
                (add-face-text-property
                 pos (1+ pos)
                 'completions-first-difference

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 8267 bytes --]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Hi all!
>
> Time flies, doesn't it?

Indeed.  First, thanks for working on this.  I tried your patches and
could reproduce all your numbers with a recent master (643c67cf239).

After checking the 2021 version again (254dc6ab4) I've discovered that
2023 Emacs doesn't respond as well to the lazy-hilit patch as 2021
Emacs.  It's as if the cost of string copies (that the patch optimized)
has gone down significantly.  In addition, in 2021, we never had anyone
perform the needed changes for icomplete.el to work with Daniel's API.
It was only known that without those changes, icomplete.el actually
performed worse under the new patch

So, backed by the new ability to conduct good benchmarks, I looked at
the problem anew. I found some insight in the problem, and came up with
a new "lazy-hilit" patch which performs just as well, if not slightly
better, than Daniel's, while keeping the changes to lisp/minibuffer.el
much more minimal and not adding replacement for the longstanding
completion-all-completions.

Before I go into benchmarks, it's obvious to me that "lazy" or
"deferred" highlighting mean basically the same thing, which is "late"
or "just-in-time" highlighting.

I also think, as I did in 2021, that we should be careful to separate
what are performance-motivated changes from style-motivated changes.
The former are easy to discuss objectively, while the latter are much,
much more subjective.

Whatever the outcome, we're on our way to at least a faster
icomplete.el, which is of course good.

So here are the benchmarks.  The setup is the following, we start Emacs
like:

   src/emacs -nw -Q -f fido-vertical-mode -l ~/Downloads/benchmark.el

And make sure to put 300 000 symbols in the obarray.  The symbols are
prefixed "yoyo" deliberately.

    (cl-loop repeat 300000 do (intern (symbol-name (gensym "yoyo"))))

First a micro-benchmark:

   ;; Daniel's patch worked by Dmitry (v3)
   (benchmark-run 50
    (let ((completion-styles '(flex)))
      (completion-filter-completions "" obarray 'fboundp 0 nil)
      (completion-filter-completions "yo" obarray 'fboundp 0 nil)
      (completion-filter-completions "yoo" obarray 'fboundp 0 nil)
      ));; => (12.192422429999999 3 0.107881004)


  ;; lazy-hilit v4 patch attached in this email
  (benchmark-run 50
    (let ((completion-styles '(flex))
          (completion-lazy-hilit (cl-gensym)))
      (completion-all-completions "" obarray 'fboundp 0 nil)
      (completion-all-completions "yo" obarray 'fboundp 0 nil)
      (completion-all-completions "yoo" obarray 'fboundp 0 nil)
      ));; => (12.267915333 4 0.14799709099999991)

Now, tests specific to icomplete.el using Dmitry's instrumentation.
This is the "yoyo" test.  Evaluate:

   (completing-read "" obarray)

This starts a fido-vertical-mode minibuffer.  First type "yo", then
repeatedly insert and backspace a "o" to make "yoo" and "yo" in
alternating fashion.

The number of matches should be exactly 300696 for "yoo" and 301721 for
"yo".  Highlighting should be correct, of course.

Observe the results printed by the instrumentation in `icomplete.el`.
Collect them from *Messages* after 18 alternations.

The results:

  ;; with Daniel's patch to minibuffer
  ;; 
  ;; Elapsed time: 0.967481s (0.401923s in 8 GCs)
  ;; Elapsed time: 0.703229s (0.252157s in 5 GCs)
  ;; Elapsed time: 0.945053s (0.401540s in 8 GCs)
  ;; Elapsed time: 0.721198s (0.252657s in 5 GCs)
  ;; Elapsed time: 0.951377s (0.394238s in 8 GCs)
  ;; Elapsed time: 0.699232s (0.254524s in 5 GCs)
  ;; Elapsed time: 0.940497s (0.400292s in 8 GCs)
  ;; Elapsed time: 0.709986s (0.253635s in 5 GCs)
  ;; Elapsed time: 0.943063s (0.399020s in 8 GCs)
  ;; Elapsed time: 0.720825s (0.251619s in 5 GCs)
  ;; Elapsed time: 0.972146s (0.407665s in 8 GCs)
  ;; Elapsed time: 0.709619s (0.255678s in 5 GCs)
  ;; Elapsed time: 0.947108s (0.397916s in 8 GCs)
  ;; Elapsed time: 0.727231s (0.254040s in 5 GCs)
  ;; Elapsed time: 0.966196s (0.398492s in 8 GCs)
  ;; Elapsed time: 0.701558s (0.252168s in 5 GCs)
  ;; Elapsed time: 0.936269s (0.388110s in 8 GCs)
  ;; Elapsed time: 0.694050s (0.249759s in 5 GCs)

  ;; with my lazy-hilit patch worked minimally by Dmitry
  ;; 
  ;; Elapsed time: 1.779906s (0.975332s in 15 GCs)
  ;; Elapsed time: 1.342160s (0.490314s in 5 GCs)
  ;; Elapsed time: 1.235759s (0.420019s in 4 GCs)
  ;; Elapsed time: 1.363909s (0.519521s in 5 GCs)
  ;; Elapsed time: 1.175773s (0.423938s in 4 GCs)
  ;; Elapsed time: 1.340017s (0.508744s in 5 GCs)
  ;; Elapsed time: 1.124552s (0.404149s in 4 GCs)
  ;; Elapsed time: 1.327419s (0.499433s in 5 GCs)
  ;; Elapsed time: 1.121927s (0.400499s in 4 GCs)
  ;; Elapsed time: 1.308526s (0.493652s in 5 GCs)
  ;; Elapsed time: 1.159132s (0.404612s in 4 GCs)
  ;; Elapsed time: 1.323803s (0.500754s in 5 GCs)
  ;; Elapsed time: 1.128562s (0.406496s in 4 GCs)
  ;; Elapsed time: 1.345577s (0.503971s in 5 GCs)
  ;; Elapsed time: 1.121691s (0.401876s in 4 GCs)
  ;; Elapsed time: 1.304913s (0.492255s in 5 GCs)
  ;; Elapsed time: 1.141926s (0.399154s in 4 GCs)
  ;; Elapsed time: 1.312480s (0.498205s in 5 GCs)
  ;; Elapsed time: 1.125095s (0.403174s in 4 GCs)
  ;; Elapsed time: 1.332119s (0.503671s in 5 GCs)
  ;; Elapsed time: 1.131561s (0.402268s in 4 GCs)

  ;; New lazy-hilit patch attached:
  ;;
  ;; Elapsed time: 0.902985s (0.224307s in 3 GCs)
  ;; Elapsed time: 0.696391s (0.079687s in 1 GCs)
  ;; Elapsed time: 0.896176s (0.219964s in 3 GCs)
  ;; Elapsed time: 0.648318s (0.074765s in 1 GCs)
  ;; Elapsed time: 0.906288s (0.221534s in 3 GCs)
  ;; Elapsed time: 0.679141s (0.079102s in 1 GCs)
  ;; Elapsed time: 0.889320s (0.222668s in 3 GCs)
  ;; Elapsed time: 0.690199s (0.076926s in 1 GCs)
  ;; Elapsed time: 0.912206s (0.220297s in 3 GCs)
  ;; Elapsed time: 0.675524s (0.078875s in 1 GCs)
  ;; Elapsed time: 0.907111s (0.226627s in 3 GCs)
  ;; Elapsed time: 0.697139s (0.079571s in 1 GCs)
  ;; Elapsed time: 0.906650s (0.220946s in 3 GCs)
  ;; Elapsed time: 0.683808s (0.078001s in 1 GCs)
  ;; Elapsed time: 0.912471s (0.221977s in 3 GCs)
  ;; Elapsed time: 0.699742s (0.079009s in 1 GCs)
  ;; Elapsed time: 0.897566s (0.217786s in 3 GCs)
  ;; Elapsed time: 0.659043s (0.076046s in 1 GCs)

  Daniel+Dmitry patch v3:        0.8308954444444444s avg
  Old lazy-hilit patch:          1.2390678333333334s avg
  New lazy-hilit patch attached: 0.7922265555555554s avg

In conclusion:

* I think the two approaches are basically evenly matched in terms of
  performance, at least for this symbol completion scenario.

* In the completion-{sorted|all}-completions micro-benchmark my patch
  does very marginally worse (0.6%).  Probably because of the use of a
  hash table.  I believe I can fix this, though.

* In the icomplete.el usability test, my new patch probably does
  slightly better (4.6%).  Probably because it doesn't recalculate the
  regexp from the pattern every time it needs to late highlight.

* My patch doesn't suffer from the 'completion--unquoted' property
  complication in Daniel+Dmitry's patch."  It's possible/likely that the
  additional memory needed by this property will introduce an additional
  slowdown which isn't visible in the simpler symbol completion
  scenario.

Both patches propose what are effectively extensions to the large, old
and complicated completion API, in my case an additional variable to
bind (or set), in Daniel's patch a brand new API entry point and the
deprecation of an existing one.

The benchmarks show that Daniel's patch is not absolutely necessary to
reap the benefits of deferred/lazy/late/just-in-time highlighting.

Looking at the two patches side-by-side it seems evident to me that one
patch is much simpler than the other.  The "maybe-alist-maybe-not" flag
dedicated to completely changing the meaning of a number of
minibuffer.el functions while keeping backward compatibility is one such
item of complexity.

Therefore, since we can come up with simpler alternatives that bring
these now well-understood benefits, it won't surprise anyone that I
think we should go for the simpler choice.

João



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-completion-frontends-to-highlight-completion-s.patch --]
[-- Type: text/x-patch, Size: 15903 bytes --]

From 78057f40e53f39f0b26f4b9bf5d950b72f1c3d99 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20T=C3=A1vora?= <joaotavora@gmail.com>
Date: Wed, 25 Oct 2023 13:45:01 +0100
Subject: [PATCH] Allow completion frontends to highlight completion strings
 just-in-time

This allows completion-pcm--hilit-commonality to be sped up
substantially.

Introduce a new variable completion-lazy-hilit that allows for
completion frontends to opt-in an time-saving optimization by some
completions styles, such as the 'flex' and 'pcm' styles.

The variable must be bound or set by the frontend to a unique value
around a completion attempt/session.  See completion-lazy-hilit
docstring for more info.

* lisp/icomplete.el (icomplete-minibuffer-setup): Set completion-lazy-hilit.
(icomplete--render-vertical): Call completion-lazy-hilit.
(icomplete-completions): Call completion-lazy-hilit.

* lisp/minibuffer.el (completion-lazy-hilit): New variable.
(completion-lazy-hilit)
(completion--hilit-from-re): New functions.
(completion--lazy-hilit-table): New variable.
(completion--flex-score-1): New helper.
(completion-pcm--hilit-commonality): Use completion-lazy-hilit.
---
 lisp/icomplete.el  |   9 +-
 lisp/minibuffer.el | 262 +++++++++++++++++++++++++++++----------------
 2 files changed, 173 insertions(+), 98 deletions(-)

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..a9ac0b3f040 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -545,6 +545,7 @@ icomplete-minibuffer-setup
     (setq-local icomplete--initial-input (icomplete--field-string))
     (setq-local completion-show-inline-help nil)
     (setq icomplete--scrolled-completions nil)
+    (setq completion-lazy-hilit (cl-gensym))
     (use-local-map (make-composed-keymap icomplete-minibuffer-map
     					 (current-local-map)))
     (add-hook 'post-command-hook #'icomplete-post-command-hook nil t)
@@ -754,12 +755,13 @@ icomplete-exhibit
                            (overlay-end rfn-eshadow-overlay)))
           (let* ((field-string (icomplete--field-string))
                  (text (while-no-input
+                         (benchmark-progn
                          (icomplete-completions
                           field-string
                           (icomplete--completion-table)
                           (icomplete--completion-predicate)
                           (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                              (eq minibuffer--require-match t))))))
                  (buffer-undo-list t)
                  deactivate-mark)
             ;; Do nothing if while-no-input was aborted.
@@ -901,7 +903,7 @@ icomplete--render-vertical
                                 'icomplete-selected-match 'append comp)
      collect (concat prefix
                      (make-string (- max-prefix-len (length prefix)) ? )
-                     comp
+                     (completion-lazy-hilit comp)
                      (make-string (- max-comp-len (length comp)) ? )
                      suffix)
      into lines-aux
@@ -1067,7 +1069,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..4591f1145c8 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3749,108 +3749,180 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request completion lazy hilighting.
+
+Completion-presenting frontends may opt to bind this variable to
+a unique non-nil value in the context of completion-producing
+calls (such as `completion-all-sorted-completions').  This hints
+the intervening completion styles that they do not need to
+propertize completion strings with the `face' property.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible for `face'-propertizing only the completion strings
+that are meant to be displayed to the user.  This can be done by
+calling the function `completion-lazy-hilit' which returns a
+`face'-propertized string.
+
+The value stored in this variable by the completion frontend
+should be unique to each completion attempt or session that
+utilizes the same completion style in `completion-styles-alist'.
+For frontends using the minibuffer as the locus of completion
+calls and display, setting it to a buffer-local value given by
+`gensym' is appropriate.  For frontends operating entirely in a
+single command, let-binding it to `gensym' is appropriate.
+
+Note that the optimization enabled by variable is only actually
+performed some completions styles.  To others, it is a harmless
+and useless hint.  To author a completion style that takes
+advantage of this, look in the source of
+`completion-pcm--hilit-commonality'.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (completion--hilit-from-re
+   (copy-sequence str)
+   (gethash completion-lazy-hilit completion--lazy-hilit-table)))
+
+(defun completion--hilit-from-re (string regexp)
+  "Fontify STRING with `completions-common-part' using REGEXP."
+  (let* ((md (and regexp (string-match regexp string) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil string)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil string))
+    string))
+
+(defun completion--flex-score-1 (md match-end len)
+  "Compute matching score of completion.
+The score lies in the range between 0 and 1, where 1 corresponds to
+the full match.
+MD is the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by (0..1] (i.e., larger than (but not equal
+         ;; to) zero, and smaller or equal to one): the higher
+         ;; the better and only a perfect match (pattern equals
+         ;; string) will have score 1.  The formula takes the
+         ;; form of a quotient.  For the numerator, we use the
+         ;; number of +, i.e. the length of the pattern.  For
+         ;; the denominator, it first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while md
+      (let ((a from)
+            (b (pop md)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (/ score-numerator (* len (1+ score-denominator)) 1.0)))
+
+(defvar completion--lazy-hilit-table (make-hash-table :weakness 'key))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
-string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
-`completions-first-difference' in the relevant segments."
+string in COMPLETIONS.
+
+If `completion-lazy-hilit' is nil, return a deep copy of
+COMPLETIONS where each string is propertized with
+`completion-score', a number between 0 and 1, and with faces
+`completions-common-part', `completions-first-difference' in the
+relevant segments.
+
+Else, if `completion-lazy-hilit' is t, return COMPLETIONS where
+each string now has a `completion-score' property and no
+highlighting."
   (cond
    ((and completions (cl-loop for e in pattern thereis (stringp e)))
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
-           (point-idx (completion-pcm--pattern-point-idx pattern))
-           (case-fold-search completion-ignore-case)
-           last-md)
-      (mapcar
-       (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
-         (unless (string-match re str)
-           (error "Internal error: %s does not match %s" re str))
-         (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
-                (match-end (match-end 0))
-                (md (cddr (setq last-md (match-data t last-md))))
-                (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by (0..1] (i.e., larger than (but not equal
-                ;; to) zero, and smaller or equal to one): the higher
-                ;; the better and only a perfect match (pattern equals
-                ;; string) will have score 1.  The formula takes the
-                ;; form of a quotient.  For the numerator, we use the
-                ;; number of +, i.e. the length of the pattern.  For
-                ;; the denominator, it first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
-             (setq from (pop md)))
-           ;; If `pattern' doesn't have an explicit trailing any, the
-           ;; regex `re' won't produce match data representing the
-           ;; region after the match.  We need to account to account
-           ;; for that extra bit of match (bug#42149).
-           (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
-       completions)))
+           last-md
+           (score (lambda (str)
+                    (unless (string-match re str)
+                      (error "Internal error: %s does not match %s" re str))
+                    (let* ((match-end (match-end 0))
+                           (md (cddr (setq last-md (match-data t last-md)))))
+                      (completion--flex-score-1 md match-end (length str))))))
+      (cond (completion-lazy-hilit
+             (puthash completion-lazy-hilit re completion--lazy-hilit-table)
+             (mapc (lambda (str)
+                     (put-text-property 0 1 'completion-score (funcall score str) str))
+                   completions))
+            (t
+             (mapcar
+              (lambda (str)
+                (setq str (copy-sequence str))
+                (put-text-property 0 1 'completion-score (funcall score str) str)
+                (completion--hilit-from-re str re)
+                str)
+              completions)))))
    (t completions)))
 
 (defun completion-pcm--find-all-completions (string table pred point
-- 
2.39.2

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

This sounds fairly reasonable: the worst-case breakage seems to be that
we may occasionally lose highlighting because the var was non-nil at the
wrong time.

Sidenote: since the hash-table uses `eq` we don't need to use `gensym`,
we can use something like `cons` instead, which is cheaper and doesn't
risk making its way into the `obarray`.

> +(defun completion-lazy-hilit (str)
> +  "Return a copy of completion STR that is `face'-propertized.
> +See documentation for variable `completion-lazy-hilit' for more
> +details."
> +  (completion--hilit-from-re
> +   (copy-sequence str)
> +   (gethash completion-lazy-hilit completion--lazy-hilit-table)))

Hmm... in order to get the right result you need to call
`completion-lazy-hilit` sometime after calling
`completion-all-completions` and before the next call to
`completion-all-completions` done with the same value of
`completion-lazy-hilit`, right?

So how important is it to use a hash-table rather than a variable
holding just "the info about the last call to
`completion-all-completions`"?

> +           last-md
> +           (score (lambda (str)
> +                    (unless (string-match re str)
> +                      (error "Internal error: %s does not match %s" re str))
> +                    (let* ((match-end (match-end 0))
> +                           (md (cddr (setq last-md (match-data t last-md)))))
> +                      (completion--flex-score-1 md match-end (length str))))))
> +      (cond (completion-lazy-hilit
> +             (puthash completion-lazy-hilit re completion--lazy-hilit-table)
> +             (mapc (lambda (str)
> +                     (put-text-property 0 1 'completion-score (funcall score str) str))
> +                   completions))
> +            (t
> +             (mapcar
> +              (lambda (str)
> +                (setq str (copy-sequence str))
> +                (put-text-property 0 1 'completion-score (funcall score str) str)
> +                (completion--hilit-from-re str re)
> +                str)
> +              completions)))))

How much more expensive is it to replace the 

    (mapc (lambda (str)
            (put-text-property 0 1 'completion-score (funcall score str) str))
          completions))

with something like

    (let ((tail `(completion-lazy-hilit (completion--hilit-from-re ,re))))
      (mapc (lambda (str)
              (add-text-properties
               0 1 `(completion-score ,(funcall score str) ,@tail) str))
            completions))

and then get rid of the hash-table altogether?
                 

        Stefan

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 245 bytes --]

On Wed, Oct 25, 2023, 21:52 Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>
>
> and then get rid of the hash-table altogether?
>

Stefan,  again we think alike, I'm taking care of that, the hash table and
the gensym aren't needed at all.

>

[-- Attachment #2: Type: text/html, Size: 749 bytes --]

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

João Távora <joaotavora@gmail.com> writes:

>>  and then get rid of the hash-table altogether?
>
> Stefan, again we think alike, I'm taking care of that, the hash table
> and the gensym aren't needed at all.

Here's a revised patch without the hash table (actually a patch on top
of the other one, but I send the two of them again).

completion-lazy-hilit is simply bound in icomplete.el's
icomplete-exhibit.

I don't recall exactly why I used to set it in the hook to a gensym in
2021.  I think it was because I was setting face properties on reused
strings (not just score) and that caused problems when switching
completion styles.  But I don't think that can happen anymore, so no
need for that.

>> Hmm... in order to get the right result you need to call
>> `completion-lazy-hilit` sometime after calling
>> `completion-all-completions` and before the next call to
>> `completion-all-completions` done with the same value of
>> `completion-lazy-hilit`, right?

Right, it used to be something like that, but not anymore I think.  Now
the semantics, could be described informally as

   "called by the frontend in the hopes that the style got the hint,
    which will speed things up significantly -- but if the hint wasn't
    caught that's OK too".

>> How much more expensive is it to replace the 

>>     (mapc (lambda (str)
>>             (put-text-property 0 1 'completion-score (funcall score str) str))
>>           completions))

>> with something like

>>     (let ((tail `(completion-lazy-hilit (completion--hilit-from-re ,re))))
>>       (mapc (lambda (str)
>>               (add-text-properties
>>                0 1 `(completion-score ,(funcall score str) ,@tail) str))
>>             completions))


I get the idea, I think but I think it would be somewhat more expensive,
at it is similar to my earlier patch which stored more stuff in every
string which Dmitry (and then I) measured to be slower.  But feel free
to experiment, of course.

João


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-completion-frontends-to-highlight-completion-s.patch --]
[-- Type: text/x-patch, Size: 15907 bytes --]

From 09ac2a87cb95d31acdefa3b4920449da2cb848fb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20T=C3=A1vora?= <joaotavora@gmail.com>
Date: Wed, 25 Oct 2023 13:45:01 +0100
Subject: [PATCH 1/2] Allow completion frontends to highlight completion
 strings just-in-time

This allows completion-pcm--hilit-commonality to be sped up
substantially.

Introduce a new variable completion-lazy-hilit that allows for
completion frontends to opt-in an time-saving optimization by some
completions styles, such as the 'flex' and 'pcm' styles.

The variable must be bound or set by the frontend to a unique value
around a completion attempt/session.  See completion-lazy-hilit
docstring for more info.

* lisp/icomplete.el (icomplete-minibuffer-setup): Set completion-lazy-hilit.
(icomplete--render-vertical): Call completion-lazy-hilit.
(icomplete-completions): Call completion-lazy-hilit.

* lisp/minibuffer.el (completion-lazy-hilit): New variable.
(completion-lazy-hilit)
(completion--hilit-from-re): New functions.
(completion--lazy-hilit-table): New variable.
(completion--flex-score-1): New helper.
(completion-pcm--hilit-commonality): Use completion-lazy-hilit.
---
 lisp/icomplete.el  |   9 +-
 lisp/minibuffer.el | 262 +++++++++++++++++++++++++++++----------------
 2 files changed, 173 insertions(+), 98 deletions(-)

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..a9ac0b3f040 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -545,6 +545,7 @@ icomplete-minibuffer-setup
     (setq-local icomplete--initial-input (icomplete--field-string))
     (setq-local completion-show-inline-help nil)
     (setq icomplete--scrolled-completions nil)
+    (setq completion-lazy-hilit (cl-gensym))
     (use-local-map (make-composed-keymap icomplete-minibuffer-map
     					 (current-local-map)))
     (add-hook 'post-command-hook #'icomplete-post-command-hook nil t)
@@ -754,12 +755,13 @@ icomplete-exhibit
                            (overlay-end rfn-eshadow-overlay)))
           (let* ((field-string (icomplete--field-string))
                  (text (while-no-input
+                         (benchmark-progn
                          (icomplete-completions
                           field-string
                           (icomplete--completion-table)
                           (icomplete--completion-predicate)
                           (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                              (eq minibuffer--require-match t))))))
                  (buffer-undo-list t)
                  deactivate-mark)
             ;; Do nothing if while-no-input was aborted.
@@ -901,7 +903,7 @@ icomplete--render-vertical
                                 'icomplete-selected-match 'append comp)
      collect (concat prefix
                      (make-string (- max-prefix-len (length prefix)) ? )
-                     comp
+                     (completion-lazy-hilit comp)
                      (make-string (- max-comp-len (length comp)) ? )
                      suffix)
      into lines-aux
@@ -1067,7 +1069,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..4591f1145c8 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3749,108 +3749,180 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request completion lazy hilighting.
+
+Completion-presenting frontends may opt to bind this variable to
+a unique non-nil value in the context of completion-producing
+calls (such as `completion-all-sorted-completions').  This hints
+the intervening completion styles that they do not need to
+propertize completion strings with the `face' property.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible for `face'-propertizing only the completion strings
+that are meant to be displayed to the user.  This can be done by
+calling the function `completion-lazy-hilit' which returns a
+`face'-propertized string.
+
+The value stored in this variable by the completion frontend
+should be unique to each completion attempt or session that
+utilizes the same completion style in `completion-styles-alist'.
+For frontends using the minibuffer as the locus of completion
+calls and display, setting it to a buffer-local value given by
+`gensym' is appropriate.  For frontends operating entirely in a
+single command, let-binding it to `gensym' is appropriate.
+
+Note that the optimization enabled by variable is only actually
+performed some completions styles.  To others, it is a harmless
+and useless hint.  To author a completion style that takes
+advantage of this, look in the source of
+`completion-pcm--hilit-commonality'.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (completion--hilit-from-re
+   (copy-sequence str)
+   (gethash completion-lazy-hilit completion--lazy-hilit-table)))
+
+(defun completion--hilit-from-re (string regexp)
+  "Fontify STRING with `completions-common-part' using REGEXP."
+  (let* ((md (and regexp (string-match regexp string) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil string)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil string))
+    string))
+
+(defun completion--flex-score-1 (md match-end len)
+  "Compute matching score of completion.
+The score lies in the range between 0 and 1, where 1 corresponds to
+the full match.
+MD is the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by (0..1] (i.e., larger than (but not equal
+         ;; to) zero, and smaller or equal to one): the higher
+         ;; the better and only a perfect match (pattern equals
+         ;; string) will have score 1.  The formula takes the
+         ;; form of a quotient.  For the numerator, we use the
+         ;; number of +, i.e. the length of the pattern.  For
+         ;; the denominator, it first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while md
+      (let ((a from)
+            (b (pop md)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (/ score-numerator (* len (1+ score-denominator)) 1.0)))
+
+(defvar completion--lazy-hilit-table (make-hash-table :weakness 'key))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
-string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
-`completions-first-difference' in the relevant segments."
+string in COMPLETIONS.
+
+If `completion-lazy-hilit' is nil, return a deep copy of
+COMPLETIONS where each string is propertized with
+`completion-score', a number between 0 and 1, and with faces
+`completions-common-part', `completions-first-difference' in the
+relevant segments.
+
+Else, if `completion-lazy-hilit' is t, return COMPLETIONS where
+each string now has a `completion-score' property and no
+highlighting."
   (cond
    ((and completions (cl-loop for e in pattern thereis (stringp e)))
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
-           (point-idx (completion-pcm--pattern-point-idx pattern))
-           (case-fold-search completion-ignore-case)
-           last-md)
-      (mapcar
-       (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
-         (unless (string-match re str)
-           (error "Internal error: %s does not match %s" re str))
-         (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
-                (match-end (match-end 0))
-                (md (cddr (setq last-md (match-data t last-md))))
-                (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by (0..1] (i.e., larger than (but not equal
-                ;; to) zero, and smaller or equal to one): the higher
-                ;; the better and only a perfect match (pattern equals
-                ;; string) will have score 1.  The formula takes the
-                ;; form of a quotient.  For the numerator, we use the
-                ;; number of +, i.e. the length of the pattern.  For
-                ;; the denominator, it first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
-             (setq from (pop md)))
-           ;; If `pattern' doesn't have an explicit trailing any, the
-           ;; regex `re' won't produce match data representing the
-           ;; region after the match.  We need to account to account
-           ;; for that extra bit of match (bug#42149).
-           (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
-       completions)))
+           last-md
+           (score (lambda (str)
+                    (unless (string-match re str)
+                      (error "Internal error: %s does not match %s" re str))
+                    (let* ((match-end (match-end 0))
+                           (md (cddr (setq last-md (match-data t last-md)))))
+                      (completion--flex-score-1 md match-end (length str))))))
+      (cond (completion-lazy-hilit
+             (puthash completion-lazy-hilit re completion--lazy-hilit-table)
+             (mapc (lambda (str)
+                     (put-text-property 0 1 'completion-score (funcall score str) str))
+                   completions))
+            (t
+             (mapcar
+              (lambda (str)
+                (setq str (copy-sequence str))
+                (put-text-property 0 1 'completion-score (funcall score str) str)
+                (completion--hilit-from-re str re)
+                str)
+              completions)))))
    (t completions)))
 
 (defun completion-pcm--find-all-completions (string table pred point
-- 
2.39.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Replace-lazy-hilit-hash-table-with-something-simpler.patch --]
[-- Type: text/x-patch, Size: 6064 bytes --]

From 5c4449dc578903314f400461d13c4c08e02a18ef Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jo=C3=A3o=20T=C3=A1vora?= <joaotavora@gmail.com>
Date: Wed, 25 Oct 2023 22:36:15 +0100
Subject: [PATCH 2/2] Replace lazy hilit hash table with something simpler

* lisp/icomplete.el (icomplete-minibuffer-setup): Don't set
completion-lazy-hilit.
(icomplete-exhibit): Set it here.

* lisp/minibuffer.el (completion-lazy-hilit): Rework docstring.
(completion-lazy-hilit-fn): Rework from completion--lazy-hilit-re.
(completion--lazy-hilit-table): Delete.
(completion-pcm--hilit-commonality): Rework.
---
 lisp/icomplete.el  |  4 ++--
 lisp/minibuffer.el | 42 +++++++++++++++++++++---------------------
 2 files changed, 23 insertions(+), 23 deletions(-)

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index a9ac0b3f040..3e888c8b06a 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -545,7 +545,6 @@ icomplete-minibuffer-setup
     (setq-local icomplete--initial-input (icomplete--field-string))
     (setq-local completion-show-inline-help nil)
     (setq icomplete--scrolled-completions nil)
-    (setq completion-lazy-hilit (cl-gensym))
     (use-local-map (make-composed-keymap icomplete-minibuffer-map
     					 (current-local-map)))
     (add-hook 'post-command-hook #'icomplete-post-command-hook nil t)
@@ -723,7 +722,8 @@ icomplete-exhibit
              ;; Check if still in the right buffer (bug#61308)
              (or (window-minibuffer-p) completion-in-region--data)
              (icomplete-simple-completing-p)) ;Shouldn't be necessary.
-    (let ((saved-point (point)))
+    (let ((saved-point (point))
+          (completion-lazy-hilit t))
       (save-excursion
         (goto-char (icomplete--field-end))
         ;; Insert the match-status information:
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 4591f1145c8..4a727615afb 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1234,6 +1234,7 @@ completion-all-completions
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
 in the last `cdr'."
+  (setq completion-lazy-hilit-fn nil)
   ;; FIXME: We need to additionally return the info needed for the
   ;; second part of completion-base-position.
   (completion--nth-completion 2 string table pred point metadata))
@@ -3753,24 +3754,16 @@ completion-lazy-hilit
   "If non-nil, request completion lazy hilighting.
 
 Completion-presenting frontends may opt to bind this variable to
-a unique non-nil value in the context of completion-producing
-calls (such as `completion-all-sorted-completions').  This hints
-the intervening completion styles that they do not need to
-propertize completion strings with the `face' property.
+non-nil value in the context of completion-producing calls (such
+as `completion-all-sorted-completions').  This hints the
+intervening completion styles that they do not need to propertize
+completion strings with the `face' property.
 
 When doing so, it is the frontend -- not the style -- who becomes
 responsible for `face'-propertizing only the completion strings
-that are meant to be displayed to the user.  This can be done by
-calling the function `completion-lazy-hilit' which returns a
-`face'-propertized string.
-
-The value stored in this variable by the completion frontend
-should be unique to each completion attempt or session that
-utilizes the same completion style in `completion-styles-alist'.
-For frontends using the minibuffer as the locus of completion
-calls and display, setting it to a buffer-local value given by
-`gensym' is appropriate.  For frontends operating entirely in a
-single command, let-binding it to `gensym' is appropriate.
+that are meant to be displayed to the user.  This is done by
+calling `completion-lazy-hilit' on each such string, which
+produces the suitably propertized string.
 
 Note that the optimization enabled by variable is only actually
 performed some completions styles.  To others, it is a harmless
@@ -3778,13 +3771,21 @@ completion-lazy-hilit
 advantage of this, look in the source of
 `completion-pcm--hilit-commonality'.")
 
+(defvar completion-lazy-hilit-fn nil
+  "Used by completions styles to honouring `completion-lazy-hilit'.
+When a given style wants to enable support for
+`completion-lazy-hilit' (which see), that style should set this
+variable to a function of one argument, a fresh string to be
+displayed to the user.  The function is responsible for
+destructively highlighting the string.")
+
 (defun completion-lazy-hilit (str)
   "Return a copy of completion STR that is `face'-propertized.
 See documentation for variable `completion-lazy-hilit' for more
 details."
-  (completion--hilit-from-re
-   (copy-sequence str)
-   (gethash completion-lazy-hilit completion--lazy-hilit-table)))
+  (if (and completion-lazy-hilit completion-lazy-hilit-fn)
+      (funcall completion-lazy-hilit-fn (copy-sequence str))
+    str))
 
 (defun completion--hilit-from-re (string regexp)
   "Fontify STRING with `completions-common-part' using REGEXP."
@@ -3883,8 +3884,6 @@ completion--flex-score-1
          last-b              b)))
     (/ score-numerator (* len (1+ score-denominator)) 1.0)))
 
-(defvar completion--lazy-hilit-table (make-hash-table :weakness 'key))
-
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
@@ -3911,7 +3910,8 @@ completion-pcm--hilit-commonality
                            (md (cddr (setq last-md (match-data t last-md)))))
                       (completion--flex-score-1 md match-end (length str))))))
       (cond (completion-lazy-hilit
-             (puthash completion-lazy-hilit re completion--lazy-hilit-table)
+             (setq completion-lazy-hilit-fn
+                   (lambda (str) (completion--hilit-from-re str re)))
              (mapc (lambda (str)
                      (put-text-property 0 1 'completion-score (funcall score str) str))
                    completions))
-- 
2.39.2

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]

João Távora <joaotavora@gmail.com> writes:

> So, backed by the new ability to conduct good benchmarks, I looked at
> the problem anew. I found some insight in the problem, and came up with
> a new "lazy-hilit" patch which performs just as well, if not slightly
> better, than Daniel's, while keeping the changes to lisp/minibuffer.el
> much more minimal and not adding replacement for the longstanding
> completion-all-completions.

Working on this a bit more, I've now been able to optimize the "lazy
hilit" patch even further by recognizing that in many situations we
don't need to match the "PCM" pattern to each string twice.  The first
time we do it, we can calculate a score immediately and store it in the
string.  The average response times for the "yo-yoo" test described
previously:

   master:                 2.604s
   2021 lazy-hilit patch:  1.240s
   2023 Daniel+Dmitry:     0.831s
   2023 lazy-hilit v1:     0.792s

And the new one:

   2023 lazy-hilit v2:     0.518s

I'm now keeping my work in a branch called
feature/completion-lazy-hilit, but I still attach the diff here.

João

PS: for the flex style in particular, there are even more optimizations
possible.  For example one could take advantage of the fact that in
flex, a longer pattern should always yield a subset of the completions
produced by a shorter pattern in the same completion session.  But this
requires solid concepts of a "completion session".



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: lazy-hilit-2023-v2.diff --]
[-- Type: text/x-patch, Size: 18040 bytes --]

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..3e888c8b06a 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -722,7 +722,8 @@ icomplete-exhibit
              ;; Check if still in the right buffer (bug#61308)
              (or (window-minibuffer-p) completion-in-region--data)
              (icomplete-simple-completing-p)) ;Shouldn't be necessary.
-    (let ((saved-point (point)))
+    (let ((saved-point (point))
+          (completion-lazy-hilit t))
       (save-excursion
         (goto-char (icomplete--field-end))
         ;; Insert the match-status information:
@@ -754,12 +755,13 @@ icomplete-exhibit
                            (overlay-end rfn-eshadow-overlay)))
           (let* ((field-string (icomplete--field-string))
                  (text (while-no-input
+                         (benchmark-progn
                          (icomplete-completions
                           field-string
                           (icomplete--completion-table)
                           (icomplete--completion-predicate)
                           (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                              (eq minibuffer--require-match t))))))
                  (buffer-undo-list t)
                  deactivate-mark)
             ;; Do nothing if while-no-input was aborted.
@@ -901,7 +903,7 @@ icomplete--render-vertical
                                 'icomplete-selected-match 'append comp)
      collect (concat prefix
                      (make-string (- max-prefix-len (length prefix)) ? )
-                     comp
+                     (completion-lazy-hilit comp)
                      (make-string (- max-comp-len (length comp)) ? )
                      suffix)
      into lines-aux
@@ -1067,7 +1069,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..b38eb49aba8 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1234,6 +1234,7 @@ completion-all-completions
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
 in the last `cdr'."
+  (setq completion-lazy-hilit-fn nil)
   ;; FIXME: We need to additionally return the info needed for the
   ;; second part of completion-base-position.
   (completion--nth-completion 2 string table pred point metadata))
@@ -3720,21 +3721,33 @@ completion-pcm--all-completions
 
     ;; Use all-completions to do an initial cull.  This is a big win,
     ;; since all-completions is written in C!
-    (let* (;; Convert search pattern to a standard regular expression.
-	   (regex (completion-pcm--pattern->regex pattern))
-           (case-fold-search completion-ignore-case)
-           (completion-regexp-list (cons regex completion-regexp-list))
-	   (compl (all-completions
-                   (concat prefix
-                           (if (stringp (car pattern)) (car pattern) ""))
-		   table pred)))
-      (if (not (functionp table))
-	  ;; The internal functions already obeyed completion-regexp-list.
-	  compl
-	(let ((poss ()))
-	  (dolist (c compl)
-	    (when (string-match-p regex c) (push c poss)))
-	  (nreverse poss))))))
+    (let* ((case-fold-search completion-ignore-case)
+           (completion-regexp-list (cons
+                                    ;; Convert search pattern to a
+                                    ;; standard regular expression.
+                                    (completion-pcm--pattern->regex pattern)
+                                    completion-regexp-list))
+	   (completions (all-completions
+                         (concat prefix
+                                 (if (stringp (car pattern)) (car pattern) ""))
+		         table pred)))
+      (cond ((or (not (functionp table))
+                 (cl-loop for e in pattern never (stringp e)))
+	     ;; The internal functions already obeyed completion-regexp-list.
+	     completions)
+            (t
+             ;; The pattern has something interesting to match, in
+             ;; which case we take the opportunity to add an early
+             ;; completion-score cookie to each completion.
+             (cl-loop with re = (completion-pcm--pattern->regex pattern 'group)
+                      for orig in completions
+                      for comp = (copy-sequence orig)
+                      for score = (completion--flex-score comp re t)
+                      when score
+                      do (put-text-property 0 1 'completion-score
+                                      score
+                                      comp)
+                      and collect comp))))))
 
 (defvar flex-score-match-tightness 3
   "Controls how the `flex' completion style scores its matches.
@@ -3749,108 +3762,195 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request completion lazy highlighting.
+
+Completion-presenting frontends may opt to bind this variable to
+non-nil value in the context of completion-producing calls (such
+as `completion-all-sorted-completions').  This hints the
+intervening completion styles that they do not need to
+fontify (i.e. propertize with the `face' property) completion
+strings with highlights of the matching parts.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible this fontification.  The frontend binds this variable
+to non-nil, and calls the function with the same name
+`completion-lazy-hilit' on each completion string that is to be
+displayed to the user.
+
+Note that only some completion styles take advantage of this
+variable for optimization purposes.  Other styles will ignore the
+hint and greedily fontify as usual.  It is still safe for a
+frontend to call `completion-lazy-hilit' in these situations.
+
+To author a completion style that takes advantage see
+`completion-lazy-hilit-fn' and look in the source of
+`completion-pcm--hilit-commonality'.")
+
+(defvar completion-lazy-hilit-fn nil
+  "Used by completions styles to honouring `completion-lazy-hilit'.
+When a given style wants to enable support for
+`completion-lazy-hilit' (which see), that style should set this
+variable to a function of one argument, a fresh string to be
+displayed to the user.  The function is responsible for
+destructively highlighting the string.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (if (and completion-lazy-hilit completion-lazy-hilit-fn)
+      (funcall completion-lazy-hilit-fn (copy-sequence str))
+    str))
+
+(defun completion--hilit-from-re (string regexp)
+  "Fontify STRING with `completions-common-part' using REGEXP."
+  (let* ((md (and regexp (string-match regexp string) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil string)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil string))
+    string))
+
+(defun completion--flex-score-1 (md-groups match-end len)
+  "Compute matching score of completion.
+The score lies in the range between 0 and 1, where 1 corresponds to
+the full match.
+MD-GROUPS is the \"group\"  part of the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by (0..1] (i.e., larger than (but not equal
+         ;; to) zero, and smaller or equal to one): the higher
+         ;; the better and only a perfect match (pattern equals
+         ;; string) will have score 1.  The formula takes the
+         ;; form of a quotient.  For the numerator, we use the
+         ;; number of +, i.e. the length of the pattern.  For
+         ;; the denominator, it first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while (and md-groups (car md-groups))
+      (let ((a from)
+            (b (pop md-groups)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md-groups)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (/ score-numerator (* len (1+ score-denominator)) 1.0)))
+
+(defvar completion--flex-score-last-md nil
+  "Helper variable for `completion--flex-score'.")
+
+(defun completion--flex-score (str re &optional dont-error)
+  "Compute flex score of completion STR based on RE.
+If DONT-ERROR, just return nil if RE doesn't match STR."
+  (cond ((string-match re str)
+         (let* ((match-end (match-end 0))
+                (md (cddr
+                     (setq
+                      completion--flex-score-last-md
+                      (match-data t completion--flex-score-last-md)))))
+           (completion--flex-score-1 md match-end (length str))))
+        ((not dont-error)
+         (error "Internal error: %s does not match %s" re str))))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
-string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
-`completions-first-difference' in the relevant segments."
+string in COMPLETIONS.
+
+If `completion-lazy-hilit' is nil, return a deep copy of
+COMPLETIONS where each string is propertized with
+`completion-score', a number between 0 and 1, and with faces
+`completions-common-part', `completions-first-difference' in the
+relevant segments.
+
+Else, if `completion-lazy-hilit' is t, return COMPLETIONS where
+each string now has a `completion-score' property and no
+highlighting."
   (cond
    ((and completions (cl-loop for e in pattern thereis (stringp e)))
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
-           (point-idx (completion-pcm--pattern-point-idx pattern))
-           (case-fold-search completion-ignore-case)
-           last-md)
-      (mapcar
-       (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
-         (unless (string-match re str)
-           (error "Internal error: %s does not match %s" re str))
-         (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
-                (match-end (match-end 0))
-                (md (cddr (setq last-md (match-data t last-md))))
-                (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by (0..1] (i.e., larger than (but not equal
-                ;; to) zero, and smaller or equal to one): the higher
-                ;; the better and only a perfect match (pattern equals
-                ;; string) will have score 1.  The formula takes the
-                ;; form of a quotient.  For the numerator, we use the
-                ;; number of +, i.e. the length of the pattern.  For
-                ;; the denominator, it first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
-             (setq from (pop md)))
-           ;; If `pattern' doesn't have an explicit trailing any, the
-           ;; regex `re' won't produce match data representing the
-           ;; region after the match.  We need to account to account
-           ;; for that extra bit of match (bug#42149).
-           (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
-       completions)))
+           (score-maybe (lambda (str)
+                          (unless (get-text-property 0 'completion-score str)
+                            (put-text-property 0 1 'completion-score
+                                               (completion--flex-score str re)
+                                               str)))))
+      (cond (completion-lazy-hilit
+             (setq completion-lazy-hilit-fn
+                   (lambda (str) (completion--hilit-from-re str re)))
+             (mapc score-maybe completions))
+            (t
+             (mapcar
+              (lambda (str)
+                (setq str (copy-sequence str))
+                (funcall score-maybe str)
+                (completion--hilit-from-re str re)
+                str)
+              completions)))))
    (t completions)))
 
 (defun completion-pcm--find-all-completions (string table pred point

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

Hi Joao,

Thanks for the updates and the numbers.

On 27/10/2023 00:49, João Távora wrote:
> João Távora<joaotavora@gmail.com>  writes:
> 
>> So, backed by the new ability to conduct good benchmarks, I looked at
>> the problem anew. I found some insight in the problem, and came up with
>> a new "lazy-hilit" patch which performs just as well, if not slightly
>> better, than Daniel's, while keeping the changes to lisp/minibuffer.el
>> much more minimal and not adding replacement for the longstanding
>> completion-all-completions.
> Working on this a bit more, I've now been able to optimize the "lazy
> hilit" patch even further by recognizing that in many situations we
> don't need to match the "PCM" pattern to each string twice.  The first
> time we do it, we can calculate a score immediately and store it in the
> string.  The average response times for the "yo-yoo" test described
> previously:

This latest change in particular regressed this related scenario:

;; Use 'set' to ensure that the variables are bound.
(cl-loop repeat 300000 do (set (intern (symbol-name (gensym "yoyo"))) 4))

C-h v <then type y o <backspace> o <backspace> ...>

It increased the timings for that scenario from ~0.600s (with either of 
the latest filter-deferred patches and your previous version) to ~1s.

My understanding is it's due to the judicious call (copy-sequence orig) 
that you added before 'put-text-property' is called. While it seems like 
a good idea to preserve the original value, when almost all of obarray 
matches the current input (which is the current scenario), a lot of 
strings will be copied.

But (completing-read "" obarray) is not affected, and I don't see why. 
Maybe because it re-sorts less than icomplete? And the consing triggers 
the GC threshold less often (thanks to the other changes).

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 25/10/2023 20:52, João Távora wrote:
> And make sure to put 300 000 symbols in the obarray.  The symbols are
> prefixed "yoyo" deliberately.
> 
>      (cl-loop repeat 300000 do (intern (symbol-name (gensym "yoyo"))))
> 
> First a micro-benchmark:
> 
>     ;; Daniel's patch worked by Dmitry (v3)
>     (benchmark-run 50
>      (let ((completion-styles '(flex)))
>        (completion-filter-completions "" obarray 'fboundp 0 nil)
>        (completion-filter-completions "yo" obarray 'fboundp 0 nil)
>        (completion-filter-completions "yoo" obarray 'fboundp 0 nil)
>        ));; => (12.192422429999999 3 0.107881004)
> 
> 
>    ;; lazy-hilit v4 patch attached in this email
>    (benchmark-run 50
>      (let ((completion-styles '(flex))
>            (completion-lazy-hilit (cl-gensym)))
>        (completion-all-completions "" obarray 'fboundp 0 nil)
>        (completion-all-completions "yo" obarray 'fboundp 0 nil)
>        (completion-all-completions "yoo" obarray 'fboundp 0 nil)
>        ));; => (12.267915333 4 0.14799709099999991)

Note on this particular test:

The loop on the first line only creates the symbols in the obarray, but 
not functions. As a result, all the completion-all-completions calls 
return nil because of the 'fboundp' predicate. When you change the 
predicate argument to nil, these timings change considerably (so it's 
wiser to reduce the number of repetitions to 1 or 5 at most), with 
completion-filter-completions being ~2.5x faster than the other.

It is slower in the sorting step, though: mostly due to the extra 
consing created with the alist to-be-sorted, I guess, but also because 
of the repeated string-match call (which, while fast and much faster 
than the match-data call, is still not free).

That's how when compared in practice using fido-vertical-mode the 
results were about the same.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

Dmitry Gutov <dmitry@gutov.dev> writes:

> My understanding is it's due to the judicious call (copy-sequence
> orig) that you added before 'put-text-property' is called. While it
> seems like a good idea to preserve the original value, when almost all
> of obarray matches the current input (which is the current scenario),
> a lot of strings will be copied.

You're right, I reproduced the regression.  I thought I had taken out
the copy-sequence, but forgot it there.  In an earlier stage I suspected
that I needed the copy, but I don't think I do.  Please try this new
patch that removes it.  I've also pushed it to the
feature/completion-lazy-hilit branch.

João


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: lazy-hilit-2023-v3.diff --]
[-- Type: text/x-patch, Size: 17985 bytes --]

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..3e888c8b06a 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -722,7 +722,8 @@ icomplete-exhibit
              ;; Check if still in the right buffer (bug#61308)
              (or (window-minibuffer-p) completion-in-region--data)
              (icomplete-simple-completing-p)) ;Shouldn't be necessary.
-    (let ((saved-point (point)))
+    (let ((saved-point (point))
+          (completion-lazy-hilit t))
       (save-excursion
         (goto-char (icomplete--field-end))
         ;; Insert the match-status information:
@@ -754,12 +755,13 @@ icomplete-exhibit
                            (overlay-end rfn-eshadow-overlay)))
           (let* ((field-string (icomplete--field-string))
                  (text (while-no-input
+                         (benchmark-progn
                          (icomplete-completions
                           field-string
                           (icomplete--completion-table)
                           (icomplete--completion-predicate)
                           (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                              (eq minibuffer--require-match t))))))
                  (buffer-undo-list t)
                  deactivate-mark)
             ;; Do nothing if while-no-input was aborted.
@@ -901,7 +903,7 @@ icomplete--render-vertical
                                 'icomplete-selected-match 'append comp)
      collect (concat prefix
                      (make-string (- max-prefix-len (length prefix)) ? )
-                     comp
+                     (completion-lazy-hilit comp)
                      (make-string (- max-comp-len (length comp)) ? )
                      suffix)
      into lines-aux
@@ -1067,7 +1069,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..ecde00dd28d 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1234,6 +1234,7 @@ completion-all-completions
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
 in the last `cdr'."
+  (setq completion-lazy-hilit-fn nil)
   ;; FIXME: We need to additionally return the info needed for the
   ;; second part of completion-base-position.
   (completion--nth-completion 2 string table pred point metadata))
@@ -3720,21 +3721,32 @@ completion-pcm--all-completions
 
     ;; Use all-completions to do an initial cull.  This is a big win,
     ;; since all-completions is written in C!
-    (let* (;; Convert search pattern to a standard regular expression.
-	   (regex (completion-pcm--pattern->regex pattern))
-           (case-fold-search completion-ignore-case)
-           (completion-regexp-list (cons regex completion-regexp-list))
-	   (compl (all-completions
-                   (concat prefix
-                           (if (stringp (car pattern)) (car pattern) ""))
-		   table pred)))
-      (if (not (functionp table))
-	  ;; The internal functions already obeyed completion-regexp-list.
-	  compl
-	(let ((poss ()))
-	  (dolist (c compl)
-	    (when (string-match-p regex c) (push c poss)))
-	  (nreverse poss))))))
+    (let* ((case-fold-search completion-ignore-case)
+           (completion-regexp-list (cons
+                                    ;; Convert search pattern to a
+                                    ;; standard regular expression.
+                                    (completion-pcm--pattern->regex pattern)
+                                    completion-regexp-list))
+	   (completions (all-completions
+                         (concat prefix
+                                 (if (stringp (car pattern)) (car pattern) ""))
+		         table pred)))
+      (cond ((or (not (functionp table))
+                 (cl-loop for e in pattern never (stringp e)))
+	     ;; The internal functions already obeyed completion-regexp-list.
+	     completions)
+            (t
+             ;; The pattern has something interesting to match, in
+             ;; which case we take the opportunity to add an early
+             ;; completion-score cookie to each completion.
+             (cl-loop with re = (completion-pcm--pattern->regex pattern 'group)
+                      for comp in completions
+                      for score = (completion--flex-score comp re t)
+                      when score
+                      do (put-text-property 0 1 'completion-score
+                                      score
+                                      comp)
+                      and collect comp))))))
 
 (defvar flex-score-match-tightness 3
   "Controls how the `flex' completion style scores its matches.
@@ -3749,108 +3761,195 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request completion lazy highlighting.
+
+Completion-presenting frontends may opt to bind this variable to
+non-nil value in the context of completion-producing calls (such
+as `completion-all-sorted-completions').  This hints the
+intervening completion styles that they do not need to
+fontify (i.e. propertize with the `face' property) completion
+strings with highlights of the matching parts.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible this fontification.  The frontend binds this variable
+to non-nil, and calls the function with the same name
+`completion-lazy-hilit' on each completion string that is to be
+displayed to the user.
+
+Note that only some completion styles take advantage of this
+variable for optimization purposes.  Other styles will ignore the
+hint and greedily fontify as usual.  It is still safe for a
+frontend to call `completion-lazy-hilit' in these situations.
+
+To author a completion style that takes advantage see
+`completion-lazy-hilit-fn' and look in the source of
+`completion-pcm--hilit-commonality'.")
+
+(defvar completion-lazy-hilit-fn nil
+  "Used by completions styles to honouring `completion-lazy-hilit'.
+When a given style wants to enable support for
+`completion-lazy-hilit' (which see), that style should set this
+variable to a function of one argument, a fresh string to be
+displayed to the user.  The function is responsible for
+destructively highlighting the string.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (if (and completion-lazy-hilit completion-lazy-hilit-fn)
+      (funcall completion-lazy-hilit-fn (copy-sequence str))
+    str))
+
+(defun completion--hilit-from-re (string regexp)
+  "Fontify STRING with `completions-common-part' using REGEXP."
+  (let* ((md (and regexp (string-match regexp string) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil string)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil string))
+    string))
+
+(defun completion--flex-score-1 (md-groups match-end len)
+  "Compute matching score of completion.
+The score lies in the range between 0 and 1, where 1 corresponds to
+the full match.
+MD-GROUPS is the \"group\"  part of the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by (0..1] (i.e., larger than (but not equal
+         ;; to) zero, and smaller or equal to one): the higher
+         ;; the better and only a perfect match (pattern equals
+         ;; string) will have score 1.  The formula takes the
+         ;; form of a quotient.  For the numerator, we use the
+         ;; number of +, i.e. the length of the pattern.  For
+         ;; the denominator, it first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while (and md-groups (car md-groups))
+      (let ((a from)
+            (b (pop md-groups)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md-groups)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (/ score-numerator (* len (1+ score-denominator)) 1.0)))
+
+(defvar completion--flex-score-last-md nil
+  "Helper variable for `completion--flex-score'.")
+
+(defun completion--flex-score (str re &optional dont-error)
+  "Compute flex score of completion STR based on RE.
+If DONT-ERROR, just return nil if RE doesn't match STR."
+  (cond ((string-match re str)
+         (let* ((match-end (match-end 0))
+                (md (cddr
+                     (setq
+                      completion--flex-score-last-md
+                      (match-data t completion--flex-score-last-md)))))
+           (completion--flex-score-1 md match-end (length str))))
+        ((not dont-error)
+         (error "Internal error: %s does not match %s" re str))))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
-string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
-`completions-first-difference' in the relevant segments."
+string in COMPLETIONS.
+
+If `completion-lazy-hilit' is nil, return a deep copy of
+COMPLETIONS where each string is propertized with
+`completion-score', a number between 0 and 1, and with faces
+`completions-common-part', `completions-first-difference' in the
+relevant segments.
+
+Else, if `completion-lazy-hilit' is t, return COMPLETIONS where
+each string now has a `completion-score' property and no
+highlighting."
   (cond
    ((and completions (cl-loop for e in pattern thereis (stringp e)))
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
-           (point-idx (completion-pcm--pattern-point-idx pattern))
-           (case-fold-search completion-ignore-case)
-           last-md)
-      (mapcar
-       (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
-         (unless (string-match re str)
-           (error "Internal error: %s does not match %s" re str))
-         (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
-                (match-end (match-end 0))
-                (md (cddr (setq last-md (match-data t last-md))))
-                (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by (0..1] (i.e., larger than (but not equal
-                ;; to) zero, and smaller or equal to one): the higher
-                ;; the better and only a perfect match (pattern equals
-                ;; string) will have score 1.  The formula takes the
-                ;; form of a quotient.  For the numerator, we use the
-                ;; number of +, i.e. the length of the pattern.  For
-                ;; the denominator, it first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
-             (setq from (pop md)))
-           ;; If `pattern' doesn't have an explicit trailing any, the
-           ;; regex `re' won't produce match data representing the
-           ;; region after the match.  We need to account to account
-           ;; for that extra bit of match (bug#42149).
-           (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
-       completions)))
+           (score-maybe (lambda (str)
+                          (unless (get-text-property 0 'completion-score str)
+                            (put-text-property 0 1 'completion-score
+                                               (completion--flex-score str re)
+                                               str)))))
+      (cond (completion-lazy-hilit
+             (setq completion-lazy-hilit-fn
+                   (lambda (str) (completion--hilit-from-re str re)))
+             (mapc score-maybe completions))
+            (t
+             (mapcar
+              (lambda (str)
+                (setq str (copy-sequence str))
+                (funcall score-maybe str)
+                (completion--hilit-from-re str re)
+                str)
+              completions)))))
    (t completions)))
 
 (defun completion-pcm--find-all-completions (string table pred point

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 27/10/2023 02:27, João Távora wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> My understanding is it's due to the judicious call (copy-sequence
>> orig) that you added before 'put-text-property' is called. While it
>> seems like a good idea to preserve the original value, when almost all
>> of obarray matches the current input (which is the current scenario),
>> a lot of strings will be copied.
> You're right, I reproduced the regression.  I thought I had taken out
> the copy-sequence, but forgot it there.  In an earlier stage I suspected
> that I needed the copy, but I don't think I do.  Please try this new
> patch that removes it.  I've also pushed it to the
> feature/completion-lazy-hilit branch.

Yep, without copy-sequence the regression is gone. Now the input strings 
are routinely mutated, though. ;-(

You could do a copy a little later -- after the match succeeds and the 
score is computed. But for short widely-matching inputs like 'a' that 
would make little difference.

I also experimented with hash-tables for "external" score storage. But 
it still comes out a little slower than either of the proposed solutions.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Fri, Oct 27, 2023 at 12:25 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 25/10/2023 20:52, João Távora wrote:
> > And make sure to put 300 000 symbols in the obarray.  The symbols are
> > prefixed "yoyo" deliberately.
> >
> >      (cl-loop repeat 300000 do (intern (symbol-name (gensym "yoyo"))))
> >
> > First a micro-benchmark:
> >
> >     ;; Daniel's patch worked by Dmitry (v3)
> >     (benchmark-run 50
> >      (let ((completion-styles '(flex)))
> >        (completion-filter-completions "" obarray 'fboundp 0 nil)
> >        (completion-filter-completions "yo" obarray 'fboundp 0 nil)
> >        (completion-filter-completions "yoo" obarray 'fboundp 0 nil)
> >        ));; => (12.192422429999999 3 0.107881004)
> >
> >
> >    ;; lazy-hilit v4 patch attached in this email
> >    (benchmark-run 50
> >      (let ((completion-styles '(flex))
> >            (completion-lazy-hilit (cl-gensym)))
> >        (completion-all-completions "" obarray 'fboundp 0 nil)
> >        (completion-all-completions "yo" obarray 'fboundp 0 nil)
> >        (completion-all-completions "yoo" obarray 'fboundp 0 nil)
> >        ));; => (12.267915333 4 0.14799709099999991)
>
> Note on this particular test:
>
> The loop on the first line only creates the symbols in the obarray, but
> not functions. As a result, all the completion-all-completions calls
> return nil because of the 'fboundp' predicate. When you change the
> predicate argument to nil, these timings change considerably (so it's
> wiser to reduce the number of repetitions to 1 or 5 at most), with
> completion-filter-completions being ~2.5x faster than the other.

Right, I missed this.  I can reproduce it, though only around 2.0x faster here.

> It is slower in the sorting step, though: mostly due to the extra
> consing created with the alist to-be-sorted, I guess, but also because
> of the repeated string-match call (which, while fast and much faster
> than the match-data call, is still not free).

And is the sorting step not included in the full call to
completion-filter-completions?  I don't fully understand how it works.

> That's how when compared in practice using fido-vertical-mode the
> results were about the same.

But that's what matters right?

Also in the last iteration of the "yoyo" fido-vertical-mode test,
results with my latest patch are quite a bit faster.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Fri, Oct 27, 2023 at 12:35 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 27/10/2023 02:27, João Távora wrote:
> > Dmitry Gutov<dmitry@gutov.dev>  writes:
> >
> >> My understanding is it's due to the judicious call (copy-sequence
> >> orig) that you added before 'put-text-property' is called. While it
> >> seems like a good idea to preserve the original value, when almost all
> >> of obarray matches the current input (which is the current scenario),
> >> a lot of strings will be copied.
> > You're right, I reproduced the regression.  I thought I had taken out
> > the copy-sequence, but forgot it there.  In an earlier stage I suspected
> > that I needed the copy, but I don't think I do.  Please try this new
> > patch that removes it.  I've also pushed it to the
> > feature/completion-lazy-hilit branch.
>
> Yep, without copy-sequence the regression is gone. Now the input strings
> are routinely mutated, though. ;-(

Not sure what you mean by mutated, the strings look fine to me,
does this make any visible problem?  I couldn't detect it.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 27/10/2023 02:44, João Távora wrote:
>> It is slower in the sorting step, though: mostly due to the extra
>> consing created with the alist to-be-sorted, I guess, but also because
>> of the repeated string-match call (which, while fast and much faster
>> than the match-data call, is still not free).
> And is the sorting step not included in the full call to
> completion-filter-completions?  I don't fully understand how it works.

It recomputes all the scores inside the display-sort-function.

>> That's how when compared in practice using fido-vertical-mode the
>> results were about the same.
> But that's what matters right?

Pretty much, yes. Except for some potential exotic cases where the UI or 
the user would want to override the sorting logic. Corfu and Vertico 
have such capability, but I'm not sure if it's used often.

> Also in the last iteration of the "yoyo" fido-vertical-mode test,
> results with my latest patch are quite a bit faster.

Hmm, your latest (lazy-hilit-2023-v3.diff) improves the (completing-read 
"" obarray) scenario a lot (over the original two 2023 solutions), but 
not the the 'C-h v' scenario. I don't have an explanation.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Fri, Oct 27, 2023 at 12:44 AM João Távora <joaotavora@gmail.com> wrote:

> > It is slower in the sorting step, though: mostly due to the extra
> > consing created with the alist to-be-sorted, I guess, but also because
> > of the repeated string-match call (which, while fast and much faster
> > than the match-data call, is still not free).
>
> And is the sorting step not included in the full call to
> completion-filter-completions?  I don't fully understand how it works.

I get it now.  Neither completion-sorted-completions
or completion-all-completions do the sorting.  It's just that
if sorting is bound to be required, completion-all-completions
will do most of the work upfront, and thus sorting will be much
faster in that case, compensating completely.

So now I don't think that micro-benchmark is of much use.  A useful
such benchmark would have to be representative of the full cycle.
Maybe we don't even need this benchmark, given the instrumentation
in icomplete.el works well and seems precise and deterministic (just
not as convenient to run).

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Fri, Oct 27, 2023 at 1:11 AM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > Also in the last iteration of the "yoyo" fido-vertical-mode test,
> > results with my latest patch are quite a bit faster.
>
> Hmm, your latest (lazy-hilit-2023-v3.diff) improves the (completing-read
> "" obarray) scenario a lot (over the original two 2023 solutions), but
> not the the 'C-h v' scenario. I don't have an explanation.

The improvement was due to running string-match only once per completion,
if you look at the changes to completion-pcm--all-completions.

It could be this code doesn't kick in in the C-h v scenario.  Notice
that that function already has some optimizations: for example, when
the regexp match is performed by all-completions and its use of
completion-regexp-alist, there's no way to get the regex match data
to compute the score, so it'll have to be postponed to
completion-pcm--hilit-commonality as in the v2.diff case.

Then again, that doesn't explain why in the C-h v scenario, the regression
was fixed precisely by adjust that code that I was conjecturing might
not kick in.

Anyway, I think it's safer to say that my latest patch is at least as fast,
sometimes faster, than the 2023 completion-filter-completions solution.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 27/10/2023 03:26, João Távora wrote:
> On Fri, Oct 27, 2023 at 1:11 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>>> Also in the last iteration of the "yoyo" fido-vertical-mode test,
>>> results with my latest patch are quite a bit faster.
>>
>> Hmm, your latest (lazy-hilit-2023-v3.diff) improves the (completing-read
>> "" obarray) scenario a lot (over the original two 2023 solutions), but
>> not the the 'C-h v' scenario. I don't have an explanation.
> 
> The improvement was due to running string-match only once per completion,
> if you look at the changes to completion-pcm--all-completions.

Right. I just don't (didn't?) have an explanation for the difference in 
the improvement in performance (or the lack thereof) between the two 
scenarios.

> It could be this code doesn't kick in in the C-h v scenario.  Notice
> that that function already has some optimizations: for example, when
> the regexp match is performed by all-completions and its use of
> completion-regexp-alist, there's no way to get the regex match data
> to compute the score, so it'll have to be postponed to
> completion-pcm--hilit-commonality as in the v2.diff case.
> 
> Then again, that doesn't explain why in the C-h v scenario, the regression
> was fixed precisely by adjust that code that I was conjecturing might
> not kick in.

According to my print-debugging, the situation is the reverse: 
(completing-read "" obarray) goes the "The internal functions already 
obeyed" route (because obarray is not a function?), and the scenario 
that didn't get better (C-h v) goes down the clause "pattern has 
something interesting to match". Unless the input is empty, then it also 
goes down the first clause.

So it seems like we might misunderstand the reason why the former got 
faster. I see the check changed from

   (not (functionp table)

to

   (or (not (functionp table))
       (cl-loop for e in pattern never (stringp e)))

but that can't be that reason.

BTW, all-completion's docstring also says that a COLLECTION that is a 
function should itself handle completion-regexp-list, so we could try to 
rely on that too and drop the additional check. That's risky, though, so 
something for a later follow-up.

> Anyway, I think it's safer to say that my latest patch is at least as fast,
> sometimes faster, than the 2023 completion-filter-completions solution.

All other things equal, I also prefer a smaller change, and thank you 
for producing it. Functionally, too, it's almost equivalent to 
completion-filter-completions. So one could easily write a wrapper like 
that with the same performance.

But there are differences. The first is that the highlighter function 
takes one string as an argument instead of a collection. I mentioned 
this before, this will be much handier to use in company-capf.

Second, in Daniel's patch the "adjust metadata" function got a 
different, arguably better, calling convention. That's not dependent on 
the rest of the patch, so it can be considered separately.

Third, it made a principled stance to avoid altering the original 
strings, even the non-visual text properties. This approach could be 
adopted piecewise from Daniel's patch, especially if the performance 
ends up the same or insignificantly different in practical usage.

As for whether we migrate to the completion-filter-completions API, I 
don't have a strong opinion. If we still think that the next revision of 
the completion API will be radically different, then there is not much 
point in making medium-sized steps like that. OTOH, if we end up with 
the same API for the next decade and more, completion-filter-completions 
does look more meaningful, and more easily extensible (e.g. next we 
could add a pair (completion-scorer . fn) to its return value; and so 
on). And again, the implementation could be a simple wrapper at first.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> BTW, all-completion's docstring also says that a COLLECTION that is
> a function should itself handle completion-regexp-list,

The key here is "should": we know how well ELisp coders follow
such recommendations.


        Stefan

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 27/10/2023 16:46, Stefan Monnier wrote:
>> BTW, all-completion's docstring also says that a COLLECTION that is
>> a function should itself handle completion-regexp-list,
> The key here is "should": we know how well ELisp coders follow
> such recommendations.

We could, for example, have a period when we warn about returned 
non-matches. string-match-p is not free, but it's not very expensive either.

I think we should fix the discrepancy between the doc and the behavior, 
one way or another. If the function does obey the current phrasing, it 
seems like it's doing extra work if we post-filter anyway, producing 
extra cosing (though the result might still be beneficial if that 
filtering results in a much smaller list on the first try).

But third-party callers of all-completions might also see the doc and 
decide to use completion-regexp-list, with mixed results. That seems 
like a pure downside.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> We could, for example, have a period when we warn about returned
> non-matches. string-match-p is not free, but it's not very expensive either.

The problem is that I dislike `completion-regexp-list` :-)

More seriously, since it's a dynbound variable it can have unwanted
effects in nested calls to `all/try-completions`, so it's safer to
ignore that variable because its binding is not always "meant for us" :-(


        Stefan

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 27/10/2023 19:19, Stefan Monnier wrote:
>> We could, for example, have a period when we warn about returned
>> non-matches. string-match-p is not free, but it's not very expensive either.
> 
> The problem is that I dislike `completion-regexp-list` :-)

When we do use it, we can avoid copying all the strings to a new list. 
Skipping consing this way can really move the needle at the level of 
optimization we're discussing now.

> More seriously, since it's a dynbound variable it can have unwanted
> effects in nested calls to `all/try-completions`, so it's safer to
> ignore that variable because its binding is not always "meant for us" :-(

I guess it would be more precise if it was a function argument, e.g. the 
first argument to 'fancy-all-completions' or somesuch that all 
completion tables are supposed to use inside. OTOH, I suppose that might 
hinder those that use external programs.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 5010 bytes --]

Dmitry Gutov <dmitry@gutov.dev> writes:

> but that can't be that reason.

Indeed it isn't.  But you're not far off, I think.  Anyway, in the
(completing-read "" obarray) scenario, the shortcut is taken, and that
and the fact that the optimization then completing skips scoring for it
-- which is blatantly wrong -- is why it is much faster.

Looking at this, it seems that string-match with a non-group-capturing
regexp over all the matches (the thing that the change was optimizing
away) isn't that expensive in the overall gist of things.

Thus in the C-h v case, the shortcut isn't taken and the optimization
does what it should, but nothing gargantuan, around 8-9%.  Also keep in
mind this yoyo case is pretty contrived and I suspect more realistic
cases would have even more modest gains.

My attempts to fix the optimization to work correctly in all cases
complicated the code and then slowed down the bare completion-read case.
Not worth it IMO, therefore I have reverted it in the branch and in the
latest patch I attach (lazy-hilit-2023-v4.diff).  Here are the latest
measurements for the "yoyo" test, now considering the C-h v scenario:

;; Daniel+Dmitry completin-read: 0.834s avg
;; Daniel+Dmitry C-h v:          0.988s avg
                                           
;; lazy-hilit completing-read:   0.825s avg
;; lazy-hilit C-h v:             0.946s avg

Again, this shows my patch to be about 2-4% faster, though not really
relevant.

Again, the optimization I removed, if it were done correctly (which it
wasn't) could maybe shave off another 8-9% off that.

> But there are differences. The first is that the highlighter function
> takes one string as an argument instead of a collection. I mentioned
> this before, this will be much handier to use in company-capf.

Don't fully follow this.  Can you perhaps show two versions (or two
snippets) of the company-capf code, the handy and the non-handy version?

> Second, in Daniel's patch the "adjust metadata" function got a
> different, arguably better, calling convention. That's not dependent
> on the rest of the patch, so it can be considered separately.

Maybe.  Changes to calling convention can be argued for but, when
possible, they shoudn't be in with performance improvements.

> Third, it made a principled stance to avoid altering the original
> strings, even the non-visual text properties. This approach could be
> adopted piecewise from Daniel's patch, especially if the performance
> ends up the same or insignificantly different in practical usage.

If we really wanted to, we could also adopt the non-propertizing
approach in my lazy-hilit patch, by calculating the score "just in
time", much like Daniel's patch does it.  But it should be clear that
what we save in allocation in completion-pcm--hilit-commonality, we'll
pay for in the same amount in consing later.  So no performance benefit.

And if we do that, don't forget there will be the ugly "unquoted"
complication to deal with.  Then again, in my understanding that
complication is due to similar problem of mixing business and display
logic.  That's assuming I understand this comment in minibuffer.el
correctly:

   ;; Here we choose to quote all elements returned, but a better option
   ;; would be to return unquoted elements together with a function to
   ;; requote them, so that *Completions* can show nicer unquoted values
   ;; which only get quoted when needed by choose-completion.

So I would look into solving that first, instead of allowing the
"unquoted" hacks to spread even further in minibuffer.el

> As for whether we migrate to the completion-filter-completions API, I
> don't have a strong opinion. If we still think that the next revision
> of the completion API will be radically different, then there is not
> much point in making medium-sized steps like that. OTOH, if we end up
> with the same API for the next decade and more,
> completion-filter-completions does look more meaningful, and more
> easily extensible (e.g. next we could add a pair (completion-scorer
> . fn) to its return value; and so on). And again, the implementation
> could be a simple wrapper at first.

I agree on some points, and looking at this from a API-convenience
standpoint I note two things:

* To get late highlighting in Daniel's patch, one has to use a whole new
  API entry function to gather completions, and another to fontify
  completions.  One also deprecates an existing member of the API.

* In my patch, one binds a new API varaible and uses a function to fontify
  completions.  No deprecations.

Even if it wasn't a much simpler change to minibuffer.el without any
backward-compatibility gymnastics, I still think the latter is much
cleaner and easier to implement for completion frontends.  The fact that
there's no deprecation of one of the most important members of the API
to date makes it more confortable to understand IMO.

João


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: lazy-hilit-2023-v4.diff --]
[-- Type: text/x-patch, Size: 15535 bytes --]

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..3e888c8b06a 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -722,7 +722,8 @@ icomplete-exhibit
              ;; Check if still in the right buffer (bug#61308)
              (or (window-minibuffer-p) completion-in-region--data)
              (icomplete-simple-completing-p)) ;Shouldn't be necessary.
-    (let ((saved-point (point)))
+    (let ((saved-point (point))
+          (completion-lazy-hilit t))
       (save-excursion
         (goto-char (icomplete--field-end))
         ;; Insert the match-status information:
@@ -754,12 +755,13 @@ icomplete-exhibit
                            (overlay-end rfn-eshadow-overlay)))
           (let* ((field-string (icomplete--field-string))
                  (text (while-no-input
+                         (benchmark-progn
                          (icomplete-completions
                           field-string
                           (icomplete--completion-table)
                           (icomplete--completion-predicate)
                           (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                              (eq minibuffer--require-match t))))))
                  (buffer-undo-list t)
                  deactivate-mark)
             ;; Do nothing if while-no-input was aborted.
@@ -901,7 +903,7 @@ icomplete--render-vertical
                                 'icomplete-selected-match 'append comp)
      collect (concat prefix
                      (make-string (- max-prefix-len (length prefix)) ? )
-                     comp
+                     (completion-lazy-hilit comp)
                      (make-string (- max-comp-len (length comp)) ? )
                      suffix)
      into lines-aux
@@ -1067,7 +1069,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..cd8eeee2c78 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1234,6 +1234,7 @@ completion-all-completions
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
 in the last `cdr'."
+  (setq completion-lazy-hilit-fn nil)
   ;; FIXME: We need to additionally return the info needed for the
   ;; second part of completion-base-position.
   (completion--nth-completion 2 string table pred point metadata))
@@ -3749,108 +3750,194 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request completion lazy highlighting.
+
+Completion-presenting frontends may opt to bind this variable to
+non-nil value in the context of completion-producing calls (such
+as `completion-all-sorted-completions').  This hints the
+intervening completion styles that they do not need to
+fontify (i.e. propertize with the `face' property) completion
+strings with highlights of the matching parts.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible this fontification.  The frontend binds this variable
+to non-nil, and calls the function with the same name
+`completion-lazy-hilit' on each completion string that is to be
+displayed to the user.
+
+Note that only some completion styles take advantage of this
+variable for optimization purposes.  Other styles will ignore the
+hint and greedily fontify as usual.  It is still safe for a
+frontend to call `completion-lazy-hilit' in these situations.
+
+To author a completion style that takes advantage see
+`completion-lazy-hilit-fn' and look in the source of
+`completion-pcm--hilit-commonality'.")
+
+(defvar completion-lazy-hilit-fn nil
+  "Used by completions styles to honouring `completion-lazy-hilit'.
+When a given style wants to enable support for
+`completion-lazy-hilit' (which see), that style should set this
+variable to a function of one argument, a fresh string to be
+displayed to the user.  The function is responsible for
+destructively highlighting the string.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (if (and completion-lazy-hilit completion-lazy-hilit-fn)
+      (funcall completion-lazy-hilit-fn (copy-sequence str))
+    str))
+
+(defun completion--hilit-from-re (string regexp)
+  "Fontify STRING with `completions-common-part' using REGEXP."
+  (let* ((md (and regexp (string-match regexp string) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil string)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil string))
+    string))
+
+(defun completion--flex-score-1 (md-groups match-end len)
+  "Compute matching score of completion.
+The score lies in the range between 0 and 1, where 1 corresponds to
+the full match.
+MD-GROUPS is the \"group\"  part of the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by (0..1] (i.e., larger than (but not equal
+         ;; to) zero, and smaller or equal to one): the higher
+         ;; the better and only a perfect match (pattern equals
+         ;; string) will have score 1.  The formula takes the
+         ;; form of a quotient.  For the numerator, we use the
+         ;; number of +, i.e. the length of the pattern.  For
+         ;; the denominator, it first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while (and md-groups (car md-groups))
+      (let ((a from)
+            (b (pop md-groups)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md-groups)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (/ score-numerator (* len (1+ score-denominator)) 1.0)))
+
+(defvar completion--flex-score-last-md nil
+  "Helper variable for `completion--flex-score'.")
+
+(defun completion--flex-score (str re &optional dont-error)
+  "Compute flex score of completion STR based on RE.
+If DONT-ERROR, just return nil if RE doesn't match STR."
+  (cond ((string-match re str)
+         (let* ((match-end (match-end 0))
+                (md (cddr
+                     (setq
+                      completion--flex-score-last-md
+                      (match-data t completion--flex-score-last-md)))))
+           (completion--flex-score-1 md match-end (length str))))
+        ((not dont-error)
+         (error "Internal error: %s does not match %s" re str))))
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
-string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
-`completions-first-difference' in the relevant segments."
+string in COMPLETIONS.
+
+If `completion-lazy-hilit' is nil, return a deep copy of
+COMPLETIONS where each string is propertized with
+`completion-score', a number between 0 and 1, and with faces
+`completions-common-part', `completions-first-difference' in the
+relevant segments.
+
+Else, if `completion-lazy-hilit' is t, return COMPLETIONS where
+each string now has a `completion-score' property and no
+highlighting."
   (cond
    ((and completions (cl-loop for e in pattern thereis (stringp e)))
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
-           (point-idx (completion-pcm--pattern-point-idx pattern))
-           (case-fold-search completion-ignore-case)
-           last-md)
-      (mapcar
-       (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
-         (unless (string-match re str)
-           (error "Internal error: %s does not match %s" re str))
-         (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
-                (match-end (match-end 0))
-                (md (cddr (setq last-md (match-data t last-md))))
-                (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by (0..1] (i.e., larger than (but not equal
-                ;; to) zero, and smaller or equal to one): the higher
-                ;; the better and only a perfect match (pattern equals
-                ;; string) will have score 1.  The formula takes the
-                ;; form of a quotient.  For the numerator, we use the
-                ;; number of +, i.e. the length of the pattern.  For
-                ;; the denominator, it first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
-             (setq from (pop md)))
-           ;; If `pattern' doesn't have an explicit trailing any, the
-           ;; regex `re' won't produce match data representing the
-           ;; region after the match.  We need to account to account
-           ;; for that extra bit of match (bug#42149).
-           (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
-       completions)))
+           (score (lambda (str)
+                    (put-text-property 0 1 'completion-score
+                                       (completion--flex-score str re)
+                                       str))))
+      (cond (completion-lazy-hilit
+             (setq completion-lazy-hilit-fn
+                   (lambda (str) (completion--hilit-from-re str re)))
+             (mapc score completions))
+            (t
+             (mapcar
+              (lambda (str)
+                (setq str (copy-sequence str))
+                (funcall score str)
+                (completion--hilit-from-re str re)
+                str)
+              completions)))))
    (t completions)))
 
 (defun completion-pcm--find-all-completions (string table pred point

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> We could, for example, have a period when we warn about returned
>>> non-matches. string-match-p is not free, but it's not very expensive either.
>> The problem is that I dislike `completion-regexp-list` :-)
> When we do use it, we can avoid copying all the strings to a new
> list. Skipping consing this way can really move the needle at the level of
> optimization we're discussing now.

Oh, don't get me wrong, I like the functionality it offers, I just
dislike the way it works.

>> More seriously, since it's a dynbound variable it can have unwanted
>> effects in nested calls to `all/try-completions`, so it's safer to
>> ignore that variable because its binding is not always "meant for us" :-(
> I guess it would be more precise if it was a function argument, e.g. the
> first argument to 'fancy-all-completions' or somesuch that all completion
> tables are supposed to use inside. OTOH, I suppose that might hinder those
> that use external programs.

In my "work in progress" (not touched since last December :-( ),
I replace `all-completions` with:

    (cl-defgeneric completion-table-fetch-matches ( table pattern
                                                    &optional pre session)
      "Return candidates matching PATTERN in the completion TABLE.
    For tables with subfields, PRE is the text found before PATTERN such that
       (let ((len (length PRE)))
         (equal (completion-table-boundaries TABLE PRE len) (cons len len)))
    
    Return a list of strings or a list of cons cells whose car is a string.
    SESSION if non-nil is a hash-table where we can stash arbitrary auxiliary
    info to avoid recomputing it between calls of the same \"session\".")

`pattern`s can take various shapes.  In my WiP code, I implement 4 kinds
of patterns: prefix, glob, regexp, and predicate.  Now, we don't want
completion tables to have to handle each and every one of those pattern
kinds (the set of which is extensible via CLOS methods), so there's
a middleman:

    (cl-defgeneric completion-pattern-convert (to pattern)
      "Convert PATTERN to be of type TO.
    Returns a pair (PRED . NEWPATTERN) where NEWPATTERN is of type TO
    and should match everything that PATTERN matches.  PRED is nil
    if NEWPATTERN matches exactly the same candidates as PATTERN
    and otherwise it is a function that takes a candidate and returns non-nil if the
    candidate also matches PATTERN.  PRED should not presume that the candidate
    has already been filtered by NEWPATTERN."

So the fallback definition of `completion-table-fetch-matches`, which
relies on the old API looks like:

    (defun completion-table--fetch-legacy (table pattern &optional pre _session)
      (pcase-let ((`(,pred . ,regexp)
                   (completion-pattern-convert 'regexp pattern))
                  (`(,ppred . ,prefix)
                   (completion-pattern-convert 'prefix pattern)))
        (let ((matches
               (let ((completion-regexp-list (if ppred (list regexp)))
                     (case-fold-search completion-ignore-case))
                 (all-completions (concat pre prefix) table))))
          (if (null pred)
              matches
            (seq-filter pred matches)))))

This is of course incorrect because `all-completions` could ignore
`completion-regexp-list`, in which case we should use `ppred` instead of
`pred` on the last 3 lines :-)

It has the disadvantage that every completion-table basically needs to
start by calling `completion-pattern-convert` so as to convert the
pattern to the format that it supports.  But I think it's still better
than the current API where completion tables "have to" obey the prefix
string, the `completion-regexp-list`, and the predicate (and where the
latter two are often nil so tables tends to ignore them, and since
tables ignore them callers don't use them, etc...).


        Stefan

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 27/10/2023 20:16, João Távora wrote:

> My attempts to fix the optimization to work correctly in all cases
> complicated the code and then slowed down the bare completion-read case.
> Not worth it IMO, therefore I have reverted it in the branch and in the
> latest patch I attach (lazy-hilit-2023-v4.diff).  Here are the latest
> measurements for the "yoyo" test, now considering the C-h v scenario:
> 
> ;; Daniel+Dmitry completin-read: 0.834s avg
> ;; Daniel+Dmitry C-h v:          0.988s avg
>                                             
> ;; lazy-hilit completing-read:   0.825s avg
> ;; lazy-hilit C-h v:             0.946s avg
> 
> Again, this shows my patch to be about 2-4% faster, though not really
> relevant.
> 
> Again, the optimization I removed, if it were done correctly (which it
> wasn't) could maybe shave off another 8-9% off that.

Thanks. My measurements are similar, except the difference switch the 
other way a little bit. It might depend on the particulars of the 
individual machine anyway. I also tried to compare the perf for 150K 
symbols, in case either the filtering or the sorting (which should have 
different complexities) would come to the front, but no such luck:

lazy-hilit

300000
completing-read 0.6063497
C-h v 0.72435995

150000
completing-read 0.316961
C-h v 0.39933575

d+d

300000
completing-read 0.571481
C-h v 0.69817995

150000
completing-read 0.308940
C-h v 0.350437

All averages made using 'M-x calc-grab-region' followed by 'u M'.

>> But there are differences. The first is that the highlighter function
>> takes one string as an argument instead of a collection. I mentioned
>> this before, this will be much handier to use in company-capf.
> 
> Don't fully follow this.  Can you perhaps show two versions (or two
> snippets) of the company-capf code, the handy and the non-handy version?

I just meant that your version will be easier to use in 
company--match-from-capf-face (because it works on individual completions).

>> Third, it made a principled stance to avoid altering the original
>> strings, even the non-visual text properties. This approach could be
>> adopted piecewise from Daniel's patch, especially if the performance
>> ends up the same or insignificantly different in practical usage.
> 
> If we really wanted to, we could also adopt the non-propertizing
> approach in my lazy-hilit patch, by calculating the score "just in
> time", much like Daniel's patch does it.  But it should be clear that
> what we save in allocation in completion-pcm--hilit-commonality, we'll
> pay for in the same amount in consing later.  So no performance benefit.

Not for performance, no. Although the way it separates the sorting into 
its own phase makes it easier to reason about that particular cost. And 
for 300000 symbols, scoring and sorting really take the most time, e.g. 
about 2/3rds. Which might help with optimizing it further down in the 
future, somehow,

> And if we do that, don't forget there will be the ugly "unquoted"
> complication to deal with.  Then again, in my understanding that
> complication is due to similar problem of mixing business and display
> logic.  That's assuming I understand this comment in minibuffer.el
> correctly:
> 
>     ;; Here we choose to quote all elements returned, but a better option
>     ;; would be to return unquoted elements together with a function to
>     ;; requote them, so that *Completions* can show nicer unquoted values
>     ;; which only get quoted when needed by choose-completion.
> 
> So I would look into solving that first, instead of allowing the
> "unquoted" hacks to spread even further in minibuffer.el

I don't really understand this quoting-requoting business, never dug 
into the feature or the code. But perhaps keeping the original string 
might even help avoid the "requoting" step? Though that would depend on 
which version of the string the scoring and highligher functions expect 
to work on.

Speaking of the comment, it sounds like the said "requote function" 
would need to be passed up the call stack and used according to some 
protocol. The idea itself reminds me of the proposal described in 
https://github.com/company-mode/company-mode/discussions/1422 (it was 
also previously brought up on the lsp-mode tracker). It seems like 
either the completion tables would need a new method, or the capf tuples 
- a new function property, which all UIs using would need to start 
supporting in lock-step. At least that's the case if we use that to 
solve the requoting issue (the LSP clients' migration to use 
complete-function can be done more easily).

So far you advocated toward avoiding breaking changes while implementing 
the present performance improvement. Daniel's argument that the quoting 
completion tables are already slow enough sounds very reasonable to me, 
that an extra text property round-trip won't be noticeable. And the 
current version of the patch is functional and passes all tests, so it's 
not clear which other places would the "hack" need to spead into. Unless 
it helps with reducing code, etc, as per my vague guess above.

But if course it would be nice to avoid the wart, so if you have any 
better ideas, they are welcome.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 27/10/2023 21:12, Stefan Monnier wrote:
>>> More seriously, since it's a dynbound variable it can have unwanted
>>> effects in nested calls to `all/try-completions`, so it's safer to
>>> ignore that variable because its binding is not always "meant for us" 🙁
>> I guess it would be more precise if it was a function argument, e.g. the
>> first argument to 'fancy-all-completions' or somesuch that all completion
>> tables are supposed to use inside. OTOH, I suppose that might hinder those
>> that use external programs.
> In my "work in progress" (not touched since last December 🙁 ),
> I replace `all-completions` with:
> 
>      (cl-defgeneric completion-table-fetch-matches ( table pattern
>                                                      &optional pre session)
>        "Return candidates matching PATTERN in the completion TABLE.
>      For tables with subfields, PRE is the text found before PATTERN such that
>         (let ((len (length PRE)))
>           (equal (completion-table-boundaries TABLE PRE len) (cons len len)))
>      
>      Return a list of strings or a list of cons cells whose car is a string.
>      SESSION if non-nil is a hash-table where we can stash arbitrary auxiliary
>      info to avoid recomputing it between calls of the same \"session\".")
> 
> `pattern`s can take various shapes.  In my WiP code, I implement 4 kinds
> of patterns: prefix, glob, regexp, and predicate.  Now, we don't want
> completion tables to have to handle each and every one of those pattern
> kinds (the set of which is extensible via CLOS methods), so there's
> a middleman:
> 
>      (cl-defgeneric completion-pattern-convert (to pattern)
>        "Convert PATTERN to be of type TO.
>      Returns a pair (PRED . NEWPATTERN) where NEWPATTERN is of type TO
>      and should match everything that PATTERN matches.  PRED is nil
>      if NEWPATTERN matches exactly the same candidates as PATTERN
>      and otherwise it is a function that takes a candidate and returns non-nil if the
>      candidate also matches PATTERN.  PRED should not presume that the candidate
>      has already been filtered by NEWPATTERN."

FWIW, this neat structure might not help too much: the most popular 
external completion backend (the LSP language servers, collectively) 
don't accept regexps or globs, they just send you the lists of 
completions available at point. With the name matching method sometimes 
configurable per-server.

As such, the most useful methods currently are: 1) Emacs Regexp, 2) 
asking server for whatever it thinks is suitable (the "backend" 
completion style).

I would also probably want to standardize on the recommended type of TO 
anyway: some of them are likely going to result in better performance 
than others.

BTW, this reminds me about urgrep in GNU ELPA, which I think includes 
converters between different flavors of regexp. Something to keep in 
mind for the occasional completion table that's based on a Grep-like tool.

> So the fallback definition of `completion-table-fetch-matches`, which
> relies on the old API looks like:
> 
>      (defun completion-table--fetch-legacy (table pattern &optional pre _session)
>        (pcase-let ((`(,pred . ,regexp)
>                     (completion-pattern-convert 'regexp pattern))
>                    (`(,ppred . ,prefix)
>                     (completion-pattern-convert 'prefix pattern)))
>          (let ((matches
>                 (let ((completion-regexp-list (if ppred (list regexp)))
>                       (case-fold-search completion-ignore-case))
>                   (all-completions (concat pre prefix) table))))
>            (if (null pred)
>                matches
>              (seq-filter pred matches)))))
> 
> This is of course incorrect because `all-completions` could ignore
> `completion-regexp-list`, in which case we should use `ppred` instead of
> `pred` on the last 3 lines 😄
> 
> It has the disadvantage that every completion-table basically needs to
> start by calling `completion-pattern-convert` so as to convert the
> pattern to the format that it supports.  But I think it's still better
> than the current API where completion tables "have to" obey the prefix
> string, the `completion-regexp-list`, and the predicate (and where the
> latter two are often nil so tables tends to ignore them, and since
> tables ignore them callers don't use them, etc...).

So I guess it's also a way to make every completion table aware of PRED?

That should work; though it might be hard to reach the same raw 
performance as the current all-completions + completion-regexp-list.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> FWIW, this neat structure might not help too much: the most popular external
> completion backend (the LSP language servers, collectively) don't accept
> regexps or globs, they just send you the lists of completions available at
> point. With the name matching method sometimes configurable per-server.

Largely agreed.  The main benefit tho is that you get just *one*
pattern, rather than three (one being the prefix argument, the second
being the `pred` arg which historically was so unused that it was abused
to hold the directory for file-name completion, so lots of tables don't
obey it, and the third being the `completion-regexp-list` that most coders
forget, and those who don't end up regretting not forgetting when it was
not meant for them), so it's much more clear.

> As such, the most useful methods currently are: 1) Emacs Regexp, 2) asking
> server for whatever it thinks is suitable (the "backend" completion style).

For the backends: agreed.
For the frontends (i.e. `completion-styles`), `glob` is the more useful
one, I'd say (except for the "external" style, of course).

We might also want support for things like `or` and `and` patterns, but
I haven't managed to fit them nicely in that structure :-(

> I would also probably want to standardize on the recommended type of TO
> anyway: some of them are likely going to result in better performance
> than others.

The TO is chosen by the specific completion table, based on what it can
handle best.  So it should always be "optimal".

> So I guess it's also a way to make every completion table aware of PRED?

Note also that these `pred` patterns are expected to be exclusively
looking at the string (they're used for `completion-styles` kind of
functionality), so nothing like `file-directory-p` or `fboundp` kind of
predicates here.

The `pred`s used in things like `completing-read` and `read-file-name`
would be handled elsewhere such as `completion-table-with-predicate`.
This part is still up in the air, tho.

> That should work; though it might be hard to reach the same raw performance
> as the current all-completions + completion-regexp-list.

I don't see why: currently my code actually uses `all-completions` and
`completion-regexp-list`, so as long as the pattern can be turned into
a regexp without requiring an additional PRED (that's usually the case),
it should be just as fast (at least for large tables; for small tables
we have the overhead of the various method calls and pattern
conversions, of course).


        Stefan

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 27/10/2023 20:16, João Távora wrote:

> Thanks. My measurements are similar, except the difference switch the
> other way a little bit. It might depend on the particulars of the
> individual machine anyway.

Yes, it could, but I've reproduced this in different hardware.

Check that you're taking enough samples, I take about 15-20 samples.
Maybe the lazy-hilit patch pays a an extra cost upfront for the very
first C-h v or completing-read, to create the properties keys on the
strings, which are then reused.  This cost is ammortized very quickly,
of course, but if you're taking the measurement immediately after Emacs
-Q and with few samples, it skews the numbers.

> All averages made using 'M-x calc-grab-region' followed by 'u M'.

Wow, thanks for this tip.  I wondered if there was an easier way than
M-x cua-rectangle-mark-mode + hand rolled avg function.

> I just meant that your version will be easier to use in
> company--match-from-capf-face (because it works on individual
> completions).

Ah, that was my intution too.  I misunderstood you then, I thought you
meant the list version would be easier and I found that odd.

>>> Third, it made a principled stance to avoid altering the original
>>> strings, even the non-visual text properties. This approach could be
>>> adopted piecewise from Daniel's patch, especially if the performance
>>> ends up the same or insignificantly different in practical usage.
>> If we really wanted to, we could also adopt the non-propertizing
>> approach in my lazy-hilit patch, by calculating the score "just in
>> time", much like Daniel's patch does it.  But it should be clear that
>> what we save in allocation in completion-pcm--hilit-commonality, we'll
>> pay for in the same amount in consing later.  So no performance benefit.
>
> Not for performance, no. Although the way it separates the sorting
> into its own phase makes it easier to reason about that particular
> cost. And for 300000 symbols, scoring and sorting really take the most
> time, e.g. about 2/3rds. Which might help with optimizing it further
> down in the future, somehow,

I think further optimization would be localized to the scoring function
itself, not to the place where it is performed.

> But if course it would be nice to avoid the wart, so if you have any
> better ideas, they are welcome.

I'm not saying it would necessarily spread even further, but if you want
to do scoring "just in time" like I suggested -- presumably to
completely avoid propertizing strings -- that particular wart spreads a
little more and thus becomes something that is slightly harder to remove
later on.

> So far you advocated toward avoiding breaking changes while
> implementing the present performance improvement.

Both patches do that, so what I've been arguing for is simplicity and
coherence.  I don't think completion-sorted-completions and the
complexity it brings to minibuffer.el is a step in the rigth direction,
and the performance benefits it does undeniably bring can be achieved
with less drastic means.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

Hi Joao,

On 30/10/2023 01:12, João Távora wrote:

>> Thanks. My measurements are similar, except the difference switch the
>> other way a little bit. It might depend on the particulars of the
>> individual machine anyway.
> 
> Yes, it could, but I've reproduced this in different hardware.
> 
> Check that you're taking enough samples, I take about 15-20 samples.
> Maybe the lazy-hilit patch pays a an extra cost upfront for the very
> first C-h v or completing-read, to create the properties keys on the
> strings, which are then reused.

I actually do see that. At first I didn't pay much attention to such 
outliers. They usually look like the second measurement here:

Elapsed time: 0.541624s (0.164396s in 5 GCs)
Elapsed time: 0.861175s (0.415142s in 10 GCs)
Elapsed time: 0.486012s (0.057915s in 1 GCs)
Elapsed time: 0.505339s (0.055759s in 1 GCs)
Elapsed time: 0.481024s (0.057757s in 1 GCs)
Elapsed time: 0.471350s (0.056383s in 1 GCs)
Elapsed time: 0.495125s (0.056129s in 1 GCs)
Elapsed time: 0.513310s (0.058437s in 1 GCs)
Elapsed time: 0.491978s (0.057144s in 1 GCs)

Which keys are those? I only know about one key - 'completion-score'.

> This cost is ammortized very quickly,
> of course, but if you're taking the measurement immediately after Emacs
> -Q and with few samples, it skews the numbers.

I always took around 16 samples, and now made sure to take exactly that 
number, starting with "y" and then "yo", "y", etc. Although the timing 
for the empty input is usually included (but it doesn't look too 
different from the rest).

Anyway, I retook the numbers a couple of times. One of the patches (the 
same one) still looks a little faster, but the fluctuations between runs 
are large enough to avoid making any big conclusions:

d+d
300000
completing-read 0.504
C-h v 0.630

lazy-hilit-2023-v4
300000
completing-read 0.517
C-h v 0.661

d+d again
300000
completing-read 0.486
C-h v 0.587

lazy-hilit-2023-v4 again
300000
completing-read 0.519
C-h v 0.651

And to double-check, these are comparison between 
0001-Add-new-completion-filter-completions-API-and-deferr-v3.diff and 
lazy-hilit-2023-v4.diff.

16 samples every time. And I think I dropped the run with the "spike" in 
all or most of the above. The first of the patches doesn't seem to cause 
it, though.

>> All averages made using 'M-x calc-grab-region' followed by 'u M'.
> 
> Wow, thanks for this tip.  I wondered if there was an easier way than
> M-x cua-rectangle-mark-mode + hand rolled avg function.

No problem! I lost your snippet for avg, so had to google. :-/

rectangle-mark-mode (C-x SPC) is still part of the recipe, though.

>>>> Third, it made a principled stance to avoid altering the original
>>>> strings, even the non-visual text properties. This approach could be
>>>> adopted piecewise from Daniel's patch, especially if the performance
>>>> ends up the same or insignificantly different in practical usage.
>>> If we really wanted to, we could also adopt the non-propertizing
>>> approach in my lazy-hilit patch, by calculating the score "just in
>>> time", much like Daniel's patch does it.  But it should be clear that
>>> what we save in allocation in completion-pcm--hilit-commonality, we'll
>>> pay for in the same amount in consing later.  So no performance benefit.
>>
>> Not for performance, no. Although the way it separates the sorting
>> into its own phase makes it easier to reason about that particular
>> cost. And for 300000 symbols, scoring and sorting really take the most
>> time, e.g. about 2/3rds. Which might help with optimizing it further
>> down in the future, somehow,
> 
> I think further optimization would be localized to the scoring function
> itself, not to the place where it is performed.

Most likely, yes. It seems to be the most expensive part. But it still 
seems easier to measure/tweak when it happens during a separate step, 
rather than mixed in with the rest of completion-all-completions' business.

>> But if course it would be nice to avoid the wart, so if you have any
>> better ideas, they are welcome.
> 
> I'm not saying it would necessarily spread even further, but if you want
> to do scoring "just in time" like I suggested -- presumably to
> completely avoid propertizing strings -- that particular wart spreads a
> little more and thus becomes something that is slightly harder to remove
> later on.

Could you describe the other places you think it might spread to? Other 
completion styles like Orderless?

As long as there's only one place producing this property (as opposed to 
consuming it), it seems straightforward enough to remove anyway.

>> So far you advocated toward avoiding breaking changes while
>> implementing the present performance improvement.
> 
> Both patches do that, so what I've been arguing for is simplicity and
> coherence.  I don't think completion-sorted-completions and the
> complexity it brings to minibuffer.el is a step in the rigth direction,
> and the performance benefits it does undeniably bring can be achieved
> with less drastic means.

What I meant is, solving the quote-unquote conundrum might require a 
larger breaking change than the one that you wanted for this discussion.

Anyway, have you looked into what it would take to solve it? Such text 
propertization might actually work as a cheap replacement to returning 
"a function to ... requote them". If both highlighting and scoring 
functions work fine on the "unquoted" strings, then we would only need 
to make sure the "quoted" is used when the completion is inserted. Could 
we make a rule that every table-with-quoting would have to call a 
particular exit-function? Perhaps before the existing exit-function.

That might solve a bunch of things, though I don't see a robust way to 
enforce this practice, given that completion-table-with-quoting works on 
the level of completion tables, whereas :exit-function is only specified 
in the capf tuple.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Tue, Oct 31, 2023 at 3:20 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> Hi Joao,
>
> On 30/10/2023 01:12, João Távora wrote:
>
> >> Thanks. My measurements are similar, except the difference switch the
> >> other way a little bit. It might depend on the particulars of the
> >> individual machine anyway.
> >
> > Yes, it could, but I've reproduced this in different hardware.
> >
> > Check that you're taking enough samples, I take about 15-20 samples.
> > Maybe the lazy-hilit patch pays a an extra cost upfront for the very
> > first C-h v or completing-read, to create the properties keys on the
> > strings, which are then reused.
>
> I actually do see that. At first I didn't pay much attention to such
> outliers. They usually look like the second measurement here:
>
> Elapsed time: 0.541624s (0.164396s in 5 GCs)
> Elapsed time: 0.861175s (0.415142s in 10 GCs)
> Elapsed time: 0.486012s (0.057915s in 1 GCs)
> Elapsed time: 0.505339s (0.055759s in 1 GCs)
> Elapsed time: 0.481024s (0.057757s in 1 GCs)
> Elapsed time: 0.471350s (0.056383s in 1 GCs)
> Elapsed time: 0.495125s (0.056129s in 1 GCs)
> Elapsed time: 0.513310s (0.058437s in 1 GCs)
> Elapsed time: 0.491978s (0.057144s in 1 GCs)
>
> Which keys are those? I only know about one key - 'completion-score'.

Yes, that's the one.  I suppose adding this key to the property lists of
large number of strings, which is only done once, is what's causing
the anomaly.

>
> > This cost is ammortized very quickly,
> > of course, but if you're taking the measurement immediately after Emacs
> > -Q and with few samples, it skews the numbers.
>
> I always took around 16 samples, and now made sure to take exactly that
> number, starting with "y" and then "yo", "y", etc. Although the timing
> for the empty input is usually included (but it doesn't look too
> different from the rest).
>
> Anyway, I retook the numbers a couple of times. One of the patches (the
> same one) still looks a little faster, but the fluctuations between runs
> are large enough to avoid making any big conclusions:

As did I, and I get the same results I posted :-/

> > I think further optimization would be localized to the scoring function
> > itself, not to the place where it is performed.
>
> Most likely, yes. It seems to be the most expensive part. But it still
> seems easier to measure/tweak when it happens during a separate step,
> rather than mixed in with the rest of completion-all-completions' business.
>
> >> But if course it would be nice to avoid the wart, so if you have any
> >> better ideas, they are welcome.
> >
> > I'm not saying it would necessarily spread even further, but if you want
> > to do scoring "just in time" like I suggested -- presumably to
> > completely avoid propertizing strings -- that particular wart spreads a
> > little more and thus becomes something that is slightly harder to remove
> > later on.
>
> Could you describe the other places you think it might spread to? Other
> completion styles like Orderless?

Maybe, I don't know.  But here I just meant that to do that idea it spreads
only one further degree.  I'm not saying it would necessarily spread even
further.

>
> As long as there's only one place producing this property (as opposed to
> consuming it), it seems straightforward enough to remove anyway.
>
> >> So far you advocated toward avoiding breaking changes while
> >> implementing the present performance improvement.
> >
> > Both patches do that, so what I've been arguing for is simplicity and
> > coherence.  I don't think completion-sorted-completions and the
> > complexity it brings to minibuffer.el is a step in the rigth direction,
> > and the performance benefits it does undeniably bring can be achieved
> > with less drastic means.
>
> What I meant is, solving the quote-unquote conundrum might require a
> larger breaking change than the one that you wanted for this discussion.

A larger change sure, not sure if "breaking" though.

> Anyway, have you looked into what it would take to solve it?

No, naively, I just think it's a similar situation of display and business
logic being mixed up.  Presumably the quoted stuff is just for insertion
(and display?), and the unquoted stuff is what patterns/scoring should
operate on.

But, IMO, there's no need to tackle it right now.

If the thing holding you back from the lazy-hilit-2023-v4.patch is the
completion-score propertization, I can move it to the sorting step
in a future v5 and add spread the completion--unquoted thing a little
bit more.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 31/10/2023 12:55, João Távora wrote:
>> Which keys are those? I only know about one key - 'completion-score'.
> 
> Yes, that's the one.  I suppose adding this key to the property lists of
> large number of strings, which is only done once, is what's causing
> the anomaly.

Ah, I didn't realize that the text property/value pairs were stored as 
plists internally as well. And that also triggers consing.

>>>> But if course it would be nice to avoid the wart, so if you have any
>>>> better ideas, they are welcome.
>>>
>>> I'm not saying it would necessarily spread even further, but if you want
>>> to do scoring "just in time" like I suggested -- presumably to
>>> completely avoid propertizing strings -- that particular wart spreads a
>>> little more and thus becomes something that is slightly harder to remove
>>> later on.
>>
>> Could you describe the other places you think it might spread to? Other
>> completion styles like Orderless?
> 
> Maybe, I don't know.  But here I just meant that to do that idea it spreads
> only one further degree.  I'm not saying it would necessarily spread even
> further.

It seems like the only code that would be concerned with it are 
completion styles that also do sorting, or completion tables that would 
do similar things to this "with quoting" business. But I'm not aware of 
any other examples of the latter aside from what is inside Emacs itself.

>> Anyway, have you looked into what it would take to solve it?
> 
> No, naively, I just think it's a similar situation of display and business
> logic being mixed up.  Presumably the quoted stuff is just for insertion
> (and display?), and the unquoted stuff is what patterns/scoring should
> operate on.

Apparently it's good for insertion, but according to that comment inside 
the function, the unquoted stuff might actually be better for display.

I'm not 100% clear which of the versions is better for 
scoring/highlighting, but apparently the unquoted one.

> But, IMO, there's no need to tackle it right now.
> 
> If the thing holding you back from the lazy-hilit-2023-v4.patch is the
> completion-score propertization, I can move it to the sorting step
> in a future v5 and add spread the completion--unquoted thing a little
> bit more.

I think that's the main blocker, yes.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2226 bytes --]

On Tue, Oct 31, 2023 at 8:52 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> It seems like the only code that would be concerned with it are
> completion styles that also do sorting, or completion tables that would
> do similar things to this "with quoting" business. But I'm not aware of
> any other examples of the latter aside from what is inside Emacs itself.

If orderless (which I've never tried), does some kind of scoring of
completions, it probably also needs the same complications of flex.

> >> Anyway, have you looked into what it would take to solve it?
> >
> > No, naively, I just think it's a similar situation of display and business
> > logic being mixed up.  Presumably the quoted stuff is just for insertion
> > (and display?), and the unquoted stuff is what patterns/scoring should
> > operate on.
>
> Apparently it's good for insertion, but according to that comment inside
> the function, the unquoted stuff might actually be better for display.

No idea what the unquoted stuff is for, so I haven't really tested it.

> I'm not 100% clear which of the versions is better for
> scoring/highlighting, but apparently the unquoted one.
>
> > But, IMO, there's no need to tackle it right now.
> >
> > If the thing holding you back from the lazy-hilit-2023-v4.patch is the
> > completion-score propertization, I can move it to the sorting step
> > in a future v5 and add spread the completion--unquoted thing a little
> > bit more.
>
> I think that's the main blocker, yes.

Alright, here goes v5 then, with this change.  Note I've implemented
this unquoted thing which kicks in in C-x f but I haven't actually
seen  any strings that have different "quoted" "non-quoted" versions.

The performance of the three main patches as measured in yet
another machine:

;; C-h v
;;
;; Daniel+Dmitry: 0.696340454545
;; lazy hilit v4: 0.692849642852
;; lazy hilit v5: 0.683088541667
;;
;; completing-read
;;
;; Daniel+Dmitry: 0.590994909091
;; lazy hilit v4: 0.586523307692
;; lazy hilit v5: 0.586165466667

Nothing unexpected.

So if you're satisfied with the general design now, maybe
we should start looking at finer details, docstrings, style,
etc.

João

[-- Attachment #2: lazy-hilit-2023-v5.diff --]
[-- Type: application/octet-stream, Size: 19094 bytes --]

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..3e888c8b06a 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -722,7 +722,8 @@ icomplete-exhibit
              ;; Check if still in the right buffer (bug#61308)
              (or (window-minibuffer-p) completion-in-region--data)
              (icomplete-simple-completing-p)) ;Shouldn't be necessary.
-    (let ((saved-point (point)))
+    (let ((saved-point (point))
+          (completion-lazy-hilit t))
       (save-excursion
         (goto-char (icomplete--field-end))
         ;; Insert the match-status information:
@@ -754,12 +755,13 @@ icomplete-exhibit
                            (overlay-end rfn-eshadow-overlay)))
           (let* ((field-string (icomplete--field-string))
                  (text (while-no-input
+                         (benchmark-progn
                          (icomplete-completions
                           field-string
                           (icomplete--completion-table)
                           (icomplete--completion-predicate)
                           (if (window-minibuffer-p)
-                              (eq minibuffer--require-match t)))))
+                              (eq minibuffer--require-match t))))))
                  (buffer-undo-list t)
                  deactivate-mark)
             ;; Do nothing if while-no-input was aborted.
@@ -901,7 +903,7 @@ icomplete--render-vertical
                                 'icomplete-selected-match 'append comp)
      collect (concat prefix
                      (make-string (- max-prefix-len (length prefix)) ? )
-                     comp
+                     (completion-lazy-hilit comp)
                      (make-string (- max-comp-len (length comp)) ? )
                      suffix)
      into lines-aux
@@ -1067,7 +1069,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..2b0ff5c1c3c 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -677,6 +677,10 @@ completion--twq-all
                                              'completions-common-part)
                                qprefix))))
                         (qcompletion (concat qprefix qnew)))
+                   ;; Attach unquoted completion string, which is needed
+                   ;; to score the completion in `completion--flex-score'.
+                   (put-text-property 0 1 'completion--unquoted
+                                      completion qcompletion)
 		   ;; FIXME: Similarly here, Cygwin's mapping trips this
 		   ;; assertion.
                    ;;(cl-assert
@@ -1234,6 +1238,7 @@ completion-all-completions
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
 in the last `cdr'."
+  (setq completion-lazy-hilit-fn nil)
   ;; FIXME: We need to additionally return the info needed for the
   ;; second part of completion-base-position.
   (completion--nth-completion 2 string table pred point metadata))
@@ -3749,108 +3754,200 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request completion lazy highlighting.
+
+Completion-presenting frontends may opt to bind this variable to
+non-nil value in the context of completion-producing calls (such
+as `completion-all-sorted-completions').  This hints the
+intervening completion styles that they do not need to
+fontify (i.e. propertize with the `face' property) completion
+strings with highlights of the matching parts.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible this fontification.  The frontend binds this variable
+to non-nil, and calls the function with the same name
+`completion-lazy-hilit' on each completion string that is to be
+displayed to the user.
+
+Note that only some completion styles take advantage of this
+variable for optimization purposes.  Other styles will ignore the
+hint and greedily fontify as usual.  It is still safe for a
+frontend to call `completion-lazy-hilit' in these situations.
+
+To author a completion style that takes advantage see
+`completion-lazy-hilit-fn' and look in the source of
+`completion-pcm--hilit-commonality'.")
+
+(defvar completion-lazy-hilit-fn nil
+  "Used by completions styles to honouring `completion-lazy-hilit'.
+When a given style wants to enable support for
+`completion-lazy-hilit' (which see), that style should set this
+variable to a function of one argument, a fresh string to be
+displayed to the user.  The function is responsible for
+destructively highlighting the string.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (if (and completion-lazy-hilit completion-lazy-hilit-fn)
+      (funcall completion-lazy-hilit-fn (copy-sequence str))
+    str))
+
+(defun completion--hilit-from-re (string regexp)
+  "Fontify STRING with `completions-common-part' using REGEXP."
+  (let* ((md (and regexp (string-match regexp string) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil string)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil string))
+    string))
+
+(defun completion--flex-score-1 (md-groups match-end len)
+  "Compute matching score of completion.
+The score lies in the range between 0 and 1, where 1 corresponds to
+the full match.
+MD-GROUPS is the \"group\"  part of the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by (0..1] (i.e., larger than (but not equal
+         ;; to) zero, and smaller or equal to one): the higher
+         ;; the better and only a perfect match (pattern equals
+         ;; string) will have score 1.  The formula takes the
+         ;; form of a quotient.  For the numerator, we use the
+         ;; number of +, i.e. the length of the pattern.  For
+         ;; the denominator, it first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while (and md-groups (car md-groups))
+      (let ((a from)
+            (b (pop md-groups)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md-groups)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (/ score-numerator (* len (1+ score-denominator)) 1.0)))
+
+(defvar completion--flex-score-last-md nil
+  "Helper variable for `completion--flex-score'.")
+
+(defun completion--flex-score (str re &optional dont-error)
+  "Compute flex score of completion STR based on RE.
+If DONT-ERROR, just return nil if RE doesn't match STR."
+  (cond ((string-match re str)
+         (let* ((match-end (match-end 0))
+                (md (cddr
+                     (setq
+                      completion--flex-score-last-md
+                      (match-data t completion--flex-score-last-md)))))
+           (completion--flex-score-1 md match-end (length str))))
+        ((not dont-error)
+         (error "Internal error: %s does not match %s" re str))))
+
+(defvar completion-pcm--regexp nil
+  "Regexp from PCM pattern in `completion-pcm--hilit-commonality'.")
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
-string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
-`completions-first-difference' in the relevant segments."
+string in COMPLETIONS.
+
+If `completion-lazy-hilit' is nil, return a deep copy of
+COMPLETIONS where each string is propertized with
+`completion-score', a number between 0 and 1, and with faces
+`completions-common-part', `completions-first-difference' in the
+relevant segments.
+
+Else, if `completion-lazy-hilit' is t, return COMPLETIONS
+unchanged, but setup a suitable `completion-lazy-hilit-fn' (which
+see) for later lazy highlighting"
+  (setq completion-pcm--regexp nil
+        completion-lazy-hilit-fn nil)
   (cond
    ((and completions (cl-loop for e in pattern thereis (stringp e)))
     (let* ((re (completion-pcm--pattern->regex pattern 'group))
-           (point-idx (completion-pcm--pattern-point-idx pattern))
-           (case-fold-search completion-ignore-case)
-           last-md)
-      (mapcar
-       (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
-         (unless (string-match re str)
-           (error "Internal error: %s does not match %s" re str))
-         (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
-                (match-end (match-end 0))
-                (md (cddr (setq last-md (match-data t last-md))))
-                (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by (0..1] (i.e., larger than (but not equal
-                ;; to) zero, and smaller or equal to one): the higher
-                ;; the better and only a perfect match (pattern equals
-                ;; string) will have score 1.  The formula takes the
-                ;; form of a quotient.  For the numerator, we use the
-                ;; number of +, i.e. the length of the pattern.  For
-                ;; the denominator, it first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
-             (setq from (pop md)))
-           ;; If `pattern' doesn't have an explicit trailing any, the
-           ;; regex `re' won't produce match data representing the
-           ;; region after the match.  We need to account to account
-           ;; for that extra bit of match (bug#42149).
-           (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
-       completions)))
+           (score (lambda (str)
+                    (put-text-property 0 1 'completion-score
+                                       (completion--flex-score str re)
+                                       str))))
+      (cond (completion-lazy-hilit
+             (setq completion-lazy-hilit-fn
+                   (lambda (str) (completion--hilit-from-re str re))
+                   completion-pcm--regexp re)
+             completions)
+            (t
+             (mapcar
+              (lambda (str)
+                (setq str (copy-sequence str))
+                (funcall score str)
+                (completion--hilit-from-re str re)
+                str)
+              completions)))))
    (t completions)))
 
 (defun completion-pcm--find-all-completions (string table pred point
@@ -4201,15 +4298,44 @@ completion--flex-adjust-metadata
         (existing-csf
          (completion-metadata-get metadata 'cycle-sort-function)))
     (cl-flet
-        ((compose-flex-sort-fn
-          (existing-sort-fn) ; wish `cl-flet' had proper indentation...
-          (lambda (completions)
-            (sort
-             (funcall existing-sort-fn completions)
-             (lambda (c1 c2)
-               (let ((s1 (get-text-property 0 'completion-score c1))
-                     (s2 (get-text-property 0 'completion-score c2)))
-                 (> (or s1 0) (or s2 0))))))))
+        ((compose-flex-sort-fn (existing-sort-fn)
+           (lambda (completions)
+             (let ((pre-sorted (funcall existing-sort-fn completions)))
+               (cond (;; There's no useful scoring to apply, since the
+                      ;; pattern is empty
+                      (null completion-pcm--regexp)
+                      pre-sorted)
+                     (completion-lazy-hilit
+                      ;; Lazy highlight has been requested, so do the
+                      ;; scoring and sorting now.
+                      (let* ((sorted (sort
+                                      (mapcar
+                                       (lambda (str)
+                                         (cons
+                                          (- (completion--flex-score
+                                              (or (get-text-property
+                                                   0 'completion--unquoted str)
+                                                  str)
+                                              completion-pcm--regexp))
+                                          str))
+                                       pre-sorted)
+                                      #'car-less-than-car))
+                             (cell sorted))
+                        ;; Reuse the list
+                        (while cell
+                          (setcar cell (cdar cell))
+                          (pop cell))
+                        sorted))
+                     (t
+                      ;; Lazy highlight not requested, so strings are
+                      ;; assumed to already contain `completion-score'
+                      ;; (and highlighting) and we can freely destroy
+                      ;; list.
+                      (sort
+                       pre-sorted
+                       (lambda (c1 c2)
+                         (> (or (get-text-property 0 'completion-score c1) 0)
+                            (or (get-text-property 0 'completion-score c2) 0))))))))))
       `(metadata
         ,@(and flex-is-filtering-p
                `((display-sort-function

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 01/11/2023 20:47, João Távora wrote:
> On Tue, Oct 31, 2023 at 8:52 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> It seems like the only code that would be concerned with it are
>> completion styles that also do sorting, or completion tables that would
>> do similar things to this "with quoting" business. But I'm not aware of
>> any other examples of the latter aside from what is inside Emacs itself.
> 
> If orderless (which I've never tried), does some kind of scoring of
> completions, it probably also needs the same complications of flex.

Turns out, Orderless doesn't do any scoring or sorting. Only filtering.

>>>> Anyway, have you looked into what it would take to solve it?
>>>
>>> No, naively, I just think it's a similar situation of display and business
>>> logic being mixed up.  Presumably the quoted stuff is just for insertion
>>> (and display?), and the unquoted stuff is what patterns/scoring should
>>> operate on.
>>
>> Apparently it's good for insertion, but according to that comment inside
>> the function, the unquoted stuff might actually be better for display.
> 
> No idea what the unquoted stuff is for, so I haven't really tested it.

A couple of scenarios:

First:
1. sudo mkdir /home/${USER}-foobar
2. C-x C-f /home/${USER} TAB ; it shows both directories inside home as

   ${USER}-foo/
   ${USER}/

Second:
1. mkdir ~/examples/test\ test\ test/
2. mkdir ~/examples/test\ test/
3. M-x shell
4. In the shell buffer, type 'ls ~/examples/test\ ' and TAB. See:

   test\ test/
   test\ test\ test/

In the current implementation, both the inputs and the text in the 
completions buffer that we see, are "quoted". The "unquoted" versions 
would be the directory name with the variable substitution performed, 
and the directory names without backslashes.

>> I'm not 100% clear which of the versions is better for
>> scoring/highlighting, but apparently the unquoted one.
>>
>>> But, IMO, there's no need to tackle it right now.
>>>
>>> If the thing holding you back from the lazy-hilit-2023-v4.patch is the
>>> completion-score propertization, I can move it to the sorting step
>>> in a future v5 and add spread the completion--unquoted thing a little
>>> bit more.
>>
>> I think that's the main blocker, yes.
> 
> Alright, here goes v5 then, with this change.  Note I've implemented
> this unquoted thing which kicks in in C-x f but I haven't actually
> seen  any strings that have different "quoted" "non-quoted" versions.
> 
> The performance of the three main patches as measured in yet
> another machine:
> 
> ;; C-h v
> ;;
> ;; Daniel+Dmitry: 0.696340454545
> ;; lazy hilit v4: 0.692849642852
> ;; lazy hilit v5: 0.683088541667
> ;;
> ;; completing-read
> ;;
> ;; Daniel+Dmitry: 0.590994909091
> ;; lazy hilit v4: 0.586523307692
> ;; lazy hilit v5: 0.586165466667
> 
> Nothing unexpected.

Confirm. The "property allocation" spikes are gone too.

> So if you're satisfied with the general design now, maybe
> we should start looking at finer details, docstrings, style,
> etc.

LGTM overall, and I see that you compressed the sorting code a little.

Both quoting/unquoting scenarios also seem to work as expected (for 
highlighting, that seems to be thanks to completion--twq-all applying 
the faces eagerly anyway).

Though given the examples (and I think others should be similar) it 
wouldn't be an end of the world if scoring didn't really work for them 
-- filtering should have already done most of the job. All of this is to 
say that any new 3rd party completion styles, even those that do 
sorting, would be okay without knowing about this text property.

Some minor nits for the patch:

 > +Completion-presenting frontends may opt to bind this variable to
 > +non-nil value in the context of completion-producing calls (such
 > +as `completion-all-sorted-completions').  This hints the

I suggest mentioning `completion-all-completions' instead, as it is more 
often used directly by the frontends.

 > +responsible this fontification.  The frontend binds this variable

responsible for

 > +hint and greedily fontify as usual.  It is still safe for a

"fontify eagerly"? I think that's a more common term than "greedily".

 > +  "Used by completions styles to honouring `completion-lazy-hilit'.

"to honour", or "styles honouring"

 > +(defun completion--flex-score (str re &optional dont-error)

Looks like the third argument is unused in both callers. I think it was 
intended for compose-flex-sort-fn.

 > +see) for later lazy highlighting"

Missing period.

 > +                      ;; Lazy highlight not requested, so strings are
 > +                      ;; assumed to already contain `completion-> score'
 > +                      ;; (and highlighting) and we can freely destroy
 > +                      ;; list.

Perhaps drop the last two lines, since IIUC the list can be 
destructively sorted in both cases, lazy highlighting or not.

I guess we should wait a few days to see if anyone has more comments, 
and then install this?

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]

On Wed, Nov 1, 2023 at 10:45 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > If orderless (which I've never tried), does some kind of scoring of
> > completions, it probably also needs the same complications of flex.
>
> Turns out, Orderless doesn't do any scoring or sorting. Only filtering.

Interesting, so if I M-x d i f f with orderless I don't get results in
any particular order?

> >>>> Anyway, have you looked into what it would take to solve it?
> >>>
> >>> No, naively, I just think it's a similar situation of display and business
> >>> logic being mixed up.  Presumably the quoted stuff is just for insertion
> >>> (and display?), and the unquoted stuff is what patterns/scoring should
> >>> operate on.
> >>
> >> Apparently it's good for insertion, but according to that comment inside
> >> the function, the unquoted stuff might actually be better for display.
> >
> > No idea what the unquoted stuff is for, so I haven't really tested it.
>
> A couple of scenarios:

Thanks.  Then I think it is working OK, but it would be safer if you were
to double-check yourself, as I really never use this functionality.

> LGTM overall, and I see that you compressed the sorting code a little.
>
> Both quoting/unquoting scenarios also seem to work as expected (for
> highlighting, that seems to be thanks to completion--twq-all applying
> the faces eagerly anyway).

That's good.

> Though given the examples (and I think others should be similar) it
> wouldn't be an end of the world if scoring didn't really work for them
> -- filtering should have already done most of the job. All of this is to
> say that any new 3rd party completion styles, even those that do
> sorting, would be okay without knowing about this text property.

Maybe.

> Some minor nits for the patch:

Thanks.

> I guess we should wait a few days to see if anyone has more comments,
> and then install this?

I addressed all your docstring suggestions, fixed a bug and significantly
simplified the code in the latest version of the patch.  I also
removed the instrumentation in icomplete.el.  Patch attached here
and pushed to feature/completion-lazy-hilit.

Stefan, Eli, would you like to chime in?

João

[-- Attachment #2: lazy-hilit-2023-v6.diff --]
[-- Type: application/octet-stream, Size: 18263 bytes --]

diff --git a/lisp/icomplete.el b/lisp/icomplete.el
index e6fdd1f1836..f4c4feb7304 100644
--- a/lisp/icomplete.el
+++ b/lisp/icomplete.el
@@ -722,7 +722,8 @@ icomplete-exhibit
              ;; Check if still in the right buffer (bug#61308)
              (or (window-minibuffer-p) completion-in-region--data)
              (icomplete-simple-completing-p)) ;Shouldn't be necessary.
-    (let ((saved-point (point)))
+    (let ((saved-point (point))
+          (completion-lazy-hilit t))
       (save-excursion
         (goto-char (icomplete--field-end))
         ;; Insert the match-status information:
@@ -901,7 +902,7 @@ icomplete--render-vertical
                                 'icomplete-selected-match 'append comp)
      collect (concat prefix
                      (make-string (- max-prefix-len (length prefix)) ? )
-                     comp
+                     (completion-lazy-hilit comp)
                      (make-string (- max-comp-len (length comp)) ? )
                      suffix)
      into lines-aux
@@ -1067,7 +1068,8 @@ icomplete-completions
                   (if (< prospects-len prospects-max)
                       (push comp prospects)
                     (setq limit t)))
-                (setq prospects (nreverse prospects))
+                (setq prospects
+                      (nreverse (mapcar #'completion-lazy-hilit prospects)))
                 ;; Decorate first of the prospects.
                 (when prospects
                   (let ((first (copy-sequence (pop prospects))))
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..ee0a547fe9b 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -677,6 +677,10 @@ completion--twq-all
                                              'completions-common-part)
                                qprefix))))
                         (qcompletion (concat qprefix qnew)))
+                   ;; Attach unquoted completion string, which is needed
+                   ;; to score the completion in `completion--flex-score'.
+                   (put-text-property 0 1 'completion--unquoted
+                                      completion qcompletion)
 		   ;; FIXME: Similarly here, Cygwin's mapping trips this
 		   ;; assertion.
                    ;;(cl-assert
@@ -1234,6 +1238,7 @@ completion-all-completions
 POINT is the position of point within STRING.
 The return value is a list of completions and may contain the base-size
 in the last `cdr'."
+  (setq completion-lazy-hilit-fn nil)
   ;; FIXME: We need to additionally return the info needed for the
   ;; second part of completion-base-position.
   (completion--nth-completion 2 string table pred point metadata))
@@ -3749,108 +3754,193 @@ flex-score-match-tightness
 than the latter (which has two \"holes\" and three
 one-letter-long matches).")
 
+(defvar-local completion-lazy-hilit nil
+  "If non-nil, request completion lazy highlighting.
+
+Completion-presenting frontends may opt to bind this variable to
+non-nil value in the context of completion-producing calls (such
+as `completion-all-completions').  This hints the intervening
+completion styles that they do not need to
+fontify (i.e. propertize with the `face' property) completion
+strings with highlights of the matching parts.
+
+When doing so, it is the frontend -- not the style -- who becomes
+responsible for this fontification.  The frontend binds this
+variable to non-nil, and calls the function with the same name
+`completion-lazy-hilit' on each completion string that is to be
+displayed to the user.
+
+Note that only some completion styles take advantage of this
+variable for optimization purposes.  Other styles will ignore the
+hint and fontify eagerly as usual.  It is still safe for a
+frontend to call `completion-lazy-hilit' in these situations.
+
+To author a completion style that takes advantage see
+`completion-lazy-hilit-fn' and look in the source of
+`completion-pcm--hilit-commonality'.")
+
+(defvar completion-lazy-hilit-fn nil
+  "Used by completions styles honoring `completion-lazy-hilit'.
+When a given style wants to enable support for
+`completion-lazy-hilit' (which see), that style should set this
+variable to a function of one argument, a fresh string to be
+displayed to the user.  The function is responsible for
+destructively highlighting the string.")
+
+(defun completion-lazy-hilit (str)
+  "Return a copy of completion STR that is `face'-propertized.
+See documentation for variable `completion-lazy-hilit' for more
+details."
+  (if (and completion-lazy-hilit completion-lazy-hilit-fn)
+      (funcall completion-lazy-hilit-fn (copy-sequence str))
+    str))
+
+(defun completion--hilit-from-re (string regexp)
+  "Fontify STRING with `completions-common-part' using REGEXP."
+  (let* ((md (and regexp (string-match regexp string) (cddr (match-data t))))
+         (me (and md (match-end 0)))
+         (from 0))
+    (while md
+      (add-face-text-property from (pop md) 'completions-common-part nil string)
+      (setq from (pop md)))
+    (unless (or (not me) (= from me))
+      (add-face-text-property from me 'completions-common-part nil string))
+    string))
+
+(defun completion--flex-score-1 (md-groups match-end len)
+  "Compute matching score of completion.
+The score lies in the range between 0 and 1, where 1 corresponds to
+the full match.
+MD-GROUPS is the \"group\"  part of the match data.
+MATCH-END is the end of the match.
+LEN is the length of the completion string."
+  (let* ((from 0)
+         ;; To understand how this works, consider these simple
+         ;; ascii diagrams showing how the pattern "foo"
+         ;; flex-matches "fabrobazo", "fbarbazoo" and
+         ;; "barfoobaz":
+
+         ;;      f abr o baz o
+         ;;      + --- + --- +
+
+         ;;      f barbaz oo
+         ;;      + ------ ++
+
+         ;;      bar foo baz
+         ;;          +++
+
+         ;; "+" indicates parts where the pattern matched.  A
+         ;; "hole" in the middle of the string is indicated by
+         ;; "-".  Note that there are no "holes" near the edges
+         ;; of the string.  The completion score is a number
+         ;; bound by (0..1] (i.e., larger than (but not equal
+         ;; to) zero, and smaller or equal to one): the higher
+         ;; the better and only a perfect match (pattern equals
+         ;; string) will have score 1.  The formula takes the
+         ;; form of a quotient.  For the numerator, we use the
+         ;; number of +, i.e. the length of the pattern.  For
+         ;; the denominator, it first computes
+         ;;
+         ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
+         ;;
+         ;; , for each hole "i" of length "Li", where tightness
+         ;; is given by `flex-score-match-tightness'.  The
+         ;; final value for the denominator is then given by:
+         ;;
+         ;;    (SUM_across_i(hole_i_contrib) + 1) * len
+         ;;
+         ;; , where "len" is the string's length.
+         (score-numerator 0)
+         (score-denominator 0)
+         (last-b 0))
+    (while (and md-groups (car md-groups))
+      (let ((a from)
+            (b (pop md-groups)))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b))
+      (setq from (pop md-groups)))
+    ;; If `pattern' doesn't have an explicit trailing any, the
+    ;; regex `re' won't produce match data representing the
+    ;; region after the match.  We need to account to account
+    ;; for that extra bit of match (bug#42149).
+    (unless (= from match-end)
+      (let ((a from)
+            (b match-end))
+        (setq
+         score-numerator   (+ score-numerator (- b a)))
+        (unless (or (= a last-b)
+                    (zerop last-b)
+                    (= a len))
+          (setq
+           score-denominator (+ score-denominator
+                                1
+                                (expt (- a last-b 1)
+                                      (/ 1.0
+                                         flex-score-match-tightness)))))
+        (setq
+         last-b              b)))
+    (/ score-numerator (* len (1+ score-denominator)) 1.0)))
+
+(defvar completion--flex-score-last-md nil
+  "Helper variable for `completion--flex-score'.")
+
+(defun completion--flex-score (str re &optional dont-error)
+  "Compute flex score of completion STR based on RE.
+If DONT-ERROR, just return nil if RE doesn't match STR."
+  (cond ((string-match re str)
+         (let* ((match-end (match-end 0))
+                (md (cddr
+                     (setq
+                      completion--flex-score-last-md
+                      (match-data t completion--flex-score-last-md)))))
+           (completion--flex-score-1 md match-end (length str))))
+        ((not dont-error)
+         (error "Internal error: %s does not match %s" re str))))
+
+(defvar completion-pcm--regexp nil
+  "Regexp from PCM pattern in `completion-pcm--hilit-commonality'.")
+
 (defun completion-pcm--hilit-commonality (pattern completions)
   "Show where and how well PATTERN matches COMPLETIONS.
 PATTERN, a list of symbols and strings as seen
 `completion-pcm--merge-completions', is assumed to match every
-string in COMPLETIONS.  Return a deep copy of COMPLETIONS where
-each string is propertized with `completion-score', a number
-between 0 and 1, and with faces `completions-common-part',
-`completions-first-difference' in the relevant segments."
+string in COMPLETIONS.
+
+If `completion-lazy-hilit' is nil, return a deep copy of
+COMPLETIONS where each string is propertized with
+`completion-score', a number between 0 and 1, and with faces
+`completions-common-part', `completions-first-difference' in the
+relevant segments.
+
+Else, if `completion-lazy-hilit' is t, return COMPLETIONS
+unchanged, but setup a suitable `completion-lazy-hilit-fn' (which
+see) for later lazy highlighting."
+  (setq completion-pcm--regexp nil
+        completion-lazy-hilit-fn nil)
   (cond
    ((and completions (cl-loop for e in pattern thereis (stringp e)))
-    (let* ((re (completion-pcm--pattern->regex pattern 'group))
-           (point-idx (completion-pcm--pattern-point-idx pattern))
-           (case-fold-search completion-ignore-case)
-           last-md)
-      (mapcar
-       (lambda (str)
-	 ;; Don't modify the string itself.
-         (setq str (copy-sequence str))
-         (unless (string-match re str)
-           (error "Internal error: %s does not match %s" re str))
-         (let* ((pos (if point-idx (match-beginning point-idx) (match-end 0)))
-                (match-end (match-end 0))
-                (md (cddr (setq last-md (match-data t last-md))))
-                (from 0)
-                (end (length str))
-                ;; To understand how this works, consider these simple
-                ;; ascii diagrams showing how the pattern "foo"
-                ;; flex-matches "fabrobazo", "fbarbazoo" and
-                ;; "barfoobaz":
-
-                ;;      f abr o baz o
-                ;;      + --- + --- +
-
-                ;;      f barbaz oo
-                ;;      + ------ ++
-
-                ;;      bar foo baz
-                ;;          +++
-
-                ;; "+" indicates parts where the pattern matched.  A
-                ;; "hole" in the middle of the string is indicated by
-                ;; "-".  Note that there are no "holes" near the edges
-                ;; of the string.  The completion score is a number
-                ;; bound by (0..1] (i.e., larger than (but not equal
-                ;; to) zero, and smaller or equal to one): the higher
-                ;; the better and only a perfect match (pattern equals
-                ;; string) will have score 1.  The formula takes the
-                ;; form of a quotient.  For the numerator, we use the
-                ;; number of +, i.e. the length of the pattern.  For
-                ;; the denominator, it first computes
-                ;;
-                ;;     hole_i_contrib = 1 + (Li-1)^(1/tightness)
-                ;;
-                ;; , for each hole "i" of length "Li", where tightness
-                ;; is given by `flex-score-match-tightness'.  The
-                ;; final value for the denominator is then given by:
-                ;;
-                ;;    (SUM_across_i(hole_i_contrib) + 1) * len
-                ;;
-                ;; , where "len" is the string's length.
-                (score-numerator 0)
-                (score-denominator 0)
-                (last-b 0)
-                (update-score-and-face
-                 (lambda (a b)
-                   "Update score and face given match range (A B)."
-                   (add-face-text-property a b
-                                           'completions-common-part
-                                           nil str)
-                   (setq
-                    score-numerator   (+ score-numerator (- b a)))
-                   (unless (or (= a last-b)
-                               (zerop last-b)
-                               (= a (length str)))
-                     (setq
-                      score-denominator (+ score-denominator
-                                           1
-                                           (expt (- a last-b 1)
-                                                 (/ 1.0
-                                                    flex-score-match-tightness)))))
-                   (setq
-                    last-b              b))))
-           (while md
-             (funcall update-score-and-face from (pop md))
-             (setq from (pop md)))
-           ;; If `pattern' doesn't have an explicit trailing any, the
-           ;; regex `re' won't produce match data representing the
-           ;; region after the match.  We need to account to account
-           ;; for that extra bit of match (bug#42149).
-           (unless (= from match-end)
-             (funcall update-score-and-face from match-end))
-           (if (> (length str) pos)
-               (add-face-text-property
-                pos (1+ pos)
-                'completions-first-difference
-                nil str))
-           (unless (zerop (length str))
-             (put-text-property
-              0 1 'completion-score
-              (/ score-numerator (* end (1+ score-denominator)) 1.0) str)))
-         str)
-       completions)))
+    (let* ((re (completion-pcm--pattern->regex pattern 'group)))
+      (setq completion-pcm--regexp re)
+      (cond (completion-lazy-hilit
+             (setq completion-lazy-hilit-fn
+                   (lambda (str) (completion--hilit-from-re str re)))
+             completions)
+            (t
+             (mapcar
+              (lambda (str)
+                (completion--hilit-from-re (copy-sequence str) re))
+              completions)))))
    (t completions)))
 
 (defun completion-pcm--find-all-completions (string table pred point
@@ -4187,36 +4277,39 @@ completion-flex-nospace
 
 (defun completion--flex-adjust-metadata (metadata)
   "If `flex' is actually doing filtering, adjust sorting."
-  (let ((flex-is-filtering-p
-         ;; JT@2019-12-23: FIXME: this is kinda wrong.  What we need
-         ;; to test here is "some input that actually leads/led to
-         ;; flex filtering", not "something after the minibuffer
-         ;; prompt".  E.g. The latter is always true for file
-         ;; searches, meaning we'll be doing extra work when we
-         ;; needn't.
-         (or (not (window-minibuffer-p))
-             (> (point-max) (minibuffer-prompt-end))))
+  (let ((flex-is-filtering-p completion-pcm--regexp)
         (existing-dsf
          (completion-metadata-get metadata 'display-sort-function))
         (existing-csf
          (completion-metadata-get metadata 'cycle-sort-function)))
     (cl-flet
-        ((compose-flex-sort-fn
-          (existing-sort-fn) ; wish `cl-flet' had proper indentation...
-          (lambda (completions)
-            (sort
-             (funcall existing-sort-fn completions)
-             (lambda (c1 c2)
-               (let ((s1 (get-text-property 0 'completion-score c1))
-                     (s2 (get-text-property 0 'completion-score c2)))
-                 (> (or s1 0) (or s2 0))))))))
+        ((compose-flex-sort-fn (existing-sort-fn)
+           (lambda (completions)
+             (let* ((sorted (sort
+                             (mapcar
+                              (lambda (str)
+                                (cons
+                                 (- (completion--flex-score
+                                     (or (get-text-property
+                                          0 'completion--unquoted str)
+                                         str)
+                                     completion-pcm--regexp))
+                                 str))
+                              (if existing-sort-fn
+                                  (funcall existing-sort-fn completions)
+                                completions))
+                             #'car-less-than-car))
+                    (cell sorted))
+               ;; Reuse the list
+               (while cell
+                 (setcar cell (cdar cell))
+                 (pop cell))
+               sorted))))
       `(metadata
         ,@(and flex-is-filtering-p
-               `((display-sort-function
-                  . ,(compose-flex-sort-fn (or existing-dsf #'identity)))))
+               `((display-sort-function . ,(compose-flex-sort-fn existing-dsf))))
         ,@(and flex-is-filtering-p
-               `((cycle-sort-function
-                  . ,(compose-flex-sort-fn (or existing-csf #'identity)))))
+               `((cycle-sort-function . ,(compose-flex-sort-fn existing-csf))))
         ,@(cdr metadata)))))
 
 (defun completion-flex--make-flex-pattern (pattern)

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 2 Nov 2023 09:48:51 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, Daniel Mendler <mail@daniel-mendler.de>, 
> 	Stefan Monnier <monnier@iro.umontreal.ca>, 47711@debbugs.gnu.org
> 
> Stefan, Eli, would you like to chime in?

Chime in on what aspect(s) of this discussion (or the patch)?

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Thu, Nov 2, 2023 at 10:11 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: João Távora <joaotavora@gmail.com>
> > Date: Thu, 2 Nov 2023 09:48:51 +0000
> > Cc: Eli Zaretskii <eliz@gnu.org>, Daniel Mendler <mail@daniel-mendler.de>,
> >       Stefan Monnier <monnier@iro.umontreal.ca>, 47711@debbugs.gnu.org
> >
> > Stefan, Eli, would you like to chime in?
>
> Chime in on what aspect(s) of this discussion (or the patch)?

The patch condenses the results of the discussion of last week,
as resuscitated by Dmitry after a 2-year long hiatus.

After some rounds of benchmarking and discussion, Dmitry and I
think the latest version of the patch should be installed.

If you've not been following closely, the variables the patch
introduces have docstrings explaining the functionality added, as
do the commit messages to feature/completion-lazy-hilit.

In a nutshell it solves the performance problem of overly eager
completion highlighting with minimal changes to the completion API.
The original patch proposed by Daniel Mendler was also partly about
solving this problem, but with much more extensive changes.

NEWS entries -- and possibly manual entries -- are also needed, not
done yet.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Thu, 2 Nov 2023 10:39:53 +0000
> Cc: dmitry@gutov.dev, mail@daniel-mendler.de, monnier@iro.umontreal.ca, 
> 	47711@debbugs.gnu.org
> 
> > > Stefan, Eli, would you like to chime in?
> >
> > Chime in on what aspect(s) of this discussion (or the patch)?
> 
> The patch condenses the results of the discussion of last week,
> as resuscitated by Dmitry after a 2-year long hiatus.
> 
> After some rounds of benchmarking and discussion, Dmitry and I
> think the latest version of the patch should be installed.

Are there any problematic aspects of the patch that need to be
discussed or considered before installing the patch?

IOW, why are you soliciting our opinions, instead of just going ahead
and installing?

> In a nutshell it solves the performance problem of overly eager
> completion highlighting with minimal changes to the completion API.

It looks to me like it adds a new feature, not just solves a
performance problem?

Some minor comments to the patch itself:

> +(defvar-local completion-lazy-hilit nil
> +  "If non-nil, request completion lazy highlighting.
> +
> +Completion-presenting frontends may opt to bind this variable to
> +non-nil value in the context of completion-producing calls (such
> +as `completion-all-completions').  This hints the intervening
> +completion styles that they do not need to
> +fontify (i.e. propertize with the `face' property) completion
> +strings with highlights of the matching parts.

If this is intended to be bound by frontends, why is it defvar-local?
I thought let-binding buffer-local variables is a tricky business that
could have unexpected results?

Also, I think this doc string should reference
completion-lazy-hilit-fn.

> +(defvar completion-lazy-hilit-fn nil
> +  "Used by completions styles honoring `completion-lazy-hilit'.

This should mention "function", since just "Used to..." doesn't convey
that, and "-fn" could also mean "file name", not just "function".

> +(defun completion-lazy-hilit (str)
> +  "Return a copy of completion STR that is `face'-propertized.
                                              ^^^^^^^^^^^^^^^^^^
Strange quoting.  "face" is not a symbol that we want to have a link
to, is it?

I see a few more places in the doc strings that will "need work", but
that can be done later.

What did you want to say in NEWS about this?  If it's just a
performance improvement, we don't normally mention them in NEWS.

Thanks.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Thu, Nov 2, 2023 at 10:58 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > After some rounds of benchmarking and discussion, Dmitry and I
> > think the latest version of the patch should be installed.
>
> Are there any problematic aspects of the patch that need to be
> discussed or considered before installing the patch?
>
> IOW, why are you soliciting our opinions, instead of just going ahead
> and installing?

No particular reason.  Dmitry suggested that we do, and you
participated in this discussion a while back, and I know you
normally have some useful comment or two.

> > In a nutshell it solves the performance problem of overly eager
> > completion highlighting with minimal changes to the completion API.
>
> It looks to me like it adds a new feature, not just solves a
> performance problem?
>
> Some minor comments to the patch itself:
>
> > +(defvar-local completion-lazy-hilit nil
> > +  "If non-nil, request completion lazy highlighting.
> > +
> > +Completion-presenting frontends may opt to bind this variable to
> > +non-nil value in the context of completion-producing calls (such
> > +as `completion-all-completions').  This hints the intervening
> > +completion styles that they do not need to
> > +fontify (i.e. propertize with the `face' property) completion
> > +strings with highlights of the matching parts.
>
> If this is intended to be bound by frontends, why is it defvar-local?
> I thought let-binding buffer-local variables is a tricky business that
> could have unexpected results?

Good catch!  It shouldn't be defvar-local indeed, not with this latest
version.  See, glad I called you ;-)

> Also, I think this doc string should reference
> completion-lazy-hilit-fn.
>
> > +(defvar completion-lazy-hilit-fn nil
> > +  "Used by completions styles honoring `completion-lazy-hilit'.
>
> This should mention "function", since just "Used to..." doesn't convey
> that, and "-fn" could also mean "file name", not just "function".

Makes sense.

> > +(defun completion-lazy-hilit (str)
> > +  "Return a copy of completion STR that is `face'-propertized.
>                                               ^^^^^^^^^^^^^^^^^^
> Strange quoting.  "face" is not a symbol that we want to have a link
> to, is it?

It's not a symbol we'll be referencing, but it's a symbol.  I'll
rewrite it.

>
> I see a few more places in the doc strings that will "need work", but
> that can be done later.
>
> What did you want to say in NEWS about this?  If it's just a
> performance improvement, we don't normally mention them in NEWS.

But it needs frontends to opt in, and that requires an non-breaking
addition to completion API.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 02/11/2023 11:48, João Távora wrote:
> On Wed, Nov 1, 2023 at 10:45 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>>> If orderless (which I've never tried), does some kind of scoring of
>>> completions, it probably also needs the same complications of flex.
>>
>> Turns out, Orderless doesn't do any scoring or sorting. Only filtering.
> 
> Interesting, so if I M-x d i f f with orderless I don't get results in
> any particular order?

Right.

And input "diff" actually results in prefix-only matching. But if the 
input contains space (or any other pre-configured delimiter character), 
then it's translated to a set of substring matches, performed in any 
order. So "diff mode" does not translate to ".*diff.*mode", but a more 
complex regexp with all permutations.

IOW, this whole approach results in stricter matching with fewer 
results, so a smarter sort isn't that necessary.

>>>>>> Anyway, have you looked into what it would take to solve it?
>>>>>
>>>>> No, naively, I just think it's a similar situation of display and business
>>>>> logic being mixed up.  Presumably the quoted stuff is just for insertion
>>>>> (and display?), and the unquoted stuff is what patterns/scoring should
>>>>> operate on.
>>>>
>>>> Apparently it's good for insertion, but according to that comment inside
>>>> the function, the unquoted stuff might actually be better for display.
>>>
>>> No idea what the unquoted stuff is for, so I haven't really tested it.
>>
>> A couple of scenarios:
> 
> Thanks.  Then I think it is working OK, but it would be safer if you were
> to double-check yourself, as I really never use this functionality.

They look okay, yes. I imagine Daniel also tested the 
'completion--unquoted' solution before offering his patches.

> I addressed all your docstring suggestions, fixed a bug and significantly
> simplified the code in the latest version of the patch.  I also
> removed the instrumentation in icomplete.el.  Patch attached here
> and pushed to feature/completion-lazy-hilit.

Thank you. I don't see that the bug was, but deferring the scoring even 
in the eager highlighting case looks sensible (why not, indeed, if the 
performance is no worse).

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Thu, Nov 2, 2023 at 2:40 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> IOW, this whole approach results in stricter matching with fewer
> results, so a smarter sort isn't that necessary.

Just curious, so in orderless, what do I type to quickly select
M-x vc-diff or M-x vc-version-diff or M-x vc-ediff?

In flex I just type "vcdiff" and these results normally bubble to the top.

> > I addressed all your docstring suggestions, fixed a bug and significantly
> > simplified the code in the latest version of the patch.  I also
> > removed the instrumentation in icomplete.el.  Patch attached here
> > and pushed to feature/completion-lazy-hilit.
>
> Thank you. I don't see that the bug was, but deferring the scoring even
> in the eager highlighting case looks sensible (why not, indeed, if the
> performance is no worse).

The bug was in the eager highlighting case.  Wasn't sorting at all.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 02/11/2023 17:24, João Távora wrote:
> On Thu, Nov 2, 2023 at 2:40 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> 
>> IOW, this whole approach results in stricter matching with fewer
>> results, so a smarter sort isn't that necessary.
> Just curious, so in orderless, what do I type to quickly select
> M-x vc-diff or M-x vc-version-diff or M-x vc-ediff?
> 
> In flex I just type "vcdiff" and these results normally bubble to the top.

I'm not really a user of it (yet?), but

"vc-dif" or "vc dif" matches the first one, and "vc-ver" or "vc vers" 
matches the second one. Not a lot of difference in the amount of typing, 
but a little more control on the part of the user.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Thu, Nov 2, 2023 at 3:36 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 02/11/2023 17:24, João Távora wrote:
> > On Thu, Nov 2, 2023 at 2:40 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> >
> >> IOW, this whole approach results in stricter matching with fewer
> >> results, so a smarter sort isn't that necessary.
> > Just curious, so in orderless, what do I type to quickly select
> > M-x vc-diff or M-x vc-version-diff or M-x vc-ediff?
> >
> > In flex I just type "vcdiff" and these results normally bubble to the top.
>
> I'm not really a user of it (yet?), but
>
> "vc-dif" or "vc dif" matches the first one, and "vc-ver" or "vc vers"
> matches the second one.

So "vc dif" doesn't also match 'vc-ediff' and 'vc-version-diff'?

> Not a lot of difference in the amount of typing,
> but a little more control on the part of the user.

That's debatable, I like to be able to type 'vcdiff' and
see all the commands that vc.el offers for diffing things.

But to each their own, of course
João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 02/11/2023 17:58, João Távora wrote:
> On Thu, Nov 2, 2023 at 3:36 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>>
>> On 02/11/2023 17:24, João Távora wrote:
>>> On Thu, Nov 2, 2023 at 2:40 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
>>>
>>>> IOW, this whole approach results in stricter matching with fewer
>>>> results, so a smarter sort isn't that necessary.
>>> Just curious, so in orderless, what do I type to quickly select
>>> M-x vc-diff or M-x vc-version-diff or M-x vc-ediff?
>>>
>>> In flex I just type "vcdiff" and these results normally bubble to the top.
>>
>> I'm not really a user of it (yet?), but
>>
>> "vc-dif" or "vc dif" matches the first one, and "vc-ver" or "vc vers"
>> matches the second one.
> 
> So "vc dif" doesn't also match 'vc-ediff' and 'vc-version-diff'?

It does, of course. I just described what I imagine is the more common 
scenario: continue typing until the command you want is at the top, so 
you don't have to reach for C-n/C-p or the arrow keys.

>> Not a lot of difference in the amount of typing,
>> but a little more control on the part of the user.
> 
> That's debatable, I like to be able to type 'vcdiff' and
> see all the commands that vc.el offers for diffing things.

That also works for Orderless and "vc dif".

And from that you can continue to "vc dif ver", which brings 
"vc-version-diff" to the top, that's something unique to Orderless, I 
suppose.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Thu, Nov 2, 2023 at 4:03 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > That's debatable, I like to be able to type 'vcdiff' and
> > see all the commands that vc.el offers for diffing things.
>
> That also works for Orderless and "vc dif".

Ah, but the order in which all these commands appear is
arbitrary, if I understand correctly.  It must be, since
there's no sorting, right?

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 02/11/2023 18:09, João Távora wrote:
> On Thu, Nov 2, 2023 at 4:03 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> 
>>> That's debatable, I like to be able to type 'vcdiff' and
>>> see all the commands that vc.el offers for diffing things.
>> That also works for Orderless and "vc dif".
> Ah, but the order in which all these commands appear is
> arbitrary, if I understand correctly.  It must be, since
> there's no sorting, right?

Seems so.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

Hi Stefan,

Sorry for the pause, let's continue this digression. Though perhaps it 
would be better moved to emacs-devel or somewhere else. ;-/

On 29/10/2023 06:41, Stefan Monnier wrote:
>> FWIW, this neat structure might not help too much: the most popular external
>> completion backend (the LSP language servers, collectively) don't accept
>> regexps or globs, they just send you the lists of completions available at
>> point. With the name matching method sometimes configurable per-server.
> 
> Largely agreed.  The main benefit tho is that you get just *one*
> pattern, rather than three (one being the prefix argument, the second
> being the `pred` arg which historically was so unused that it was abused
> to hold the directory for file-name completion, so lots of tables don't
> obey it, and the third being the `completion-regexp-list` that most coders
> forget, and those who don't end up regretting not forgetting when it was
> not meant for them), so it's much more clear.

The motivation all makes sense.

>> As such, the most useful methods currently are: 1) Emacs Regexp, 2) asking
>> server for whatever it thinks is suitable (the "backend" completion style).
> 
> For the backends: agreed.
> For the frontends (i.e. `completion-styles`), `glob` is the more useful
> one, I'd say (except for the "external" style, of course).
> 
> We might also want support for things like `or` and `and` patterns, but
> I haven't managed to fit them nicely in that structure :-(

Possibly, but which code would produce such patterns?

>> I would also probably want to standardize on the recommended type of TO
>> anyway: some of them are likely going to result in better performance
>> than others.
> 
> The TO is chosen by the specific completion table, based on what it can
> handle best.  So it should always be "optimal".

Sorry, I meant the recommended type of FROM. Because if the original 
caller passes an arbitrary regexp, it will often get turned into a pair 
with a predicate where the latter calls string-match-p.

And if the type of FROM is standardized, there likely would be no need 
for a four-way bidirectional conversion. Maybe just a helper that 
converts from the original "main" type into any of the available that is 
currently required.

>> So I guess it's also a way to make every completion table aware of PRED?
> 
> Note also that these `pred` patterns are expected to be exclusively
> looking at the string (they're used for `completion-styles` kind of
> functionality), so nothing like `file-directory-p` or `fboundp` kind of
> predicates here.

Would we consider those pred's "fast enough"?

I've done a little comparison of completion-regexp-list vs pred:

(setq sss (cl-loop repeat 300000 collect (symbol-name (gensym "yoyo"))))

(benchmark-run 10 (let* ((re "yo[^o]*o")
                          (completion-regexp-list (list re)))
                     (all-completions "" sss)))
;; => 0.60s


(benchmark-run 10 (let* ((re "yo[^o]*o"))
                     (all-completions "" sss
                              (lambda (s) (string-match-p re s)))))
;; => 1.14s

> The `pred`s used in things like `completing-read` and `read-file-name`
> would be handled elsewhere such as `completion-table-with-predicate`.
> This part is still up in the air, tho.
> 
>> That should work; though it might be hard to reach the same raw performance
>> as the current all-completions + completion-regexp-list.
> 
> I don't see why: currently my code actually uses `all-completions` and
> `completion-regexp-list`, so as long as the pattern can be turned into
> a regexp without requiring an additional PRED (that's usually the case), <...>

Right, I was talking about the possible exceptions.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Sorry for the pause, let's continue this digression. Though perhaps it would
> be better moved to emacs-devel or somewhere else. ;-/

I see this API as an experiment.  I have no idea if I'll like
the result.  It's definitely far from being something ready to submit as
a proposal for a new design.

>>> As such, the most useful methods currently are: 1) Emacs Regexp, 2) asking
>>> server for whatever it thinks is suitable (the "backend" completion style).
>> For the backends: agreed.
>> For the frontends (i.e. `completion-styles`), `glob` is the more useful
>> one, I'd say (except for the "external" style, of course).
>> We might also want support for things like `or` and `and` patterns, but
>> I haven't managed to fit them nicely in that structure :-(
> Possibly, but which code would produce such patterns?

The `or` pattern?  No idea :-)
The `and` pattern?  Well, the `orderless` style, for one.

But indeed, I'm not sure it'd be useful to handle things like or/and
directly in there rather than by using union/intersection on
the resulting completions.  It's just an aspect of the design
I considered and I noticed that I had trouble extending it in
that direction.

Obviously, the caller which needs to collect a set of matching
candidates always has a choice between using a more refined pattern
or using a simpler pattern (including various calls with various
different patterns).

>>> I would also probably want to standardize on the recommended type of TO
>>> anyway: some of them are likely going to result in better performance
>>> than others.
>> The TO is chosen by the specific completion table, based on what it can
>> handle best.  So it should always be "optimal".
> Sorry, I meant the recommended type of FROM. Because if the original caller
> passes an arbitrary regexp, it will often get turned into a pair with
> a predicate where the latter calls string-match-p.

The caller should use the most primitive pattern they can.

> And if the type of FROM is standardized, there likely would be no need for
> a four-way bidirectional conversion. Maybe just a helper that converts from
> the original "main" type into any of the available that is
> currently required.

Indeed the default method does:

  (cond
   ((eq to (car pattern)) (cons nil (cdr pattern)))
   ((eq 'glob to) (cl-call-next-method))
   (t
    ;; Most conversions can be performed by going through `glob'.
    (pcase-let* ((`(,gpred . ,glob)
                  (completion-pattern-convert 'glob pattern))
                 (`(,tpred . ,newpattern)
                  (completion-pattern-convert to glob)))
      (cons (or gpred tpred) newpattern)))))

>>> So I guess it's also a way to make every completion table aware of PRED?
>> Note also that these `pred` patterns are expected to be exclusively
>> looking at the string (they're used for `completion-styles` kind of
>> functionality), so nothing like `file-directory-p` or `fboundp` kind of
>> predicates here.
> Would we consider those pred's "fast enough"?

I don't know.  I haven't had a use for a `pred` pattern yet, to
be honest.  As for the predicates returned by
`completion-pattern-convert`, they're currently just fancy booleans
indicating if the returned pattern is faithful or not :-)
So I'm not sure those predicates will survive this experiment.

> (benchmark-run 10 (let* ((re "yo[^o]*o")
>                          (completion-regexp-list (list re)))
>                     (all-completions "" sss)))
> ;; => 0.60s
>
>
> (benchmark-run 10 (let* ((re "yo[^o]*o"))
>                     (all-completions "" sss
>                              (lambda (s) (string-match-p re s)))))
> ;; => 1.14s

>>> That should work; though it might be hard to reach the same raw performance
>>> as the current all-completions + completion-regexp-list.
>> I don't see why: currently my code actually uses `all-completions` and
>> `completion-regexp-list`, so as long as the pattern can be turned into
>> a regexp without requiring an additional PRED (that's usually the case), <...>
> Right, I was talking about the possible exceptions.

I don't know what you're getting at.


        Stefan

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Howard Melman]

Dmitry Gutov <dmitry@gutov.dev> writes:

> On 02/11/2023 18:09, João Távora wrote:
>> On Thu, Nov 2, 2023 at 4:03 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
>> 
>>>> That's debatable, I like to be able to type 'vcdiff' and
>>>> see all the commands that vc.el offers for diffing things.
>>> That also works for Orderless and "vc dif".
>> Ah, but the order in which all these commands appear is
>> arbitrary, if I understand correctly.  It must be, since
>> there's no sorting, right?
>
> Seems so.

FYI, it's true that orderless just does filtering, it's just a
completion-style and leaves sorting to the completion UI
(I use it with vertico, but others work too).

It's configurable so what people can input can vary a lot,
but the main feature is that each space-separated bit is used
in any order.  So in the above example "vc dif" and "dif vc"
would both work.  The second is unlikely in this example
but it's far more useful when searching for function or
variable names.

The other great feature is that each "word" can be evaluated
in different ways, I typically use them as regexps, but it's
also easy to add syntax.  It's common to make a leading ! in
a word mean "without".  So an input of "file !--" would
match all things that include file anywhere and doesn't
have -- anywhere.  "^rx- !--" matches everything in the
public rx API (well anything beginning with rx- without --).

-- 

Howard

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Wed, Nov 1, 2023 at 10:45 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> I guess we should wait a few days to see if anyone has more comments,
> and then install this?

Five days elapsed, and no more comments came in, so I addressed your
comments and Eli's and I pushed this to master as
dfffb91a70532ac0021648ba692336331cbe0499.

Thanks,
João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 06/11/2023 18:20, João Távora wrote:
> On Wed, Nov 1, 2023 at 10:45 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> 
>> I guess we should wait a few days to see if anyone has more comments,
>> and then install this?
> Five days elapsed, and no more comments came in, so I addressed your
> comments and Eli's and I pushed this to master as
> dfffb91a70532ac0021648ba692336331cbe0499.

Thanks!

And thanks to Daniel for the original proposal and design work.

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Mon, Nov 6, 2023 at 7:39 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 06/11/2023 18:20, João Távora wrote:
> > On Wed, Nov 1, 2023 at 10:45 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> >
> >> I guess we should wait a few days to see if anyone has more comments,
> >> and then install this?
> > Five days elapsed, and no more comments came in, so I addressed your
> > comments and Eli's and I pushed this to master as
> > dfffb91a70532ac0021648ba692336331cbe0499.

Here's another place where completion-lazy-hilit could be leveraged:

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index ca2b25415f1..bb2670bccf6 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -2067,7 +2067,7 @@ completion--insert-strings
         ;; when the caller uses tabs inside prefix.
         (setq colwidth (- colwidth (mod colwidth completion-tab-width))))
       (funcall (intern (format "completion--insert-%s" completions-format))
-               strings group-fun length wwidth colwidth columns))))
+               (mapcar #'completion-lazy-hilit strings) group-fun
length wwidth colwidth columns))))

 (defun completion--insert-horizontal (strings group-fun
                                               length wwidth
@@ -2378,6 +2378,7 @@ minibuffer-completion-help
          (end (or end (point-max)))
          (string (buffer-substring start end))
          (md (completion--field-metadata start))
+         (completion-lazy-hilit t)
          (completions (completion-all-completions
                        string
                        minibuffer-completion-table

Any objections?  Seems to speed it up when flex is the preferred
completion style outside icomplete.  These are average times
collected from the instrumentation of the above
completion-all-completions when doing a M-: (setq i TAB)

I just used my normal Emacs session for this.

with lazy hilit:      0.104536125
without lazy hilit:   0.172522571

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 07/11/2023 14:13, João Távora wrote:
> Any objections?  Seems to speed it up when flex is the preferred
> completion style outside icomplete.  These are average times
> collected from the instrumentation of the above
> completion-all-completions when doing a M-: (setq i TAB)
> 
> I just used my normal Emacs session for this.
> 
> with lazy hilit:      0.104536125
> without lazy hilit:   0.172522571

IIUC the problem with the default completion-at-point UI here is that is 
prints all completions anyway, in the buffer *Completions*. And so it 
applies syntax highlighting to them as well, and does that eagerly (as 
opposed to e.g. doing that via jit-lock).

If you instrumented only the 'completion-all-completions' call, then 
that might miss the subsequent time spent in sorting.

But speaking of the case when the *Completions* buffer isn't shown yet, 
the code calls something similar to completion-try-completion, which 
ultimately goes through completion-pcm--find-all-completions. Does it 
currently apply faces too (in the default, non-lazy scenario)?

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[João Távora]

On Wed, Nov 8, 2023 at 1:06 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 07/11/2023 14:13, João Távora wrote:
> > Any objections?  Seems to speed it up when flex is the preferred
> > completion style outside icomplete.  These are average times
> > collected from the instrumentation of the above
> > completion-all-completions when doing a M-: (setq i TAB)
> >
> > I just used my normal Emacs session for this.
> >
> > with lazy hilit:      0.104536125
> > without lazy hilit:   0.172522571
>
> IIUC the problem with the default completion-at-point UI here is that is
> prints all completions anyway, in the buffer *Completions*. And so it
> applies syntax highlighting to them as well, and does that eagerly (as
> opposed to e.g. doing that via jit-lock).
>
> If you instrumented only the 'completion-all-completions' call, then
> that might miss the subsequent time spent in sorting.

You probably mean highlighting: that's the saving being made here,
not sorting.

Anyway, it _felt_ snappier, but maybe I was dreaming.  Got any
better suggestions for places where to place `benchmark-progn`?

Also, I don't think *Completions* has _every_ matching completion,
does it?  Doesn't it display more completions as you keep TABing?
That's what I supposed was providing the speedup.

> But speaking of the case when the *Completions* buffer isn't shown yet,
> the code calls something similar to completion-try-completion, which
> ultimately goes through completion-pcm--find-all-completions. Does it
> currently apply faces too (in the default, non-lazy scenario)?

No idea. just looking to optimize low-hanging fruit I can find.  If you
can find more, go ahead.

João

bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov]

On 08/11/2023 03:24, João Távora wrote:
> On Wed, Nov 8, 2023 at 1:06 AM Dmitry Gutov <dmitry@gutov.dev> wrote:
>>
>> On 07/11/2023 14:13, João Távora wrote:
>>> Any objections?  Seems to speed it up when flex is the preferred
>>> completion style outside icomplete.  These are average times
>>> collected from the instrumentation of the above
>>> completion-all-completions when doing a M-: (setq i TAB)
>>>
>>> I just used my normal Emacs session for this.
>>>
>>> with lazy hilit:      0.104536125
>>> without lazy hilit:   0.172522571
>>
>> IIUC the problem with the default completion-at-point UI here is that is
>> prints all completions anyway, in the buffer *Completions*. And so it
>> applies syntax highlighting to them as well, and does that eagerly (as
>> opposed to e.g. doing that via jit-lock).
>>
>> If you instrumented only the 'completion-all-completions' call, then
>> that might miss the subsequent time spent in sorting.
> 
> You probably mean highlighting: that's the saving being made here,
> not sorting.

I meant both but I forgot that we moved scoring to later in both cases.

> Anyway, it _felt_ snappier, but maybe I was dreaming.  Got any
> better suggestions for places where to place `benchmark-progn`?

Perhaps around the whole minibuffer-completion-help.

> Also, I don't think *Completions* has _every_ matching completion,
> does it?  Doesn't it display more completions as you keep TABing?
> That's what I supposed was providing the speedup.

Well, I type (d, press C-M-i, then select the window showing 
*Completions* and scroll to the end of the buffer. The last completion 
displayed there starts with "dy", so that's probably all of them.

That's with the default completion-styles config, but with flex it's the 
same: all completions shown in the buffer. Sorted and highlighted.

bug#47711: 27.1; Deferred highlighting support in `completion-all-completions', `vertico--all-completions`

[Dmitry Gutov]

On 11/04/2021 23:51, Daniel Mendler wrote:
> Emacs is lacking a possibility to defer the completion highlighting when
> computing completions via `completion-all-completions'. This feature is
> important for the performance of completion UIs when the set of all
> completions is much larger than the set of completions which are
> displayed.

I think this bug can be closed, with commit dfffb91a70532 we installed 
last year, and subsequent refinements.

Those might not cover all of the features that the patch proposed in 
this thread did, but the needs described in the first message (the 
original bug description) should be covered.