bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov <dmitry@gutov.dev>]

Hi Stefan,

Sorry for the pause, let's continue this digression. Though perhaps it 
would be better moved to emacs-devel or somewhere else. ;-/

On 29/10/2023 06:41, Stefan Monnier wrote:
>> FWIW, this neat structure might not help too much: the most popular external
>> completion backend (the LSP language servers, collectively) don't accept
>> regexps or globs, they just send you the lists of completions available at
>> point. With the name matching method sometimes configurable per-server.
> 
> Largely agreed.  The main benefit tho is that you get just *one*
> pattern, rather than three (one being the prefix argument, the second
> being the `pred` arg which historically was so unused that it was abused
> to hold the directory for file-name completion, so lots of tables don't
> obey it, and the third being the `completion-regexp-list` that most coders
> forget, and those who don't end up regretting not forgetting when it was
> not meant for them), so it's much more clear.

The motivation all makes sense.

>> As such, the most useful methods currently are: 1) Emacs Regexp, 2) asking
>> server for whatever it thinks is suitable (the "backend" completion style).
> 
> For the backends: agreed.
> For the frontends (i.e. `completion-styles`), `glob` is the more useful
> one, I'd say (except for the "external" style, of course).
> 
> We might also want support for things like `or` and `and` patterns, but
> I haven't managed to fit them nicely in that structure :-(

Possibly, but which code would produce such patterns?

>> I would also probably want to standardize on the recommended type of TO
>> anyway: some of them are likely going to result in better performance
>> than others.
> 
> The TO is chosen by the specific completion table, based on what it can
> handle best.  So it should always be "optimal".

Sorry, I meant the recommended type of FROM. Because if the original 
caller passes an arbitrary regexp, it will often get turned into a pair 
with a predicate where the latter calls string-match-p.

And if the type of FROM is standardized, there likely would be no need 
for a four-way bidirectional conversion. Maybe just a helper that 
converts from the original "main" type into any of the available that is 
currently required.

>> So I guess it's also a way to make every completion table aware of PRED?
> 
> Note also that these `pred` patterns are expected to be exclusively
> looking at the string (they're used for `completion-styles` kind of
> functionality), so nothing like `file-directory-p` or `fboundp` kind of
> predicates here.

Would we consider those pred's "fast enough"?

I've done a little comparison of completion-regexp-list vs pred:

(setq sss (cl-loop repeat 300000 collect (symbol-name (gensym "yoyo"))))

(benchmark-run 10 (let* ((re "yo[^o]*o")
                          (completion-regexp-list (list re)))
                     (all-completions "" sss)))
;; => 0.60s


(benchmark-run 10 (let* ((re "yo[^o]*o"))
                     (all-completions "" sss
                              (lambda (s) (string-match-p re s)))))
;; => 1.14s

> The `pred`s used in things like `completing-read` and `read-file-name`
> would be handled elsewhere such as `completion-table-with-predicate`.
> This part is still up in the air, tho.
> 
>> That should work; though it might be hard to reach the same raw performance
>> as the current all-completions + completion-regexp-list.
> 
> I don't see why: currently my code actually uses `all-completions` and
> `completion-regexp-list`, so as long as the pattern can be turned into
> a regexp without requiring an additional PRED (that's usually the case), <...>

Right, I was talking about the possible exceptions.