bug#47711: bug#48841: bug#47711: bug#48841: bug#47711: [PATCH VERSION 2] Add new `completion-filter-completions` API and deferred highlighting

[Dmitry Gutov <dmitry@gutov.dev>]

On 27/10/2023 21:12, Stefan Monnier wrote:
>>> More seriously, since it's a dynbound variable it can have unwanted
>>> effects in nested calls to `all/try-completions`, so it's safer to
>>> ignore that variable because its binding is not always "meant for us" 🙁
>> I guess it would be more precise if it was a function argument, e.g. the
>> first argument to 'fancy-all-completions' or somesuch that all completion
>> tables are supposed to use inside. OTOH, I suppose that might hinder those
>> that use external programs.
> In my "work in progress" (not touched since last December 🙁 ),
> I replace `all-completions` with:
> 
>      (cl-defgeneric completion-table-fetch-matches ( table pattern
>                                                      &optional pre session)
>        "Return candidates matching PATTERN in the completion TABLE.
>      For tables with subfields, PRE is the text found before PATTERN such that
>         (let ((len (length PRE)))
>           (equal (completion-table-boundaries TABLE PRE len) (cons len len)))
>      
>      Return a list of strings or a list of cons cells whose car is a string.
>      SESSION if non-nil is a hash-table where we can stash arbitrary auxiliary
>      info to avoid recomputing it between calls of the same \"session\".")
> 
> `pattern`s can take various shapes.  In my WiP code, I implement 4 kinds
> of patterns: prefix, glob, regexp, and predicate.  Now, we don't want
> completion tables to have to handle each and every one of those pattern
> kinds (the set of which is extensible via CLOS methods), so there's
> a middleman:
> 
>      (cl-defgeneric completion-pattern-convert (to pattern)
>        "Convert PATTERN to be of type TO.
>      Returns a pair (PRED . NEWPATTERN) where NEWPATTERN is of type TO
>      and should match everything that PATTERN matches.  PRED is nil
>      if NEWPATTERN matches exactly the same candidates as PATTERN
>      and otherwise it is a function that takes a candidate and returns non-nil if the
>      candidate also matches PATTERN.  PRED should not presume that the candidate
>      has already been filtered by NEWPATTERN."

FWIW, this neat structure might not help too much: the most popular 
external completion backend (the LSP language servers, collectively) 
don't accept regexps or globs, they just send you the lists of 
completions available at point. With the name matching method sometimes 
configurable per-server.

As such, the most useful methods currently are: 1) Emacs Regexp, 2) 
asking server for whatever it thinks is suitable (the "backend" 
completion style).

I would also probably want to standardize on the recommended type of TO 
anyway: some of them are likely going to result in better performance 
than others.

BTW, this reminds me about urgrep in GNU ELPA, which I think includes 
converters between different flavors of regexp. Something to keep in 
mind for the occasional completion table that's based on a Grep-like tool.

> So the fallback definition of `completion-table-fetch-matches`, which
> relies on the old API looks like:
> 
>      (defun completion-table--fetch-legacy (table pattern &optional pre _session)
>        (pcase-let ((`(,pred . ,regexp)
>                     (completion-pattern-convert 'regexp pattern))
>                    (`(,ppred . ,prefix)
>                     (completion-pattern-convert 'prefix pattern)))
>          (let ((matches
>                 (let ((completion-regexp-list (if ppred (list regexp)))
>                       (case-fold-search completion-ignore-case))
>                   (all-completions (concat pre prefix) table))))
>            (if (null pred)
>                matches
>              (seq-filter pred matches)))))
> 
> This is of course incorrect because `all-completions` could ignore
> `completion-regexp-list`, in which case we should use `ppred` instead of
> `pred` on the last 3 lines 😄
> 
> It has the disadvantage that every completion-table basically needs to
> start by calling `completion-pattern-convert` so as to convert the
> pattern to the format that it supports.  But I think it's still better
> than the current API where completion tables "have to" obey the prefix
> string, the `completion-regexp-list`, and the predicate (and where the
> latter two are often nil so tables tends to ignore them, and since
> tables ignore them callers don't use them, etc...).

So I guess it's also a way to make every completion table aware of PRED?

That should work; though it might be hard to reach the same raw 
performance as the current all-completions + completion-regexp-list.