bug#48841: [PATCH] Add new `completion-filter-completions` API and deferred highlighting

[Daniel Mendler <mail@daniel-mendler.de>]

On 8/11/21 4:16 PM, Daniel Mendler wrote:
> I prepared a patch which provides the API
> `completion-filter-completions`. This function supports deferred
> highlighting and returns additional data with the list of matching
> completion candidates. The API supersedes the existing function
> `completion-all-completions`.
> 
> The main goal of the new API is to avoid expensive string allocations
> and highlighting during completion. This is particularly relevant for
> continuously updating completion UIs like Icomplete or Vertico.
> Furthermore the end position of the completion boundaries is returned
> with the completion results. This information is not provided by the
> existing `completion-all-completions` API.
> 
> See also the relevant bugs bug#47711 and bug#48841. I am looking forward
> to your feedback. Thank you!

There are currently two issues with the patch with regards to backward
compatibility. Fortunately they are fixable with a little effort.

1. I would like to deprecate `completion-score' or remove it altogether,
   but unfortunately `completion-score' is used in the wild. In order to
   preserve `completion-score', bind `completion--filter-completions' in
   the highlighting functions. Add `completion-score' in
   `completion-pcm--hilit-commonality' when
   `completion--filter-completions' is nil.

2. In `completion--nth-completion' set `completion--filter-completions'
   to nil, unless `(memq style '(emacs21 emacs22 basic
   partial-completion initials flex))' such that custom completion
   styles which wrap the completion functions don't see the new return
   value format, except if the custom style opts in explicitly by
   binding `completion--filter-completions'. An alternative criterion is
   `(memq fun '(completion-emacs22-all-completions) ...)'. Unfortunately
   this approach will still not work if the user has advised a
   `completion-x-all-completions' function. The only 100% safe approach
   seems to transparently redirect calls to
   `completion-x-all-completions' to `completion--x-filter-completions',
   which returns the results in the new format.

With these changes the patch should be 100% backward compatible.

Daniel