Simplification of `affixation-function`

Simplification of `affixation-function`

[Daniel Mendler]

The `affixation-function`, which can be specified in the completion 
table metadata is allowed to return a list with elements of either two 
or three elements depending on if you want to add only a suffix or both 
a prefix and a suffix.

Would it make sense to use a simpler definition for this function? What 
is the downside of only allowing triples? In case you don't want to use 
a prefix or suffix, specify the empty string.

Or is the idea to make the function extensible over time, such that 
completion UIs should ignore if more than three elements are specified? 
But if that is the case the current definition is problematic. The first 
element is always the candidate, the second element is either suffix or 
prefix and the third element if it exists is the suffix.

It is kind of understandable that this definition has been chosen since 
suffix before prefix is confusing. Still, if extensibility is the goal 
it would be preferable if the position of the strings does not change 
depending on the length of the list.

Daniel

Re: Simplification of `affixation-function`

[Juri Linkov]

> The `affixation-function`, which can be specified in the completion table
> metadata is allowed to return a list with elements of either two or three
> elements depending on if you want to add only a suffix or both a prefix and
> a suffix.
>
> Would it make sense to use a simpler definition for this function? What is
> the downside of only allowing triples? In case you don't want to use
> a prefix or suffix, specify the empty string.

If the logic of specifying the elements is too confusing,
then such simplification is welcome.

Please note that only the documentation could be changed
to remove mentions of the ability to specify only suffix
for `affixation-function`.  But code will remain unchanged
because internally it depends on the predefined order of elements:
“completion + suffix” or “completion + prefix + suffix”.
There is no backward-compatible way to change this order.

> Or is the idea to make the function extensible over time, such that
> completion UIs should ignore if more than three elements are specified? But
> if that is the case the current definition is problematic. The first
> element is always the candidate, the second element is either suffix or
> prefix and the third element if it exists is the suffix.

More than three elements are already supported since e.g. the
total length is calculated as a sum of lengths of all strings:

  (apply #'+ (mapcar #'string-width str))

Re: Simplification of `affixation-function`

[Daniel Mendler]

On 4/24/21 10:22 PM, Juri Linkov wrote:
> If the logic of specifying the elements is too confusing,
> then such simplification is welcome.

> Please note that only the documentation could be changed
> to remove mentions of the ability to specify only suffix
> for `affixation-function`.  But code will remain unchanged
> because internally it depends on the predefined order of elements:
> “completion + suffix” or “completion + prefix + suffix”.
> There is no backward-compatible way to change this order.

I think it is slightly confusing and unnecessarily complicated. But if 
you want to retain backward compatibility, there is no way back. However 
since the `affixation-function` has only been introduced recently, there 
should still be a time window to change the definition? There is no 
released version of Emacs with `affixation-function` support.

(If it is decided to change the definition it should be feasible to go 
over all ELPA/MELPA and apply patches to all packages which already make 
use of the `affixation-function`.)

> More than three elements are already supported since e.g. the
> total length is calculated as a sum of lengths of all strings:
> 
>    (apply #'+ (mapcar #'string-width str))

I see. So all the later elements are suffixes and concatenated? This 
seems like an unnecessary completion given that the completion-table 
implementor can already concatenate all the suffixes. Or is the idea 
that the UI can show all these suffixes in some kind of table-like view?
It seems that there is at least need for better documentation.

My proposal is to only allow three-element lists and remove support to 
specify only the suffix.

And one could keep the door open by allowing more than three element 
lists for additional annotations. Let's say you want to specify some 
additional help/tool-tip text for example.

Daniel

Re: Simplification of `affixation-function`

[Juri Linkov]

> I think it is slightly confusing and unnecessarily complicated. But if you
> want to retain backward compatibility, there is no way back. However since
> the `affixation-function` has only been introduced recently, there should
> still be a time window to change the definition? There is no released
> version of Emacs with `affixation-function` support.
>
> (If it is decided to change the definition it should be feasible to go over
> all ELPA/MELPA and apply patches to all packages which already make use of
> the `affixation-function`.)

I'm not sure if breaking the existing API is worth the trouble.

We could just change the documentation of ‘affixation-function’ to define
only one supported way of using the “completion + prefix + suffix” format
to reduce confusion.

>> More than three elements are already supported since e.g. the
>> total length is calculated as a sum of lengths of all strings:
>>    (apply #'+ (mapcar #'string-width str))
>
> I see. So all the later elements are suffixes and concatenated? This seems
> like an unnecessary completion given that the completion-table implementor
> can already concatenate all the suffixes.

This is just implementation detail, there is no intention to support
more suffixes.

> Or is the idea that the UI can show all these suffixes in some kind of
> table-like view?

Currently I'm trying to use tabulated-list-mode in the completions buffer.
But even in this case there is no need to add more suffixes for more
table columns.

> My proposal is to only allow three-element lists and remove support to
> specify only the suffix.

Supporting only the suffix still is necessary for the ‘annotation-function’.

> And one could keep the door open by allowing more than three element lists
> for additional annotations. Let's say you want to specify some additional
> help/tool-tip text for example.

I'd prefer to support additional text using keywords, e.g.

  ("completion" "prefix" "suffix" :help "Help text")

Re: Simplification of `affixation-function`

[Daniel Mendler]

On 4/24/21 11:41 PM, Juri Linkov wrote:> We could just change the 
documentation of ‘affixation-function’ to define
> only one supported way of using the “completion + prefix + suffix” format
> to reduce confusion.

This is a "resolution" I would not be happy with. We still keep around 
an obsolete format (which has only been introduced a few months ago) and 
then keep it out of the documentation. I think it is preferable to 
correct the definition then.

But it is okay to keep the current definition if you think the costs of 
a change outweigh the benefits.

 > I'd prefer to support additional text using keywords, e.g.
 > ("completion" "prefix" "suffix" :help "Help text")

The suggestion with the keywords is nice. If you plan to allow such 
additional keywords, this should be documented then, since completion 
UIs rely on the documentation when they implement support for the 
function. Which formats will be allowed? We already have `(candidate 
suffix)` and `(candidate prefix suffix)`.

For extensibility one may want to add

(candidate :prefix prefix :suffix suffix :other-keys...)

or as you proposed

(candidate prefix suffix :other-keys...).

Daniel

Re: Simplification of `affixation-function`

[Stefan Monnier]

> because internally it depends on the predefined order of elements:
> “completion + suffix” or “completion + prefix + suffix”.
> There is no backward-compatible way to change this order.

We don't need to worry about backward compatibility here since this
functionality has not yet been included in any released Emacs.


        Stefan

Re: Simplification of `affixation-function`

[Dmitry Gutov]

On 25.04.2021 01:34, Stefan Monnier wrote:
>> because internally it depends on the predefined order of elements:
>> “completion + suffix” or “completion + prefix + suffix”.
>> There is no backward-compatible way to change this order.
> We don't need to worry about backward compatibility here since this
> functionality has not yet been included in any released Emacs.

Speaking of not being able to change it after the release, am I right to 
assume that this feature's main target is completing-read and not 
completion-at-point-functions?

Nobody proposed a patch for Company, and I'm not sure how to handle such 
non-semantic affixations in there (icons generally require more care 
since they often take up more than single character width).

Speaking of it's used in help--symbol-completion-table-affixation, we 
have recently added a related feature called 'kind' in Company which 
assigns each completion a icon (using a user-customizable mapping and/or 
algorithm).

You can see its integration in elisp-completion-at-point (the 
:company-kind attributes).

Re: Simplification of `affixation-function`

[Daniel Mendler]

On 4/25/21 12:48 AM, Dmitry Gutov wrote:
> On 25.04.2021 01:34, Stefan Monnier wrote:
>>> because internally it depends on the predefined order of elements:
>>> “completion + suffix” or “completion + prefix + suffix”.
>>> There is no backward-compatible way to change this order.
>> We don't need to worry about backward compatibility here since this
>> functionality has not yet been included in any released Emacs.
> 
> Speaking of not being able to change it after the release, am I right to 
> assume that this feature's main target is completing-read and not 
> completion-at-point-functions?

I support affixation-functions in my small Corfu package, which shows 
overlay popups like Company. However the result will not be good if the 
affixation-functions return long suffix strings and are written with the 
*Completions* buffer in mind.

Daniel

Re: Simplification of `affixation-function`

[Dmitry Gutov]

On 25.04.2021 01:56, Daniel Mendler wrote:

> I support affixation-functions in my small Corfu package, which shows 
> overlay popups like Company. However the result will not be good if the 
> affixation-functions return long suffix strings and are written with the 
> *Completions* buffer in mind.

I guess my problem with it is, with the apparent end goal of 
flexibility, it ends up targeting only the default completion UI, and 
the more an alternative UI is different from it, the worse the result 
can look.

And without semantic information, it fails to take advantage of the 
additional features the alternative UIs might provide.

Re: Simplification of `affixation-function`

[Daniel Mendler]

On 4/25/21 7:58 PM, Dmitry Gutov wrote:
> I guess my problem with it is, with the apparent end goal of 
> flexibility, it ends up targeting only the default completion UI, and 
> the more an alternative UI is different from it, the worse the result 
> can look.
> 
> And without semantic information, it fails to take advantage of the 
> additional features the alternative UIs might provide.

You are right about that. As things stand now the 
annotation/affixation-function only work well for the default 
completions buffer and the vertical completion UIs. The 
completion-at-point popups are very different from that, so it is only 
fair if they try to invent their own metadata functions as you are doing 
in Company with `company-kind` etc.

However I am a bit critical with regards to the "semantic information". 
As far as I understood you use some icon names for that, but this 
restricts the purpose of the annotations. Maybe it would be sufficient 
to use a more crude solution, where you only specify that the 
annotations/affixations must be sufficiently short in order to display 
well in the popups, without going the full way to the semantic kinds.

Daniel

Re: Simplification of `affixation-function`

[Dmitry Gutov]

On 25.04.2021 21:08, Daniel Mendler wrote:
> On 4/25/21 7:58 PM, Dmitry Gutov wrote:
>> I guess my problem with it is, with the apparent end goal of 
>> flexibility, it ends up targeting only the default completion UI, and 
>> the more an alternative UI is different from it, the worse the result 
>> can look.
>>
>> And without semantic information, it fails to take advantage of the 
>> additional features the alternative UIs might provide.
> 
> You are right about that. As things stand now the 
> annotation/affixation-function only work well for the default 
> completions buffer and the vertical completion UIs.

Isn't Company a vertical completion UI?

> The 
> completion-at-point popups are very different from that, so it is only 
> fair if they try to invent their own metadata functions as you are doing 
> in Company with `company-kind` etc.

It's fair that they try to invent new stuff, but it's more economical to 
reuse the stuff when feasible.

> However I am a bit critical with regards to the "semantic information". 
> As far as I understood you use some icon names for that, but this 
> restricts the purpose of the annotations. Maybe it would be sufficient 
> to use a more crude solution, where you only specify that the 
> annotations/affixations must be sufficiently short in order to display 
> well in the popups, without going the full way to the semantic kinds.

There are currently only two places where affixation-function is used in 
the core.

One of them is the aforementioned help--symbol-completion-table.

Regardless of whether there will ever be a completing-read-function 
based on Company (it's not hard to do, actually; mostly the popup 
rendering method is holding it back, but I had it working with 
company-posframe), if help--symbol-completion-table-affixation plugged 
into :company-kind instead, the users could see the icons already 
familiar from completion code, from other backends, they wouldn't need 
to puzzle out what the "u", "a" and "c" characters mean. And other 
completion frameworks can pick up this feature too (maybe even reuse the 
icon mappings, we'll see). The user can also customize the icons' look, 
choose whether they should be SVG, text, icon font characters, or so on.

The other use is in read-char-by-name. And I can admit, seeing the 
characters in the first column is slightly better than following each 
completion. Not by much, they still work well enough as annotations, but 
seeing them in the first column is a bit tidier.

What would I do? First of all, keep them in annotations, because just 
one slightly better-looking use case does not justify the added 
complexity. If someone really wanted to improve positioning of 
annotations for that completion table, there are other options:

- Add a custom var that aligns all annotations to the right, then 
they'll also be in one column. Company has such an option already.

- Create a var specific for the default UI that would move the 
annotations to the left of the completions. read-char-by-name could bind 
it to t before calling completing-read. Then the *Completions* rendering 
code would do whatever is necessary to render it correctly at the 
requested side.

But since mule--ucs-names-affixation, aside from information, also 
contains presentation (spacing and alignment chars), it would be 
infeasible for a completion function that doesn't want extra stuff on 
the left to render its result on the right.

You can also note that the third (suffix) element is always an empty 
string in both of these use cases. So affixation-function was really 
added to be able to add a prefix.

Re: Simplification of `affixation-function`

[Juri Linkov]

> You can also note that the third (suffix) element is always an empty string
> in both of these use cases. So affixation-function was really added to be
> able to add a prefix.

affixation-function was an improvement over annotation-function,
but nonetheless it has limitations too.  What would be a better thing is
like Daniel proposed a new meta `group-function`, I'd imagine a similar
meta `format-function` that could receive a candidate and return
a string to insert to the completions buffer.

Then the caller e.g. help--symbol-completion-table could define whether
to append "u", "a" and "c" in parens by using on a candidate something like
(format "%s (%s)" cand (cond ((fboundp (intern cand)) "f")) ...),
or prepend a dimmed letter as a prefix, or to use an icon.

The same `format-function` could be used to remove the group prefix
when `group-function` is in use, instead of providing an additional
argument `transform` for `group-function`.

Re: Simplification of `affixation-function`

[Daniel Mendler]

On 4/27/21 6:48 PM, Juri Linkov wrote:
>> You can also note that the third (suffix) element is always an empty string
>> in both of these use cases. So affixation-function was really added to be
>> able to add a prefix.
> 
> affixation-function was an improvement over annotation-function,
> but nonetheless it has limitations too.  What would be a better thing is
> like Daniel proposed a new meta `group-function`, I'd imagine a similar
> meta `format-function` that could receive a candidate and return
> a string to insert to the completions buffer.
> 
> Then the caller e.g. help--symbol-completion-table could define whether
> to append "u", "a" and "c" in parens by using on a candidate something like
> (format "%s (%s)" cand (cond ((fboundp (intern cand)) "f")) ...),
> or prepend a dimmed letter as a prefix, or to use an icon.
> 
> The same `format-function` could be used to remove the group prefix
> when `group-function` is in use, instead of providing an additional
> argument `transform` for `group-function`.

There has been some confusion regarding this already. Some people (me
included) had assumed at some point that the affixation function is
allowed to transform the candidate, since the candidate is part of the
returned list elements (Now I know that this is not the case, but the
function should be allowed to add faces). It should be possible to relax
the affixation function to allow a candidate transformation, then we
avoid the addition of another format-function function. One may ask -
why not change the affixation function then such that it only returns a
single string? We could do that but I think it is valuable to separate
prefix/candidate/suffix for display in a tablist.

The group function candidate transformation is also to be distinguished
from a potential transformation performed by a
format/affixation-function, since the group transformation should be
only applied when grouping is active. It is therefore better to keep the
transformations separate.

Daniel

Re: Simplification of `affixation-function`

[Juri Linkov]

>> affixation-function was an improvement over annotation-function,
>> but nonetheless it has limitations too.  What would be a better thing is
>> like Daniel proposed a new meta `group-function`, I'd imagine a similar
>> meta `format-function` that could receive a candidate and return
>> a string to insert to the completions buffer.
>
> There has been some confusion regarding this already. Some people (me
> included) had assumed at some point that the affixation function is
> allowed to transform the candidate, since the candidate is part of the
> returned list elements (Now I know that this is not the case, but the
> function should be allowed to add faces). It should be possible to relax
> the affixation function to allow a candidate transformation, then we
> avoid the addition of another format-function function. One may ask -

affixation-function is called immediately after sorting candidates.
format-function could be called immediately before inserting candidates
only when they are displayed.

> why not change the affixation function then such that it only returns a
> single string? We could do that but I think it is valuable to separate
> prefix/candidate/suffix for display in a tablist.

Because completion--insert-strings needs to highlight the untransformed
candidate with mouse-face=highlight.

>> The same `format-function` could be used to remove the group prefix
>> when `group-function` is in use, instead of providing an additional
>> argument `transform` for `group-function`.
>
> The group function candidate transformation is also to be distinguished
> from a potential transformation performed by a
> format/affixation-function, since the group transformation should be
> only applied when grouping is active. It is therefore better to keep the
> transformations separate.

API would be more clear when transformation is separated from grouping.
So if there is no need to transform group candidates, only group-function
is provided.  When group candidates need transformation, then additionally
format/affixation-function can be provided as well.

Re: Simplification of `affixation-function`

[Daniel Mendler]

On 4/27/21 8:11 PM, Juri Linkov wrote:
> affixation-function is called immediately after sorting candidates.
> format-function could be called immediately before inserting candidates
> only when they are displayed.

I don't see why using the affixation function is not possible for the
transformation. The affixation function is basically just called right
before insertion, right before the candidates are passed to
`display-completion-list`. I don't see the need for an additional format
function here.

>> why not change the affixation function then such that it only returns a
>> single string? We could do that but I think it is valuable to separate
>> prefix/candidate/suffix for display in a tablist.
> 
> Because completion--insert-strings needs to highlight the untransformed
> candidate with mouse-face=highlight.

Yes, that's right. `choose-completion` uses the mouse highlight to
locate the candidate. But this trick must be relaxed as soon as you
bring transformed candidates into the game. In my `group-function` patch
I am attaching the untransformed candidate to the property
`completion--string`, which is then read by `choose-completion`. Note
that the highlighting is still preserved, if no grouping/transformation
is used. In that case the highlighting still reflects the untransformed
candidate. As such the change is backward compatible as long as
grouping/transformation is guarded behind a feature flag.
>> The group function candidate transformation is also to be distinguished
>> from a potential transformation performed by a
>> format/affixation-function, since the group transformation should be
>> only applied when grouping is active. It is therefore better to keep the
>> transformations separate.
> 
> API would be more clear when transformation is separated from grouping.
> So if there is no need to transform group candidates, only group-function
> is provided.  When group candidates need transformation, then additionally
> format/affixation-function can be provided as well.

I considered exactly this, but it is problematic. The idea of grouping
is that you can optionally turn it on or optionally support it in the
UI. The grouping transformation should only be applied when grouping is
enabled. This is important since the grouping candidate transformation
may remove some redundant part of the candidate which is then displayed
in the group title. This means we cannot conflate the grouping candidate
transformation with the `affixation/format-function`.

We can either go with two functions
`group-sort-function+group-transform-function`, where
`group-transform-function` is optional or with the boolean `transform`
argument I am having now in the patch. I think it is easier to have a
single `group-function` instead of splitting this small feature into two
functions.

Daniel

Re: Simplification of `affixation-function`

[Dmitry Gutov]

On 27.04.2021 19:48, Juri Linkov wrote:
>> You can also note that the third (suffix) element is always an empty string
>> in both of these use cases. So affixation-function was really added to be
>> able to add a prefix.
> 
> affixation-function was an improvement over annotation-function,
> but nonetheless it has limitations too.  What would be a better thing is
> like Daniel proposed a new meta `group-function`, I'd imagine a similar
> meta `format-function` that could receive a candidate and return
> a string to insert to the completions buffer.

That would also conflate information with presentation, and thus only be 
usable for the default completion UI. And not ideal even for that purpose.

> Then the caller e.g. help--symbol-completion-table could define whether
> to append "u", "a" and "c" in parens by using on a candidate something like
> (format "%s (%s)" cand (cond ((fboundp (intern cand)) "f")) ...),
> or prepend a dimmed letter as a prefix, or to use an icon.

Do you use Company? Take a look at what elisp--company-kind does.

The fact that the space of allowed values is constrained makes the 
returned information really more useful.

And here since you do not document which values (strings, characters, 
etc) the prefix returned by affixation-function can take, their 
usefulness is limited. The user can't be sure what "c" means: it's 
"command" here, but could be meant as "constant" by another affixation 
function. Is "m" a "macro" or "method"? "f" stands for "function" or 
"field"?

And also because of that no completing-read UI can replace these 
characters with any alternative (e.g. graphical) representation.

Re: Simplification of `affixation-function`

[Juri Linkov]

>> Then the caller e.g. help--symbol-completion-table could define whether
>> to append "u", "a" and "c" in parens by using on a candidate something like
>> (format "%s (%s)" cand (cond ((fboundp (intern cand)) "f")) ...),
>> or prepend a dimmed letter as a prefix, or to use an icon.
>
> Do you use Company? Take a look at what elisp--company-kind does.
>
> The fact that the space of allowed values is constrained makes the returned
> information really more useful.
>
> And here since you do not document which values (strings, characters, etc)
> the prefix returned by affixation-function can take, their usefulness is
> limited. The user can't be sure what "c" means: it's "command" here, but
> could be meant as "constant" by another affixation function. Is "m"
> a "macro" or "method"? "f" stands for "function" or "field"?
>
> And also because of that no completing-read UI can replace these characters
> with any alternative (e.g. graphical) representation.

I took a look at elisp--company-kind and see such code:

                           :annotation-function
                           (lambda (str) (if (fboundp (intern-soft str)) " <f>"))
                           :company-kind #'elisp--company-kind

There are still hard-coded letters where "f" stands for "function".

Re: Simplification of `affixation-function`

[Dmitry Gutov]

On 28.04.2021 22:59, Juri Linkov wrote:

> I took a look at elisp--company-kind and see such code:
> 
>                             :annotation-function
>                             (lambda (str) (if (fboundp (intern-soft str)) " <f>"))
>                             :company-kind #'elisp--company-kind
> 
> There are still hard-coded letters where "f" stands for "function".

You're looking at the "old" approach to that feature, using 
annotation-function. I couldn't just remove it here because the default 
completion UI doesn't know to use :company-kind (elisp--company-kind's 
definition is below in the same file).

The approach above has worked okay-ish over the years (so it hopefully 
illustrates that the use of affixation-function in read-char-by-name is 
not essential), but has the same flaws that I already described, for the 
purpose of using it seriously to show icons/kinds/types in different 
commands and completion tables.

Until now it has only been used for Elisp, and only to indicate one 
specific kind, so it was less of a problem.