bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

As mentioned in
http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg00445.html,
the docstring of completion-category-defaults, as well as its manual
documentation do not describe its behavior accurately.

(See what `completion--styles' does.)

In GNU Emacs 25.0.50.5 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.7)
 of 2016-01-06 built on axl
Repository revision: 50575b1bdd7fcb4d1bf525fb5ca635fe7ab7d8c6

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 07 Jan 2016 23:27:23 +0300
> 
> As mentioned in
> http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg00445.html,
> the docstring of completion-category-defaults, as well as its manual
> documentation do not describe its behavior accurately.
> 
> (See what `completion--styles' does.)

Would adding `partial-completion' to the list there fix this problem?
Or is there anything else that needs to be done?

Thanks.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 01/08/2016 03:17 PM, Eli Zaretskii wrote:

> Would adding `partial-completion' to the list there fix this problem?

It would make no difference WRT to resulting behavior.

> Or is there anything else that needs to be done?

The problem is that the actual behavior and the documented one are 
different, and should be reconciled.

FTR, I'm not sure it's the documentation that needs fixing.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Eli Zaretskii]

> Cc: 22324@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 8 Jan 2016 15:21:36 +0300
> 
> FTR, I'm not sure it's the documentation that needs fixing.

Thanks, that wasn't clear from the problem description.  Normally,
when someone says "the documentation doesn't describe the behavior
accurately", they mean the documentation should be updated.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 01/08/2016 03:26 PM, Eli Zaretskii wrote:

> Thanks, that wasn't clear from the problem description.  Normally,
> when someone says "the documentation doesn't describe the behavior
> accurately", they mean the documentation should be updated.

Right, sorry. Poor wording on my part.

But if you were talking about a documentation fix, changing the list in 
`(emacs) Completion Styles' to include `partial-completion' is not a 
good fix semantically, because the result would still contain the word 
"only".

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Eli Zaretskii]

> Cc: 22324@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 8 Jan 2016 15:30:52 +0300
> 
> On 01/08/2016 03:26 PM, Eli Zaretskii wrote:
> 
> > Thanks, that wasn't clear from the problem description.  Normally,
> > when someone says "the documentation doesn't describe the behavior
> > accurately", they mean the documentation should be updated.
> 
> Right, sorry. Poor wording on my part.

No harm done.

> But if you were talking about a documentation fix, changing the list in 
> `(emacs) Completion Styles' to include `partial-completion' is not a 
> good fix semantically, because the result would still contain the word 
> "only".

Who said I intended to leave that word there? ;-)

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 01/08/2016 03:26 PM, Eli Zaretskii wrote:
>
>> Thanks, that wasn't clear from the problem description.  Normally,
>> when someone says "the documentation doesn't describe the behavior
>> accurately", they mean the documentation should be updated.
>
> Right, sorry. Poor wording on my part.
>
> But if you were talking about a documentation fix, changing the list
> in `(emacs) Completion Styles' to include `partial-completion' is not
> a good fix semantically, because the result would still contain the
> word "only".

This bug report is a bit on the vague side.

It looks like `partial-completion' is documented in that node?

The subject mentions "doesn't override", but is that about
`completion-category-overrides' instead?

---
This overrides the defaults specified in `completion-category-defaults'."
---

And it does indeed prepend:

(defun completion--category-override (category tag)
  (or (assq tag (cdr (assq category completion-category-overrides)))
      (assq tag (cdr (assq category completion-category-defaults)))))

But...  I think saying that that "overrides" is fine?

So it's unclear what's suggested in this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  22324@debbugs.gnu.org
> Date: Thu, 02 Dec 2021 10:10:40 +0100
> 
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
> > On 01/08/2016 03:26 PM, Eli Zaretskii wrote:
> >
> >> Thanks, that wasn't clear from the problem description.  Normally,
> >> when someone says "the documentation doesn't describe the behavior
> >> accurately", they mean the documentation should be updated.
> >
> > Right, sorry. Poor wording on my part.
> >
> > But if you were talking about a documentation fix, changing the list
> > in `(emacs) Completion Styles' to include `partial-completion' is not
> > a good fix semantically, because the result would still contain the
> > word "only".
> 
> This bug report is a bit on the vague side.
> 
> It looks like `partial-completion' is documented in that node?
> 
> The subject mentions "doesn't override", but is that about
> `completion-category-overrides' instead?
> 
> ---
> This overrides the defaults specified in `completion-category-defaults'."
> ---
> 
> And it does indeed prepend:
> 
> (defun completion--category-override (category tag)
>   (or (assq tag (cdr (assq category completion-category-overrides)))
>       (assq tag (cdr (assq category completion-category-defaults)))))
> 
> But...  I think saying that that "overrides" is fine?
> 
> So it's unclear what's suggested in this bug report.

I think the problem is that we don't clearly document how the list of
styles to be actually used for some CATEGORY is obtained from the
various contributions, like completion-category-defaults and
completion-category-overrides, and what is the priority of each
contribution in the resulting list of styles.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 02.12.2021 12:10, Lars Ingebrigtsen wrote:
> ---
> This overrides the defaults specified in `completion-category-defaults'."
> ---
> 
> And it does indeed prepend:
> 
> (defun completion--category-override (category tag)
>    (or (assq tag (cdr (assq category completion-category-overrides)))
>        (assq tag (cdr (assq category completion-category-defaults)))))
> 
> But...  I think saying that that "overrides" is fine?

I think an "override" has a particular meaning, and that's replacing 
something that was there before. Not prepending to it.

Why is it important? Suppose the "category defaults" entry has a 
"permissive" style set up for a certain completion category.

As a user, I might try to override it with an entry in 
completion-category-overrides, to use a stricter style like 
'partial-completion' or even 'basic'.

But what happens when my input fails to find any completions with the 
style I specified? It will fall back the default one, which is both 
surprising, given the current documentation, and can be problematic with 
respect to performance ('flex' is slower than 'partial-completion') and 
behavior (bringing lots of probably irrelevant completions which match 
my input because 'flex' is quite lax).

Whether one enjoys the lax behavior of 'flex', is more or less a user 
preference, and it seems one can't configure it entirely through 
completion-category-overrides. And since that variable is a defcustom 
and completion-category-defaults is not, it seems like we're limiting 
customization this way.

Though, of course, a user that's motivated enough can change the value 
of completion-category-defaults too.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> (defun completion--category-override (category tag)
>>    (or (assq tag (cdr (assq category completion-category-overrides)))
>>        (assq tag (cdr (assq category completion-category-defaults)))))
>> But...  I think saying that that "overrides" is fine?
>
> I think an "override" has a particular meaning, and that's replacing
> something that was there before. Not prepending to it.
>
> Why is it important? Suppose the "category defaults" entry has a
> "permissive" style set up for a certain completion category.
>
> As a user, I might try to override it with an entry in
> completion-category-overrides, to use a stricter style like
> 'partial-completion' or even 'basic'.

Well, it overrides the specific tag/category -- but
completion-category-overrides doesn't (by being non-nil) override the
entirety of completion-category-defaults.

So it's two types of overrides.

> But what happens when my input fails to find any completions with the
> style I specified? It will fall back the default one, which is both
> surprising, given the current documentation, and can be problematic
> with respect to performance ('flex' is slower than
> 'partial-completion') and behavior (bringing lots of probably
> irrelevant completions which match my input because 'flex' is quite
> lax).

Yes, it doesn't seem to allow dropping any of them out completely.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 06.12.2021 05:25, Lars Ingebrigtsen wrote:

>> As a user, I might try to override it with an entry in
>> completion-category-overrides, to use a stricter style like
>> 'partial-completion' or even 'basic'.
> 
> Well, it overrides the specific tag/category -- but
> completion-category-overrides doesn't (by being non-nil) override the
> entirety of completion-category-defaults.

Perhaps a better word is "prepends".

> So it's two types of overrides.
If we use the language from the nadvice package, it performs the 
"before-until" combination, rather than the "override" one.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Perhaps a better word is "prepends".
>
>> So it's two types of overrides.
> If we use the language from the nadvice package, it performs the
> "before-until" combination, rather than the "override" one.

Right.  But I don't think that'll make things clearer.

Perhaps instead of trying to find the perfect word here, we could just
write a paragraph that explains what's happening?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 07.12.2021 23:28, Lars Ingebrigtsen wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> Perhaps a better word is "prepends".
>>
>>> So it's two types of overrides.
>> If we use the language from the nadvice package, it performs the
>> "before-until" combination, rather than the "override" one.
> Right.  But I don't think that'll make things clearer.
> 
> Perhaps instead of trying to find the perfect word here, we could just
> write a paragraph that explains what's happening?

Consider changing the behavior instead, though.

Yes, it's been like this for a long time, but I imagine most users won't 
notice the change.

We could experiment on master.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Consider changing the behavior instead, though.
>
> Yes, it's been like this for a long time, but I imagine most users
> won't notice the change.
>
> We could experiment on master.

I'd rather not change something as subtle as this (especially in a
mechanism that's been around for a while like this as).  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> Consider changing the behavior instead, though.
>>
>> Yes, it's been like this for a long time, but I imagine most users
>> won't notice the change.
>>
>> We could experiment on master.
>
> I'd rather not change something as subtle as this (especially in a
> mechanism that's been around for a while like this as).  

Nobody had any further opinions here in a month, so I went ahead and
changed the doc string.  If somebody feels strongly that the semantics
should be tweaked, I don't really have a strong opinion either way.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 21.01.2022 15:46, Lars Ingebrigtsen wrote:
> Lars Ingebrigtsen<larsi@gnus.org>  writes:
> 
>> Dmitry Gutov<dgutov@yandex.ru>  writes:
>>
>>> Consider changing the behavior instead, though.
>>>
>>> Yes, it's been like this for a long time, but I imagine most users
>>> won't notice the change.
>>>
>>> We could experiment on master.
>> I'd rather not change something as subtle as this (especially in a
>> mechanism that's been around for a while like this as).
> Nobody had any further opinions here in a month, so I went ahead and
> changed the doc string.  If somebody feels strongly that the semantics
> should be tweaked, I don't really have a strong opinion either way.

Hi Lars,

The doc change you have pushed in 62a84eea34c33bd1d4b1 misses the point, 
which leads me to believe that I have not explained the problem well.

The issue is not that an entry in completion-category-overrides doesn't 
override all properties wholesale, that is falls back to defaults for 
properties not specified among the overrides.

The issue is that when the 'styles' property is looked up (which is 99% 
of the uses of this variable), the override value is not used as-is. 
Instead, the default value is appended.

So the fix I had in mind looks like:

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index d58c23af8f..0aee55f33c 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1043,7 +1043,7 @@ completion--styles
    (let* ((cat (completion-metadata-get metadata 'category))
           (over (completion--category-override cat 'styles)))
      (if over
-        (delete-dups (append (cdr over) (copy-sequence completion-styles)))
+        (cdr over)
         completion-styles)))

  (defun completion--nth-completion (n string table pred point metadata)

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> So the fix I had in mind looks like:
>
> diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
> index d58c23af8f..0aee55f33c 100644
> --- a/lisp/minibuffer.el
> +++ b/lisp/minibuffer.el
> @@ -1043,7 +1043,7 @@ completion--styles
>    (let* ((cat (completion-metadata-get metadata 'category))
>           (over (completion--category-override cat 'styles)))
>      (if over
> -        (delete-dups (append (cdr over) (copy-sequence completion-styles)))
> +        (cdr over)
>         completion-styles)))

Oh, I see.  Hm...  that would change the behaviour for those that depend
on this working the old way.

Perhaps it'd make more sense to add a new variable to allow real overrides?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 24.01.2022 11:46, Lars Ingebrigtsen wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> So the fix I had in mind looks like:
>>
>> diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
>> index d58c23af8f..0aee55f33c 100644
>> --- a/lisp/minibuffer.el
>> +++ b/lisp/minibuffer.el
>> @@ -1043,7 +1043,7 @@ completion--styles
>>     (let* ((cat (completion-metadata-get metadata 'category))
>>            (over (completion--category-override cat 'styles)))
>>       (if over
>> -        (delete-dups (append (cdr over) (copy-sequence completion-styles)))
>> +        (cdr over)
>>          completion-styles)))
> 
> Oh, I see.  Hm...  that would change the behaviour for those that depend
> on this working the old way.

My theory is that this has never been reported because people never 
really notice the distinction in practice since if the completions 
styles are customized, that's usually done in favor of more lax ones.

So appending the default styles at the end doesn't affect the behavior 
in 99% of the cases. But it does add some CPU usage/latency in the "no 
matches" case, because that's the only case when the failover happens.

There might be a few users out there, of course, who rely on the current 
behavior, but I'm not sure what their use cases would be, and there 
can't be many people like that either because the completion styles 
stuff is fairly obscure. People have been exploring alternative 
front-ends recently that plug into CAPF, but 
completion-category-overrides is probably not a popular variable to 
customize still.

> Perhaps it'd make more sense to add a new variable to allow real overrides?

I don't know. What would it be called? 
completion-category-overrides-for-real-now?

Given that the only times people are likely to notice the distinction 
are some odd edge cases (and the extra lag is not so big or obvious), 
there will be even fewer occasions for people to learn about and 
customize the new var.

Unless we obsolete the previous one, which would be a fair approach, if 
we knew that it actually has a fair amount of users who need to migrate.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Given that the only times people are likely to notice the distinction
> are some odd edge cases (and the extra lag is not so big or obvious),
> there will be even fewer occasions for people to learn about and
> customize the new var.
>
> Unless we obsolete the previous one, which would be a fair approach,
> if we knew that it actually has a fair amount of users who need to
> migrate.

Hard to say.  Perhaps searching on Github for usages of the variable
might give us a clue?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 25.01.2022 14:19, Lars Ingebrigtsen wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> Given that the only times people are likely to notice the distinction
>> are some odd edge cases (and the extra lag is not so big or obvious),
>> there will be even fewer occasions for people to learn about and
>> customize the new var.
>>
>> Unless we obsolete the previous one, which would be a fair approach,
>> if we knew that it actually has a fair amount of users who need to
>> migrate.
> 
> Hard to say.  Perhaps searching on Github for usages of the variable
> might give us a clue?

Sure.

https://github.com/search?q=%22completion-category-overrides%22&type=code should 
a grand total of 2,433 matches, but the vast majority of those seem to 
be users of Orderless who use this scheme:

   (setq completion-styles '(orderless))
   (setq completion-category-defaults nil)
   (setq completion-category-overrides '((file (styles 
partial-completion))))

Some add 'basic' before 'partial-completion' for better "Tramp hostname 
completion".

Do these users expect the failover to the orderless style when 
partial-completion fails to complete? Possible. Orderless is more lax.

But also kinda doubtful since partial-completion is fairly powerful by 
itself, and entering the segments of a file name in a different order is 
not something does often or intentionally.

Apparently this config is recommended in the README of both Corfu and 
Vertico (https://github.com/minad/vertico#configuration), both projects 
by Daniel Mendler.

I guess we should ask Daniel whether he has been aware of the 
completion-styles failover mechanic, and what he thinks about it.

***

And there are occasionally configs like

   (setq completion-styles
         '(partial-completion substring initials basic))
   ;; override default completion style of specific modes
   (setq completion-category-overrides
         '((file (styles . (flex)))
           (buffer (styles . (flex)))
           (project-file (styles . (flex)))
           (info-menu (styles . (flex)))))

where the override is explicitly more powerful, so we know that failover 
is not desirable, just wasted CPU cycles.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Daniel Mendler]

Hello Dmitry,

thank you for pinging me.

> Apparently this config is recommended in the README of both Corfu and 
> Vertico (https://github.com/minad/vertico#configuration), both projects 
> by Daniel Mendler.
> 
> I guess we should ask Daniel whether he has been aware of the 
> completion-styles failover mechanic, and what he thinks about it.

Yes, I've been aware of the failover mechanic and I think it is not
good. From my experience, I would prefer if the override is a real
override, which this issue is about. Users can still opt-in to the
default styles on a case by case basis if this is desired.

The assessment that the variable `completion-category-overrides` is
modified rarely might have been true, but this is not the case anymore.
In the context of the GNU ELPA packages Orderless, Vertico, Consult,
Embark, Marginalia etc., we educate users to make heavy use of this
variable since it permits fine tuning of the completion behavior
depending on the completion category. We also use the completion
category heavily in Marginalia and Embark to determine the type of the
candidates for annotations and minibuffer actions.

Unfortunately the configurations you mentioned, Dmitry, make use of the
failover mechanism. I indeed want to use partial-completion and then
failover to the orderless default. If there wouldn't be a failover I
would have of course recommended another override (partial-completion +
orderless).

Nevertheless I would appreciate the removal (or replacing) of the
failover mechanism. I noticed that it has lead to confusion for some
users before. It leads to performance issues when the slow default sets
in as has been pointed out already. Since currently there is no
possibility to really override the completion styles except via an
advice, removing the failover seems like a good idea.

Furthermore we've also got `completion-category-defaults`. It may make
sense to distinguish them by making the override a real override and
keep the current behavior for the defaults. The alternative would be a
deprecation of the override variable and the introduction of new
variable which is a real override.

One final statement regarding making a breaking change here: You should
consider that the packages orderless etc are fairly new and still have
breaking changes from time to time, so even if these configurations are
widespread or see growing adoption, this should not hold you back from
making a breaking change. I will then promptly adapt the documentation
of these projects and add a warning note, which will soon propagate to
the users who use Emacs master, which is still young in the development
cycle. Of course my statement applies only if the aforementioned are
truly the only widespread packages affected by this.

Daniel

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Apparently this config is recommended in the README of both Corfu and
>> Vertico (https://github.com/minad/vertico#configuration), both projects
>> by Daniel Mendler.
>> I guess we should ask Daniel whether he has been aware of the
>> completion-styles failover mechanic, and what he thinks about it.
> Yes, I've been aware of the failover mechanic and I think it is not
> good. From my experience, I would prefer if the override is a real
> override, which this issue is about. Users can still opt-in to the
> default styles on a case by case basis if this is desired.

We can probably have our cake and eat it too by adding a `fail`
completion style.  Such a style would always take responsibility
(i.e. would never return nil to delegate to subsequent styles), but
would always return a "useless" value that gives no completions.

Then entries in `completion-category-override/defaults` could choose to
either be mere additions to the global default (as now) or be a full
override (which they'd get by having this `fail` style at the end).

> Furthermore we've also got `completion-category-defaults`. It may make
> sense to distinguish them by making the override a real override and
> keep the current behavior for the defaults.

No, the role of those two is already quite different:
`completion-category-defaults` is for packages to set, whereas
`completion-category-override` is to be set by end-users.


        Stefan

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Daniel Mendler]

> We can probably have our cake and eat it too by adding a `fail`
> completion style.  Such a style would always take responsibility
> (i.e. would never return nil to delegate to subsequent styles), but
> would always return a "useless" value that gives no completions.
> 
> Then entries in `completion-category-override/defaults` could choose to
> either be mere additions to the global default (as now) or be a full
> override (which they'd get by having this `fail` style at the end).

This sounds like a good idea to solve the issue and retain backward
compatibility. It is like eating the cake backwards :) A `fail'
completion style is pretty trivial. It boils down to adding `(fail
ignore ignore "Fail with no completions")` to the
`completion-styles-alist`, or did I miss something?

>> Furthermore we've also got `completion-category-defaults`. It may make
>> sense to distinguish them by making the override a real override and
>> keep the current behavior for the defaults.
> 
> No, the role of those two is already quite different:
> `completion-category-defaults` is for packages to set, whereas
> `completion-category-override` is to be set by end-users.

I am aware of this distinction, but I chose to ignore it. Calling it
"quite different" feels like an exaggeration, given that Emacs is
supposed to be configurable throughout by the user - of course this is
only my interpretation. I usually override
`completion-category-defaults` since I want to control the completion
precisely myself and I don't like if packages interfere with that. But
this is probably a special preference of someone who wrote multiple
completion UIs and likes to tweak the Orderless matching behavior... ;)
As far as I know `completion-category-defaults` is not used widely. I
don't have a single package installed which makes use of this
functionality, but once again, this is probably not representative.

Daniel

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> This sounds like a good idea to solve the issue and retain backward
> compatibility. It is like eating the cake backwards :) A `fail'
> completion style is pretty trivial. It boils down to adding `(fail
> ignore ignore "Fail with no completions")` to the
> `completion-styles-alist`, or did I miss something?

No, because `ignore` will return nil and so we'll just keep going to the
next style.  We need try/all-completion functions for this style to
return a non-nil value but that is like "no completion".

I suspect it can't be done quite right without changing `minibuffer.el`,
but we can probably get close enough to be tolerable with older Emacsen.

E.g. for the try-completion case, I think we can return (STRING . POINT)
and for all-completions maybe returning `0` will do the trick.

> I am aware of this distinction, but I chose to ignore it. Calling it
> "quite different" feels like an exaggeration, given that Emacs is
> supposed to be configurable throughout by the user - of course this is
> only my interpretation.

The users are indeed free to mess with anything they like, of course.
But packages are allowed to change `completion-category-defaults`
whereas they're not allowed to change `completion-category-override`.

While we're here, let's not forget the other problem with
`completion-category-defaults` which is when a package puts something
like `substring` in it because `partial-completion` is not "good
enough": this can end up taking precedence over the users's
customization of the default to something like `flex` which is probably
not what's wanted.

I'm not completely sure how to fix that one.  An cheap solution would be
to allow `completion-category-defaults` to specify a function rather
than a list of styles, and then this function will return the list of
styles to use so it can dynamically adapt to the user's chosen default.
But it's kind of a cop out because that function will need to "guess"
what is the relationship between the various styles, which is the crux
of the matter.

This problem doesn't apply to `completion-category-override` since we
can consider it to be the user's responsability to take `completion-styles`
into account when setting `completion-category-override`.


        Stefan

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Daniel Mendler]

On 1/26/22 18:19, Stefan Monnier wrote:
> No, because `ignore` will return nil and so we'll just keep going to the
> next style.  We need try/all-completion functions for this style to
> return a non-nil value but that is like "no completion".
> 
> I suspect it can't be done quite right without changing `minibuffer.el`,
> but we can probably get close enough to be tolerable with older Emacsen.
> 
> E.g. for the try-completion case, I think we can return (STRING . POINT)
> and for all-completions maybe returning `0` will do the trick.

Okay, right. This makes the proposal a bit less appealing to be honest,
since we end up with a hack, where the result is something like a
non-nil invalid completion result. Hmm. So maybe we shouldn't do this
and fix the problem at the root? Remove the failover mechanism? I am not
fond of introducing a hack to work around the problematic failover design.
> While we're here, let's not forget the other problem with
> `completion-category-defaults` which is when a package puts something
> like `substring` in it because `partial-completion` is not "good
> enough": this can end up taking precedence over the users's
> customization of the default to something like `flex` which is probably
> not what's wanted.

Exactly. This is the reason why I reset `completion-category-defaults`
in my configuration and I also recommend this in the README of the
packages. I should probably describe the problem more explicitly. Often
users copy example configurations from packages without investigating
the implications. Furthermore in the case of completion styles the
implications are often not that obvious. One more reason to remove
complexity if possible...

> I'm not completely sure how to fix that one.  An cheap solution would be
> to allow `completion-category-defaults` to specify a function rather
> than a list of styles, and then this function will return the list of
> styles to use so it can dynamically adapt to the user's chosen default.
> But it's kind of a cop out because that function will need to "guess"
> what is the relationship between the various styles, which is the crux
> of the matter.
> 
> This problem doesn't apply to `completion-category-override` since we
> can consider it to be the user's responsability to take `completion-styles`
> into account when setting `completion-category-override`.

My cheap proposal would be the removal of `completion-category-defaults`
and the removal of the failover mechanism of
`completion-category-overrides`. Then the user can adjust the
`completion-styles` entirely in their configuration. This is maybe not
the most user friendly solution in the first place. But as soon as you
start to tweak the completion styles it makes things simpler and easier
to understand. My opinion in this case is admittedly a bit radical since
I propose that users adjust their completion styles heavily to unlock a
lot of potential (for example look at orderless and the flexible
orderless style dispatchers). But such adjustments may not be for everyone.

Daniel

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> No, because `ignore` will return nil and so we'll just keep going to the
>> next style.  We need try/all-completion functions for this style to
>> return a non-nil value but that is like "no completion".
>> 
>> I suspect it can't be done quite right without changing `minibuffer.el`,
>> but we can probably get close enough to be tolerable with older Emacsen.
>> 
>> E.g. for the try-completion case, I think we can return (STRING . POINT)
>> and for all-completions maybe returning `0` will do the trick.
>
> Okay, right. This makes the proposal a bit less appealing to be honest,
> since we end up with a hack, where the result is something like a
> non-nil invalid completion result. Hmm.

I think it should be easy enough to make the various UIs handle it (in
the worst case, add an advice to `completion-try-completion` and
`completion-all-completions`).

Of course it's a hack, but only one needed during the transition.
Like my gym coach used to say when we had an injury: you won't remember
it in 10 years.

> So maybe we shouldn't do this and fix the problem at the root?
> Remove the failover mechanism?

Not sure how that'll help with Emacs<29 ;-)

> My cheap proposal would be the removal of `completion-category-defaults`
> and the removal of the failover mechanism of
> `completion-category-overrides`. Then the user can adjust the
> `completion-styles` entirely in their configuration.

The purpose of `completion-category-defaults` is to improve the OOTB
behavior for those who don't customize very much.
Those who do customize heavily can easily set it to nil or override it
with `completion-category-overrides`.


        Stefan

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Daniel Mendler]

On 1/26/22 23:57, Stefan Monnier wrote:
>> So maybe we shouldn't do this and fix the problem at the root?
>> Remove the failover mechanism?
> 
> Not sure how that'll help with Emacs<29 ;-)

Easy. If advices are allowed as temporary workaround, I will just advise
`completion--styles` or `completion--category-override` appropriately
such that the fail over is gone.

Let's remove the fail over mechanism! :)

Daniel

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Daniel Mendler [2022-01-27 00:32:29] wrote:
> On 1/26/22 23:57, Stefan Monnier wrote:
>>> So maybe we shouldn't do this and fix the problem at the root?
>>> Remove the failover mechanism?
>> Not sure how that'll help with Emacs<29 ;-)
> Easy. If advices are allowed as temporary workaround, I will just advise
> `completion--styles` or `completion--category-override` appropriately
> such that the fail over is gone.

But then you can use the same approach to implement proper support for
a new `fail` pseudo-style ;-)


        Stefan

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 26.01.2022 20:59, Daniel Mendler wrote:
> On 1/26/22 18:19, Stefan Monnier wrote:
>> No, because `ignore` will return nil and so we'll just keep going to the
>> next style.  We need try/all-completion functions for this style to
>> return a non-nil value but that is like "no completion".
>>
>> I suspect it can't be done quite right without changing `minibuffer.el`,
>> but we can probably get close enough to be tolerable with older Emacsen.
>>
>> E.g. for the try-completion case, I think we can return (STRING . POINT)
>> and for all-completions maybe returning `0` will do the trick.
> Okay, right. This makes the proposal a bit less appealing to be honest,
> since we end up with a hack, where the result is something like a
> non-nil invalid completion result. Hmm. So maybe we shouldn't do this
> and fix the problem at the root? Remove the failover mechanism? I am not
> fond of introducing a hack to work around the problematic failover design.

Let's remove it, I think.

IMHO, backward compatibility hacks are the reason of >50% of existing 
CAPF's problems: you add one special case, then another, then another.

Each step isn't bad by itself (just like Stefan's current suggestion 
sounds workable), but every one of them complicates reading the code, 
and writing code to it, even if by a little.

If we were designing it from the ground up, we probably wouldn't add an 
'ignore' style. We could have added a special value like 't' which would 
mean the opposite (*do* the fallback, for those users who would want 
their configs to be just a little bit more terse), just like in the 
local values of hook variables. But I'm not sure how kludgy the 
implementation of this will turn out either, and terseness is not the 
primary end goal for this part of user customization, I think.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 26.01.2022 15:36, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
> We can probably have our cake and eat it too by adding a `fail`
> completion style.  Such a style would always take responsibility
> (i.e. would never return nil to delegate to subsequent styles), but
> would always return a "useless" value that gives no completions.

*If* we did end up adding such style, would the default value of 
completion-category-defaults include 'fail' at the end of its every line?

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 26.01.2022 04:31, Daniel Mendler wrote:
> One final statement regarding making a breaking change here: You should
> consider that the packages orderless etc are fairly new and still have
> breaking changes from time to time, so even if these configurations are
> widespread or see growing adoption, this should not hold you back from
> making a breaking change. I will then promptly adapt the documentation
> of these projects and add a warning note, which will soon propagate to
> the users who use Emacs master, which is still young in the development
> cycle. Of course my statement applies only if the aforementioned are
> truly the only widespread packages affected by this.

Yeah, I think the current userbase of Orderless should be pretty 
dynamic, not too set in their ways.

So they should be up for checking out new versions and documentation 
changes and adapting to them.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Daniel Mendler]

On 1/28/22 03:35, Dmitry Gutov wrote:
> Let's remove it, I think.
> 
> IMHO, backward compatibility hacks are the reason of >50% of existing 
> CAPF's problems: you add one special case, then another, then another.
> 
> Each step isn't bad by itself (just like Stefan's current suggestion 
> sounds workable), but every one of them complicates reading the code, 
> and writing code to it, even if by a little.
> 
> If we were designing it from the ground up, we probably wouldn't add an 
> 'ignore' style. We could have added a special value like 't' which would 
> mean the opposite (*do* the fallback, for those users who would want 
> their configs to be just a little bit more terse), just like in the 
> local values of hook variables. But I'm not sure how kludgy the 
> implementation of this will turn out either, and terseness is not the 
> primary end goal for this part of user customization, I think.

I agree with you. Removing the fail over seems better in the long term.
For the users of Orderless and other custom documentation styles etc, we
can document that the fail over mechanism has been removed and provide
an advice in the documentation, which can be used to upgrade the behavior.

The user base of Orderless etc is dynamic as you point out, so they will
adapt to changes. If we assume that these users make up the majority of
users who make use of the fail over, this should not be a blocker for
this minor breaking change. Furthermore I've got multiple reports where
people wondered about the failover, so the current behavior is more
confusing than having no failover. Using `t` as last element to trigger
fail over seems like a good idea.

Daniel

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> If we were designing it from the ground up, we probably wouldn't add an
> 'ignore' style. We could have added a special value like 't' which would
> mean the opposite (*do* the fallback, for those users who would want their
> configs to be just a little bit more terse),

FWIW, the choice of using a fallback to `completion-style` was made for
`completion-category-defaults` so that those package-choices don't
unilaterally override the user's choice in `completion-style`.

For `completion-category-override` there is indeed not much need for
a fallback, since it's set by the same person as `completion-style`.


        Stefan

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Dmitry Gutov [2022-01-28 04:37:36] wrote:
> On 26.01.2022 15:36, Stefan Monnier via Bug reports for GNU Emacs, the Swiss
> army knife of text editors wrote:
>> We can probably have our cake and eat it too by adding a `fail`
>> completion style.  Such a style would always take responsibility
>> (i.e. would never return nil to delegate to subsequent styles), but
>> would always return a "useless" value that gives no completions.
> *If* we did end up adding such style, would the default value of
>  completion-category-defaults include 'fail' at the end of its every line?

No, since the fallback was chosen specifically for
`completion-category-defaults`, we'd want to keep it for the cases that
are already present at least.


        Stefan

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 28.01.2022 18:59, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
> Dmitry Gutov [2022-01-28 04:37:36] wrote:
>> On 26.01.2022 15:36, Stefan Monnier via Bug reports for GNU Emacs, the Swiss
>> army knife of text editors wrote:
>>> We can probably have our cake and eat it too by adding a `fail`
>>> completion style.  Such a style would always take responsibility
>>> (i.e. would never return nil to delegate to subsequent styles), but
>>> would always return a "useless" value that gives no completions.
>> *If*  we did end up adding such style, would the default value of
>>   completion-category-defaults include 'fail' at the end of its every line?
> No, since the fallback was chosen specifically for
> `completion-category-defaults`, we'd want to keep it for the cases that
> are already present at least.

Depends on how you look at it: out of 6 entries in there currently, I 
added two of them, and since I was unaware of the fallback mechanism, it 
wasn't quite intentional. Even if certain edge cases ended up using the 
fallback to their benefit.

Knowing that now, I would probably rewrite those 
completion-category-defaults to include all styles that would be tried 
explicitly (meaning, add 'partial-completion' after 'substring').

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 28.01.2022 18:56, Stefan Monnier wrote:
>> If we were designing it from the ground up, we probably wouldn't add an
>> 'ignore' style. We could have added a special value like 't' which would
>> mean the opposite (*do* the fallback, for those users who would want their
>> configs to be just a little bit more terse),
> 
> FWIW, the choice of using a fallback to `completion-style` was made for
> `completion-category-defaults` so that those package-choices don't
> unilaterally override the user's choice in `completion-style`.
> 
> For `completion-category-override` there is indeed not much need for
> a fallback, since it's set by the same person as `completion-style`.

That seems to argue for Daniel's original suggestion: to make 
'-overrides' a "real" override and keep the composition behavior for the 
'-defaults' variable.

Trying to honor the user's customization of 'completion-styles' makes a 
certain amount of sense. Though I don't know how much we honor it this 
way: if the user is relatively new, they might not even know to keep 
typing to see the fallback, after noting that their input does not give 
them the matches they expected.

It's more of a critique of the whole "list of styles" design, admittedly.

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Trying to honor the user's customization of 'completion-styles' makes
> a certain amount of sense. Though I don't know how much we honor it this
> way: if the user is relatively new, they might not even know to keep typing
> to see the fallback, after noting that their input does not give them the
> matches they expected.

Actually, I suspect it works better for new users than for old ones: new
users don't yet have a clear mental model of how Emacs's completion
works so they might expect something more like what you see with a web
browser search where adding more data can completely change the
proposed completions.

In contrast old-timers may indeed "know" that there won't be any
completions further down and will never reach the second style (to some
extent, that's how I got Richard to accept `partial-completion` in the
default).

> It's more of a critique of the whole "list of styles" design, admittedly.

I don't regret doing it because I don't think there was any other
way to activate `partial-completion` by default, but yes it has
its downsides.


        Stefan

bug#22324: 25.0.50; completion-category-defaults style doesn't override completion-styles (gets prepended instead)

[Dmitry Gutov]

On 29.01.2022 01:18, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
>> Trying to honor the user's customization of 'completion-styles' makes
>> a certain amount of sense. Though I don't know how much we honor it this
>> way: if the user is relatively new, they might not even know to keep typing
>> to see the fallback, after noting that their input does not give them the
>> matches they expected.
> 
> Actually, I suspect it works better for new users than for old ones: new
> users don't yet have a clear mental model of how Emacs's completion
> works so they might expect something more like what you see with a web
> browser search where adding more data can completely change the
> proposed completions.

But a web browser model is more like a single 'flex' completion style. 
You get fuzzy matching and a score-based searching.

After my input in the address bar makes the completions list shrink to 
just a few entries, it never occurs to me to keep typing to see 
something else that is not in that list.

In Firefox that will never work -- but it works with completion styles 
in Emacs.

> In contrast old-timers may indeed "know" that there won't be any
> completions further down and will never reach the second style (to some
> extent, that's how I got Richard to accept `partial-completion` in the
> default).
> 
>> It's more of a critique of the whole "list of styles" design, admittedly.
> 
> I don't regret doing it because I don't think there was any other
> way to activate `partial-completion` by default, but yes it has
> its downsides.

Sure. I'm just not sure where we go from here.

A "single completion style" approach (with sorting and scoring) would 
make things easier and familiar in the long run. But if we stay with the 
current approach (which has its upsides), seems like Daniel's suggestion 
can be a good option for this particular bug.