bug#39484: 26.3; try-completion bug

bug#39484: 26.3; try-completion bug

[Wanrong Lin]

try-completion does not work as expected when completion-ignore-case is t.

Examples:

(let ((completion-ignore-case t))
        (try-completion "xy" '("XyzD" "XyzAbc"))
        )
==> "Xyz"

Expected. The return value has uppercase "X", since the return value 
should be the maximum substring among all the *candidates*.

(let ((completion-ignore-case t))
        (try-completion "xyz" '("XyzD" "XyzAbc"))
        )
==> "xyz"

*Unexpected*. Should be "Xyz".

Thanks for looking into this.

Wanrong

bug#39484: 26.3; try-completion bug

[Lars Ingebrigtsen]

Wanrong Lin <wrglin@gmail.com> writes:

> (let ((completion-ignore-case t))
>        (try-completion "xyz" '("XyzD" "XyzAbc"))
>        )
> ==> "xyz"
>
> *Unexpected*. Should be "Xyz".

That is somewhat odd, especially considering:

(let ((completion-ignore-case t))
  (try-completion "xyz" '("XyzaD" "Xyzabc")))
=> "Xyza"

Stefan?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#39484: 26.3; try-completion bug

[Andreas Schwab]

On Okt 27 2020, Lars Ingebrigtsen wrote:

> Wanrong Lin <wrglin@gmail.com> writes:
>
>> (let ((completion-ignore-case t))
>>        (try-completion "xyz" '("XyzD" "XyzAbc"))
>>        )
>> ==> "xyz"
>>
>> *Unexpected*. Should be "Xyz".
>
> That is somewhat odd

  /* If we are ignoring case, and there is no exact match,
     and no additional text was supplied,
     don't change the case of what the user typed.  */

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#39484: 26.3; try-completion bug

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>   /* If we are ignoring case, and there is no exact match,
>      and no additional text was supplied,
>      don't change the case of what the user typed.  */

Should this be mentioned in the doc string?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#39484: 26.3; try-completion bug

[Wanrong Lin]

I am curious what is the rationale behind this behavior?

I remember this behavior gives me problem in "ido" trying to expand the 
user input to the max common matched text among all candidates, because 
as you can see, the "expansion" using this method may end up returning 
something that is not common at all for all the candidates.

On 10/27/2020 3:17 PM, Lars Ingebrigtsen wrote:
> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>>    /* If we are ignoring case, and there is no exact match,
>>       and no additional text was supplied,
>>       don't change the case of what the user typed.  */
> Should this be mentioned in the doc string?
>

bug#39484: 26.3; try-completion bug

[Andreas Schwab]

On Okt 27 2020, Wanrong Lin wrote:

> I remember this behavior gives me problem in "ido" trying to expand the
> user input to the max common matched text among all candidates, because 
> as you can see, the "expansion" using this method may end up returning
> something that is not common at all for all the candidates.

It isn't?  "xyz" surely is a common prefix of "XyzD" and "XyzAbc",
ignoring case.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#39484: 26.3; try-completion bug

[Wanrong Lin]

My understanding is: "ignoring case" here applies to the comparison 
part, not the returning part. What makes me think so:

1. That variable name is "completion-ignore-case", meaning, complete as 
much as possible ignoring case in comparison. But still, the action is 
to "complete", and "complete" should use the original candidate text 
when possible.
2.  the first example did return the original text in the candidate 
instead of the user input.
(let ((completion-ignore-case t))
        (try-completion "xy" '("XyzD" "XyzAbc"))
        )
==> "Xyz"

I am not saying other behaviors are "wrong", but not ideal in my view, 
and there should be some rationale (performance?) behind it.

Wanrong

On 10/27/2020 4:28 PM, Andreas Schwab wrote:
> On Okt 27 2020, Wanrong Lin wrote:
>
>> I remember this behavior gives me problem in "ido" trying to expand the
>> user input to the max common matched text among all candidates, because
>> as you can see, the "expansion" using this method may end up returning
>> something that is not common at all for all the candidates.
> It isn't?  "xyz" surely is a common prefix of "XyzD" and "XyzAbc",
> ignoring case.
>
> Andreas.
>

bug#39484: 26.3; try-completion bug

[Andreas Schwab]

So what should (try-completion "abc" '("Abcde" "aBcdf" "abCdg")) return?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#39484: 26.3; try-completion bug

[Wanrong Lin]

For this case, "abc" might be OK, unless you think there is another 
better answer. But in the example I gave, don't you think "Xyz" is 
obviously a better answer than "xyz"?

The argument here is for some examples, the function does not return the 
best answer, and also, does not seem to be consistent to me (for those 
two examples I gave). There might be a reason behind this "compromise", 
or, it might be just an arbitrary decision as documented in the comments 
you gave. If it is the latter, can we improve on it? I think those are 
valid questions.

Wanrong

On 10/27/2020 5:21 PM, Andreas Schwab wrote:
> So what should (try-completion "abc" '("Abcde" "aBcdf" "abCdg")) return?
>
> Andreas.
>

bug#39484: 26.3; try-completion bug

[Wanrong Lin]

Sorry,  I meant "abcd" as the answer for your example.

On 10/27/2020 8:44 PM, Wanrong Lin wrote:
> For this case, "abc" might be OK, unless you think there is another 
> better answer. But in the example I gave, don't you think "Xyz" is 
> obviously a better answer than "xyz"?
>
> The argument here is for some examples, the function does not return 
> the best answer, and also, does not seem to be consistent to me (for 
> those two examples I gave). There might be a reason behind this 
> "compromise", or, it might be just an arbitrary decision as documented 
> in the comments you gave. If it is the latter, can we improve on it? I 
> think those are valid questions.
>
> Wanrong
>
> On 10/27/2020 5:21 PM, Andreas Schwab wrote:
>> So what should (try-completion "abc" '("Abcde" "aBcdf" "abCdg")) return?
>>
>> Andreas.
>>
>

bug#39484: 26.3; try-completion bug

[Wanrong Lin]

I just tried this in Emacs 26.3:

(let ((completion-ignore-case t))
   (try-completion "abc" '("Abcde" "aBcdf" "abCdg"))) -> "Abcd"

The answer is not bad, matching at least one of the candidates. That 
makes the answer to my original example even more odd:
(let ((completion-ignore-case t))
        (try-completion "xyz" '("XyzD" "XyzAbc"))
        )
==> "xyz"

Why does it matter whether any additional chars follow the user input in 
the match?

Wanrong

On 10/27/2020 8:47 PM, Wanrong Lin wrote:
> Sorry, I meant "abcd" as the answer for your example.
>
> On 10/27/2020 8:44 PM, Wanrong Lin wrote:
>> For this case, "abc" might be OK, unless you think there is another 
>> better answer. But in the example I gave, don't you think "Xyz" is 
>> obviously a better answer than "xyz"?
>>
>> The argument here is for some examples, the function does not return 
>> the best answer, and also, does not seem to be consistent to me (for 
>> those two examples I gave). There might be a reason behind this 
>> "compromise", or, it might be just an arbitrary decision as 
>> documented in the comments you gave. If it is the latter, can we 
>> improve on it? I think those are valid questions.
>>
>> Wanrong
>>
>> On 10/27/2020 5:21 PM, Andreas Schwab wrote:
>>> So what should (try-completion "abc" '("Abcde" "aBcdf" "abCdg")) 
>>> return?
>>>
>>> Andreas.
>>>
>>
>

bug#39484: 26.3; try-completion bug

[Andreas Schwab]

On Okt 27 2020, Wanrong Lin wrote:

> The answer is not bad, matching at least one of the candidates. That makes
> the answer to my original example even more odd:
> (let ((completion-ignore-case t))
>        (try-completion "xyz" '("XyzD" "XyzAbc"))
>        )
> ==> "xyz"

That also matches one of the candidates (since case doesn't matter).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#39484: 26.3; try-completion bug

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> On Okt 27 2020, Wanrong Lin wrote:
>
>> The answer is not bad, matching at least one of the candidates. That makes
>> the answer to my original example even more odd:
>> (let ((completion-ignore-case t))
>>        (try-completion "xyz" '("XyzD" "XyzAbc"))
>>        )
>> ==> "xyz"
>
> That also matches one of the candidates (since case doesn't matter).

It does, but it's weird.  I mean, this pair of return values don't make
sense on any level:

(let ((completion-ignore-case t))
  (try-completion "xyz" '("XyzaD" "XyZabc")))
=> "Xyza"

(let ((completion-ignore-case t))
  (try-completion "xyz" '("XyZbD" "XyZabc")))
=> "xyz"  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#39484: 26.3; try-completion bug

[Andreas Schwab]

On Okt 28 2020, Lars Ingebrigtsen wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> On Okt 27 2020, Wanrong Lin wrote:
>>
>>> The answer is not bad, matching at least one of the candidates. That makes
>>> the answer to my original example even more odd:
>>> (let ((completion-ignore-case t))
>>>        (try-completion "xyz" '("XyzD" "XyzAbc"))
>>>        )
>>> ==> "xyz"
>>
>> That also matches one of the candidates (since case doesn't matter).
>
> It does, but it's weird.  I mean, this pair of return values don't make
> sense on any level:
>
> (let ((completion-ignore-case t))
>   (try-completion "xyz" '("XyzaD" "XyZabc")))
> => "Xyza"
>
> (let ((completion-ignore-case t))
>   (try-completion "xyz" '("XyZbD" "XyZabc")))
> => "xyz"  

It makes a whole lot sense.  If the candidate is not modified, it
should not receive any case changes either.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#39484: 26.3; try-completion bug

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> It does, but it's weird.  I mean, this pair of return values don't make
>> sense on any level:
>>
>> (let ((completion-ignore-case t))
>>   (try-completion "xyz" '("XyzaD" "XyZabc")))
>> => "Xyza"
>>
>> (let ((completion-ignore-case t))
>>   (try-completion "xyz" '("XyZbD" "XyZabc")))
>> => "xyz"  
>
> It makes a whole lot sense.  If the candidate is not modified, it
> should not receive any case changes either.

But the case change in the modified case is essentially arbitrary, while
the unmodified case has a unique, non-arbitrary match.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#39484: 26.3; try-completion bug

[Andreas Schwab]

On Okt 28 2020, Lars Ingebrigtsen wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>>> It does, but it's weird.  I mean, this pair of return values don't make
>>> sense on any level:
>>>
>>> (let ((completion-ignore-case t))
>>>   (try-completion "xyz" '("XyzaD" "XyZabc")))
>>> => "Xyza"
>>>
>>> (let ((completion-ignore-case t))
>>>   (try-completion "xyz" '("XyZbD" "XyZabc")))
>>> => "xyz"  
>>
>> It makes a whole lot sense.  If the candidate is not modified, it
>> should not receive any case changes either.
>
> But the case change in the modified case is essentially arbitrary, while
> the unmodified case has a unique, non-arbitrary match.

The case always matches one of the candidates.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#39484: 26.3; try-completion bug

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> The case always matches one of the candidates.

Yes.  An arbitrary candidate (well, the first one, but it's not
documented, which means that it's arbitrary).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#39484: 26.3; try-completion bug

[Andreas Schwab]

On Okt 28 2020, Lars Ingebrigtsen wrote:

> Andreas Schwab <schwab@linux-m68k.org> writes:
>
>> The case always matches one of the candidates.
>
> Yes.  An arbitrary candidate (well, the first one, but it's not
> documented, which means that it's arbitrary).

It is not nessesary to document each and every implementation detail.
That makes it difficult to change the implementation later, and it also
unduly restricts the way a collection function can handle that
situation.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#39484: 26.3; try-completion bug

[Wanrong Lin]

This is not implementation detail. This is the interface of this 
function. User/caller should know exactly what are the rules for the 
return value.

So in summary, my two points:

1. Return value is not ideal. You can argue it is still not wrong, but I 
think we can improve.
2. Even if we don't change anything here, we need to make this 
less-than-ideal behavior clear in the documentation (doc string).

I hope I convinced at least some of you.

Wanrong

On 10/28/2020 7:44 AM, Andreas Schwab wrote:
> On Okt 28 2020, Lars Ingebrigtsen wrote:
>
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>>
>>> The case always matches one of the candidates.
>> Yes.  An arbitrary candidate (well, the first one, but it's not
>> documented, which means that it's arbitrary).
> It is not nessesary to document each and every implementation detail.
> That makes it difficult to change the implementation later, and it also
> unduly restricts the way a collection function can handle that
> situation.
>
> Andreas.
>

bug#39484: 26.3; try-completion bug

[Stefan Monnier]

> 1. Return value is not ideal. You can argue it is still not wrong, but
>    I think we can improve.

Indeed, it can be improved, but we should not try to be too clever about
it, because some choices might seem obvious in some circumstances but
would result in rather poor answers in other cases.

So rather than hypothetical cases like what we've seen here, I'm much
more interested in real life situations.

The current design is trying to be conservative, in the sense that it
tries to avoid returning a poor result, at the cost of sometimes failing
to return a better result.  It does it by refraining from mix-and-match:
either the whole result comes from the user input or the whole result
comes from *one* of the candidates.

There are cases where `completion-try-completion` (as opposed to
`try-completion`) doesn't actually follow this rule correctly, and it's
been a source of suboptimal results.


        Stefan

bug#39484: 26.3; try-completion bug

[Wanrong Lin]

The examples I gave are real life situations. I discovered this issue 
from an unexpected behavior with ido. The "bug" has real negative 
consequences, at least for me.

As for this:

"It does it by refraining from mix-and-match:
either the whole result comes from the user input or the whole result
comes from *one* of the candidates."

It still sounds quite arbitrary to me, as I failed to understand why it 
is bad if the whole result comes from *all* of the candidates if that 
happens to be possible.

I will try to give a version which I think is better (up to debate, of 
course)

For the user input x, return a string y (or nil if impossible) so that 
it satisfies all three conditions below:

1. x is a prefix of y, ignoring case.

2. y is the maximum common prefix, ignoring case, among all candidates

3. y is the *exact* (including case) prefix of at least one of the 
candidates

Wanrong

On 10/28/2020 10:45 AM, Stefan Monnier wrote:
>> 1. Return value is not ideal. You can argue it is still not wrong, but
>>     I think we can improve.
> Indeed, it can be improved, but we should not try to be too clever about
> it, because some choices might seem obvious in some circumstances but
> would result in rather poor answers in other cases.
>
> So rather than hypothetical cases like what we've seen here, I'm much
> more interested in real life situations.
>
> The current design is trying to be conservative, in the sense that it
> tries to avoid returning a poor result, at the cost of sometimes failing
> to return a better result.  It does it by refraining from mix-and-match:
> either the whole result comes from the user input or the whole result
> comes from *one* of the candidates.
>
> There are cases where `completion-try-completion` (as opposed to
> `try-completion`) doesn't actually follow this rule correctly, and it's
> been a source of suboptimal results.
>
>
>          Stefan
>

bug#39484: 26.3; try-completion bug

[Stefan Monnier]

> The examples I gave are real life situations.

All the examples I see are of the "abcd" or "xyz", i.e. made up examples.
I understand they are meant to illustrate real-life examples, but they
are not themselves real life.

> 1. x is a prefix of y, ignoring case.
> 2. y is the maximum common prefix, ignoring case, among all candidates

That's what is guaranteed by the semantics, so our code already does
that (barring bugs).

> 3. y is the *exact* (including case) prefix of at least one of the
>    candidates

We could try that, indeed.  It still satisfies the constraint
I described as "conservative", and it wouldn't cost extra
efforts either.

If someone wants to try that, feel free to do so, but I personally would
prefer if we only do that when all the matching candidates agree on
the capitalization.

BTW, we currently have the following tests in test/src/minibuffer-tests.el:

    (ert-deftest test-try-completion-ignore-case ()
      (let ((completion-ignore-case t))
        (should (equal (try-completion "bar" '("bAr" "barfoo")) "bAr"))
        (should (equal (try-completion "bar" '("bArfoo" "barbaz")) "bar"))
        (should (equal (try-completion "bar" '("bArfoo" "barbaz"))
                       (try-completion "bar" '("barbaz" "bArfoo"))))
        ;; bug#11339
        (should (equal (try-completion "baz" '("baz" "bAz")) "baz")) ;And not `t'!
        (should (equal (try-completion "baz" '("bAz" "baz"))
                       (try-completion "baz" '("baz" "bAz"))))))


-- Stefan

bug#39484: 26.3; try-completion bug

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The current design is trying to be conservative, in the sense that it
> tries to avoid returning a poor result, at the cost of sometimes failing
> to return a better result.  It does it by refraining from mix-and-match:
> either the whole result comes from the user input or the whole result
> comes from *one* of the candidates.

I've now mentioned something like this in the doc string of the function
in Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no