bug#61532: 30.0.50; [PATCH]: Make completions without sortText fall to back of the list

[Theodor Thornhill via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

João Távora <joaotavora@gmail.com> writes:

> On Sun, Feb 19, 2023 at 4:08 PM Theodor Thornhill <theo@thornhill.no> wrote:
>>
>>
>> >bug tracker list.  Next time, you can add me to the special X-Debbugs-CC:
>> >header]
>> Pretty sure I did that, but it doesn't matter :)
>
> [ I didn't get the message. I went to look in the mbox file off Debbugs and
> I did
> find a 'X-Debbugs-Cc' header, but maybe it has to be 'X-Debbugs-CC'?]
>
>> I don't think it's a bug, really. Isn't it just the flex style greediness?
>
> The 'flex' completion style isn't really doing (or at shouldn't be doing)
> what it does normally. Its purpose in Eglot is only to allow for flex-style
> fontification of the pattern to happen. Nothing more, and that includes
> no sorting.
>
> That's because, contrary to the normal uses of flex, here it's the
> server which does all the selection and the filtering for whatever
> it thinks is a pattern.  It turns out that a very common style of filtering
> among servers is akin to 'flex', so using flex on our side to "paint"
> the pattern in the completion candidate is usually, though not always,
> a good bet.  If the server happens to use 'prefix' ,then 'flex' will also
> paint it correctly, in principle. This is of course presuming we guess
> the filter pattern that the server used, which we're not guaranteed
> to, but more or less always do by looking for a 'symbol' thing-at-point.
>
> Anyway, flex shouldn't be doing any kind of completion sorting for
> eglot-completion-at-point. So if it is doing that, it's IMO a bug (though
> perhaps not a serious one, as it wouldn't be a very absurd sorting
> anyway).
>

I spent some time trying to understand what was happening inside this
function, which seems to sort, and is triggered upon completion.  Maybe
that's a clue?  I have no idea, honestly, but I remember getting
different results when changing > to < in the middle of it:

```

(defun completion--flex-adjust-metadata (metadata)
  "If `flex' is actually doing filtering, adjust sorting."
  (let ((flex-is-filtering-p
         ;; JT@2019-12-23: FIXME: this is kinda wrong.  What we need
         ;; to test here is "some input that actually leads/led to
         ;; flex filtering", not "something after the minibuffer
         ;; prompt".  E.g. The latter is always true for file
         ;; searches, meaning we'll be doing extra work when we
         ;; needn't.
         (or (not (window-minibuffer-p))
             (> (point-max) (minibuffer-prompt-end))))
        (existing-dsf
         (completion-metadata-get metadata 'display-sort-function))
        (existing-csf
         (completion-metadata-get metadata 'cycle-sort-function)))
    (cl-flet
        ((compose-flex-sort-fn
          (existing-sort-fn) ; wish `cl-flet' had proper indentation...
          (lambda (completions)
            (sort
             (funcall existing-sort-fn completions)
             (lambda (c1 c2)
               (let ((s1 (get-text-property 0 'completion-score c1))
                     (s2 (get-text-property 0 'completion-score c2)))
                 (> (or s1 0) (or s2 0)))))))) ;; This line here
      `(metadata
        ,@(and flex-is-filtering-p
               `((display-sort-function
                  . ,(compose-flex-sort-fn (or existing-dsf #'identity)))))
        ,@(and flex-is-filtering-p
               `((cycle-sort-function
                  . ,(compose-flex-sort-fn (or existing-csf #'identity)))))
        ,@(cdr metadata)))))
```

>> It feels like it tries to match the longest string possible
> alphabetically? It's
>> just unintuitive because the json results doesn't match the output, and
>> debug stepping over was very unhelpful. We could maybe just add
>> some docs explaining that eglot default, which for many really is an
> eglot
>> hard-coding.
>
> I hope I explained why it's there.  This is what I recall of it, though
> I may be misremembering.  You could help improve the documentation
> by confirming my recollection hypothesis and adding comments to the
> code.
>
> Anyway, this boils down to a limitation of LSP, that it doesn't report on
> what
> kind of matching style it uses for textDocument/completion.  At least it
> used to be a limitation of LSP, maybe someone else has fixed it in the
> meantime, or added something that we can use.
>
> João

Yeah, that makes sense.

>
> PS: Added Stefan and Augusto to the discussion since I think they are
> already familiar with this LSP/Emacs discrepancy regarding completion
> systems and completion philosophy.

:thumbsup:

Theo