bug#62694: 30.0.50; eglot-tests fails with recent pylsp

["João Távora" <joaotavora@gmail.com>]

Michael Albinus <michael.albinus@gmx.de> writes:

>>> I've shown you the Eglot traces for one test case on both Debian pylsp
>>> (failed) and Fedora pylsp (succeeded). I still have no idea whether the
>>> Debian flavour is inside the LSP specs or not. But if it returns
>>> out-of-spec replies, I guess eglot should raise an Emacs error
>>> indicating this fact.
>>
>> It's _not_ an out-of-spec reply.  It's just a insuficient in-spec reply
>> from a poorly installed or configured server.
>
> If it is an on-spec reply, eglot shall handle this. If there isn't
> sufficient information in the reply, eglot shall err out with this
> information.

[I must admit I don't understand your use of "shall".  Is this the
German "soll", or is it really some kind of grave proclamation?]

Let's see if I can put this misunderstanding to rest.  There is a
misunderstanding here.  I hope I'm not going to be overly pedantic,
here: I do this in good faith.  First, Eglot is not the matter here,
we're talking about eglot-tests.el.  Now, for any software test to be
effective in predicting if the thing under test is functioning
correctly, there must be assumptions made.  Sometimes they are called
fixtures, the "fixed" thing you attach your system to.  Here, the server
is part of the fixture.  We have to assume that the server will react in
certain way when exposed to certain eglot-tests.el stimulae to be able
to say things about eglot.el's correctness.  If the pylsp server
responds with no completions when completions should have been available
-- as your log showed -- eglot-tests.el has to assume this is eglot.el's
fault, which it very well might be if Eglot failed to properly
communicate buffer changes or mangled some of its request or a million
other things.  This is exactly what these end-to-end tests are checking
against: that the multiple Eglot actions leading up to completions are
being taken correctly.

In tests, we have to doubt something: we cannot doubt the
fixture and the code all at once.  The fixture is, by definition, beyond
doubt.

It's like if you were desining a sorting algorithm.  You apply that
sorting algorithm to a known randomly sorted list, check the first
element to see if it's the smallest one indeed.  You check with the
function 'car'.  If it's not the smallest element, will you doubt or
algorithm or a misbehaving implementation of the function 'car'?  How
can you even know if you have a mischievous implementation of 'car'?

It's certainly true that external language servers, being "external" to
Emacs, are not as much under our control as the implementation of the
car primitive.  So they are not perfect fixtures.  Someone suggested
here that there should be an Emacs language server under our control,
just like the 'car' is under our control.  I agree.  But this is an
enormous task, so we have to surround ourselves with minimally stable
fixtures.  And pylsp is very reasonably stable when installed in the
recommended fashion.  Hopefully clangd is much better as a fixture (but
it's not perfect).

>> The point of these tests, as I've explained multiple times, is not to
>> test the servers, rather eglot.el's particularly its interactions with
>> other emacs facilities, such as xref, completion, flymake etc.  Any
>> server will do, as long as it is reasonably well-behaved and
>> predictable.  That's why I switched to clangd and all this discussion
>> is moot now.
>
> It isn't only the tests. You cannot prevent that a curious user, with
> the brand new Emacs 29 installed, reads the NEWS and knows there's
> eglot. She installs the Debian pylsp server (because that's what Debian
> offers), tries it, and fails. And you'll ghet a bug report.

First, you speak as if Eglot is a brand new thing in Emacs 29.  It's
not.  People have been using it since Emacs 26.3.

But sure, and I get a zillion million of those (not for pylsp by the
way).  This is only natural: servers are very frequently buggy and users
don't know about that unless they use a userr-facing client such as
Eglot.  So they, quite understandibly, report the problem to Eglot.  I
have to analyze them and refer the user to the server.  It's happened so
many times I've lost count.  The server maintainer discusses the issue
with me and we come to conclusion as to where the bug lies.  There's
nothing else I can do about that.  LSP server bugs are super-frequent.

But here, it's not even a pylsp bug.  It's a Debian packaging bug, by
all acounts.

But 0 users except for you have reported bugs in the pylsp server, in
the 5 years that Eglot has existed.  And if they had, I really doubt
they would have so obstinately resisted the tip to install pylsp in the
recommeded way or report this prolem to the Debian packagers, at least
to get their input on this matter.

>>> Based on this fact, you could always catch this specific error in the
>>> tests, and say that the server is not suited. Whether you shall skip
>>> or err out the test then is something else; until now it isn't obvious
>>> that a failed eglot test is due to the (possible) server misbehavior,
>>> or due to an eglot error. At least this information should be shown.
>>
>> It's impossible to know that.  You can design perfectly in-spec naughty
>> servers breaking all of eglot tests.
>
> Eglot shall fail the gracefully. The error messages I have seen so far
> don't tell me anything.

These are end-to-end tests.  What better messages would you like me to
put there?  Just submit a patch.

>> Or you can follow Eglot's maintainer advice to install versions of
>> servers known to be working correctly.  Heroically complexifying Eglot
>> to detect misbehaving servers is a completely futile exercise.
>
> That's an illusion. People don't follow advices, people don't read
> manuals. Believe me with 20+ years of Tramp experience, 40+ years
> experience in developing and maintaining large projects.

Well, you certainly don't, that's for sure.

João