bug#41531: 27.0.91; Better handle asynchronous eldoc backends

[Dmitry Gutov <dgutov@yandex.ru>]

On 07.07.2020 13:56, João Távora wrote:

> You talked and talked and presented your favourite async handling
> techniques, like having functions return functions that return
> functions.  You contented that I -- not you -- should work with it to
> solve the problem.

I showed a solution to the part of the problem that we managed to define 
at that time. You added some more features now.

If you like, I can rewrite your branch in terms of that proposal. Or in 
terms of eldoc-future (as suggested by Stefan). Your pick.

(But we still need to discuss the extra changes in this branch).

> For the quintillionth time: I AM NOT AGAINST FUTURES IN PRINCIPLE: I
> just don't use these Eldoc fixes to shoehorn something rushedly into
> Emacs.  Make a new thread, or join the existing one:
> 
>      https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00935.html

Thanks for the link. I seem to have missed it. Not exactly the 
discussion to join, but the projects mentioned there could be helpful.

> Afterwards, propose a change to the technique, not only in Eldoc but
> elsewhere, too.  This idea is so simple that it boggles the mind that
> you don't grasp it.

Sure. As long as this discussion waits.

>> urgent endeavor. E.g., Flymake is stable, and I don't have any
>> particular bugs in mind that need solving.
> 
> Great.  I'll just note here that it uses exactly the same technique I'm
> proposing in Eldoc to communicate with multiple backends: it's curious
> how that doesn't bother you.  I would reasonably expect "futures" or
> something else its implementation much simpler too.

It doesn't have existing clients that return results synchronously. It 
almost always works with external processes. It doesn't care if a 
checker never calls back. It doesn't compose results from multiple 
checkers. Basically: it's simple, and whatever legacy it did have, you 
cordoned that behind flymake-proc (which everybody promptly migrated off).

And as you can note, the interface you proposed here is not exactly the 
same: due to some of the requirements above, it's slightly different. A 
flymake backend is considered running whenever it didn't signal an 
error. The convention you proposed here involves returning a non-string, 
non-nil value.

I imagine having different interfaces between facilities is better than 
having similar-by-subtly-different ones.

>> Aside: given that this discussion has user interface in mind, it needs
>> more consideration of actual user experiences we'd want to allow. Ones
>> not provided by eldoc itself as well. For instance, I think we sooner
>> or later should have a callsig floating popup (in the vein of MS's
>> editors) as an alternative to showing the signature in the echo area
>> only.
> 
> That is addressed in a comment in eldoc-docs
> 
>        ;; Finally, output to the echo area.  We handle the
>        ;; `truncate-sym-name-if-fit' special case first, by selecting a
>        ;; top-section of the `*eldoc' buffer.  I'm pretty sure nicer
>        ;; strategies can be used here, probably by splitting this
>        ;; function into some `eldoc-display-functions' special hook.

I think the first question to ask here, is whether the eldoc clients are 
not ultimately the best place to decide how the doc should be formatted 
and displayed.

So the alternative to your current proposal (with composing strategies) 
would be for us to provide a set of utility functions to clients: ones 
to manipulate async computations, ones to truncate the doc to desired 
dimensions, ones to combine several docs in one, etc, while honoring the 
value of eldoc-echo-area-use-multiline-p.

The advantage to that approach is that we don't limit the display to 
particular format (e.g. the truncate-sym-name-if-fit requires the symbol 
to be at the beginning of the line, and for it to be specified as the 
:thing property as well). The downside: mainly being unable to compose 
eldoc results from multiple clients. But I wonder how well we can manage 
to handle that anyway.

> I agree there is ample space for improvement, including a
> "floating popup" (which I wouldn't limit to callsigs though).  Make
> another bug report to study this.

Callsigs are a whole other matter, actually. The best experience seems 
impossible to implement through eldoc. The LSP protocol shows an example 
of a good data structure required for it (a list of overloads, the 
number of the current guessed one, and a documentation string for the 
function). Eldoc can be used in the absence of other options, but a 
different hook which requires richer values seems more appropriate.

>> The new API is incompatible with the previous one, it requires
>> changing a lot of functions (though admittedly in a minor way).
> 
> This is demonstrably false. As I've explained to Eli there is no
> incompatibility in practice.  0 (zero) third-party extensions make use
> of the protocol being changed from Eldoc 1.0.0 to Eldoc 1.1.0, unless
> you're stalling here to secretly work on one.

So what happens if we merge this to master now? Either you expect no 
clients, and there is no need to hurry, or we expect some new eldoc 
clients to use this interface. But then the same clients will experience 
breakage as soon as we switch over to "proper" futures.

> But if we really, really wanted to, it's easy to get rid of the
> arguments, too, with a variation to the callback technique.  I just
> don't think it's worth it: a technique is a technique.

The variation that I showed in my little patch a month ago?

>> is easy to miss, as evidenced by describe-char-eldoc which still
>> doesn't accept any arguments.
> 
> Oh, an actual useful comment! Easily solved, thanks.  And it was only
> "missed" because it wasn't used anywhere.

Well, you corrected its docstring in this branch.

>> The choice to require a return of a non-nil value if the callback is
>> going to be used is also kinda odd (+1 control flow to test). What's
>> the problem with using the callback synchronously if the doc is
>> available right away?
> 
> Nothing, you can do it.  As long as you return non-nil, non-string.  But
> if you are are going to synchronously call the callback on a string, you
> might as well return that string, just seems simpler.

Seems like a missed opportunity to simplify (or not).

>> First of all, if I understand the main motivation behind it, it's to
>> delegate the implementation of asynchronous primitives to Eldoc.
> 
> It's rather to make clients don't worry about the synchronization.

Because it's fairly difficult, right? Especially in the absence of 
helpful standard libraries.

>> We don't want to bother with that in Eglot, etc. But you mentioned "a
>> type of docstring" to be returned in the discussion (and the callbacks
>> have the plist argument as a future proofing). If the type is a
>> buffer, its contents might as well be created from several async calls
> 
> If you want to support "buffers" as a "type of docstring", just do that:
> make buffers a type of docstring.  The obvious way is to have multiple
> sources produce multiple buffers.

That doesn't really jive with what I imagined. If I want to produce a 
"pretty" buffer in a client, I will print to it. Possibly making several 
HTTP calls in the process, if that's not hard to do.

I would also probably prefer not to worry about some other eldoc client 
leaving its documentation function in the list before mine. Or having 
them later in the list, leaving some unrelated stuff in the resulting 
buffer. As a client, I would probably know which calls are made, which 
data is returned, and how I want it to be rendered.

> Thing: why would you ever want to put buffer-joining complexity in the
> client?

Because it knows in which order to put them? Or how to render the seams 
best? Or it could mix the HTTP responses in other ways than simply 
joining them with "\n\n". Having generic code do that seems limiting.

Having individual clients do that should encourage more attention to detail.

>> The strategies themselves:
>>
>> - eldoc-documentation-enthusiast: What's the difference compared to
>> the 'default' one? Sacrificing CPU load for lower latency? It's an odd
>> choice to force the user to make. The only people who can make an
>> ideal decision regarding this are probably the authors of
>> eldoc-documentation-functions, but it wouldn't help if
>> eldoc-documentation-functions has functions coming from different
>> authors, facilities, etc.
> 
> Has nothing to do with CPU.  This is the way Eglot works now, or at
> least tries to: there are two delayed doc-producing backends, and
> neither is guaranteed to complete.

Why not? HTTP responses normally arrive reliably.

> One has priority over the other (and
> special hooks are a decent, standard Emacs way to manage priority)
> 
> Eglot shows the lower-priority one if it shows it can survive for more
> than x seconds (currently x = 0.3, uncustomizable).  No more doc
> blinking.

Why not just wait for the first one, if its documentation function 
returned non-nil?

I considered commenting on the 0.3 magic number, but dropped it in the 
first review.

>> - eldoc-documentation-compose: Okay, this is kinda interesting (though
>>    not essential),
> 
>> I think the only reasonably predictable behavior would be to truncate
>> each of them to one line, always.
> 
> That's a display problem, not a composition problem For now it works OK
> for one liners and also multiple multi-liners.  Displaying doc is not
> the main goal of these patches, there is certainly room for improvement,
> as I said above.

Whether we can reliably display these docs in a "composed" way, from any 
sources, or not, should probably factor into the design. Because if not, 
do we really need different strategies?

>> - eldoc-documentation-compose-eagerly: Ooh, now I think I see why
>> Returning futures instead (when async is needed) would provide this
>> kind of guarantee without the seeming duplication of signals.
> 
> Can't let go of that futures drum, can you?  It'd be a pleasure to see
> you walk the walk.

There's not much inherent difficulty in extracting the future-merge code 
from here or elsewhere. The actual problems I mentioned in the email 
with the (tiny) future.el come from elsewhere (e.g. from requirements 
for using them for completion), but they need to be decided on, in order 
to minimize breakage later.

>> On a related note, I believe some facilities might want to use only
>> certain "kinds" of documentation functions (as indicated by the plist
>> arguments). If the plist is only provided together with the response,
>> there is no way to avoid unnecessary computations (e.g. making HTTP
>> requests that return some other kinds of values). If the plist was
>> returned together with a future value, however, it would be easy to
>> do, as long as we document that the futures should start the
>> computation until a callback is provided (if possible, at least).
> 
> Save it for your future futures implementation.

My point was, again, adopting futures here would create a structural 
change. A more incompatible one than if we adopted a more compatible 
API. Or straight away used eldoc-local futures.

>> And in a different high-level argument: you stated that you intend to
>> have Eglot use a non-default value of eldoc-documentation-strategy.
> 
> OK, but this has nothing to do with Eldoc, does it?  Make a bug report
> for Eglot, I'll explain why it does this, and maybe I'll change it..

It does, if the actual requirements here mean that Eglot could just as 
fine perform combining its documentation results itself, in its 
documentation function. Then Eldoc could eventually do away with the 
eldoc-documentation-function/strategy variable altogether.

And the current change to the API would be minimal.

>> idea). This should very well be possible to untangle in the future,
>> but I'd rather not have code like this in Emacs if we can help it.
> 
> You're such an ace programmer that you code alternatives that are so
> brief that they occupy no code at all!

Nice punch.

I hope you haven't missed the implication that it's _hard_ for me to 
make heads or tails of your code there. But I could take a shot.

>> Further, having all strategies basically delegate to hardcoded options
>> inside eldoc-print-current-symbol-info seems to confirm that the set
>> of available strategies is a closed one. Which is not what I would
>> expect from a variable called eldoc-documentation-strategy.
> 
> There are four strategies to choose from but you can make more.  What
> did you have in mind?

Well... if I were thinking further in the direction of strategies, 
perhaps some would first request/wait the documentation from sources 
that return buffers, and then if none of those return any, then query 
the rest of functions. Or order the sources based on their kind before 
doing the calls, using user-supplied algo. Or perhaps skip buffers which 
are already displayed in some window.

My point here was, though, that a strategy sounds like something 
customizable and extensible. So their semantics, docstrings and 
implementations will need to be more accessible to an average Lisp 
developer.

>> These are my objections, for now. I'll have to study
>> eldoc--handle-docs a bit later, but any problems it has are probably
>> orthogonal to the rest of the list.
> 
> Thanks.

Having looked at it, the only problems there I can report on are 
practical, the same as I described when talking about 
eldoc-documentation-compose in the previous email: blinking after every 
user command, missing truncation when composing several docstrings, and 
undefined behavior with multiple multiline docstrings.

Is it at all possible to get rid of blinking? One-line eldoc doesn't blink.

Also, umm... it seems to truncate the contents of a long doc buffer to 
its bottom part. At least that's what I get when trying the related 
branch of Eglot.