bug#26710: Fwd: 25.2; project-find-regexp makes emacs use 100% cpu

[Dmitry Gutov <dgutov@yandex.ru>]

On 01.05.2017 10:20, Eli Zaretskii wrote:

> In my testing, find-grep finishes almost instantaneously.  The
> exception is when you have a cold cache, but even then it takes about
> 10% of the total run time, for the Emacs source tree (which yields
> about 100,000 hits in the test case).

This particular example, uses a very frequent term. I get 61000 hits or 
so, and it's still a lot, the search never finishes here (probably 
because I have more minor modes and customizations enabled).

I don't think this is the common case, but let's try to remove some 
unnecessary work in Elisp first.

See commit c99a3b9. Please take a look at 
xref--regexp-syntax-dependent-p specifically, and see if any significant 
false negatives come to mind.

With this, project-find-regexp for 'emacs' finally completes in ~10 
seconds on my machine. That's still more than 10 times longer than the 
external process takes, but I'm out of big optimization ideas at this point.

> I thought the request was to allow the user do something in the
> foreground, while this processing runs in the background.  If that's
> not what was requested, then I guess I no longer understand the
> request.

If the project is huge, and there are only a few hits, parallelizing the 
search and processing will allow the user to do whatever they want in 
the foreground. Because processing in Elisp, while slow, will still take 
a small fraction of the time.

If the search term returns a lot of hits (compared to the size of the 
project), processing might indeed take a lot of time, and the UI might 
appear sluggish (not sure how sluggish, though, that should depend on 
the scheduling of the main and background threads).

Even if it's sluggish, at least the user will see that the search has 
started, and there is some progress. We could even allow them to stop 
the search midway, and still do something with the first results.

These are some of the advantages 'M-x rgrep' has over project-find-regexp.

>> What we _can_ manage to run in parallel, in the find-grep process in the
>> background, and the post-processing of the results in Elisp.
> 
> Yes, you can -- if you invoke find-grep asynchronously and move the
> processing of the hits to the filter function.

Yes, these parts are necessary either way. What I was describing would 
go on top of them, as an abstraction.

> But that doesn't need
> to involve threads, and is being done in many packages/features out
> there, so I'm not sure what did you ask me to do with this.

I imagined that the xref API that allows this kind of asynchronous 
results might look better and more readable if it's implemented with 
threads underneath.

> IOW, it
> should be "trivial", at least in principle, to make this command work
> in the background, just like, say, "M-x grep".

In Compilation buffers (of which Grep is one example), the sentinel code 
has access to the buffer where the results are displayed. And the 
process outputs to that buffer as well. And 'M-x rgrep' doesn't have to 
abstract over possible way to obtain search results.

None of those are the case with the xref API, or the results rendering 
code, which has to work with the values returned by an arbitrary xref 
backend, as documented.

Right now, an xref backend implements several methods that are allowed 
to return the same type of value: "a list of xref items".

Our task, as I see it, is to generalize that return value type for 
asynchronous work, and to do that as sanely as possible.

Threads are not strictly necessary for this (see the last paragraph of 
my previous email), but this case seems like it could be a good, limited 
in scope, showcase for the threading functionality.

> I'm not sure I understand the need for this complexity, given that
> async subprocesses are available.  I'm probably missing something
> because I know too little about the internals of the involved code.

The main thing to understand is the xref API, not the internals of the 
package.