On 16.12.2020 22:32, Juri Linkov wrote:
>>> Another backup plan is to use ripgrep.  Its multiline handling with -U
>>> also allows to search words ignoring any whitespace, even newlines.
>>> This is like isearch-lax-whitespace using search-whitespace-regexp
>>> when it contains a newline, e.g. "[ \t\r\n]+".
>>
>> Right. It has a problem of its own, though: it still outputs a file name
>> per line, even when a match is spread across several lines (unlike
>> pcregrep). So we're left guessing where a given multiline match ends.
>>
>> Also, 'sort' doesn't seem to be able to treat both : and \0 as separators
>> at the same time.
>>
>> Here's a rough patch, for illustration.
> 
> Thanks, now finally it's possible to search text ignoring whitespace
> between words, for example:
> 
>    Find regexp: file[ 	
> ]+names
> 
> finds everything correctly, even though current implementation maybe
> not the most elegant.
> 
>> It's kind of working, but I'm not loving it.
> 
> What do you think about using the option `rg --json`?
> Emacs has the fast JSON parsing library now, so using
> JSON output would be more reliable.

Very interesting. It returns better data, each multiline match is wholly 
in one entry instead of being spread across lines. Even the matches are 
annotated with match string/length/absolute position.

We should really investigate it, but perhaps a bit later, including our 
capability to parse it quickly when there are a lot of matches (>1000), 
how said byte offsets interact with different file encodings.

Also, its output is not one JSON document but a series of them 
(including ones with just search statistics which we'll want to skip), 
but some re-search-forward followed by (json-parse-buffer) should do the 
trick.

In the meantime, here's a smaller patch using the traditional output 
format. I figure since there is a file name on each line anyway, --null 
doesn't help much. So it can be simplified a little (see attached).

Unfortunately, xref-replace-in-matches is broken for such multiline 
matches. And, of course, it merges together matches on adjacent lines, 
whether they are one match or several (that hasn't changed from the 
previous match). So more investigation is needed.