bug#49836: Support ripgrep in semantic-symref-tool-grep

[Dmitry Gutov <dgutov@yandex.ru>]

On 06.08.2021 03:35, Juri Linkov wrote:

> semantic-symref-grep-use-template constructs such command line:
> 
>    "rg ... -e \\\\bsoap-type-is-array\\?\\\\b"
> 
> that finds matches.

The correct one will probably look like

   "rg ... -e \\\\bsoap-type-is-array\\\\?\\\\b"

(same number of backslashes before '?' as before 'b'), and it won't find 
any. The one you mentioned will find false positives.

E.g., try searching for 'file-name-as-directory?'. Or 'carr?'.

>> See above. But also consider what happens if a user sees that grep-program
>> is now customizable and ripgrep is an officially supported value. They
>> change it to "rg", and then suddenly their 'M-x rgrep' input has to use the
>> extended regexp format?
> 
> This difference could be explained in the documentation.

If it comes to that, yes, but it's usually better to fix usability 
problems that just document them.

>> Worse than that, any third-party package that uses grep-find-template will
>> suddenly have a high chance of failing if they pass any nontrivial regexps
>> to it, especially if those have groupings or alternations.
> 
> This already happened after trying to customize grep-find-template
> to use rg broke xref-find-references, so the problem already exists
> and needs to be solved.

The problem exists, and has been for a long time: grep.el doesn't 
properly support the "alternative" search programs, which are very 
popular now. Its abstraction is leaky and doesn't work with anything but 
grep. But I think that means we need a better abstraction.

Let's try to make sure we don't create bigger problems when fixing it. 
And "packages stop working when I customize grep-program" sounds worse 
than "I can't customize grep-program to 'rg', so my searches are a bit 
slower than they could have been".

>> It's a hard problem: grep.el is not prepared for abstracting like that. If
>> we at least standardized it internally on Extended format, that would at
>> least remove one source of uncertainty and bugs. The user still can input
>> basic regexps interactively, we can translate them easily.
> 
> Is there a package that can translate between them reliably?

For the limited purpose of symref/grep, we could use 
xref--regexp-to-extended. It's already used in xref-matches-in-directory 
and xref-matches-in-files. Better name/documentation and tests are pending.

Note that it actually translates from a (subset of) Emacs regexp to 
Extended and back (it's reversible). The proper basic regexp syntax 
treats '+' and '?' as normal characters unless escaped, but they're 
special in Emacs regexps.

The above function is how one can use Emacs syntax (though only limited 
a subset, for now) in project-find-regexp.

I also saw some commits to ELPA yesterday, that show that Consult 
includes a more advanced version of this feature:

https://git.savannah.gnu.org/cgit/emacs/elpa.git/commit/?h=externals/consult&id=7bd3e44929d44cf0e17f38e943e9be2bd6014237
https://git.savannah.gnu.org/cgit/emacs/elpa.git/commit/?h=externals/consult&id=95dadd98a6a0f08955f67f1e9a7cc312435a86b8

Not sure how mature it is (seems still in development), but perhaps we 
could move it to the core sooner or later. And use it instead, if it 
does provide any improvement for our use case here.

>> Further: seeing xref-search-program-alist, people asked for support for
>> other similar programs, such as ag and pt. Any solution we end up with, we
>> should try to ensure they are valid values of grep-program as well.
> 
> Why not, semantic-symref already supports alternative tools
> such as cscope, global, idutils.  So xref could support more too.

It's easy enough for Xref, yes. It only has to support one single, 
well-defined scenario.

>>> +                         (if (equal grep-program "rg")
>>> +                             (format "(^|\\W)%s(\\W|$)"
>>> +                                     (oref tool searchfor))
>>> +                           (format "\\(^\\|\\W\\)%s\\(\\W\\|$\\)"
>>> +                                   (oref tool searchfor))))))
>>
>> This can work. Except the comparison should be with "grep", I think: all
>> other alternatives only work with the Extended format.
> 
> I'm worried about the case when the user customizes
> 'grep-program' to e.g. an absolute path "/bin/grep"
> or "/usr/local/bin/grep", etc.

(string-match "\\bgrep\\b" grep-program) could take care of this.

To sum up, I'm all for adding some clutches to symref/grep.el, to 
support your advanced scenario, right now.

As for having grep-program customizable, perhaps we should add some new 
feature/abstraction/package? To avoid breakage, and for it to be opt-in 
for any new callers from Lisp.

Or indeed have templates use Extended syntax, and grep-expand-template 
translate REGEXP to it. That can cause breakage for existing users, 
though, those who already customize grep-find-template, etc, to their 
particular programs.