[RFC] Urgrep: New ELPA submission (eventually)

[RFC] Urgrep: New ELPA submission (eventually)

[Jim Porter]

A while back, I made a package called Urgrep ("Universal Recursive Grep" 
- <https://github.com/jimporter/urgrep>), and now that it's pretty 
stable, I'm starting preparations to submit it to GNU ELPA. I'm not 
*quite* ready to submit it for real, but I'm reasonably close and wanted 
to get some wider feedback before I actually do submit it. If you have 
any suggestions, questions, etc, I'm happy to hear them.

A brief summary: Urgrep is designed to provide a universal interface on 
top of all "recursive grep"-like commands, from ripgrep to git grep to 
the classic find+grep. I made it to solve a frustration I had with other 
similar packages: they're primarily built to support a *particular* 
searching tool, and while some can be coaxed into running with other 
tools, it doesn't always work. Even when it does, it's usually not 
seamless, especially when you add Tramp to the equation.

With Urgrep, you can use *any* rgrep-like command[1]. It will 
automatically use the best tool on the system in question, so if some 
remote host doesn't have your favorite tool, it'll still work just fine.

I also added some Isearch-like bindings in the main function's prompt 
(creatively named 'urgrep'). These all start with M-s and let you set 
common search flags, like number of context lines. Maybe there's a 
better way to do this; the current method seemed reasonable to me, but 
I'm certainly open to suggestions. (I looked at transient.el, but it 
didn't seem like the best fit in this case; maybe I just don't know 
enough about transient.el though.)

Again, I'm not quite ready to submit this to GNU ELPA, but I think it's 
stabilized enough that it's worth getting some more eyes on it in case I 
missed something.

- Jim

[1] If it doesn't support your favorite one out of the box, you can add 
your own support: you just need to add to 'urgrep-tools' with the right 
mappings for the various flags.

[RFC] Urgrep: New ELPA submission (eventually)

[Jim Porter]

(Resending; sorry if you get duplicates.)

A while back, I made a package called Urgrep ("Universal Recursive Grep" 
- <https://github.com/jimporter/urgrep>), and now that it's pretty 
stable, I'm starting preparations to submit it to GNU ELPA. I'm not 
*quite* ready to submit it for real, but I'm reasonably close and wanted 
to get some wider feedback before I actually do submit it. If you have 
any suggestions, questions, etc, I'm happy to hear them.

A brief summary: Urgrep is designed to provide a universal interface on 
top of all "recursive grep"-like commands, from ripgrep to git grep to 
the classic find+grep. I made it to solve a frustration I had with other 
similar packages: they're primarily built to support a *particular* 
searching tool, and while some can be coaxed into running with other 
tools, it doesn't always work. Even when it does, it's usually not 
seamless, especially when you add Tramp to the equation.

With Urgrep, you can use *any* rgrep-like command[1]. It will 
automatically use the best tool on the system in question, so if some 
remote host doesn't have your favorite tool, it'll still work just fine.

I also added some Isearch-like bindings in the main function's prompt 
(creatively named 'urgrep'). These all start with M-s and let you set 
common search flags, like number of context lines. Maybe there's a 
better way to do this; the current method seemed reasonable to me, but 
I'm certainly open to suggestions. (I looked at transient.el, but it 
didn't seem like the best fit in this case; maybe I just don't know 
enough about transient.el though.)

Again, I'm not quite ready to submit this to GNU ELPA, but I think it's 
stabilized enough that it's worth getting some more eyes on it in case I 
missed something.

- Jim

[1] If it doesn't support your favorite one out of the box, you can add 
your own support: you just need to add to 'urgrep-tools' with the right 
mappings for the various flags.

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Stefan Kangas]

Jim Porter <jporterbugs@gmail.com> writes:

> With Urgrep, you can use *any* rgrep-like command[1]. It will
> automatically use the best tool on the system in question, so if some
> remote host doesn't have your favorite tool, it'll still work just fine.

Thanks, this looks very useful.  I will try it when I get the chance.

Is `urgrep' a full featured replacement for `rgrep'?  Is there anything
missing, and/or something that will be hard to add?

I note that the repository includes an implementation of wgrep.
It would be great to include such functionality in Emacs by default at
some point.

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Eli Zaretskii]

> From: Jim Porter <jporterbugs@gmail.com>
> Date: Sun, 11 Sep 2022 21:56:04 -0700
> 
> A while back, I made a package called Urgrep ("Universal Recursive Grep" 
> - <https://github.com/jimporter/urgrep>), and now that it's pretty 
> stable, I'm starting preparations to submit it to GNU ELPA. I'm not 
> *quite* ready to submit it for real, but I'm reasonably close and wanted 
> to get some wider feedback before I actually do submit it. If you have 
> any suggestions, questions, etc, I'm happy to hear them.
> 
> A brief summary: Urgrep is designed to provide a universal interface on 
> top of all "recursive grep"-like commands, from ripgrep to git grep to 
> the classic find+grep. I made it to solve a frustration I had with other 
> similar packages: they're primarily built to support a *particular* 
> searching tool, and while some can be coaxed into running with other 
> tools, it doesn't always work. Even when it does, it's usually not 
> seamless, especially when you add Tramp to the equation.
> 
> With Urgrep, you can use *any* rgrep-like command[1]. It will 
> automatically use the best tool on the system in question, so if some 
> remote host doesn't have your favorite tool, it'll still work just fine.

Would it make sense to add this to grep.el, instead of having an
add-on package?

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Stefan Monnier]

> Again, I'm not quite ready to submit this to GNU ELPA, but I think it's
> stabilized enough that it's worth getting some more eyes on it in case
> I missed something.

I think it's worth repeating at this point that GNU ELPA is not
restricted to established/stable/aging/dead packages.

We welcome packages in-development as well (and you can easily make
sure it stays in GNU-devel ELPA until you're ready for a real release
if you don't feel like exposing your package to unsuspecting users).


        Stefan

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Jim Porter]

On 9/12/2022 2:19 AM, Stefan Kangas wrote:
> Is `urgrep' a full featured replacement for `rgrep'?  Is there anything
> missing, and/or something that will be hard to add?

It does (or should do) everything `rgrep' does. However, I wouldn't 
really call it a replacement, since it just uses `rgrep' under the 
hood[1]. It's also only designed to work like `rgrep', so it doesn't 
handle similar commands like `lgrep' (though maybe it could).

> I note that the repository includes an implementation of wgrep.
> It would be great to include such functionality in Emacs by default at
> some point.

I agree, that would be great. However, note that urgrep-wgrep.el just 
provides the necessary glue for wgrep to understand Urgrep buffers. You 
still need to install wgrep for it to do anything.

[1] Technically, it uses `rgrep-default-command'.

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Jim Porter]

On 9/12/2022 4:11 AM, Eli Zaretskii wrote:
> Would it make sense to add this to grep.el, instead of having an
> add-on package?

That could work too. I think there's some use in having this available 
on GNU ELPA though, even if it ships with base Emacs. Since it's new and 
I expect it'll gain new features, it's probably worth letting people get 
updates sooner than the next Emacs release.

That said, having this in base Emacs would probably have some really 
nice benefits too. I haven't followed it super-closely, but I seem to 
recall some fairly recent changes to xref.el and project.el (maybe 
others too) to support a wider variety of grep-like tools. Urgrep would 
(hopefully) make this simpler for any package that wants to do this: the 
'urgrep-command' function can convert an abstract search query to the 
concrete command, e.g.

   (urgrep-command "my query" :case-fold t :color nil)
     => "ag --nocolor --group -i -Q -- my\\ query"

Then any time some Emacs code wants to do something rgreppy, it could 
call 'urgrep-command' and get the "best" command to use for the system 
in question.

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Jim Porter]

On 9/12/2022 5:58 AM, Stefan Monnier wrote:
> I think it's worth repeating at this point that GNU ELPA is not
> restricted to established/stable/aging/dead packages.
> 
> We welcome packages in-development as well (and you can easily make
> sure it stays in GNU-devel ELPA until you're ready for a real release
> if you don't feel like exposing your package to unsuspecting users).

Thanks for the explanation; I didn't realize that was possible. Then 
there's probably no major issue with putting it on GNU-devel ELPA 
sometime soon. I'll have to read up on how to get all that configured 
properly.

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Ergus]

Hi Jim:

Any progress on this? I have seen the package on github and it looks
very nice... but sadly not in elpa yet, so it won't be in my system
since then.

Is there something missing? I would like even to see this package or
similar approach in vanilla (mainly to speedup the venerable rgrep which
sadly gets too slow on big projects due to the serialization that find
adds to grep.)

Best,
Ergus


On Tue, Sep 13, 2022 at 09:52:11PM -0700, Jim Porter wrote:
>On 9/12/2022 5:58 AM, Stefan Monnier wrote:
>>I think it's worth repeating at this point that GNU ELPA is not
>>restricted to established/stable/aging/dead packages.
>>
>>We welcome packages in-development as well (and you can easily make
>>sure it stays in GNU-devel ELPA until you're ready for a real release
>>if you don't feel like exposing your package to unsuspecting users).
>
>Thanks for the explanation; I didn't realize that was possible. Then 
>there's probably no major issue with putting it on GNU-devel ELPA 
>sometime soon. I'll have to read up on how to get all that configured 
>properly.
>

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Jim Porter]

(Coincidentally, I was actually thinking about posting a followup to 
this thread this weekend myself!)

On 2/4/2023 9:12 AM, Ergus wrote:
> Any progress on this? I have seen the package on github and it looks
> very nice... but sadly not in elpa yet, so it won't be in my system
> since then.
> 
> Is there something missing?

Mostly a lack of time, and trying to spend the Emacs time I do have on 
fixing some Eshell stuff. But since any future Eshell changes will be in 
Emacs 30, the time pressure is off there.

I do have a couple of open questions that I couldn't find answers for 
though:

First, assigning copyright of Urgrep to the FSF (so that it can go in 
GNU ELPA) is just a matter of updating the copyright field in the code, 
right? Do I need to fill out any paperwork or anything too?

Second, my usual process for versioning is that I'll release a version 
like "1.0" by updating the "Version:" field and tagging it. Then, in the 
next commit, I update the "Version:" field to something like 
"2.0-snapshot". So in this case, I'd expect GNU ELPA to give users 
version 1.0, and GNU ELPA-devel to give users the latest 2.0-snapshot 
version. Does ELPA understand that? I see that after cutting a release, 
lots of Emacs packages just leave the version identifier alone until the 
next release. I'd prefer not to do that, just in case someone reports a 
bug on the snapshot version: having a different version string for the 
snapshot helps to prevent confusion about which version they're using.

I think GNU ELPA-devel solves this confusion by appending the date to 
the version identifier (so you get something like 1.0.20230204...), but 
that wouldn't help users pulling Urgrep directly from Git.

> I would like even to see this package or similar approach in vanilla
> (mainly to speedup the venerable rgrep which sadly gets too slow on
> big projects due to the serialization that find adds to grep.)
There are definitely a number of places that core Emacs would benefit 
from something like this. For example, Xref does some of this on its 
own, but has fewer options for search programs, and I'm not sure if 
Xref's search program is remote-connection aware (one of my main reasons 
for writing Urgrep is that I use Tramp extensively, and not every system 
I connect to has my favorite search program).

I could see Urgrep becoming a core Emacs package once it's had wide 
enough use that all the major bugs/design flaws have been worked out. 
That might require shuffling some things around though, since Urgrep 
comes with hooks for wgrep, but wgrep is currently in Non-GNU ELPA. I'm 
not sure it's ok for something in Emacs proper to have an optional 
dependency on a Non-GNU package.

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Ergus]

Hi Jim:

On February 4, 2023 8:44:12 PM GMT+01:00, Jim Porter <jporterbugs@gmail.com> wrote:
>(Coincidentally, I was actually thinking about posting a followup to this thread this weekend myself!)
>
>On 2/4/2023 9:12 AM, Ergus wrote:
>> Any progress on this? I have seen the package on github and it looks
>> very nice... but sadly not in elpa yet, so it won't be in my system
>> since then.
>> 
>> Is there something missing?
>
>Mostly a lack of time, and trying to spend the Emacs time I do have on fixing some Eshell stuff. But since any future Eshell changes will be in Emacs 30, the time pressure is off there.
>
>I do have a couple of open questions that I couldn't find answers for though:
>
>First, assigning copyright of Urgrep to the FSF (so that it can go in GNU ELPA) is just a matter of updating the copyright field in the code, right? Do I need to fill out any paperwork or anything too?
>

Eli?

>Second, my usual process for versioning is that I'll release a version like "1.0" by updating the "Version:" field and tagging it. Then, in the next commit, I update the "Version:" field to something like "2.0-snapshot". So in this case, I'd expect GNU ELPA to give users version 1.0, and GNU ELPA-devel to give users the latest 2.0-snapshot version. Does ELPA understand that? I see that after cutting a release, lots of Emacs packages just leave the version identifier alone until the next release. I'd prefer not to do that, just in case someone reports a bug on the snapshot version: having a different version string for the snapshot helps to prevent confusion about which version they're using.
>
>I think GNU ELPA-devel solves this confusion by appending the date to the version identifier (so you get something like 1.0.20230204...), but that wouldn't help users pulling Urgrep directly from Git.
>
>> I would like even to see this package or similar approach in vanilla
>> (mainly to speedup the venerable rgrep which sadly gets too slow on
>> big projects due to the serialization that find adds to grep.)
>There are definitely a number of places that core Emacs would benefit from something like this. For example, Xref does some of this on its own, but has fewer options for search programs, and I'm not sure if Xref's search program is remote-connection aware (one of my main reasons for writing Urgrep is that I use Tramp extensively, and not every system I connect to has my favorite search program).
>

I have exactly the same issue... So I am happy to know that the tramp compatibility is tested :). BTW, in the gtags-mode package I added a search strategy with connection local variables and some caching that is reasonable to avoid using executable-find excessively with tramp, but also minimizing the manual user customization. 

>I could see Urgrep becoming a core Emacs package once it's had wide enough use that all the major bugs/design flaws have been worked out. That might require shuffling some things around though, since Urgrep comes with hooks for wgrep, but wgrep is currently in Non-GNU ELPA. I'm not sure it's ok for something in Emacs proper to have an optional dependency on a Non-GNU package.

IMHO I would prefer a simpler version which only implements the new basic features (functions) or improves the ones in the emacs core (do one thing and do it right). Mainly to minimize external dependencies, potential conflicts and the danger of depending on something that may get unmaintained/broken in a while (simple is better than complex). All improves for other packages can be added as optional (different packages, or advises/hooks that the user can add easily to its config, so they only need a couple of lines in the readme... Like vertico does...

WDYT?

Best, Ergus.


-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] Urgrep: New ELPA submission (eventually)

[Stefan Monnier]

> First, assigning copyright of Urgrep to the FSF (so that it can go in GNU
> ELPA) is just a matter of updating the copyright field in the code, right?
> Do I need to fill out any paperwork or anything too?

You basically need to "state" that you consider that code to be covered
by your existing Emacs assignment.  We usually consider that changing
the copyright line in the files is a valid way to make this statement.

> Second, my usual process for versioning is that I'll release a version like
> "1.0" by updating the "Version:" field and tagging it. Then, in the next
> commit, I update the "Version:" field to something like "2.0-snapshot". So
> in this case, I'd expect GNU ELPA to give users version 1.0, and GNU
> ELPA-devel to give users the latest 2.0-snapshot version. Does ELPA
> understand that?

Yes.


        Stefan