grep command doc - point out that you can chain now

grep command doc - point out that you can chain now

[Drew Adams]

I just found out from Le Wang that you can now chain grep commands within
the Emacs `grep' command. I mean that you can now do `M-x grep foo *.el |
grep bar | grep toto'.

This is absolutely wonderful. I don't know when it was added (I suspect
Emacs 22) - it wasn't available in Emacs 20, in any case. (In Emacs 20, I've
been using `M-!' to chain greps, or doing it outside Emacs. BTW,
`flush-lines' doesn't work in *grep*, because it throws off the `next-error'
index.)

Chaining this way, in my experience, represents how most programmers use
grep. Instead of trying to come up with a complex regexp, they just use two
or three chained greps. The technique is so useful that I've applied it to
the use of regexps generally (e.g. regexp searching) in my own libraries.

So, I'm disappointed to see that this is not pointed out in the Emacs manual
presentation of `grep'. I really think this is worth pointing out
explicitly, with a simple example, especially since it was not possible
before.

Re: grep command doc - point out that you can chain now

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

> I just found out from Le Wang that you can now chain grep commands within
> the Emacs `grep' command. I mean that you can now do `M-x grep foo *.el |
> grep bar | grep toto'.
>
> This is absolutely wonderful. I don't know when it was added (I suspect
> Emacs 22) - it wasn't available in Emacs 20, in any case. (In Emacs 20, I've
> been using `M-!' to chain greps, or doing it outside Emacs. BTW,
> `flush-lines' doesn't work in *grep*, because it throws off the `next-error'
> index.)
>
> Chaining this way, in my experience, represents how most programmers use
> grep. Instead of trying to come up with a complex regexp, they just use two
> or three chained greps. The technique is so useful that I've applied it to
> the use of regexps generally (e.g. regexp searching) in my own libraries.
>
> So, I'm disappointed to see that this is not pointed out in the Emacs manual
> presentation of `grep'. I really think this is worth pointing out
> explicitly, with a simple example, especially since it was not possible
> before.

Why wouldn't it have been possible before?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: grep command doc - point out that you can chain now

[Nick Roberts]

 > > So, I'm disappointed to see that this is not pointed out in the Emacs
 > > manual presentation of `grep'. I really think this is worth pointing out
 > > explicitly, with a simple example, especially since it was not possible
 > > before.
 > 
 > Why wouldn't it have been possible before?

I think it relates to this change:

2001-10-05  Miles Bader  <miles@gnu.org>

	* progmodes/compile.el (grep-use-null-device): New variable.
	(grep-command): Mention `grep-use-null-device'.
	(grep-compute-defaults): Compute `grep-use-null-device' if necessary.
	Make computation of `grep-command' respect `grep-use-null-device'.

which was made just after the branch for Emacs 21.1 and so is only in CVS.

I don't think it should be pointed out in the manual because you would expect
it to work.  However it might be worth mentioning it in NEWS, so that users
who have tried it in the past know it will now work.

-- 
Nick                                           http://www.inet.net.nz/~nickrob

RE: grep command doc - point out that you can chain now

[Drew Adams]

    I don't think it should be pointed out in the manual because
    you would expect it to work.  However it might be worth mentioning
    it in NEWS, so that users who have tried it in the past know it
    will now work.

I don't think it's obvious ("you would expect it to work"), since Emacs
`grep' is asking for input to the Unix/GNU-Linux `grep' command. Do a `man'
on grep, and you won't see its arguments mentioning this, because piping
output to other commands is not part of the arguments - it is part of the
shell interpretation of Unix/GNU-Linux.

There is nothing (in the prompt, doc string etc.) that would suggest that
you can instead pass it both the args for grep and a chain of piped
commands. I imagine you could pipe anything, not just grep (haven't tried),
and that's certainly not what "you would expect" from a `grep' command
asking for input.

Aside from this consideration, I think this is important enough that it
should be in the doc, not just the NEWS, to bring it to the attention of
users of previous Emacs releases. What's wrong with adding such a "Usage
Note" to the doc?

Re: grep command doc - point out that you can chain now

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Mon, 17 Jul 2006 08:28:36 -0700
> 
>     I don't think it should be pointed out in the manual because
>     you would expect it to work.  However it might be worth mentioning
>     it in NEWS, so that users who have tried it in the past know it
>     will now work.
> 
> I don't think it's obvious ("you would expect it to work"), since Emacs
> `grep' is asking for input to the Unix/GNU-Linux `grep' command.

IMHO, it should suffice to say that "M-x grep" expects _any_ shell
command whose output resembles that of Grep.  That includes, for
example, "find SOMETHING | xargs fgrep" etc.

RE: grep command doc - point out that you can chain now

[Drew Adams]

    >     I don't think it should be pointed out in the manual because
    >     you would expect it to work.  However it might be worth mentioning
    >     it in NEWS, so that users who have tried it in the past know it
    >     will now work.
    >
    > I don't think it's obvious ("you would expect it to work"),
    > since Emacs `grep' is asking for input to the Unix/GNU-Linux `grep'
command.

    IMHO, it should suffice to say that "M-x grep" expects _any_ shell
    command whose output resembles that of Grep.  That includes, for
    example, "find SOMETHING | xargs fgrep" etc.

Yes indeed. That's even more general, and I wasn't aware of that. It's
important to include the example, to illustrate what is meant by "shell
command whose output resembles...".

Re: grep command doc - point out that you can chain now

[Richard Stallman]

    I don't think it should be pointed out in the manual because you would expect
    it to work.

It never occurred to me, so I think it is worth mentioning in the
Emacs Manual.  I will add it to the manual.

Re: grep command doc - point out that you can chain now

[Nick Roberts]

  >     I don't think it should be pointed out in the manual because you would
 >     expect it to work.
 > 
 > It never occurred to me, so I think it is worth mentioning in the
 > Emacs Manual.  I will add it to the manual.

Probably because you knew it didn't work in the past.  It still doesn't work
for users with versions of grep which don't have the -H option, so they will
be confused by your description.


-- 
Nick                                           http://www.inet.net.nz/~nickrob

RE: grep command doc - point out that you can chain now

[Drew Adams]

     >     I don't think it should be pointed out in the manual
     >     because you would expect it to work.
     >
     > It never occurred to me, so I think it is worth mentioning in the
     > Emacs Manual.  I will add it to the manual.

    Probably because you knew it didn't work in the past.

Yes, I said that myself.

And, I added, also because what you type is apparently not the arguments to
the Unix/GNU-Linux `grep' command, but something possibly much richer. Even
if someone were not used to Emacs 20 `grep' behavior, if s?he were used to
the Unix/GNU-Linux `grep' command, then s?he would expect to be inputting
just the `grep' args, no?

    It still doesn't work for users with versions of grep which don't
    have the -H option, so they will be confused by your description.

Do you mean Eli's description? I didn't describe it, beyond mentioning
piping. Eli gave a real description of what's going on, presumably.

I didn't know that either (-H), BTW. That too is worth pointing out. IOW,
what you type is different for different `grep's - in one case it is the
`grep' args; in another case it is "_any_ shell command whose output
resembles that of Grep."

Those behavior differences are certainly worth pointing out, unless they are
part of the `grep -H' doc. IOW, is what you type the expected arguments for
`grep -H', or is it something special that Emacs (only) allows? Does
`grep -H'  _itself_ allow "_any_ shell command whose output resembles that
of Grep."?

If this is part of the behavior of `grep -H' itself, then I guess it
wouldn't need to be documented in Emacs. It's not clear to me what the story
is. When I do `M-x man RET grep', and look for -H, it says "Print the
filename for each match." That can't be what this is about, can it? Yet,
with that same `grep' (it says "GNU Project 2002/01/22" in the footer), I
get the magical piping. Isn't some explanation necessary? Is this an Emacs
`grep' feature or a Unix/GNU-Linux `grep' feature that Emacs manifests?

RE: grep command doc - point out that you can chain now

[Nick Roberts]

Drew Adams writes:
 >      >     I don't think it should be pointed out in the manual
 >      >     because you would expect it to work.
 >      >
 >      > It never occurred to me, so I think it is worth mentioning in the
 >      > Emacs Manual.  I will add it to the manual.
 > 
 >     Probably because you knew it didn't work in the past.
 > 
 > Yes, I said that myself.

You predicted Richard's reponse?

 > And, I added, also because what you type is apparently not the arguments to
 > the Unix/GNU-Linux `grep' command, but something possibly much richer. Even
 > if someone were not used to Emacs 20 `grep' behavior, if s?he were used to
 > the Unix/GNU-Linux `grep' command, then s?he would expect to be inputting
 > just the `grep' args, no?

Grep uses start-process just like shell-command.  You saw it as natural to
use the latter with a pipe so I'm surprised that you feel it needs
explaining for grep.

Grep failed in the past because grep-use-null-device was automatically t (or
more accurately "/dev/null" was always appended to the end of the command) By
saying that I would expect it to work, I mean I can't see why it wouldn't.
Could you see a reason?  M-x grep says:

Run grep (like this): grep -nH -e 

not

Run grep (with the following arguments): -nH -e

 >     It still doesn't work for users with versions of grep which don't
 >     have the -H option, so they will be confused by your description.
 > 
 > Do you mean Eli's description? I didn't describe it, beyond mentioning
 > piping. Eli gave a real description of what's going on, presumably.

I mean the person to whom the e-mail was addressed (Richard in this case).
Since you only post to the list, I'm often not sure who you're talking to.

 > I didn't know that either (-H), BTW. That too is worth pointing out. IOW,
 > what you type is different for different `grep's - in one case it is the
 > `grep' args; in another case it is "_any_ shell command whose output
 > resembles that of Grep."
 > 
 > Those behavior differences are certainly worth pointing out, unless they are
 > part of the `grep -H' doc. IOW, is what you type the expected arguments for
 > `grep -H', or is it something special that Emacs (only) allows? Does
 > `grep -H'  _itself_ allow "_any_ shell command whose output resembles that
 > of Grep."?
 >
 > If this is part of the behavior of `grep -H' itself, then I guess it
 > wouldn't need to be documented in Emacs. It's not clear to me what the story
 > is. When I do `M-x man RET grep', and look for -H, it says "Print the
 > filename for each match." That can't be what this is about, can it? Yet,
 > with that same `grep' (it says "GNU Project 2002/01/22" in the footer), I
 > get the magical piping. Isn't some explanation necessary? Is this an Emacs
 > `grep' feature or a Unix/GNU-Linux `grep' feature that Emacs manifests?

Behaviour depends on the version of grep and the value of grep-command but
ultimately the value of grep-use-null-device.  It has to do with the fact
that without "-H" grep doesn't output the filename if only one is given
in the input (hence the dummy extra: /dev/null).  I don't which versions
of grep have "-H" or when it was added to GNU/Linux.

-- 
Nick                                           http://www.inet.net.nz/~nickrob

RE: grep command doc - point out that you can chain now

[Drew Adams]

     >     Probably because you knew it didn't work in the past.
     >
     > Yes, I said that myself.

    You predicted Richard's reponse?

Oops - I thought you were quoting me; my bad. I thought I said that one
reason I didn't expect it was that I knew it didn't work in the past.

     > And, I added, also because what you type is apparently not
     > the arguments to the Unix/GNU-Linux `grep' command, but something
     > possibly much richer. Even if someone were not used to Emacs 20
     > `grep' behavior, if s?he were used to the Unix/GNU-Linux `grep'
     > command, then s?he would expect to be inputting
     > just the `grep' args, no?

    Grep uses start-process just like shell-command.  You saw it as
    natural to use the latter with a pipe so I'm surprised that you
    feel it needs explaining for grep.

You know something? I'm learning more and more here. I was about to write
that I wouldn't naturally expect just any old Unix/GNU-Linux command used in
Emacs to accept more than the args defined for that command. Looking for
"proof", I tried `M-x man RET grep | grep grep'. My intention was to show
that `man', another Unix command, would barf and not accept the piping.

Well... I was wrong. That works too! And that one works in Emacs 20 too.

I don't know; is this obvious to others? It really wasn't to me.

Fun. And I wonder now if this shouldn't be documented generally, because
it's apparently not an Emacs `grep' thing, but it is, as you say, because
`start-process' is used.

I'd say that this should perhaps be made more clear in the Emacs-Lisp doc
for `start-process'. That doc does point out that `start-process' starts an
asynchronous process, and it says that whatever you give it as &rest args
are treated like command-line args, but, though I've read that before and
used `start-process' several times, I never interpreted "command-line
arguments for the program" (e.g. grep) to mean whatever might be on the
command line, including pipes (and redirection etc., presumably).

How about rewording this, to emphasize the generality (a shell command line
passed to the process) and de-emphasize "arguments" to the program? I guess
I wasn't thinking well enough about processes interpreting the command line;
I don't know if others might think similarly.

And, because Emacs users will not know about it, and because `grep', in
particular, is so useful when chained, I still think the grep-piping use
pattern is worth pointing out in the Emacs `grep' doc.

    Grep failed in the past because grep-use-null-device was
    automatically t (or more accurately "/dev/null" was always appended
    to the end of the command)

I see.

    By saying that I would expect it to work,
    I mean I can't see why it wouldn't.
    Could you see a reason?

No, now that I understand it, I understand it (I think) ;-).

    M-x grep says:  Run grep (like this): grep -nH -e
    not Run grep (with the following arguments): -nH -e

Yes, I noticed that prompt change, but the subtlety didn't sink in. It's not
just running `grep' with those options, however. It's starting an
asynchronous process and passing `grep -nH -e...' to it. That's not really
the same thing as executing `grep -hH -e <some grep args>', it seems to me.

    Behaviour depends on the version of grep and the value of
    grep-command but ultimately the value of grep-use-null-device.
    It has to do with the fact that without "-H" grep doesn't
    output the filename if only one is given
    in the input (hence the dummy extra: /dev/null).  I don't which versions
    of grep have "-H" or when it was added to GNU/Linux.

I see. Good to know. I'm not sure how much of this info is worth
documenting, but I'm convinced some of it is:

1. Explain better that `start-process' accepts a shell command line, not
just arguments for the command.

2. Explain that (depending on your version of grep), you might be able to
chain grep commands.

Very enlightening - thx.

Re: grep command doc - point out that you can chain now

[Richard Stallman]

    I'd say that this should perhaps be made more clear in the Emacs-Lisp doc
    for `start-process'. That doc does point out that `start-process' starts an
    asynchronous process, and it says that whatever you give it as &rest args
    are treated like command-line args, but, though I've read that before and
    used `start-process' several times, I never interpreted "command-line
    arguments for the program" (e.g. grep) to mean whatever might be on the
    command line, including pipes (and redirection etc., presumably).

Your previous interpretation was correct: args to start-process are
passed to the program.  In general, pipes and redirection are NOT
allowed.

They work in these specific contexts which use start-process to run
the shell.  The string you type is passed as argument to the shell.

Re: grep command doc - point out that you can chain now

[Richard Stallman]

    Probably because you knew it didn't work in the past.

No, it just never occurred to me to think about the question.

							   It still
    doesn't work for users with versions of grep which don't have the
    -H option, so they will be confused by your description.

The default command uses the -H option, so anyone who has trouble
with this example will have trouble with the default.  I don't think
the Emacs manual needs to cater to that.

Re: grep command doc - point out that you can chain now

[Nick Roberts]

 > 							   It still
 >     doesn't work for users with versions of grep which don't have the
 >     -H option, so they will be confused by your description.
 > 
 > The default command uses the -H option, so anyone who has trouble
 > with this example will have trouble with the default.  I don't think
 > the Emacs manual needs to cater to that.

The default is computed by grep-compute-defaults.  The default command won't
use the -H option if the user's version of grep doesn't have it.

-- 
Nick                                           http://www.inet.net.nz/~nickrob

Re: grep command doc - point out that you can chain now

[Richard Stallman]

    The default is computed by grep-compute-defaults.  The default command won't
    use the -H option if the user's version of grep doesn't have it.

What does it do?  Add /dev/null as input instead?

Re: grep command doc - point out that you can chain now

[Nick Roberts]

 >     The default is computed by grep-compute-defaults.  The default command
 >     won't use the -H option if the user's version of grep doesn't have it.
 > 
 > What does it do?  Add /dev/null as input instead?

Yes.  If grep uses -H and it's valid, grep-use-null-device is set to nil.
Otherwise it's t.  The user could get piping to work by setting
grep-use-null-device to nil directly but that would mean that the filename
doesn't get included in the output when there was only one input file.

It's a bit awkward to explain in the manual.  You say "you can use any shell
command that produces output in the same format."  Yes, you could run "make"
through the grep buffer, and "grep" through the compile buffer.  You can run a
telnet session through the GUD buffer (and debug remotely), but I don't
think these things need be described in the manual.  Why not just extend the
existing entry in NEWS?:

  *** M-x grep now tries to avoid appending `/dev/null' to the command line
  by using GNU grep `-H' option instead.  M-x grep automatically
  detects whether this is possible or not the first time it is invoked.
  When `-H' is used, the grep command line supplied by the user is passed
  unchanged to the system to execute, which allows more complicated
  command lines to be used than was possible before.

-- 
Nick                                           http://www.inet.net.nz/~nickrob

RE: grep command doc - point out that you can chain now

[Drew Adams]

    It's a bit awkward to explain in the manual....Why not
    just extend the existing entry in NEWS?:

Piping grep commands is particularly useful; the others you mention are not.
The fact that you can chain multiple greps and use the result in buffer
*grep* is an extension of grep functionality, in effect. I think it would be
useful to mention that one use pattern in the Emacs manual, where grep is
discussed.

RE: grep command doc - point out that you can chain now

[Nick Roberts]

 >     It's a bit awkward to explain in the manual....Why not
 >     just extend the existing entry in NEWS?:
 > 
 > Piping grep commands is particularly useful;

I'm not questioning its usefulness, just saying that I think its description
is more appropriate to a Unix manual.  

 >                                            the others you mention are not.

Actually remote debugging in Emacs would be rather useful to those that do it.

 > The fact that you can chain multiple greps and use the result in buffer
 > *grep* is an extension of grep functionality, in effect. I think it would be
 > useful to mention that one use pattern in the Emacs manual, where grep is
 > discussed.

Yes, as you've already said.


-- 
Nick                                           http://www.inet.net.nz/~nickrob

Re: grep command doc - point out that you can chain now

[Richard Stallman]

      Why not just extend the
    existing entry in NEWS?:

NEWS is not the most effective place to describe a feature
for users to see it.