problem using call-process with grep

problem using call-process with grep

[Drew Adams]

I hesitate whether to send this to emacs-devel or help-gnu-emacs. I doubt
that this is a bug - it's probably user ignorance, because I see the same
behavior in both Emacs 20 and 22. Perhaps someone can tell me what the
problem is.

Type this in a file foo, ending the file with a newline:

alpha,beta,gamma,delta,epsilon

Evaluate this:

 (call-process "grep" nil
               (generate-new-buffer "Result")
               nil "epsilon$" "foo")

As expected, buffer Result contains the same text as file foo. The
evaluation returns 0.

Change the regexp passed to `grep' to this: "epsilon\\($\\)" and try again.
I expect the same result, but, instead, buffer Result is empty and the
evaluation returns 1.

If I run the Emacs `grep' command like this (with or without the options),
it works:

grep -nH -e "epsilon\\($\\)" foo

This gives the same result using Emacs `grep' (single backslashes):

grep "epsilon\($\)" foo

(Since these are not Lisp strings, but input args to the Emacs `grep'
command, which it passes on to the `grep' process, I'm a bit surprised to
see that the double-backslash version works, but it does.)

I'm using Cygwin `grep' (GNU `grep'), for which the doc says that \(, $, and
\) are supported in the usual way. And if I run grep from the Cygwin command
line (with either parenthetical regexp - single or double backslashes), it
works OK. I only have the problem when I use `call-process'.

I'm sure I'm missing something simple (doing something wrong in the call to
`call-process') - I'd be grateful for the explanation.

FWIW: The coding system for the file foo is undecided-unix. The defaults for
subprocess I/O are undecided-dos (decoding) and undecided-unix (encoding).
I've loaded cywin-mount.el, version 1.2.1 - 2001-05-13. shell-file-name is
"bash". w32-quote-process-arg is ?\". process-coding-system-alist is
(("bash" raw-text-dos . raw-text-unix)).

In GNU Emacs 22.0.50.1 (i386-mingw-nt5.1.2600)
 of 2005-06-26 on NONIQPC
X server distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc
(3.3) --cflags -I../../jpeg-6b-3/include -I../../libpng-1.2.8/include -I../.
./tiff-3.6.1-2/include -I../../xpm-nox-4.2.0/include -I../../zlib-1.2.2/incl
ude'

Thanks.

Re: problem using call-process with grep

[Richard M. Stallman]

When I run

   grep 'epsilon\($\)' foo

from the shell, it finds a match.
What does your grep do?

RE: problem using call-process with grep

[Drew Adams]

    When I run grep 'epsilon\($\)' foo
    from the shell, it finds a match. What does your grep do?

I said:

  And if I run grep from the Cygwin command line (with either
  parenthetical regexp - single or double backslashes), it
  works OK. I only have the problem when I use `call-process'.

By command line, I meant shell (bash from Cygwin). (I showed the regexp with
double-quotes, but single-quotes work too.)

Summary: grep works with that regexp when run from the shell or when using
the Emacs `grep' command. The only problem is with `call-process'.

Re: problem using call-process with grep

[Kevin Rodgers]

Drew Adams wrote:
>     When I run grep 'epsilon\($\)' foo
>     from the shell, it finds a match. What does your grep do?
> 
> I said:
> 
>   And if I run grep from the Cygwin command line (with either
>   parenthetical regexp - single or double backslashes), it
>   works OK. I only have the problem when I use `call-process'.
> 
> By command line, I meant shell (bash from Cygwin). (I showed the regexp with
> double-quotes, but single-quotes work too.)
> 
> Summary: grep works with that regexp when run from the shell or when using
> the Emacs `grep' command. The only problem is with `call-process'.

The quotes are not part of the regex, and are only necessary when
running the command via a shell (`M-x grep' uses
start-process-shell-command).

The correct way to use call-process would be:

(call-process "grep" nil BUFFER nil "epsilon\\($\\)" "foo")

-- 
Kevin

RE: problem using call-process with grep

[Drew Adams]

    >     When I run grep 'epsilon\($\)' foo
    >     from the shell, it finds a match. What does your grep do?
    >
    > I said:
    >
    >   And if I run grep from the Cygwin command line (with either
    >   parenthetical regexp - single or double backslashes), it
    >   works OK. I only have the problem when I use `call-process'.
    >
    > By command line, I meant shell (bash from Cygwin). (I showed
    > the regexp with double-quotes, but single-quotes work too.)
    >
    > Summary: grep works with that regexp when run from the shell
    > or when using the Emacs `grep' command. The only problem is
    > with `call-process'.

    The quotes are not part of the regex, and are only necessary
    when running the command via a shell (`M-x grep' uses
    start-process-shell-command).

    The correct way to use call-process would be:

    (call-process "grep" nil BUFFER nil "epsilon\\($\\)" "foo")

Uh, isn't that what I wrote(?):

 (call-process "grep" nil
               (generate-new-buffer "Result")
               nil "epsilon\\($\\)" "foo")

This does *not* work for me. That's the point - that is the code I started
with. It works if I change \\($\\) to $ (but that's not what I need in the
general case).

(And yes, I realize that the quotes are not part of the regexp. By "I showed
the regexp with double-quotes" I meant that I showed it enclosed in
double-quotes.)

Re: problem using call-process with grep

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     >     When I run grep 'epsilon\($\)' foo
>     >     from the shell, it finds a match. What does your grep do?
>     >
>     > I said:
>     >
>     >   And if I run grep from the Cygwin command line (with either
>     >   parenthetical regexp - single or double backslashes), it
>     >   works OK. I only have the problem when I use `call-process'.
>     >
>     > By command line, I meant shell (bash from Cygwin). (I showed
>     > the regexp with double-quotes, but single-quotes work too.)
>     >
>     > Summary: grep works with that regexp when run from the shell
>     > or when using the Emacs `grep' command. The only problem is
>     > with `call-process'.
>
>     The quotes are not part of the regex, and are only necessary
>     when running the command via a shell (`M-x grep' uses
>     start-process-shell-command).
>
>     The correct way to use call-process would be:
>
>     (call-process "grep" nil BUFFER nil "epsilon\\($\\)" "foo")
>
> Uh, isn't that what I wrote(?):
>
>  (call-process "grep" nil
>                (generate-new-buffer "Result")
>                nil "epsilon\\($\\)" "foo")
>
> This does *not* work for me. That's the point - that is the code I started
> with. It works if I change \\($\\) to $ (but that's not what I need in the
> general case).
>
> (And yes, I realize that the quotes are not part of the regexp. By "I showed
> the regexp with double-quotes" I meant that I showed it enclosed in
> double-quotes.)

I guess you want to call some echo-like program compiled in the same
way as your "grep" to see what actually is happening here.  Probably
some part of Cygwin's Unix emulation layer is trying variable
substitution or replacing backward with forward slashes or adding
quotes or something like that.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: problem using call-process with grep

[Drew Adams]

    >  (call-process "grep" nil
    >                (generate-new-buffer "Result")
    >                nil "epsilon\\($\\)" "foo")
    >
    > This does *not* work for me.

    I guess you want to call some echo-like program compiled in the same
    way as your "grep" to see what actually is happening here.  Probably
    some part of Cygwin's Unix emulation layer is trying variable
    substitution or replacing backward with forward slashes or adding
    quotes or something like that.

Could you be more specific on what to try?

I also don't understand how this could be a Cygwin problem if there is no
problem when I call `grep' directly in a Cygwin shell (bash) or indirectly
via the Emacs `grep' command. That is, once bash gets the `grep' arguments
correctly, it does the right thing. It is `call-process' that in effect
passes the args to `grep'.

I don't claim that this is not a Cygwin (bash) problem; I just don't
understand how it could be. To me, this is a problem with `call-process'. My
assumption was that I was simply not calling `call-process' correctly (I'm
not claiming there is a bug with `call-process').

But you made me think of another possibility. Could it be that
`call-process' somehow processes its arguments in such a way that it applies
Windows-to-Unix directory-separator conversion to all of them, flipping
backslashes to slashes?

I don't think so. For instance, if I use this regexp, it works OK:
"eps\\ilon$". If backslashes were changed to slashes, then this would not
match "epsilon" in the file.

Another regexp that doesn't work: "gamma\\(,\\)" - just to show that the
problem is with \(...\), not with $ (variable substitution).

Re: problem using call-process with grep

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     >  (call-process "grep" nil
>     >                (generate-new-buffer "Result")
>     >                nil "epsilon\\($\\)" "foo")
>     >
>     > This does *not* work for me.
>
>     I guess you want to call some echo-like program compiled in the same
>     way as your "grep" to see what actually is happening here.  Probably
>     some part of Cygwin's Unix emulation layer is trying variable
>     substitution or replacing backward with forward slashes or adding
>     quotes or something like that.
>
> Could you be more specific on what to try?

Use /bin/echo instead of grep.  Something like that.

> I also don't understand how this could be a Cygwin problem

Because Emacs might be compiled with Cygwin?  Because it might try to
interact with Cygwin?

> But you made me think of another possibility. Could it be that
> `call-process' somehow processes its arguments in such a way that it
> applies Windows-to-Unix directory-separator conversion to all of
> them, flipping backslashes to slashes?
>
> I don't think so. For instance, if I use this regexp, it works OK:
> "eps\\ilon$". If backslashes were changed to slashes, then this
> would not match "epsilon" in the file.
>
> Another regexp that doesn't work: "gamma\\(,\\)" - just to show that
> the problem is with \(...\), not with $ (variable substitution).

Well, maybe what arrives at the application level is gamma(,) without
backslashed at all.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: problem using call-process with grep

[Drew Adams]

    maybe what arrives at the application level is gamma(,) without
    backslashes

Bingo!  Thank you, David.

Don't know why I didn't see this before. I just need to use "\\\\" to get \,
as is usual with a Lisp string (which this is). Sorry for wasting time.

Re: problem using call-process with grep

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     maybe what arrives at the application level is gamma(,) without
>     backslashes
>
> Bingo!  Thank you, David.
>
> Don't know why I didn't see this before. I just need to use "\\\\" to get \,
> as is usual with a Lisp string (which this is). Sorry for wasting time.

Uh, there is still something wrong here.  You wrote:

>     >  (call-process "grep" nil
>     >                (generate-new-buffer "Result")
>     >                nil "epsilon\\($\\)" "foo")
>     >

This means that the string is 'epsilon\($\)', and call-process is
supposed to pass this unmolested into grep.  So some other entity is
removing a level of backslashes that shouldn't.  It is either
call-process, or grep itself.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: problem using call-process with grep

[Drew Adams]

    >     maybe what arrives at the application level is gamma(,) without
    >     backslashes
    >
    > Bingo!  Thank you, David.
    >
    > Don't know why I didn't see this before. I just need to use
    > "\\\\" to get \, as is usual with a Lisp string (which
    > this is). Sorry for wasting time.

    Uh, there is still something wrong here.  You wrote:

    >     >  (call-process "grep" nil
    >     >                (generate-new-buffer "Result")
    >     >                nil "epsilon\\($\\)" "foo")
    >     >

    This means that the string is 'epsilon\($\)', and call-process is
    supposed to pass this unmolested into grep.  So some other entity is
    removing a level of backslashes that shouldn't.  It is either
    call-process, or grep itself.

Hm, you're right. It's as if we had to supply a regexp that would match the
literal string `\('. The Lisp string to do that would be "\\\\(". I just
re-read the doc for `grep' and `call-process'; neither mentions anything
about this.

To figure out whether this is specific to Cygwin `grep' or a property of
`call-process', could someone else (e.g. on GNU/Linux) please try these:

 (call-process "/path/to/your/echo" nil t nil  "foo\\\\(bar\\\\)")

 (call-process "/path/to/your/echo" nil t nil  "foo\\(bar\\)")

In my case, the first gives: foo\(bar\); the second gives foo(bar).

Re: problem using call-process with grep

[Giorgos Keramidas]

        >     maybe what arrives at the application level is gamma(,)
        >     without backslashes
        >
        > Bingo!  Thank you, David.
        >
        > Don't know why I didn't see this before. I just need to use
        > "\\\\" to get \, as is usual with a Lisp string (which this
        > is). Sorry for wasting time.

        Uh, there is still something wrong here.  You wrote:

        >     >  (call-process "grep" nil
        >     >                (generate-new-buffer "Result")
        >     >                nil "epsilon\\($\\)" "foo")
        >     >

        This means that the string is 'epsilon\($\)', and call-process
        is supposed to pass this unmolested into grep.  So some other
        entity is removing a level of backslashes that shouldn't.  It is
        either call-process, or grep itself.

    Hm, you're right. It's as if we had to supply a regexp that would match the
    literal string `\('. The Lisp string to do that would be "\\\\(". I just
    re-read the doc for `grep' and `call-process'; neither mentions anything
    about this.

    To figure out whether this is specific to Cygwin `grep' or a property of
    `call-process', could someone else (e.g. on GNU/Linux) please try these:

    (call-process "/path/to/your/echo" nil t nil  "foo\\\\(bar\\\\)")

Using M-: the following expression evaluates to:

    (call-process "/bin/echo" nil t nil  "foo\\\\(bar\\\\)")
    => 0 (#o0, #x0)

In my Lisp interaction buffer the following is inserted:

    foo\\(bar\\)

Evaluating the following expression:

    (call-process "/path/to/your/echo" nil t nil  "foo\\(bar\\)")
    => 0 (#o0, #x0)

Results in the following line of text:

    foo\(bar\)

This is not GNU/Linux but FreeBSD/amd64, FWIW.  The results seem
to indicate that there is only one level of escaping here, the
one done by Lisp.

Re: problem using call-process with grep

[Richard M. Stallman]

When I evaluate this:

     (call-process "grep" nil
		   (generate-new-buffer "Result")
		   nil "epsilon\\($\\)" "foo")

it does match.

    Summary: grep works with that regexp when run from the shell or when using
    the Emacs `grep' command. The only problem is with `call-process'.

Can you debug the inside of `call-process' and see what arguments
are actually passed to grep?

RE: problem using call-process with grep

[Drew Adams]

        Summary: grep works with that regexp when run from the
        shell or when using the Emacs `grep' command. The only
        problem is with `call-process'.

    Can you debug the inside of `call-process' and see what arguments
    are actually passed to grep?

No, I cannot debug the C code. Perhaps someone else can. See Eli's email for
more background info.

Re: problem using call-process with grep

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 24 Dec 2005 08:35:35 -0800
> 
>         Summary: grep works with that regexp when run from the
>         shell or when using the Emacs `grep' command. The only
>         problem is with `call-process'.
> 
>     Can you debug the inside of `call-process' and see what arguments
>     are actually passed to grep?
> 
> No, I cannot debug the C code.

Why not?  We can provide guidance, if you aren't familiar with GDB.
Right now, it seems like a simple matter of setting a breakpoint on a
single source line and when it's hit, printing a single variable.

RE: problem using call-process with grep

[Drew Adams]

    > No, I cannot debug the C code.
    
    Why not?

1. I am allergic to C.
2. C is allergic to me.
3. I don't have the time, nor does C.

Re: problem using call-process with grep

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 24 Dec 2005 09:08:53 -0800
> 
>     > No, I cannot debug the C code.
>     
>     Why not?
> 
> 1. I am allergic to C.
> 2. C is allergic to me.
> 3. I don't have the time, nor does C.

Too bad: without you, this bug could remain unsolved for a long time.

RE: problem using call-process with grep

[Drew Adams]

    To figure out whether this is specific to Cygwin `grep' or a property of
    `call-process', could someone else (e.g. on GNU/Linux) please try these:

     (call-process "/path/to/your/echo" nil t nil  "foo\\\\(bar\\\\)")

     (call-process "/path/to/your/echo" nil t nil  "foo\\(bar\\)")

    In my case, the first gives: foo\(bar\); the second gives foo(bar).

Feedback from someone on GNU/Linux:

    system-configuration
    => "i386-pc-linux-gnu"

    (call-process "/bin/echo" nil t nil
                  "\\x \\\\x \\(x\\) \\\\(x\\\\)")
    => 0
    =| \x \\x \(x\) \\(x\\)

    (call-process "/bin/sh" nil t nil "-c"
                  "echo '\\x \\\\x \\(x\\) \\\\(x\\\\)'")
    => 0
    =| \x \\x \(x\) \\(x\\)

    identical results for buffers in text and emacs lisp modes.

I misstated the test, above. This doesn't tell us anything about `grep',
obviously.

Do the tests mean that there is a problem in the native Windows
implementation of `call-process'? Could they mean that there is a problem in
all Cygwin processes?

IIUC, `call-process' doesn't spawn a shell, so if this were a problem on the
Cygwin end, then it would have to be a problem for each of `grep', `echo',
and `sh' (commands tested so far) that doesn't manifest itself from the
command line. Or am I missing something else?

Could someone who works on the Windows port perhaps take a look to see if
`call-process' is doing the right thing? It would seem surprising for this
to turn out to be a bug in that code, since this behavior is old (at least
since Emacs 20) and it occurs in such a prominent place. I'd think that this
would have already been pounded on enough to expose any problem of this
kind.

Re: problem using call-process with grep

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Fri, 23 Dec 2005 17:51:04 -0800
> 
>     To figure out whether this is specific to Cygwin `grep' or a property of
>     `call-process', could someone else (e.g. on GNU/Linux) please try these:
> 
>      (call-process "/path/to/your/echo" nil t nil  "foo\\\\(bar\\\\)")
> 
>      (call-process "/path/to/your/echo" nil t nil  "foo\\(bar\\)")
> 
>     In my case, the first gives: foo\(bar\); the second gives foo(bar).

Is your `echo.exe' also a Cygwin executable?  My native Windows
executable gives foo\\(bar\\) and foo\(bar\) respectively, as it does
on GNU/Linux.

FWIW, your original test, wiz.:

    (call-process "grep" nil
		   (generate-new-buffer "Result")
		   nil "epsilon\\($\\)" "foo")

also works for me as expected, but my Grep is not a Cygwin executable,
it's a native Windows executable.

> Do the tests mean that there is a problem in the native Windows
> implementation of `call-process'? Could they mean that there is a problem in
> all Cygwin processes?

It could be, see below.

> IIUC, `call-process' doesn't spawn a shell

It doesn't (unless the command is a shell, of course).

> Could someone who works on the Windows port perhaps take a look to see if
> `call-process' is doing the right thing?

The Windows code invoked by `call-process' does quite a lot with
argument escaping and quoting, see w32proc.c:sys_spawnve.  It does so
for a good reason: Windows executables generally have the globbing
code as part of the program itself, as stock Windows shells don't
handle that.  Thus, to DTRT with special characters, such as shell
wildcard characters, quotes and escaping, the Windows code that
launches subsidiary programs needs to know whether the program being
run is a Cygwin program or a native w32 program, and it needs then to
massage the command-line arguments such that when they wind up in the
child program, they will, after removal of quotes and escape
characters done by the application's code, be in the form you wanted
them to be.

That is quite a feat, but I think the code generally does TRT (in
particular, I see that it doubles every \ for Cygwin subsidiary
programs), although having a mix of Cygwin and non-Cygwin tools will
probably put that code to a hard test.

To see what is being passed to the sub-process, the best way is to put
a breakpoint in w32proc.c:sys_spawnve, right before it is about to
call create_child, and look at the command line it generated.  I don't
have Cygwin executables installed, so I cannot look into this myself.
Don't waste your time on experiments with `echo' and other programs:
that only adds more complexity to the problem, as those other programs
also process their command line, and what you see is a result of that
processing, not what Emacs passed.

It is also possible that w32proc.c:w32_executable_type, which tries to
determine whether the subsidiary program is a Cygwin executable, fails
for some reason, so look at that function's results as well.

On top of that, it could be that Cygwin runtime has changed the way it
parses command-line arguments, and it now expects them to be quoted
differently than what Emacs does.

Finally, AFAIK Drew is using an old build of Emacs, so perhaps trying
this in a newer build is in order.

RE: problem using call-process with grep

[Drew Adams]

    Finally, AFAIK Drew is using an old build of Emacs, so perhaps trying
    this in a newer build is in order.

FWIW, I get the same behavior with the build of June 6, 2005 (6 months old).
My Cygwin dates from January 2004, judging by some of the file dates.

Thanks for the other information. I won't be able to test the C code, but
perhaps someone will.

Re: problem using call-process with grep

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 24 Dec 2005 07:48:35 -0800
> 
>     Finally, AFAIK Drew is using an old build of Emacs, so perhaps trying
>     this in a newer build is in order.
> 
> FWIW, I get the same behavior with the build of June 6, 2005 (6 months old).

That's what I meant by ``old'' ;-)

> Thanks for the other information. I won't be able to test the C code, but
> perhaps someone will.

It will have to be someone who has Cygwin binaries.

RE: problem using call-process with grep

[Drew Adams]

    > FWIW, I get the same behavior with the build of June 6, 2005 
    > (6 months old).
    
    That's what I meant by ``old'' ;-)

Yes, I know. Just wanted to be clear about the version.