Q on call-process and grep

Q on call-process and grep

[Drew Adams]

I'm using native Emacs on Windows, and I use Cygwin for commands like
`grep'. The (large) file "myfile" has only comma-delimited lines of words
and phrases, like this:

  word1,word2,word phrase3,word4,word phrase4,word5

Some of the lines are quite long. Lines are of varying length (different
words and phrases). There are no TABs. There are no spaces either, except
between words in phrases (none before or after commas or at eol).

If I do this from the command line or using Emacs command `grep', it works
fine:

  grep -i ",someword\\($\\|,\\)" "myfile"

If, however, I do this, then some words are found and others (which are also
present in myfile) are not found:

  (call-process "grep" nil buf nil "-i" ",someword\\($\\|,\\)" "myfile")

The same words are systematically found or not found. I haven't been able to
figure out why this doesn't work for some words (only). I visited the file
with `find-file-literally' and removed all ^M's at eol. I checked that the
commas are normal commas, thinking that some might have a different encoding
or something, making the regexp miss. All characters are ASCII, I believe
(how to check that?). The problem also doesn't seem to be related to line
lengths or the positions of the target words in the lines.

When I do `C-h C RET' it says that the buffer (for the grepped file) has no
conversion (binary), that the defaults for subprocess I/O are
`undecided-dos' for decoding and `undecided-unix' for encoding, and that
process I/O with target-pattern "bash" uses coding systems `(raw-text-dos .
raw-text-unix)'. I don't know if some of that might be a problem (or how to
change it, if it is).

I've tried having the error output sent to a file (e.g. "errors") - but to
no avail:

  (call-process "grep" nil (list (get-buffer "*scratch*") "errors")
     nil "-i" ",someword\\($\\|,\\)" "myfile")

Grep apparently does not error (the return code is 1), and the output buffer
is always empty for some words, as is also the error file.

I get the same behavior in different versions of Emacs, so I'm no doubt
missing something (i.e. there is no bug, except in my understanding). Any
ideas? Any suggestions on how to debug this? Thx.

Re: Q on call-process and grep

[Eric Pement]

Drew Adams wrote:

> I'm using native Emacs on Windows, and I use Cygwin [ ... ]

> If I do this from the command line or using Emacs command `grep', it works
> fine:
>
>   grep -i ",someword\\($\\|,\\)" "myfile"
>
> If, however, I do this, then some words are found and others (which are also
> present in myfile) are not found:
>
>   (call-process "grep" nil buf nil "-i" ",someword\\($\\|,\\)" "myfile")
>
> The same words are systematically found or not found. I haven't been able to
> figure out why this doesn't work for some words (only).

If it were me, I'd make a copy of the file, and then chop it into
smaller pieces where I can illustrate the problem in a manageable
length (say, 10 or 20 lines, but the fewer the better). The sed command

   sed -n 17,35p bigfile >smallfile

will print lines 17 to 35, inclusive, so you can do your testing. But
you say that some of the lines are quite long. So try this:

   awk '{print length($0)}' smallfile

to see how long is too long. If the lines are under 4000 chars, I'd
feel safe in guessing that line length isn't a problem. If you have
lines 20,000 chars or more, then I'd start thinking about the input.

Does each line in the problem set end in a CR/LF? I've had datafiles
that gave me bad data because somehow some lines ended with CR/LF,
others with CR/CR/LF, and others with CR only. How I got the problem
isn't relevant. But to normalize the input, try

   tr -d '\r' <smallfile | sed -n p >clean_smallfile

which should remove any extraneous CRs which might be causing
corruption and restore the line endings to your Cygwin default (Unix or
DOS, whichever you picked).

[ ... ]
> All characters are ASCII, I believe (how to check that?).

   Use tr to delete all the characters that are permissible or
expected, and whatever is left must be an unexpected character. Examine
the output with cat -A or od or your tool of choice. E.g.,

   tr -d '\n\r\t\40-\176' <infile >outfile

If it were me, I might wonder about embedded backspaces or carriage
returns in the text. Just a thought. Good luck on your hunting!

--
Eric Pement

RE: Q on call-process and grep

[Drew Adams]

    If it were me, I'd make a copy of the file, and then chop it into
    smaller pieces where I can illustrate the problem in a manageable
    length (say, 10 or 20 lines, but the fewer the better). The sed command

       sed -n 17,35p bigfile >smallfile

There are over 30,000 lines.

    will print lines 17 to 35, inclusive, so you can do your testing. But
    you say that some of the lines are quite long. So try this:

       awk '{print length($0)}' smallfile

    to see how long is too long. If the lines are under 4000 chars, I'd
    feel safe in guessing that line length isn't a problem. If you have
    lines 20,000 chars or more, then I'd start thinking about the input.

I was hoping that I was missing something simple. You seem to be confirming
that I didn't miss anything obvious (to you) ;-).

The longest line is over 12,000 characters.

    Does each line in the problem set end in a CR/LF? I've had datafiles
    that gave me bad data because somehow some lines ended with CR/LF,
    others with CR/CR/LF, and others with CR only. How I got the problem
    isn't relevant. But to normalize the input, try

       tr -d '\r' <smallfile | sed -n p >clean_smallfile

    which should remove any extraneous CRs which might be causing
    corruption and restore the line endings to your Cygwin default (Unix or
    DOS, whichever you picked).

Did that on the complete original file. `ediff' shows no difference from the
original.

I tried using a small file - just a few lines of the original - no change.
Terms that can't be found still aren't; those that can be found still are.

    Use tr to delete all the characters that are permissible or
    expected, and whatever is left must be an unexpected character. Examine
    the output with cat -A or od or your tool of choice. E.g.,

       tr -d '\n\r\t\40-\176' <infile >outfile

Did that. outfile is empty, so I guess everything was ASCII.

    If it were me, I might wonder about embedded backspaces or carriage
    returns in the text. Just a thought. Good luck on your hunting!

My guess is that the line lengths and number of lines don't matter here,
because it works fine for other words, including 1) words in the longest
line and 2) words in the last line of the file. It's a mystery to me why it
doesn't work for certain words.

Thanks for your suggestions, though - they were good things to try, even if
I haven't yet solved the problem.

problem using call-process with grep

[Drew Adams]

I hesitate whether to send this to emacs-devel or help-gnu-emacs. I doubt
that this is a bug - it's probably user ignorance, because I see the same
behavior in both Emacs 20 and 22. Perhaps someone can tell me what the
problem is.

Type this in a file foo, ending the file with a newline:

alpha,beta,gamma,delta,epsilon

Evaluate this:

 (call-process "grep" nil
               (generate-new-buffer "Result")
               nil "epsilon$" "foo")

As expected, buffer Result contains the same text as file foo. The
evaluation returns 0.

Change the regexp passed to `grep' to this: "epsilon\\($\\)" and try again.
I expect the same result, but, instead, buffer Result is empty and the
evaluation returns 1.

If I run the Emacs `grep' command like this (with or without the options),
it works:

grep -nH -e "epsilon\\($\\)" foo

This gives the same result using Emacs `grep' (single backslashes):

grep "epsilon\($\)" foo

(Since these are not Lisp strings, but input args to the Emacs `grep'
command, which it passes on to the `grep' process, I'm a bit surprised to
see that the double-backslash version works, but it does.)

I'm using Cygwin `grep' (GNU `grep'), for which the doc says that \(, $, and
\) are supported in the usual way. And if I run grep from the Cygwin command
line (with either parenthetical regexp - single or double backslashes), it
works OK. I only have the problem when I use `call-process'.

I'm sure I'm missing something simple (doing something wrong in the call to
`call-process') - I'd be grateful for the explanation.

FWIW: The coding system for the file foo is undecided-unix. The defaults for
subprocess I/O are undecided-dos (decoding) and undecided-unix (encoding).
I've loaded cywin-mount.el, version 1.2.1 - 2001-05-13. shell-file-name is
"bash". w32-quote-process-arg is ?\". process-coding-system-alist is
(("bash" raw-text-dos . raw-text-unix)).

In GNU Emacs 22.0.50.1 (i386-mingw-nt5.1.2600)
 of 2005-06-26 on NONIQPC
X server distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc
(3.3) --cflags -I../../jpeg-6b-3/include -I../../libpng-1.2.8/include -I../.
./tiff-3.6.1-2/include -I../../xpm-nox-4.2.0/include -I../../zlib-1.2.2/incl
ude'

Thanks.

Re: problem using call-process with grep

[Richard M. Stallman]

When I run

   grep 'epsilon\($\)' foo

from the shell, it finds a match.
What does your grep do?

RE: problem using call-process with grep

[Drew Adams]

    When I run grep 'epsilon\($\)' foo
    from the shell, it finds a match. What does your grep do?

I said:

  And if I run grep from the Cygwin command line (with either
  parenthetical regexp - single or double backslashes), it
  works OK. I only have the problem when I use `call-process'.

By command line, I meant shell (bash from Cygwin). (I showed the regexp with
double-quotes, but single-quotes work too.)

Summary: grep works with that regexp when run from the shell or when using
the Emacs `grep' command. The only problem is with `call-process'.

Re: problem using call-process with grep

[Kevin Rodgers]

Drew Adams wrote:
>     When I run grep 'epsilon\($\)' foo
>     from the shell, it finds a match. What does your grep do?
> 
> I said:
> 
>   And if I run grep from the Cygwin command line (with either
>   parenthetical regexp - single or double backslashes), it
>   works OK. I only have the problem when I use `call-process'.
> 
> By command line, I meant shell (bash from Cygwin). (I showed the regexp with
> double-quotes, but single-quotes work too.)
> 
> Summary: grep works with that regexp when run from the shell or when using
> the Emacs `grep' command. The only problem is with `call-process'.

The quotes are not part of the regex, and are only necessary when
running the command via a shell (`M-x grep' uses
start-process-shell-command).

The correct way to use call-process would be:

(call-process "grep" nil BUFFER nil "epsilon\\($\\)" "foo")

-- 
Kevin

RE: problem using call-process with grep

[Drew Adams]

    >     When I run grep 'epsilon\($\)' foo
    >     from the shell, it finds a match. What does your grep do?
    >
    > I said:
    >
    >   And if I run grep from the Cygwin command line (with either
    >   parenthetical regexp - single or double backslashes), it
    >   works OK. I only have the problem when I use `call-process'.
    >
    > By command line, I meant shell (bash from Cygwin). (I showed
    > the regexp with double-quotes, but single-quotes work too.)
    >
    > Summary: grep works with that regexp when run from the shell
    > or when using the Emacs `grep' command. The only problem is
    > with `call-process'.

    The quotes are not part of the regex, and are only necessary
    when running the command via a shell (`M-x grep' uses
    start-process-shell-command).

    The correct way to use call-process would be:

    (call-process "grep" nil BUFFER nil "epsilon\\($\\)" "foo")

Uh, isn't that what I wrote(?):

 (call-process "grep" nil
               (generate-new-buffer "Result")
               nil "epsilon\\($\\)" "foo")

This does *not* work for me. That's the point - that is the code I started
with. It works if I change \\($\\) to $ (but that's not what I need in the
general case).

(And yes, I realize that the quotes are not part of the regexp. By "I showed
the regexp with double-quotes" I meant that I showed it enclosed in
double-quotes.)

Re: problem using call-process with grep

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     >     When I run grep 'epsilon\($\)' foo
>     >     from the shell, it finds a match. What does your grep do?
>     >
>     > I said:
>     >
>     >   And if I run grep from the Cygwin command line (with either
>     >   parenthetical regexp - single or double backslashes), it
>     >   works OK. I only have the problem when I use `call-process'.
>     >
>     > By command line, I meant shell (bash from Cygwin). (I showed
>     > the regexp with double-quotes, but single-quotes work too.)
>     >
>     > Summary: grep works with that regexp when run from the shell
>     > or when using the Emacs `grep' command. The only problem is
>     > with `call-process'.
>
>     The quotes are not part of the regex, and are only necessary
>     when running the command via a shell (`M-x grep' uses
>     start-process-shell-command).
>
>     The correct way to use call-process would be:
>
>     (call-process "grep" nil BUFFER nil "epsilon\\($\\)" "foo")
>
> Uh, isn't that what I wrote(?):
>
>  (call-process "grep" nil
>                (generate-new-buffer "Result")
>                nil "epsilon\\($\\)" "foo")
>
> This does *not* work for me. That's the point - that is the code I started
> with. It works if I change \\($\\) to $ (but that's not what I need in the
> general case).
>
> (And yes, I realize that the quotes are not part of the regexp. By "I showed
> the regexp with double-quotes" I meant that I showed it enclosed in
> double-quotes.)

I guess you want to call some echo-like program compiled in the same
way as your "grep" to see what actually is happening here.  Probably
some part of Cygwin's Unix emulation layer is trying variable
substitution or replacing backward with forward slashes or adding
quotes or something like that.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: problem using call-process with grep

[Drew Adams]

    >  (call-process "grep" nil
    >                (generate-new-buffer "Result")
    >                nil "epsilon\\($\\)" "foo")
    >
    > This does *not* work for me.

    I guess you want to call some echo-like program compiled in the same
    way as your "grep" to see what actually is happening here.  Probably
    some part of Cygwin's Unix emulation layer is trying variable
    substitution or replacing backward with forward slashes or adding
    quotes or something like that.

Could you be more specific on what to try?

I also don't understand how this could be a Cygwin problem if there is no
problem when I call `grep' directly in a Cygwin shell (bash) or indirectly
via the Emacs `grep' command. That is, once bash gets the `grep' arguments
correctly, it does the right thing. It is `call-process' that in effect
passes the args to `grep'.

I don't claim that this is not a Cygwin (bash) problem; I just don't
understand how it could be. To me, this is a problem with `call-process'. My
assumption was that I was simply not calling `call-process' correctly (I'm
not claiming there is a bug with `call-process').

But you made me think of another possibility. Could it be that
`call-process' somehow processes its arguments in such a way that it applies
Windows-to-Unix directory-separator conversion to all of them, flipping
backslashes to slashes?

I don't think so. For instance, if I use this regexp, it works OK:
"eps\\ilon$". If backslashes were changed to slashes, then this would not
match "epsilon" in the file.

Another regexp that doesn't work: "gamma\\(,\\)" - just to show that the
problem is with \(...\), not with $ (variable substitution).

Re: problem using call-process with grep

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     >  (call-process "grep" nil
>     >                (generate-new-buffer "Result")
>     >                nil "epsilon\\($\\)" "foo")
>     >
>     > This does *not* work for me.
>
>     I guess you want to call some echo-like program compiled in the same
>     way as your "grep" to see what actually is happening here.  Probably
>     some part of Cygwin's Unix emulation layer is trying variable
>     substitution or replacing backward with forward slashes or adding
>     quotes or something like that.
>
> Could you be more specific on what to try?

Use /bin/echo instead of grep.  Something like that.

> I also don't understand how this could be a Cygwin problem

Because Emacs might be compiled with Cygwin?  Because it might try to
interact with Cygwin?

> But you made me think of another possibility. Could it be that
> `call-process' somehow processes its arguments in such a way that it
> applies Windows-to-Unix directory-separator conversion to all of
> them, flipping backslashes to slashes?
>
> I don't think so. For instance, if I use this regexp, it works OK:
> "eps\\ilon$". If backslashes were changed to slashes, then this
> would not match "epsilon" in the file.
>
> Another regexp that doesn't work: "gamma\\(,\\)" - just to show that
> the problem is with \(...\), not with $ (variable substitution).

Well, maybe what arrives at the application level is gamma(,) without
backslashed at all.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: problem using call-process with grep

[Drew Adams]

    maybe what arrives at the application level is gamma(,) without
    backslashes

Bingo!  Thank you, David.

Don't know why I didn't see this before. I just need to use "\\\\" to get \,
as is usual with a Lisp string (which this is). Sorry for wasting time.

Re: problem using call-process with grep

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>     maybe what arrives at the application level is gamma(,) without
>     backslashes
>
> Bingo!  Thank you, David.
>
> Don't know why I didn't see this before. I just need to use "\\\\" to get \,
> as is usual with a Lisp string (which this is). Sorry for wasting time.

Uh, there is still something wrong here.  You wrote:

>     >  (call-process "grep" nil
>     >                (generate-new-buffer "Result")
>     >                nil "epsilon\\($\\)" "foo")
>     >

This means that the string is 'epsilon\($\)', and call-process is
supposed to pass this unmolested into grep.  So some other entity is
removing a level of backslashes that shouldn't.  It is either
call-process, or grep itself.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

RE: problem using call-process with grep

[Drew Adams]

    >     maybe what arrives at the application level is gamma(,) without
    >     backslashes
    >
    > Bingo!  Thank you, David.
    >
    > Don't know why I didn't see this before. I just need to use
    > "\\\\" to get \, as is usual with a Lisp string (which
    > this is). Sorry for wasting time.

    Uh, there is still something wrong here.  You wrote:

    >     >  (call-process "grep" nil
    >     >                (generate-new-buffer "Result")
    >     >                nil "epsilon\\($\\)" "foo")
    >     >

    This means that the string is 'epsilon\($\)', and call-process is
    supposed to pass this unmolested into grep.  So some other entity is
    removing a level of backslashes that shouldn't.  It is either
    call-process, or grep itself.

Hm, you're right. It's as if we had to supply a regexp that would match the
literal string `\('. The Lisp string to do that would be "\\\\(". I just
re-read the doc for `grep' and `call-process'; neither mentions anything
about this.

To figure out whether this is specific to Cygwin `grep' or a property of
`call-process', could someone else (e.g. on GNU/Linux) please try these:

 (call-process "/path/to/your/echo" nil t nil  "foo\\\\(bar\\\\)")

 (call-process "/path/to/your/echo" nil t nil  "foo\\(bar\\)")

In my case, the first gives: foo\(bar\); the second gives foo(bar).

Re: problem using call-process with grep

[Giorgos Keramidas]

        >     maybe what arrives at the application level is gamma(,)
        >     without backslashes
        >
        > Bingo!  Thank you, David.
        >
        > Don't know why I didn't see this before. I just need to use
        > "\\\\" to get \, as is usual with a Lisp string (which this
        > is). Sorry for wasting time.

        Uh, there is still something wrong here.  You wrote:

        >     >  (call-process "grep" nil
        >     >                (generate-new-buffer "Result")
        >     >                nil "epsilon\\($\\)" "foo")
        >     >

        This means that the string is 'epsilon\($\)', and call-process
        is supposed to pass this unmolested into grep.  So some other
        entity is removing a level of backslashes that shouldn't.  It is
        either call-process, or grep itself.

    Hm, you're right. It's as if we had to supply a regexp that would match the
    literal string `\('. The Lisp string to do that would be "\\\\(". I just
    re-read the doc for `grep' and `call-process'; neither mentions anything
    about this.

    To figure out whether this is specific to Cygwin `grep' or a property of
    `call-process', could someone else (e.g. on GNU/Linux) please try these:

    (call-process "/path/to/your/echo" nil t nil  "foo\\\\(bar\\\\)")

Using M-: the following expression evaluates to:

    (call-process "/bin/echo" nil t nil  "foo\\\\(bar\\\\)")
    => 0 (#o0, #x0)

In my Lisp interaction buffer the following is inserted:

    foo\\(bar\\)

Evaluating the following expression:

    (call-process "/path/to/your/echo" nil t nil  "foo\\(bar\\)")
    => 0 (#o0, #x0)

Results in the following line of text:

    foo\(bar\)

This is not GNU/Linux but FreeBSD/amd64, FWIW.  The results seem
to indicate that there is only one level of escaping here, the
one done by Lisp.

Re: problem using call-process with grep

[Richard M. Stallman]

When I evaluate this:

     (call-process "grep" nil
		   (generate-new-buffer "Result")
		   nil "epsilon\\($\\)" "foo")

it does match.

    Summary: grep works with that regexp when run from the shell or when using
    the Emacs `grep' command. The only problem is with `call-process'.

Can you debug the inside of `call-process' and see what arguments
are actually passed to grep?

RE: problem using call-process with grep

[Drew Adams]

        Summary: grep works with that regexp when run from the
        shell or when using the Emacs `grep' command. The only
        problem is with `call-process'.

    Can you debug the inside of `call-process' and see what arguments
    are actually passed to grep?

No, I cannot debug the C code. Perhaps someone else can. See Eli's email for
more background info.

Re: problem using call-process with grep

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 24 Dec 2005 08:35:35 -0800
> 
>         Summary: grep works with that regexp when run from the
>         shell or when using the Emacs `grep' command. The only
>         problem is with `call-process'.
> 
>     Can you debug the inside of `call-process' and see what arguments
>     are actually passed to grep?
> 
> No, I cannot debug the C code.

Why not?  We can provide guidance, if you aren't familiar with GDB.
Right now, it seems like a simple matter of setting a breakpoint on a
single source line and when it's hit, printing a single variable.

RE: problem using call-process with grep

[Drew Adams]

    > No, I cannot debug the C code.
    
    Why not?

1. I am allergic to C.
2. C is allergic to me.
3. I don't have the time, nor does C.

Re: problem using call-process with grep

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 24 Dec 2005 09:08:53 -0800
> 
>     > No, I cannot debug the C code.
>     
>     Why not?
> 
> 1. I am allergic to C.
> 2. C is allergic to me.
> 3. I don't have the time, nor does C.

Too bad: without you, this bug could remain unsolved for a long time.