bug#2397: 23.0.90; grep no longer highlights the match

bug#2397: 23.0.90; grep no longer highlights the match

[Drew Adams]

emacs -Q
 
load library cygwin-mount.el, then setup-cygwin.el:
 
http://www.emacswiki.org/emacs/cygwin-mount.el
http://www.emacswiki.org/emacs/setup-cygwin.el
 
M-x grep -nH -e pattern *.el
 
The text matching "pattern" is not highlighted. In Emacs 22 it is
highlighted using face `match' (yellow background).
 

In GNU Emacs 23.0.90.1 (i386-mingw-nt5.1.2600)
 of 2009-02-01 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'
 

bug#2397: 23.0.90; grep no longer highlights the match

[Drew Adams]

The bug seems to have been introduced here:

Revision 1.81 - (view) (download) (annotate) - [select for diffs] 
Fri Nov 23 00:32:05 2007 UTC (15 months ago) by jurta 
Branch: MAIN 
Changes since 1.80: +9 -4 lines 
Diff to previous 1.80 
(grep-process-setup): Set envvar "TERM" to "emacs-grep".
Set envvar "GREP_OPTIONS" to "--color=auto" instead of "--color=always".

That change seems to have removed the pattern highlighting, by changing this:

(defun grep-process-setup ()
  "Setup compilation variables and buffer for `grep'.
Set up `compilation-exit-message-function' and run `grep-setup-hook'."
  (unless (or (not grep-highlight-matches) (eq grep-highlight-matches t))
    (grep-compute-defaults))
  (when (eq grep-highlight-matches t)
    ;; Modify `process-environment' locally bound in `compilation-start'
    (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=always"))
    ;; for GNU grep 2.5.1
    (setenv "GREP_COLOR" "01;31")
    ;; for GNU grep 2.5.1-cvs
    (setenv "GREP_COLORS" "mt=01;31:fn=:ln=:bn=:se=:ml=:cx=:ne"))
  (set (make-local-variable 'compilation-exit-message-function)
       (lambda (status code msg)
	 (if (eq status 'exit)
	     (cond ((zerop code)
		    '("finished (matches found)\n" . "matched"))
		   ((= code 1)
		    '("finished with no matches found\n" . "no match"))
		   (t
		    (cons msg code)))
	   (cons msg code))))
  (run-hooks 'grep-setup-hook))

To this:

(defun grep-process-setup ()
  "Setup compilation variables and buffer for `grep'.
Set up `compilation-exit-message-function' and run `grep-setup-hook'."
  (unless (or (not grep-highlight-matches) (eq grep-highlight-matches t))
    (grep-compute-defaults))
  (when (eq grep-highlight-matches t)
    ;; `setenv' modifies `process-environment' let-bound in `compilation-start'
    ;; Any TERM except "dumb" allows GNU grep to use `--color=auto'
    (setenv "TERM" "emacs-grep")
    ;; `--color=auto' emits escape sequences on a tty rather than on a pipe,
    ;; thus allowing to use multiple grep filters on the command line
    ;; and to output escape sequences only on the final grep output
    (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=auto"))
    ;; GREP_COLOR is used in GNU grep 2.5.1, but deprecated in later versions
    (setenv "GREP_COLOR" "01;31")
    ;; GREP_COLORS is used in GNU grep 2.5.2 and later versions
    (setenv "GREP_COLORS" "mt=01;31:fn=:ln=:bn=:se=:ml=:cx=:ne"))
  (set (make-local-variable 'compilation-exit-message-function)
       (lambda (status code msg)
	 (if (eq status 'exit)
	     (cond ((zerop code)
		    '("finished (matches found)\n" . "matched"))
		   ((= code 1)
		    '("finished with no matches found\n" . "no match"))
		   (t
		    (cons msg code)))
	   (cons msg code))))
  (run-hooks 'grep-setup-hook))

Eval'ing the former is enough to restore the pattern highlighting.

Even more specifically, it is the change from 
(setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=always"))
to
(setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=auto"))
that introduces the bug. Putting back `always' in place of `auto' restores the
highlighting.

Why `auto' doesn't work, I don't know. 

Again, I have Cygwin on MS Windows, and my shell is bash. I don't know how to
determine my Cygwin version, but it seems to date from 2007-08-22. In any case,
Emacs should work with older versions of Cygwin also, and the version I have
does support highlighting, if `--color=always' is used. 

`man grep' shows this for me, which seems to indicate that `auto' is also
supported, but it doesn't say what `auto' means/does(!):

--colour[=WHEN], --color[=WHEN]
 Surround the matching string with the marker find in GREP_COLOR
 environment variable. WHEN may be `never', `always', or `auto'

GREP_OPTIONS
 This variable specifies default options to be placed in front of
 any  explicit  options.   For  example,   if   GREP_OPTIONS   is
 '--binary-files=without-match  --directories=skip', grep behaves
 as if the two options --binary-files=without-match and  --direc-
 tories=skip  had  been  specified  before  any explicit options.
 Option specifications are separated by whitespace.  A  backslash
 escapes  the  next  character,  so  it can be used to specify an
 option containing whitespace or a backslash.

GREP_COLOR
 Specifies the marker for highlighting.

bug#2397: 23.0.90; grep no longer highlights the match

[Juri Linkov]

> Even more specifically, it is the change from
> (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=always"))
> to
> (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=auto"))
> that introduces the bug. Putting back `always' in place of `auto' restores the
> highlighting.
>
> Why `auto' doesn't work, I don't know.

This change was the result of the following discussion:

http://thread.gmane.org/gmane.emacs.bugs/16956
http://thread.gmane.org/gmane.emacs.devel/83316

As you can see grep source code emits highlighting sequences
only when TERM is not "dumb".  So we set it to "emacs-grep".

          if(isatty(STDOUT_FILENO) && getenv("TERM") &&
	     strcmp(getenv("TERM"), "dumb"))
                  color_option = 1;
          else
            color_option = 0;

Could you please post the value of `process-connection-type'.
Also please eval `M-x grep RET set RET' and show the value of
the environment variable `TERM'.

If it is "emacs-grep" then I'm afraid it is the line
"isatty(STDOUT_FILENO)" in grep source code that fails
in your environment.

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#2397: 23.0.90; grep no longer highlights the match

[Drew Adams]

> From: Juri Linkov Sent: Sunday, February 22, 2009 11:02 AM
> > Even more specifically, it is the change from
> > (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " 
> --color=always"))
> > to
> > (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " 
> --color=auto"))
> > that introduces the bug. Putting back `always' in place of 
> `auto' restores the
> > highlighting.
> >
> > Why `auto' doesn't work, I don't know.
> 
> This change was the result of the following discussion:
> 
> http://thread.gmane.org/gmane.emacs.bugs/16956
> http://thread.gmane.org/gmane.emacs.devel/83316
> 
> As you can see grep source code emits highlighting sequences
> only when TERM is not "dumb".  So we set it to "emacs-grep".
> 
>           if(isatty(STDOUT_FILENO) && getenv("TERM") &&
> 	     strcmp(getenv("TERM"), "dumb"))
>                   color_option = 1;
>           else
>             color_option = 0;
> 
> Could you please post the value of `process-connection-type'.

`process-connection-type' has value t.

> Also please eval `M-x grep RET set RET'

Sorry, I don't understand. That does nothing; it just exits with no matches
found - there are no file arguments specified. And in which directory? With
which `grep' switches? I don't follow.

> and show the value of the environment variable `TERM'.

M-: (getenv "TERM) gives "dumb", both in emacs -Q and in my own environment
(i.e., after loading the cywin libraries I mentioned).
 
> If it is "emacs-grep" then I'm afraid it is the line
> "isatty(STDOUT_FILENO)" in grep source code that fails
> in your environment.

It's not. 

And as I mentioned, before your change highlighting works fine.

bug#2397: 23.0.90; grep no longer highlights the match

[Juri Linkov]

>> Also please eval `M-x grep RET set RET'
>
> Sorry, I don't understand. That does nothing; it just exits with no matches
> found - there are no file arguments specified. And in which directory? With
> which `grep' switches? I don't follow.
>
>> and show the value of the environment variable `TERM'.

`M-x grep RET set RET' should print a list of grep environment variables
in a Unix-like shell instead of running a grep command.  But it seems
your shell where grep runs is not bash.  Perhaps this is the cause of
your problems.

> M-: (getenv "TERM) gives "dumb", both in emacs -Q and in my own environment
> (i.e., after loading the cywin libraries I mentioned).

It is normal that (getenv "TERM") gives "dumb".  More important is to see
the value of "TERM" in the grep environment.  Could you try some other
command instead of "grep" to see the value of "TERM" after running the
`grep' command?  For example, `M-x grep RET echo $TERM RET'.

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#2397: 23.0.90; grep no longer highlights the match

[Drew Adams]

> >> Also please eval `M-x grep RET set RET'
> >
> > Sorry, I don't understand. That does nothing; it just exits 
> > with no matches found - there are no file arguments specified.
> > And in which directory? With which `grep' switches? I don't follow.
> >
> >> and show the value of the environment variable `TERM'.
> 
> `M-x grep RET set RET' should print a list of grep environment
> variables in a Unix-like shell instead of running a grep command.

grep -nH -e set
Grep finished with no matches found at Sun Feb 22 14:17:17

(Likewise, without the switches.)

This is in GNU Emacs 23.0.90.1 (i386-mingw-nt5.1.2600)
 of 2009-02-01 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)',
after loading the two files I cited: cygwin-mount.el and setup-cygwin.el.

And `grep --help' does not mention `set'. This is all it says:

------8<-------------------------
Usage: grep [OPTION]... PATTERN [FILE] ...
Search for PATTERN in each FILE or standard input.
Example: grep -i 'hello world' menu.h main.c

Regexp selection and interpretation:
  -E, --extended-regexp     PATTERN is an extended regular expression
  -F, --fixed-strings       PATTERN is a set of newline-separated strings
  -G, --basic-regexp        PATTERN is a basic regular expression
  -P, --perl-regexp         PATTERN is a Perl regular expression
  -e, --regexp=PATTERN      use PATTERN as a regular expression
  -f, --file=FILE           obtain PATTERN from FILE
  -i, --ignore-case         ignore case distinctions
  -w, --word-regexp         force PATTERN to match only whole words
  -x, --line-regexp         force PATTERN to match only whole lines
  -z, --null-data           a data line ends in 0 byte, not newline

Miscellaneous:
  -s, --no-messages         suppress error messages
  -v, --invert-match        select non-matching lines
  -V, --version             print version information and exit
      --help                display this help and exit
      --mmap                use memory-mapped input if possible

Output control:
  -m, --max-count=NUM       stop after NUM matches
  -b, --byte-offset         print the byte offset with output lines
  -n, --line-number         print line number with output lines
      --line-buffered       flush output on every line
  -H, --with-filename       print the filename for each match
  -h, --no-filename         suppress the prefixing filename on output
      --label=LABEL         print LABEL as filename for standard input
  -o, --only-matching       show only the part of a line matching PATTERN
  -q, --quiet, --silent     suppress all normal output
      --binary-files=TYPE   assume that binary files are TYPE
                            TYPE is 'binary', 'text', or 'without-match'
  -a, --text                equivalent to --binary-files=text
  -I                        equivalent to --binary-files=without-match
  -d, --directories=ACTION  how to handle directories
                            ACTION is 'read', 'recurse', or 'skip'
  -D, --devices=ACTION      how to handle devices, FIFOs and sockets
                            ACTION is 'read' or 'skip'
  -R, -r, --recursive       equivalent to --directories=recurse
      --include=PATTERN     files that match PATTERN will be examined
      --exclude=PATTERN     files that match PATTERN will be skipped.
      --exclude-from=FILE   files that match PATTERN in FILE will be skipped.
  -L, --files-without-match only print FILE names containing no match
  -l, --files-with-matches  only print FILE names containing matches
  -c, --count               only print a count of matching lines per FILE
  -Z, --null                print 0 byte after FILE name

Context control:
  -B, --before-context=NUM  print NUM lines of leading context
  -A, --after-context=NUM   print NUM lines of trailing context
  -C, --context=NUM         print NUM lines of output context
  -NUM                      same as --context=NUM
      --color[=WHEN],
      --colour[=WHEN]       use markers to distinguish the matching string
                            WHEN may be `always', `never' or `auto'.
  -U, --binary              do not strip CR characters at EOL (MSDOS)
  -u, --unix-byte-offsets   report offsets as if CRs were not there (MSDOS)

`egrep' means `grep -E'.  `fgrep' means `grep -F'.
With no FILE, or when FILE is -, read standard input.  If less than
two FILEs given, assume -h.  Exit status is 0 if match, 1 if no match,
and 2 if trouble.

Report bugs to <bug-grep@gnu.org>.
------8<-------------------------

> But it seems your shell where grep runs is not bash.  Perhaps
> this is the cause of your problems.

grep -nH -e echo $SHELL
Binary file C:/cygwin/bin/bash.exe matches

Looks like bash to me. And it's always acted like bash, AFAICT.
 
> > M-: (getenv "TERM) gives "dumb", both in emacs -Q and in my 
> > own environment (i.e., after loading the cywin libraries I
> > mentioned).
> 
> It is normal that (getenv "TERM") gives "dumb".  More important
> is to see the value of "TERM" in the grep environment.  Could
> you try some other command instead of "grep" to see the value
> of "TERM" after running the `grep' command?  For example,
> `M-x grep RET echo $TERM RET'.

That gives this:

grep -nH -e echo $TERM
grep: emacs-grep: No such file or directory

bug#2397: 23.0.90; grep no longer highlights the match

[Juri Linkov]

> grep -nH -e echo $SHELL
> Binary file C:/cygwin/bin/bash.exe matches
>
> Looks like bash to me. And it's always acted like bash, AFAICT.

> That gives this:
>
> grep -nH -e echo $TERM
> grep: emacs-grep: No such file or directory

Actually I meant removing the default input (the "grep -nH -e" part)
completely, and replacing it with these commands.  So more precise
recipes are:

`M-x grep RET C-a C-k set RET'
`M-x grep RET C-a C-k echo $TERM RET'

But anyway I see that your $SHELL is "bash.exe" and $TERM is "emacs-grep".

Could you also try running grep from Bash (not from Emacs) with options
"--color=always" and "--color=auto" and see whether it outputs highlighting
sequences for both cases in a standalone shell?

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#2397: 23.0.90; grep no longer highlights the match

[Drew Adams]

> > grep -nH -e echo $SHELL
> > Binary file C:/cygwin/bin/bash.exe matches
> >
> > Looks like bash to me. And it's always acted like bash, AFAICT.
> 
> > That gives this:
> >
> > grep -nH -e echo $TERM
> > grep: emacs-grep: No such file or directory
> 
> Actually I meant removing the default input (the "grep -nH -e" part)
> completely, and replacing it with these commands.  So more precise
> recipes are:
> 
> `M-x grep RET C-a C-k set RET'

Here:

...
SHELL=C:/cygwin/bin/bash.exe
...
TERM=emacs-grep

> `M-x grep RET C-a C-k echo $TERM RET'

echo $TERM
emacs-grep

> But anyway I see that your $SHELL is "bash.exe" and $TERM is 
> "emacs-grep".
> 
> Could you also try running grep from Bash (not from Emacs) 
> with options "--color=always" and "--color=auto" and see whether
> it outputs highlighting sequences for both cases in a standalone
> shell?

Yes, both produce the highlighting.
It is apparently only in Emacs that --color=auto does not work.

bug#2397: 23.0.90; grep no longer highlights the match

[Juri Linkov]

>> `M-x grep RET C-a C-k set RET'
>
> Here:
>
> ...
> SHELL=C:/cygwin/bin/bash.exe
> ...
> TERM=emacs-grep

Thanks.

>> Could you also try running grep from Bash (not from Emacs)
>> with options "--color=always" and "--color=auto" and see whether
>> it outputs highlighting sequences for both cases in a standalone
>> shell?
>
> Yes, both produce the highlighting.
> It is apparently only in Emacs that --color=auto does not work.

Weird.  Maybe a Windows guru could explain the difference.

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#2397: 23.0.90; grep no longer highlights the match

[Drew Adams]

> >> Could you also try running grep from Bash (not from Emacs)
> >> with options "--color=always" and "--color=auto" and see whether
> >> it outputs highlighting sequences for both cases in a standalone
> >> shell?
> >
> > Yes, both produce the highlighting.
> > It is apparently only in Emacs that --color=auto does not work.
> 
> Weird.  Maybe a Windows guru could explain the difference.

I think it's an Emacs guru that's needed (perhaps with some Windows, Cygwin, or
GNU bash knowledge).

bug#2397: marked as done (23.0.90; grep no longer highlights the match)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]


Your message dated Sat, 28 Feb 2009 19:42:33 +0200
with message-id <utz6eo3ue.fsf@gnu.org>
and subject line Re: bug#2397: 23.0.90; grep no longer highlights the match
has caused the Emacs bug report #2397,
regarding 23.0.90; grep no longer highlights the match
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@emacsbugs.donarmstrong.com
immediately.)


-- 
2397: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=2397
Emacs Bug Tracking System
Contact owner@emacsbugs.donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 3197 bytes --]

From: "Drew Adams" <drew.adams@oracle.com>
To: <emacs-pretest-bug@gnu.org>
Subject: 23.0.90; grep no longer highlights the match
Date: Thu, 19 Feb 2009 15:26:41 -0800
Message-ID: <011f01c992e9$854a3cc0$c2b22382@us.oracle.com>

emacs -Q
 
load library cygwin-mount.el, then setup-cygwin.el:
 
http://www.emacswiki.org/emacs/cygwin-mount.el
http://www.emacswiki.org/emacs/setup-cygwin.el
 
M-x grep -nH -e pattern *.el
 
The text matching "pattern" is not highlighted. In Emacs 22 it is
highlighted using face `match' (yellow background).
 

In GNU Emacs 23.0.90.1 (i386-mingw-nt5.1.2600)
 of 2009-02-01 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'
 




[-- Attachment #3: Type: message/rfc822, Size: 4614 bytes --]

From: Eli Zaretskii <eliz@gnu.org>
To: Drew Adams <drew.adams@oracle.com>, 2397-done@emacsbugs.donarmstrong.com
Cc: juri@jurta.org
Subject: Re: bug#2397: 23.0.90; grep no longer highlights the match
Date: Sat, 28 Feb 2009 19:42:33 +0200
Message-ID: <utz6eo3ue.fsf@gnu.org>

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Mon, 23 Feb 2009 16:56:14 -0800
> Cc: 2397@emacsbugs.donarmstrong.com
> 
> > >> Could you also try running grep from Bash (not from Emacs)
> > >> with options "--color=always" and "--color=auto" and see whether
> > >> it outputs highlighting sequences for both cases in a standalone
> > >> shell?
> > >
> > > Yes, both produce the highlighting.
> > > It is apparently only in Emacs that --color=auto does not work.
> > 
> > Weird.  Maybe a Windows guru could explain the difference.
> 
> I think it's an Emacs guru that's needed (perhaps with some Windows, Cygwin, or
> GNU bash knowledge).

I think this has nothing to do with either Cygwin or Bash.  Drew,
could you please verify that the same problem happens for you in
"emacs -Q" without loading cygwin-mount.el and setup-cygwin.el?

AFAICT, this problem happens because Emacs on Windows invokes
subsidiary programs through a pipe, and that pipe fails the `isatty'
test in Grep.  So "--color=auto" can never work on MS-Windows when
Grep is invoked by Emacs.

I ``fixed'' this by going back, on DOS/Windows only, to the
"--color=always" way we used before Juri installed his 2007-11-23
changes.  It's true that this will reinstate the original problem with
multiple grep invocations in a pipe (on Windows and DOS only), but I
don't see how can that use-case be solved, and having a single
instance of grep in the command is by far more frequent usage.

Here's the change I installed.  Drew, could you please see if it works
for you as well?

2009-02-28  Eli Zaretskii  <eliz@gnu.org>

	* progmodes/grep.el (grep-process-setup) [windows-nt msdos]: Use
	"--color=always".

Index: lisp/progmodes/grep.el
===================================================================
RCS file: /cvsroot/emacs/emacs/lisp/progmodes/grep.el,v
retrieving revision 1.99
retrieving revision 1.100
diff -u -r1.99 -r1.100
--- lisp/progmodes/grep.el	25 Jan 2009 00:54:53 -0000	1.99
+++ lisp/progmodes/grep.el	28 Feb 2009 17:24:29 -0000	1.100
@@ -407,7 +407,11 @@
     ;; `--color=auto' emits escape sequences on a tty rather than on a pipe,
     ;; thus allowing to use multiple grep filters on the command line
     ;; and to output escape sequences only on the final grep output
-    (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=auto"))
+    (setenv "GREP_OPTIONS"
+	    (concat (getenv "GREP_OPTIONS")
+		    ;; Windows and DOS pipes fail `isatty' detection in Grep.
+		    " --color=" (if (memq system-type '(windows-nt ms-dos))
+				    "always" "auto")))
     ;; GREP_COLOR is used in GNU grep 2.5.1, but deprecated in later versions
     (setenv "GREP_COLOR" "01;31")
     ;; GREP_COLORS is used in GNU grep 2.5.2 and later versions

bug#2397: 23.0.90; grep no longer highlights the match

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <2397-done@emacsbugs.donarmstrong.com>, <juri@jurta.org>
> Date: Sat, 28 Feb 2009 11:26:32 -0800
> 
> When I try `M-! c:/cygwin/bin/grep.exe -n buff-menu *.el', I get this error:
> shell-command-on-region: Searching for program: no such file or directory,
> /bin/bash. So is the problem is that my env var SHELL is bash, and it doesn't
> find bash? (getenv "SHELL") shows "/bin/bash". I tried (setenv "SHELL"
> "c:\cygwin\bin\bash.exe"), but it didn't help - same error messages.
> 
> What's the right way to test this?

Try unsetting SHELL in the environment, I'd guess.

bug#2397: 23.0.90; grep no longer highlights the match

[Drew Adams]

> Try unsetting SHELL in the environment, I'd guess.

OK, yes, I get the same result in emacs -Q: no highlighting of matches, and it's
fixed if I use your patch. Thx - Drew.

bug#2397: 23.0.90; grep no longer highlights the match

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <2397@emacsbugs.donarmstrong.com>, <juri@jurta.org>
> Date: Sat, 28 Feb 2009 13:09:28 -0800
> 
> > Try unsetting SHELL in the environment, I'd guess.
> 
> OK, yes, I get the same result in emacs -Q: no highlighting of matches, and it's
> fixed if I use your patch. Thx - Drew.

Thanks again for testing this.