[jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Richard Stallman]

He is arguing that setting GREP_COLOR is a mistake
and it would be better to put --color into the grep command.

What do other people think?

------- Start of forwarded message -------
X-Spam-Status: No, score=0.6 required=5.0 tests=NO_REAL_NAME autolearn=no 
	version=3.1.0
To: rms@gnu.org
Cc: bug-gnu-emacs@gnu.org
Subject: Re: find-grep makes raw terminal ANSI
From: jidanni@jidanni.org
Date: Wed, 14 Nov 2007 12:04:20 +0800
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

>>>>> "r" == Richard Stallman <rms@gnu.org> writes:

r> I am lost.

Allow us to take the case of
(find-grep "find /etc/passwd -type f -print0 | xargs -0 -e grep -nH -e roo")
which works fine unless one then smacks at the end an additional filter:
(find-grep "find /etc/passwd -type f -print0 | xargs -0 -e grep -nH -e roo|grep :root")
Grep finished with no matches found at Wed Nov 14 12:03:40
whereas the same command in the shell finds something:
$ find /etc/passwd -type f -print0 | xargs -0 -e grep -nH -e roo|grep :root
/etc/passwd:1:root:x:0:0:root:/root:/bin/bash

Thus the user walks away thinking there was nothing to be found, when
all along what is happening is that ANSI terminal control characters
are being injected, as can be seen with
(find-grep "find /etc/passwd -type f -print0 | xargs -0 -e grep -nH -e roo|cat -v")

Thus the user is mislead thinking he can merely extend the UNIX filter
by another pipe, but instead unwittingly throwing away valid matches.

ANSI terminal codes have no business being injected at this layer of
the program.

Wait, I see what is going on,
12 matches for "highlight" in buffer: grep.el.gz
7 matches for "color" in buffer: grep.el.gz,
(including "The deprecated environment variable GREP_COLOR." as noted
on grep(1).)

It would be much better to put --color right there in the command line
where the user can be aware of what is going on. But that line is
already crowded.

Anyway, just wanted to let you know "I told the boss that we have no
record of that client, when all along that color jazz
undermined my extra search filter. How was I to know?"
------- End of forwarded message -------

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Juri Linkov]

> He is arguing that setting GREP_COLOR is a mistake

GREP_COLOR is for compatibility with older grep versions.
grep.el also uses GREP_COLORS for newer versions.
So everything in this regard is already correct in grep.el.

> and it would be better to put --color into the grep command.

Currently grep.el adds " --color=always" to the environment variable
GREP_OPTIONS instead of putting this argument into the grep command.

The disadvantage of putting " --color=always" into the grep command is
that it makes the grep command line too long to edit in the minibuffer
before running.  The advantage of putting it into the command line is
that it will allow the user to be aware what is going on and to write
the correct grep command line.  For instance, with using explicit --color
argument the correct command line in the OP's example perhaps would be

(find-grep "find /etc/passwd -type f -print0 | xargs -0 -e \
grep --color=none -nH -e r|\
grep --color=always :root")

But without putting the --color argument to the minibuffer initially
it is not evident that the user can use this argument to control
highlighting.  So the presence of --color in the command line will
provide a hint for the user how to control the grep highlighting.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Stefan Monnier]

> Currently grep.el adds " --color=always" to the environment variable
> GREP_OPTIONS instead of putting this argument into the grep command.

How 'bout using --color=auto (and then setup TERM appropriately and run
the grep(&find) process in a tty rather than a pipe)?


        Stefan

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Juri Linkov]

>> Currently grep.el adds " --color=always" to the environment variable
>> GREP_OPTIONS instead of putting this argument into the grep command.
>
> How 'bout using --color=auto (and then setup TERM appropriately and run
> the grep(&find) process in a tty rather than a pipe)?

Do you mean a patch like below?  It works because grep produces escape
sequences when stdout is a tty that is not dumb.  Here is relevant part
of grep source code for --color=auto:

          if(isatty(STDOUT_FILENO) && getenv("TERM") &&
	     strcmp(getenv("TERM"), "dumb"))
                  color_option = 1;
          else
            color_option = 0;

and `process-connection-type' is t by default.

Index: lisp/progmodes/grep.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/progmodes/grep.el,v
retrieving revision 1.79
diff -c -r1.79 grep.el
*** lisp/progmodes/grep.el	30 Aug 2007 09:36:59 -0000	1.79
--- lisp/progmodes/grep.el	17 Nov 2007 15:32:27 -0000
***************
*** 366,372 ****
      (grep-compute-defaults))
    (when (eq grep-highlight-matches t)
      ;; Modify `process-environment' locally bound in `compilation-start'
!     (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=always"))
      ;; for GNU grep 2.5.1
      (setenv "GREP_COLOR" "01;31")
      ;; for GNU grep 2.5.1-cvs
--- 366,373 ----
      (grep-compute-defaults))
    (when (eq grep-highlight-matches t)
      ;; Modify `process-environment' locally bound in `compilation-start'
!     (setenv "TERM" "emacs")
!     (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=auto"))
      ;; for GNU grep 2.5.1
      (setenv "GREP_COLOR" "01;31")
      ;; for GNU grep 2.5.1-cvs

-- 
Juri Linkov
http://www.jurta.org/emacs/

RE: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Drew Adams]

> !     (setenv "TERM" "emacs")

I apologize if this reply is not very meaningful, as I haven't followed
this.

But is it good to change a user's environment variable this way? On MS
Windows, I see that my TERM is "cmd". I have no idea if this change will
break something, but I wonder.

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 17 Nov 2007 09:28:25 -0800
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> > !     (setenv "TERM" "emacs")
> 
> I apologize if this reply is not very meaningful, as I haven't followed
> this.
> 
> But is it good to change a user's environment variable this way?

setenv doesn't change user's environment, it changes the environment
passed to inferior processes.

RE: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Drew Adams]

> > > !     (setenv "TERM" "emacs")
> > 
> > But is it good to change a user's environment variable this way?
> 
> setenv doesn't change user's environment, it changes the environment
> passed to inferior processes.

Right; forgot. Thx.

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Drew Adams" <drew.adams@oracle.com>
>> Date: Sat, 17 Nov 2007 09:28:25 -0800
>> Cc: rms@gnu.org, emacs-devel@gnu.org
>> 
>> > !     (setenv "TERM" "emacs")
>> 
>> I apologize if this reply is not very meaningful, as I haven't followed
>> this.
>> 
>> But is it good to change a user's environment variable this way?
>
> setenv doesn't change user's environment, it changes the environment
> passed to inferior processes.

My first reaction to this was "Huh?".  In UNIX, where the environment
concept originated, it is _exclusively_ something which is passed to
inferior processes, quite like command line arguments.

It is only when coming from an MSDOS background (which tended to
implement UNIX concepts badly or incompletely, and so had something like
a global environment, and pipes implemented with intermediate files of
arbitrary size and other things close, but not identical to the real
thing) that something like a global "user's environment" concept does
even exist.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: "Drew Adams" <drew.adams@oracle.com>,  emacs-devel@gnu.org
> Date: Sun, 18 Nov 2007 01:29:03 +0100
> 
> > setenv doesn't change user's environment, it changes the environment
> > passed to inferior processes.
> 
> My first reaction to this was "Huh?".

You misunderstood what Drew was asking, and thus my response.  He was
asking about the environment used by Emacs itself.  In general,
`setenv' changes the environment of the program that calls it, but in
the case in point it isn't so.

> In UNIX, where the environment
> concept originated, it is _exclusively_ something which is passed to
> inferior processes, quite like command line arguments.
> 
> It is only when coming from an MSDOS background (which tended to
> implement UNIX concepts badly or incompletely, and so had something like
> a global environment, and pipes implemented with intermediate files of
> arbitrary size and other things close, but not identical to the real
> thing) that something like a global "user's environment" concept does
> even exist.

This rethoric is misplaced: there's no difference between Unix and
MS-DOS in this regard: both have separate environment for each
process, and on both systems a program can only legitimately change
its own environment, which is then copied to the children processes.
It cannot change the environment of the parent process (except by
writing to its memory, which I don't consider a legitimate behavior).
Unlike what you seem to think, there's no "global" environment on
MS-DOS.

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Juri Linkov]

>> !     (setenv "TERM" "emacs")
>
> I apologize if this reply is not very meaningful, as I haven't followed
> this.
>
> But is it good to change a user's environment variable this way? On MS
> Windows, I see that my TERM is "cmd". I have no idea if this change will
> break something, but I wonder.

Please see the comment just above the line you quoted from the previous patch:

      ;; Modify `process-environment' locally bound in `compilation-start'
!     (setenv "TERM" "emacs")

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Stefan Monnier]

> !     (setenv "TERM" "emacs")
> !     (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=auto"))

Looks about right, except that I believe `emacs' is too vague.
I'd use something like "emacs-grep".  We could then even make up
a terminfo file for it that describes its (very limited) capabilities.


        Stefan

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> !     (setenv "TERM" "emacs")
>> !     (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=auto"))
>
> Looks about right, except that I believe `emacs' is too vague.
> I'd use something like "emacs-grep".

See comint-exec-1. "emacs" is the traditional value used for TERM on
termcap systems, where you can define an arbitrary terminal in the
environment.  This doesn't work on terminfo systems, so Emacs uses
"dumb" instead.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Juri Linkov]

>> !     (setenv "TERM" "emacs")
>> !     (setenv "GREP_OPTIONS" (concat (getenv "GREP_OPTIONS") " --color=auto"))
>
> Looks about right, except that I believe `emacs' is too vague.
> I'd use something like "emacs-grep".

I'm OK with any name except "dumb", but as Andreas noted, "emacs" is
a traditional name to inform the subprocess that it is called by Emacs.

> We could then even make up a terminfo file for it that describes its
> (very limited) capabilities.

For grep there is no need to make up a terminfo entry because grep
doesn't detect terminfo capabilities.  From comments in grep source code:

      It would be impractical for GNU grep to become a full-fledged
      terminal program linked against ncurses or the like, so it will
      not detect terminfo(5) capabilities.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Stefan Monnier]

> I'm OK with any name except "dumb", but as Andreas noted, "emacs" is
> a traditional name to inform the subprocess that it is called by Emacs.

Depending on the precise package by which it is executed the terminal
emulated by Emacs is different.  So just saying "emacs" is not
good enough.  When no terminal-processing is performed (the default), it
should just say "dumb" which is indeed the default.


        Stefan

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Richard Stallman]

    Looks about right, except that I believe `emacs' is too vague.
    I'd use something like "emacs-grep".  We could then even make up
    a terminfo file for it that describes its (very limited) capabilities.

That would be a lot more work.  If `emacs' is not incorrect,
why not use it?

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Richard Stallman]

    > He is arguing that setting GREP_COLOR is a mistake

    GREP_COLOR is for compatibility with older grep versions.
    grep.el also uses GREP_COLORS for newer versions.
    So everything in this regard is already correct in grep.el.

I think we are miscommunicating.  The problem he's talking about
is that this affects grep even if grep is piped to another command,
and that gives bad results (sometimes).

I expect that the problem is the same regardless of precisely which
envvar is used.  Is that not so?

    The disadvantage of putting " --color=always" into the grep command is
    that it makes the grep command line too long to edit in the minibuffer
    before running.  The advantage of putting it into the command line is
    that it will allow the user to be aware what is going on and to write
    the correct grep command line.

That's a good summary of the issue.

I don't have much feel for what is best here.

Re: [jidanni@jidanni.org: Re: find-grep makes raw terminal ANSI]

[Juri Linkov]

>     > He is arguing that setting GREP_COLOR is a mistake
>
>     GREP_COLOR is for compatibility with older grep versions.
>     grep.el also uses GREP_COLORS for newer versions.
>     So everything in this regard is already correct in grep.el.
>
> I think we are miscommunicating.  The problem he's talking about
> is that this affects grep even if grep is piped to another command,
> and that gives bad results (sometimes).
>
> I expect that the problem is the same regardless of precisely which
> envvar is used.  Is that not so?

Yes, the problem is the same for the deprecated envvar GREP_COLOR and
a newer envvar GREP_COLORS.  I understood that part of the complaint was
using the deprecated envvar GREP_COLOR instead of newer GREP_COLORS.
Since Emacs sets both there is no problem for supporting old and new
versions of GNU grep.

As for the main problem, Stefan suggested a workable solution.

-- 
Juri Linkov
http://www.jurta.org/emacs/