bug#4982: rgrep can't deal with .gz files

bug#4982: rgrep can't deal with .gz files

[jidanni]

rgrep should be enhanced so if it encounters .gz files it knows what to do.

bug#4982: rgrep can't deal with .gz files

[Juri Linkov]

> rgrep should be enhanced so if it encounters .gz files
> it knows what to do.

This would be tricky to implement.  When searching in .gz files
we have to rely on zgrep (that has its own problems - it doesn't
highlight grep matches).

I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
and with UI like `rgrep'.

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#4982: rgrep can't deal with .gz files

[Thierry Volpiatto]

Juri Linkov <juri@jurta.org> writes:

>> rgrep should be enhanced so if it encounters .gz files
>> it knows what to do.
>
> This would be tricky to implement.  When searching in .gz files
> we have to rely on zgrep (that has its own problems - it doesn't
> highlight grep matches).

traverselisp.el handle compressed archives using avfs.
http://www.emacswiki.org/cgi-bin/emacs/traverselisp.el
You can launch traverse-deep-rfind (similar to rgrep) on any archive.(tar)gz.

You could use also avfs to handle .gz files with rgrep.
http://sourceforge.net/projects/avf/
 
> I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
> and with UI like `rgrep'.

-- 
A + Thierry Volpiatto
Location: Saint-Cyr-Sur-Mer - France

bug#4982: rgrep can't deal with .gz files

[Kevin Rodgers]

Juri Linkov wrote:
> I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
> and with UI like `rgrep'.

Clearly, it should be called `rzgrep' :-)

-- 
Kevin Rodgers
Denver, Colorado, USA

bug#4982: rgrep can't deal with .gz files

[Juri Linkov]

> > I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
> > and with UI like `rgrep'.
>
> Clearly, it should be called `rzgrep' :-)

See in the zgrep script:

  grep='${GREP-grep}'
  version='z$grep (gzip) 1.3.12'

So according to the naming convention of zgrep, it prepends the "z" prefix
to any grep command name :-)

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#4982: rgrep can't deal with .gz files

[jidanni]

>>>>> "JL" == Juri Linkov <juri@jurta.org> writes:
>> rgrep should be enhanced so if it encounters .gz files
>> it knows what to do.

JL> This would be tricky to implement.  When searching in .gz files
JL> we have to rely on zgrep

Ah, of course... OK, now in .emacs I'll just use
(setq grep-program "zgrep")
as a workaround, as zgrep also works fine on non compressed files.

JL>(that has its own problems - it doesn't highlight grep matches).

Seems to work for me.

JL> I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
JL> and with UI like `rgrep'.

bug#4982: rgrep can't deal with .gz files

[Juri Linkov]

> JL>(that has its own problems - it doesn't highlight grep matches).
>
> Seems to work for me.

Hmm, strange.  In http://thread.gmane.org/gmane.emacs.bugs/16956
you requested us to remove highlighting when there are additional
filters on the grep output.  So we changed `--color=always' into
`--color=auto' that broke the highlighting in zgrep output because
zgrep uses additional filters.

But now you say that highlighting in zgrep output works for you.
I wonder how this is possible?

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#4982: rgrep can't deal with .gz files

[jidanni]

>>>>> "JL" == Juri Linkov <juri@jurta.org> writes:
JL> But now you say that highlighting in zgrep output works for you.
JL> I wonder how this is possible?
All I know is after I changed grep-command to zgrep, when I run
(rgrep "JL" "* .*" "~" nil)
it uses
find . \( -path \*/SCCS ... xargs -0 -e zgrep  -nH -e JL
and the front of this line is blue:
./KIDPIX/n_PICT4993.JPG:Binary file (standard input) matches
and this is green:
./.ssh/known_hosts:14:|1|HHSxNOckqq
Note for me
$ env|grep -ci grep
0
and I use the latest debian emacs-snapshot.

bug#4982: rgrep can't deal with .gz files

[Juri Linkov]

> JL> But now you say that highlighting in zgrep output works for you.
> JL> I wonder how this is possible?
>
> All I know is after I changed grep-command to zgrep, when I run
> (rgrep "JL" "* .*" "~" nil)
> it uses
> find . \( -path \*/SCCS ... xargs -0 -e zgrep  -nH -e JL
> and the front of this line is blue:
> ./KIDPIX/n_PICT4993.JPG:Binary file (standard input) matches
> and this is green:
> ./.ssh/known_hosts:14:|1|HHSxNOckqq

So you don't see yellow matches?

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#4982: rgrep can't deal with .gz files

[jidanni]

>>>>> "JL" == Juri Linkov <juri@jurta.org> writes:
JL> So you don't see yellow matches?
OK, I do if I mouseover them. By the way, some ugliness now that I set
grep-command to zgrep:

gzip: stdout: Broken pipe
./libchewing3-data_0.3.0-2_i386.deb:Binary file (standard input) matches

gzip: stdout: Broken pipe
./Nokia_3315_APAC_UG_En.pdf:Binary file (standard input) matches

bug#4982: rgrep can't deal with .gz files

[Juri Linkov]

> JL> So you don't see yellow matches?
> OK, I do if I mouseover them. By the way, some ugliness now that I set
> grep-command to zgrep:
>
> gzip: stdout: Broken pipe
> ./libchewing3-data_0.3.0-2_i386.deb:Binary file (standard input) matches
>
> gzip: stdout: Broken pipe
> ./Nokia_3315_APAC_UG_En.pdf:Binary file (standard input) matches

Yes, zgrep incorrectly processes non-gzipped files.  To search
recursively in gzipped files, I created a new command `zrgrep'.
It works like `rgrep' but uses `zgrep' for `grep-program', sets the default
file name to `*.gz', and sets `grep-highlight-matches' to `always'."

Other changes in the patch below include:

1. grep-highlight-matches has two new options `always' and `auto'.
This allows `zrgrep' to override the default value with `always'.

2. Move Windows/DOS specific --colors settings handling from
`grep-process-setup' to `grep-compute-defaults'.

Index: lisp/progmodes/grep.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/progmodes/grep.el,v
retrieving revision 1.111
diff -c -r1.111 grep.el
*** lisp/progmodes/grep.el	19 Nov 2009 17:37:25 -0000	1.111
--- lisp/progmodes/grep.el	24 Nov 2009 17:04:11 -0000
***************
*** 69,90 ****
    :group 'grep)
  
  (defcustom grep-highlight-matches 'auto-detect
!   "If t, use special markers to highlight grep matches.
  
  Some grep programs are able to surround matches with special
  markers in grep output.  Such markers can be used to highlight
  matches in grep mode.
  
! This option sets the environment variable GREP_COLOR to specify
  markers for highlighting and GREP_OPTIONS to add the --color
  option in front of any explicit grep options before starting
  the grep.
  
  In interactive usage, the actual value of this variable is set up
! by `grep-compute-defaults'; to change the default value, use
! Customize or call the function `grep-apply-setting'."
    :type '(choice (const :tag "Do not highlight matches with grep markers" nil)
  		 (const :tag "Highlight matches with grep markers" t)
  		 (other :tag "Not Set" auto-detect))
    :set 'grep-apply-setting
    :version "22.1"
--- 69,102 ----
    :group 'grep)
  
  (defcustom grep-highlight-matches 'auto-detect
!   "Use special markers to highlight grep matches.
  
  Some grep programs are able to surround matches with special
  markers in grep output.  Such markers can be used to highlight
  matches in grep mode.
  
! This option sets the environment variable GREP_COLORS to specify
  markers for highlighting and GREP_OPTIONS to add the --color
  option in front of any explicit grep options before starting
  the grep.
  
+ When this option is `auto', grep uses `--color=auto' to highlight
+ matches only when it outputs to a terminal (when `grep' is the last
+ command in the pipe), thus avoiding the use of any potentially-harmful
+ escape sequences when standard output goes to a file or pipe.
+ 
+ To make grep highlight matches even into a pipe, you need the option
+ `always' that forces grep to use `--color=always' to unconditionally
+ output escape sequences.
+ 
  In interactive usage, the actual value of this variable is set up
! by `grep-compute-defaults' when the default value is `auto-detect'.
! To change the default value, use Customize or call the function
! `grep-apply-setting'."
    :type '(choice (const :tag "Do not highlight matches with grep markers" nil)
  		 (const :tag "Highlight matches with grep markers" t)
+ 		 (const :tag "Use --color=always" always)
+ 		 (const :tag "Use --color=auto" auto)
  		 (other :tag "Not Set" auto-detect))
    :set 'grep-apply-setting
    :version "22.1"
***************
*** 442,460 ****
  (defun grep-process-setup ()
    "Setup compilation variables and buffer for `grep'.
  Set up `compilation-exit-message-function' and run `grep-setup-hook'."
!   (unless (or (not grep-highlight-matches) (eq grep-highlight-matches t))
      (grep-compute-defaults))
!   (when (eq grep-highlight-matches t)
      ;; `setenv' modifies `process-environment' let-bound in `compilation-start'
      ;; Any TERM except "dumb" allows GNU grep to use `--color=auto'
      (setenv "TERM" "emacs-grep")
-     ;; `--color=auto' emits escape sequences on a tty rather than on a pipe,
-     ;; thus allowing to use multiple grep filters on the command line
-     ;; and to output escape sequences only on the final grep output
      (setenv "GREP_OPTIONS"
  	    (concat (getenv "GREP_OPTIONS")
! 		    ;; Windows and DOS pipes fail `isatty' detection in Grep.
! 		    " --color=" (if (memq system-type '(windows-nt ms-dos))
  				    "always" "auto")))
      ;; GREP_COLOR is used in GNU grep 2.5.1, but deprecated in later versions
      (setenv "GREP_COLOR" "01;31")
--- 454,469 ----
  (defun grep-process-setup ()
    "Setup compilation variables and buffer for `grep'.
  Set up `compilation-exit-message-function' and run `grep-setup-hook'."
!   (when (eq grep-highlight-matches 'auto-detect)
      (grep-compute-defaults))
!   (unless (or (eq grep-highlight-matches 'auto-detect)
! 	      (null grep-highlight-matches))
      ;; `setenv' modifies `process-environment' let-bound in `compilation-start'
      ;; Any TERM except "dumb" allows GNU grep to use `--color=auto'
      (setenv "TERM" "emacs-grep")
      (setenv "GREP_OPTIONS"
  	    (concat (getenv "GREP_OPTIONS")
! 		    " --color=" (if (eq grep-highlight-matches 'always)
  				    "always" "auto")))
      ;; GREP_COLOR is used in GNU grep 2.5.1, but deprecated in later versions
      (setenv "GREP_COLOR" "01;31")
***************
*** 579,592 ****
  			(t
  			 (format "%s . <X> -type f <F> -print | %s %s"
  				 find-program xargs-program gcmd))))))))
!     (unless (or (not grep-highlight-matches) (eq grep-highlight-matches t))
        (setq grep-highlight-matches
  	    (with-temp-buffer
  	      (and (grep-probe grep-program '(nil t nil "--help"))
  		   (progn
  		     (goto-char (point-min))
  		     (search-forward "--color" nil t))
! 		   t))))
  
      ;; Save defaults for this host.
      (setq grep-host-defaults-alist
--- 588,603 ----
  			(t
  			 (format "%s . <X> -type f <F> -print | %s %s"
  				 find-program xargs-program gcmd))))))))
!     (when (eq grep-highlight-matches 'auto-detect)
        (setq grep-highlight-matches
  	    (with-temp-buffer
  	      (and (grep-probe grep-program '(nil t nil "--help"))
  		   (progn
  		     (goto-char (point-min))
  		     (search-forward "--color" nil t))
! 		   ;; Windows and DOS pipes fail `isatty' detection in Grep.
! 		   (if (memq system-type '(windows-nt ms-dos))
! 		       'always 'auto)))))
  
      ;; Save defaults for this host.
      (setq grep-host-defaults-alist
***************
*** 978,983 ****
--- 993,1030 ----
  	  (if (eq next-error-last-buffer (current-buffer))
  	      (setq default-directory dir)))))))
  
+ ;;;###autoload
+ (defun zrgrep (regexp &optional files dir confirm grep-find-template)
+   "Recursively grep for REGEXP in gzipped FILES in tree rooted at DIR.
+ Like `rgrep' but uses `zgrep' for `grep-program', sets the default
+ file name to `*.gz', and sets `grep-highlight-matches' to `always'."
+   (interactive
+    (let ((grep-program "zgrep")
+ 	 (grep-find-command nil)
+ 	 (grep-find-template nil)
+ 	 (grep-host-defaults-alist nil)
+ 	 (grep-files-aliases '(("*.gz" . "*.gz") ; for `grep-read-files'
+ 			       ("all" . "* .*"))))
+      ;; Recompute defaults using let-bound values above.
+      (grep-compute-defaults)
+      (cond
+       ((and grep-find-command (equal current-prefix-arg '(16)))
+        (list (read-from-minibuffer "Run: " grep-find-command
+ 				   nil nil 'grep-find-history)))
+       ((not grep-find-template)
+        (error "grep.el: No `grep-find-template' available"))
+       (t (let* ((regexp (grep-read-regexp))
+ 		(files (grep-read-files regexp))
+ 		(dir (read-directory-name "Base directory: "
+ 					  nil default-directory t))
+ 		(confirm (equal current-prefix-arg '(4))))
+ 	   (list regexp files dir confirm grep-find-template))))))
+   ;; Set `grep-highlight-matches' to `always'
+   ;; since `zgrep' puts filters in the grep output.
+   (let ((grep-highlight-matches 'always))
+     ;; `rgrep' uses the dynamically bound value `grep-find-template'
+     ;; from the argument `grep-find-template' whose value is computed
+     ;; in the `interactive' spec.
+     (rgrep regexp files dir confirm)))
  
;;;###autoload
(defalias 'rzgrep 'zrgrep)
  
  (provide 'grep)
  
-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#4982: rgrep can't deal with .gz files

[Stefan Monnier]

> Hmm, strange.  In http://thread.gmane.org/gmane.emacs.bugs/16956
> you requested us to remove highlighting when there are additional
> filters on the grep output.  So we changed `--color=always' into
> `--color=auto' that broke the highlighting in zgrep output because
> zgrep uses additional filters.

Maybe zgrep is careful to distinguish filters on the input from filters
on the output.  AFAICT zgrep only uses a filter on the input, so it
should work.


        Stefan

bug#4982: rgrep can't deal with .gz files

[Juri Linkov]

> > Hmm, strange.  In http://thread.gmane.org/gmane.emacs.bugs/16956
> > you requested us to remove highlighting when there are additional
> > filters on the grep output.  So we changed `--color=always' into
> > `--color=auto' that broke the highlighting in zgrep output because
> > zgrep uses additional filters.

> Maybe zgrep is careful to distinguish filters on the input from
> filters on the output.  AFAICT zgrep only uses a filter on the input,
> so it should work.

IIUC, zgrep puts a sed filter on the grep output to replace
"(standard input)" strings with real file names.

-- 
Juri Linkov
http://www.jurta.org/emacs/

bug#4982: marked as done (rgrep can't deal with .gz files)

[Emacs bug Tracking System]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

Your message dated Wed, 25 Nov 2009 19:28:14 +0200
with message-id <87y6lu7gq6.fsf@mail.jurta.org>
and subject line Re: bug#4982: rgrep can't deal with .gz files
has caused the Emacs bug report #4982,
regarding rgrep can't deal with .gz files
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@emacsbugs.donarmstrong.com
immediately.)


-- 
4982: http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=4982
Emacs Bug Tracking System
Contact owner@emacsbugs.donarmstrong.com with problems

[-- Attachment #2: Type: message/rfc822, Size: 2117 bytes --]

From: jidanni@jidanni.org
To: bug-gnu-emacs@gnu.org
Subject: rgrep can't deal with .gz files
Date: Fri, 20 Nov 2009 09:42:45 +0800
Message-ID: <87ocmyx97u.fsf@jidanni.org>

rgrep should be enhanced so if it encounters .gz files it knows what to do.



[-- Attachment #3: Type: message/rfc822, Size: 2151 bytes --]

From: Juri Linkov <juri@jurta.org>
To: 4982-done@emacsbugs.donarmstrong.com
Cc: jidanni@jidanni.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Wed, 25 Nov 2009 19:28:14 +0200
Message-ID: <87y6lu7gq6.fsf@mail.jurta.org>

>> By the way, some ugliness now that I set grep-command to zgrep:
>>
>> gzip: stdout: Broken pipe
>> ./libchewing3-data_0.3.0-2_i386.deb:Binary file (standard input) matches
>>
>> gzip: stdout: Broken pipe
>> ./Nokia_3315_APAC_UG_En.pdf:Binary file (standard input) matches
>
> Yes, zgrep incorrectly processes non-gzipped files.  To search
> recursively in gzipped files, I created a new command `zrgrep'.
> It works like `rgrep' but uses `zgrep' for `grep-program', sets the default
> file name to `*.gz', and sets `grep-highlight-matches' to `always'."

I installed a new command `zrgrep' to search recursively in gzipped files.

-- 
Juri Linkov
http://www.jurta.org/emacs/