bug#27873: 26.0.50; M-x grep broken

bug#27873: 26.0.50; M-x grep broken

[Eric Hanchrow]

[-- Attachment #1: Type: text/plain, Size: 5037 bytes --]

I ran "grep", searching for the word "alpha" in all files in my home
directory.  I happen to know that that word appears exactly once in
those files.

     M-x [execute-extended-command]
     g [self-insert-command]
     r [self-insert-command]
     e [self-insert-command]
     p [self-insert-command]
     <return> [minibuffer-complete-and-exit]
     a [self-insert-command]
     l [self-insert-command]
     p [self-insert-command]
     h [self-insert-command]
     a [self-insert-command]
     SPC [self-insert-command]
     * [self-insert-command]
     <return> [exit-minibuffer]

This displayed a *grep* buffer that looked something like I expected,
but:

* instead of there being exactly one line with some underlining
  (indicating a "hit"), there were a bunch: the one I expected, as well
  as a few before it like this:

    grep: git-repositories: Is a directory
    grep: guix: Is a directory
    grep: homedir: Is a directory
    grep: homework: Is a directory
    grep: iTunesDSM: Is a directory
    grep: jessie64: Is a directory
    grep: local: Is a directory
    grep: log: Is a directory
    grep: mygo: Is a directory
    grep: node_modules: Is a directory
    grep: perl5: Is a directory
    phonetic-alphabet.txt1:Stolen from
http://www.fourmilab.ch/documents/phoneticalphabet/
    grep: pprof: Is a directory

  You can't tell from what I've pasted above, but the 9 "Is a directory"
  lines before the actual hit were red and underlined, just like the
  match was.

* Hitting C-x `, instead of visiting the file with the hit, put a
  nine-line-high prompt in the minibuffer, as if it was asking me which
  directory to find the file in a directory with a really weird name.
  (You can see some evedince of this in the "Recent messages" stuff below).



In GNU Emacs 26.0.50 (build 1, x86_64-apple-darwin16.7.0, NS appkit-1504.83
Version 10.12.6 (Build 16G29))
 of 2017-07-29 built on Eric-Hanchrows-MacBook-Pro.local
Repository revision: d7825cb09eae438a83ed2f5b3e0715523d4ed5b7
Windowing system distributor 'Apple', version 10.3.1504
Recent messages:
grep: jessie64: Is a directory
grep: local: Is a directory
grep: log: Is a directory
grep: mygo: Is a directory
grep: node_modules: Is a directory
grep: perl5: Is a directory
phonetic-alphabet.txt’
completing-read-default: Command attempted to use minibuffer while in
minibuffer
QuitInvalid face reference: unspecified
Invalid face reference: unspecified [5 times]

Configured features:
JPEG RSVG NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message subr-x puny seq dired
dired-loaddefs rfc822 mml easymenu mml-sec epa derived epg epg-config
gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils tramp tramp-compat
tramp-loaddefs trampver parse-time format-spec advice auth-source cl-seq
eieio byte-opt bytecomp byte-compile cconv eieio-core cl-macs gv
eieio-loaddefs cl-loaddefs cl-lib password-cache shell pcomplete
thingatpt grep compile comint ansi-color ring time-date tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/ns-win
ns-win ucs-normalize mule-util term/common-win tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow
isearch timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote kqueue cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 258110 10162)
 (symbols 48 30900 1)
 (miscs 40 48 187)
 (strings 32 53124 1653)
 (string-bytes 1 1167343)
 (vectors 16 38237)
 (vector-slots 8 750607 17056)
 (floats 8 55 150)
 (intervals 56 244 0)
 (buffers 992 12))

[-- Attachment #2: Type: text/html, Size: 6606 bytes --]

bug#27873: Probably a dupe of #27840

[Eric Hanchrow]

[-- Attachment #1: Type: text/plain, Size: 93 bytes --]

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27840 is a similar report
from a few days ago.

[-- Attachment #2: Type: text/html, Size: 186 bytes --]

bug#27873: 26.0.50; M-x grep broken

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

tags 27873 + patch
quit

Eric Hanchrow <eric.hanchrow@gmail.com> writes:

> This displayed a *grep* buffer that looked something like I expected,
> but:
>
> * instead of there being exactly one line with some underlining
> (indicating a "hit"), there were a bunch: the one I expected, as well
> as a few before it like this:
>
> grep: git-repositories: Is a directory
> grep: guix: Is a directory
> grep: homedir: Is a directory
> grep: homework: Is a directory
> grep: iTunesDSM: Is a directory
> grep: jessie64: Is a directory
> grep: local: Is a directory
> grep: log: Is a directory
> grep: mygo: Is a directory
> grep: node_modules: Is a directory
> grep: perl5: Is a directory
> phonetic-alphabet.txt1:Stolen from http://www.fourmilab.ch/documents/phoneticalphabet/
> grep: pprof: Is a directory
>
> You can't tell from what I've pasted above, but the 9 "Is a directory"
> lines before the actual hit were red and underlined, just like the
> match was.
>
> * Hitting C-x `, instead of visiting the file with the hit, put a
> nine-line-high prompt in the minibuffer, as if it was asking me which
> directory to find the file in a directory with a really weird name.
> (You can see some evedince of this in the "Recent messages" stuff below).

> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27840 is a similar report from a few days ago.

It's not a duplicate though, this bug is because I thought using --null
would make it possible to get filenames containing newlines
unambiguously, but I was wrong.  Here's a patch, also covers a couple of
minor issues that came up later in Bug#6843.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch --]
[-- Type: text/x-diff, Size: 2588 bytes --]

From a6fae71f7f1ea5774a85f4b4238c9d7e4ed4e132 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Sun, 30 Jul 2017 11:07:01 -0400
Subject: [PATCH v1] Don't take multiples lines of grep output as a single
 filename (Bug#27873)

* lisp/progmodes/grep.el (grep-with-null-regexp-alist): Exclude
newlines from the filename part of the regexp.  We must assume
filenames don't have newlines to avoid ambiguity with "foo is a
directory" messages.  Use 'face nil' instead of 'face unspecified',
the latter causes errors (albeit demoted to messsages).  Also renumber
FILE and LINE regexp groups to match the without-null regexp.
(grep-without-null-regexp-alist): Rename from
`grep-fallback-regexp-alist'.
---
 lisp/progmodes/grep.el | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/lisp/progmodes/grep.el b/lisp/progmodes/grep.el
index 4723290fbe..483a1c49ff 100644
--- a/lisp/progmodes/grep.el
+++ b/lisp/progmodes/grep.el
@@ -384,16 +384,22 @@ (defconst grep--regexp-alist-column
               (mend (and mbeg (next-single-property-change mbeg 'font-lock-face nil end))))
          (when mend
            (- mend beg)))))))
+
 (defconst grep--regexp-alist-bin-matcher
   '("^Binary file \\(.+\\) matches$" 1 nil nil 0 1))
+
 (defconst grep-with-null-regexp-alist
-  `(("^\\([^\0]+\\)\\(\0\\)\\([0-9]+\\)\\([\0:]\\)" 1 3 ,grep--regexp-alist-column nil nil
-     (2 '(face unspecified display ":"))
-     (4 '(face unspecified display ":")))
+  `(;; Use explicit numbering to keep FILE and LINE groups the same
+    ;; for both regexp alists.
+    ("^\\(?1:[^\0\n]+\\)\\(?3:\0\\)\\(?2:[0-9]+\\)\\(?4:[\0:]\\)"
+     1 2 ,grep--regexp-alist-column nil nil
+     (3 '(face nil display ":"))
+     (4 '(face nil display ":")))
     ,grep--regexp-alist-bin-matcher)
   "Regexp used to match grep hits.
 See `compilation-error-regexp-alist'.")
-(defconst grep-fallback-regexp-alist
+
+(defconst grep-without-null-regexp-alist
   `(;; Use a tight regexp to handle weird file names (with colons
     ;; in them) as well as possible.  E.g., use [1-9][0-9]* rather
     ;; than [0-9]+ so as to accept ":034:" in file names.
@@ -401,7 +407,8 @@ (defconst grep-fallback-regexp-alist
      1 2 ,grep--regexp-alist-column)
     ,grep--regexp-alist-bin-matcher)
   "Regexp used to match grep hits when `--null' is not supported.
-See `compilation-error-regexp-alist'.")
+See `compilation-error-regexp-alist' and
+`grep-use-null-filename-separator'.")
 
 (defvaralias 'grep-regex-alist 'grep-with-null-regexp-alist)
 (make-obsolete-variable
-- 
2.11.1

bug#27873: 26.0.50; M-x grep broken

[Eric Hanchrow]

[-- Attachment #1: Type: text/plain, Size: 1856 bytes --]

"git apply" refused to apply the patch, but I applied it by hand, and it
seems to work fine.  Thanks.

On Sun, Jul 30, 2017 at 8:30 AM <npostavs@users.sourceforge.net> wrote:

> tags 27873 + patch
> quit
>
> Eric Hanchrow <eric.hanchrow@gmail.com> writes:
>
> > This displayed a *grep* buffer that looked something like I expected,
> > but:
> >
> > * instead of there being exactly one line with some underlining
> > (indicating a "hit"), there were a bunch: the one I expected, as well
> > as a few before it like this:
> >
> > grep: git-repositories: Is a directory
> > grep: guix: Is a directory
> > grep: homedir: Is a directory
> > grep: homework: Is a directory
> > grep: iTunesDSM: Is a directory
> > grep: jessie64: Is a directory
> > grep: local: Is a directory
> > grep: log: Is a directory
> > grep: mygo: Is a directory
> > grep: node_modules: Is a directory
> > grep: perl5: Is a directory
> > phonetic-alphabet.txt1:Stolen from
> http://www.fourmilab.ch/documents/phoneticalphabet/
> > grep: pprof: Is a directory
> >
> > You can't tell from what I've pasted above, but the 9 "Is a directory"
> > lines before the actual hit were red and underlined, just like the
> > match was.
> >
> > * Hitting C-x `, instead of visiting the file with the hit, put a
> > nine-line-high prompt in the minibuffer, as if it was asking me which
> > directory to find the file in a directory with a really weird name.
> > (You can see some evedince of this in the "Recent messages" stuff below).
>
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27840 is a similar report
> from a few days ago.
>
> It's not a duplicate though, this bug is because I thought using --null
> would make it possible to get filenames containing newlines
> unambiguously, but I was wrong.  Here's a patch, also covers a couple of
> minor issues that came up later in Bug#6843.
>
>

[-- Attachment #2: Type: text/html, Size: 2646 bytes --]

bug#27873: 26.0.50; M-x grep broken

[npostavs]

Eric Hanchrow <eric.hanchrow@gmail.com> writes:

> "git apply" refused to apply the patch

Oops, I wrote it here on top of the patch in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27840#8, so it doesn't
apply to current master.

> , but I applied it by hand, and it seems to work fine. Thanks.

Thanks for testing.  I will try to get this grep mess sorted soon.

bug#27873: 26.0.50; M-x grep broken

[Noam Postavsky]

On Sun, Jul 30, 2017 at 4:25 PM, Eric Hanchrow <eric.hanchrow@gmail.com> wrote:
> I suspect you've subtly changed the semantics of the grep command, by
> assuming that the hits are teminated by a 0 byte.  This is a good thing, but
> it seems to have broken a command I've written myself -- that is, when I run
>
>     cd /Users/erichanchrow/git-repositories/3rd-party/emacs/ && git grep
> --perl-regexp -I -n --ignore-case -e hanchrow
>
> I see ... well, it's hard to describe, but the hits aren't highlighted in
> the color I expect, and C-x ` just says "Moved past last grep hit".  But,
> not surprisingly, if I add --null to that command line, it works nicely.
>
> So now I will need to adjust my command based on which version of grep.el
> I've got.  Is there some programmatic way that I can tell which version I've
> got?  I.e., perhaps I can write something like "if boundp
> 'grep-with-null-regexp-alist then add --null to my command, else leave it
> out"?

It should work to just customize `grep-use-null-filename-separator' to
nil. Then you should have the old behaviour again.

I'm currently leaning towards making it default to nil actually, to
avoid the hassle in cases like the one you've described above (see
also Bug#27840).