bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Richard Stallman]

When I use M-x grep through a lot of files, grep hits appear and
display all at once.  It used to be that hits in the first files
searched would appear earlier -- as soon as grep finds them, I expect
-- but I think Emacs doesn't notice that output from the subprocess any more.


In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 2.24.30, cairo version 1.14.6)
 of 2020-06-18 built on freetop
Repository revision: fbf40c1d903d18286ecd7d2c1d7b117c88a1d5dd
Repository branch: master
System Description: Trisquel GNU/Linux Etiona (9.0)

Recent messages:
next-line: End of buffer
Quit
Auto-saving...done
Sending...
Wrote /home/rms/outgoing/out-12
Sending...done
black 6 Black 0  white 0 White 0
Mark set [2 times]
Grep finished with no matches found [4 times]
M-RET is undefined

Configured using:
 'configure 'CFLAGS=-O0 -g' --with-gnutls=no'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK2 X11 XDBE XIM MODULES THREADS PDUMPER GMP

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Dired by name

Minor modes in effect:
  shell-dirtrack-mode: t
  gpm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow emacsbug etags fileloop generator xref project ispell
dired-aux shell pcomplete thingatpt grep compile comint ansi-color
ring misearch multi-isearch mule-util quail help-mode dabbrev
mailalias sendmail shr url-cookie url-domsuf url-util svg xml dom
rmailout rmailsum qp rmailmm message rmc puny rfc822 mml mml-sec epa
epg epg-config gnus-util text-property-search time-date mm-decode
mm-bodies mm-encode mailabbrev gmm-utils mailheader mail-parse rfc2231
rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils dired dired-loaddefs t-mouse term/linux view derived paren
cus-start cus-load advice finder-inf package easymenu browse-url
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese
eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic
indian cyrillic chinese composite charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 436787 50618)
 (symbols 48 11528 1)
 (strings 32 44345 7084)
 (string-bytes 1 1184832)
 (vectors 16 25745)
 (vector-slots 8 642201 100732)
 (floats 8 56 269)
 (intervals 56 70799 524)
 (buffers 992 42)
 (heap 1024 23901 1816))
[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Drew Adams]

> When I use M-x grep through a lot of files, grep hits appear and
> display all at once.  It used to be that hits in the first files
> searched would appear earlier -- as soon as grep finds them, I expect
> -- but I think Emacs doesn't notice that output from the subprocess any
> more.
> 
> In GNU Emacs 28.0.50

FWIW, I don't see that in Emacs 26.3.  I see the
"used to" behavior of showing hits as `grep' progresses.

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Drew Adams]

> FWIW, I don't see that in Emacs 26.3.  I see the
> "used to" behavior of showing hits as `grep' progresses.

Oops; sorry.  That's probably not very helpful.
I forgot that I'm using Cygwin's `grep' (on MS
Windows), and an old Cygwin at that.

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> When I use M-x grep through a lot of files, grep hits appear and
> display all at once.  It used to be that hits in the first files
> searched would appear earlier -- as soon as grep finds them, I expect
> -- but I think Emacs doesn't notice that output from the subprocess any more.

I'm unable to reproduce this bug in Emacs 28 on Debian/bullseye.

My test case:

emacs -Q
M-x grep RET grep --color -nH --null -e the `find . -name '*.el'` RET

I can then go to the *grep* buffer, jump to the end, and see all the
matches stream in.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Sun, 29 Nov 2020 00:22:20 -0500
> 
> When I use M-x grep through a lot of files, grep hits appear and
> display all at once.  It used to be that hits in the first files
> searched would appear earlier -- as soon as grep finds them, I expect
> -- but I think Emacs doesn't notice that output from the subprocess any more.

It still works for me as it ever did.  Maybe you didn't define a
search that runs for long enough?

There should be a count of matches displayed in brackets on the mode
line; with a long enough search, like using "grep -R" on a large
directory tree, I see the numbers increasing, and I can go the *grep*
buffer and see the stuff coming in.

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It still works for me as it ever did.  Maybe you didn't define a
  > search that runs for long enough?

Thes search was through 230 files with a total size of 1.2G.
So it was not that.

However, today I found what is responsible.
I am using a little script called cgrep, as follows.

   cgrep  -nH --null "From: .*ruben@" rms2*

The search got several hits in various files but they all came out at
once, just before exiting

I did he same command in an ordinary terminal shell, not under Emacs,
and the lines came out  without delay.

Here is the cgrep script:

#!/bin/bash

grep -a "$@" | cut -c -200

cut seems to be responsible for the problem by buffering output even to a tty.
So it is not Emacs's fault.

Please forgive the noise.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> #!/bin/bash
>
> grep -a "$@" | cut -c -200
>
> cut seems to be responsible for the problem by buffering output even to a tty.
> So it is not Emacs's fault.
>
> Please forgive the noise.

No problem; closing the bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: 44941@debbugs.gnu.org
> Date: Sun, 29 Nov 2020 23:46:22 -0500
> 
> Here is the cgrep script:
> 
> #!/bin/bash
> 
> grep -a "$@" | cut -c -200
> 
> cut seems to be responsible for the problem by buffering output even to a tty.

Sounds like a useful feature to ask Grep developers to add it.

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Andreas Schwab]

On Nov 30 2020, Eli Zaretskii wrote:

>> From: Richard Stallman <rms@gnu.org>
>> Cc: 44941@debbugs.gnu.org
>> Date: Sun, 29 Nov 2020 23:46:22 -0500
>> 
>> Here is the cgrep script:
>> 
>> #!/bin/bash
>> 
>> grep -a "$@" | cut -c -200
>> 
>> cut seems to be responsible for the problem by buffering output even to a tty.
>
> Sounds like a useful feature to ask Grep developers to add it.

That feature already exists: stdbuf.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: rms@gnu.org,  44941@debbugs.gnu.org
> Date: Mon, 30 Nov 2020 16:47:28 +0100
> 
> On Nov 30 2020, Eli Zaretskii wrote:
> 
> >> From: Richard Stallman <rms@gnu.org>
> >> Cc: 44941@debbugs.gnu.org
> >> Date: Sun, 29 Nov 2020 23:46:22 -0500
> >> 
> >> Here is the cgrep script:
> >> 
> >> #!/bin/bash
> >> 
> >> grep -a "$@" | cut -c -200
> >> 
> >> cut seems to be responsible for the problem by buffering output even to a tty.
> >
> > Sounds like a useful feature to ask Grep developers to add it.
> 
> That feature already exists: stdbuf.

I'm not sure I understand: I meant the feature to limit the output
lines to a given column count.

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2020-11-30 19:39]:
> > From: Andreas Schwab <schwab@linux-m68k.org>
> > Cc: rms@gnu.org,  44941@debbugs.gnu.org
> > Date: Mon, 30 Nov 2020 16:47:28 +0100
> > 
> > On Nov 30 2020, Eli Zaretskii wrote:
> > 
> > >> From: Richard Stallman <rms@gnu.org>
> > >> Cc: 44941@debbugs.gnu.org
> > >> Date: Sun, 29 Nov 2020 23:46:22 -0500
> > >> 
> > >> Here is the cgrep script:
> > >> 
> > >> #!/bin/bash
> > >> 
> > >> grep -a "$@" | cut -c -200
> > >> 
> > >> cut seems to be responsible for the problem by buffering output even to a tty.
> > >
> > > Sounds like a useful feature to ask Grep developers to add it.
> > 
> > That feature already exists: stdbuf.
> 
> I'm not sure I understand: I meant the feature to limit the output
> lines to a given column count.

Is it not this option for limiting?

	-m NUM, --max-count=NUM
		Stop reading a  file after NUM matching  lines.  If the
		input is  standard input from  a regular file,  and NUM
		matching  lines  are  output,  grep  ensures  that  the
		standard  input is  positioned to  just after  the last
		matching  line   before  exiting,  regardless   of  the
		presence  of trailing  context lines.   This enables  a
		calling process  to resume  a search.  When  grep stops
		after  NUM  matching  lines, it  outputs  any  trailing
		context lines.  When  the -c or --count  option is also
		used, grep  does not output  a count greater  than NUM.
		When the -v or --invert-match option is also used, grep
		stops after outputting NUM non-matching lines.

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Eli Zaretskii]

> Date: Mon, 30 Nov 2020 19:46:28 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: Andreas Schwab <schwab@linux-m68k.org>, 44941@debbugs.gnu.org,
>   rms@gnu.org
> 
> > > >> grep -a "$@" | cut -c -200
> > > >> 
> > > >> cut seems to be responsible for the problem by buffering output even to a tty.
> > > >
> > > > Sounds like a useful feature to ask Grep developers to add it.
> > > 
> > > That feature already exists: stdbuf.
> > 
> > I'm not sure I understand: I meant the feature to limit the output
> > lines to a given column count.
> 
> Is it not this option for limiting?
> 
> 	-m NUM, --max-count=NUM
> 		Stop reading a  file after NUM matching  lines.  If the

No.  Please read the 'cut' documentation to see what does -c do there.

bug#44941: 28.0.50; M-x grep, perhaps all asynch subprocesses

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That feature already exists: stdbuf.

Thank you very much!  I had never heard of that.  stdbuf --output=L
applying to grep fixed the problem -- grep needed this, since it was
outputting to a pipe.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)