slow processing of process output when it's huge

slow processing of process output when it's huge

[Dan Nicolaescu]

I run by mistake vc-print-log at the top of the tree of a Linux git
tree.  After 15 minutes of running I gave up and killed it.  The command
does finish after a while, I left it running overnight.

For that particular tree, the "git log" output is 44MB, 1146946 lines.
The time to run the "git log" command is about 3.5 seconds.
The interesting thing is that when saving the "git log" output to
FILE.gitlog and doing C-x C-f FILE.gitlog takes only a 3-4 seconds, even
when enabling font-locking.

So the problem seems to occur when running "git log" from vc-print-log.

Not sure this is a very interesting test case, but maybe someone wants
to take a look at what could be wrong...

Re: slow processing of process output when it's huge

[Lennart Borgman (gmail)]

Dan Nicolaescu wrote:
> I run by mistake vc-print-log at the top of the tree of a Linux git
> tree.  After 15 minutes of running I gave up and killed it.  The command
> does finish after a while, I left it running overnight.
> 
> For that particular tree, the "git log" output is 44MB, 1146946 lines.

Maybe it is 2 * 1 146 946 process switches?

> The time to run the "git log" command is about 3.5 seconds.
> The interesting thing is that when saving the "git log" output to
> FILE.gitlog and doing C-x C-f FILE.gitlog takes only a 3-4 seconds, even
> when enabling font-locking.
> 
> So the problem seems to occur when running "git log" from vc-print-log.
> 
> Not sure this is a very interesting test case, but maybe someone wants
> to take a look at what could be wrong...
> 
> 
> 

Re: slow processing of process output when it's huge

[Stefan Monnier]

> I run by mistake vc-print-log at the top of the tree of a Linux git
> tree.  After 15 minutes of running I gave up and killed it.  The command
> does finish after a while, I left it running overnight.

Does vc-git run its print-log command asynchronously or synchronously?


        Stefan

Re: slow processing of process output when it's huge

[Dan Nicolaescu]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

  > > I run by mistake vc-print-log at the top of the tree of a Linux git
  > > tree.  After 15 minutes of running I gave up and killed it.  The command
  > > does finish after a while, I left it running overnight.
  > 
  > Does vc-git run its print-log command asynchronously or synchronously?

asynchronously

Re: slow processing of process output when it's huge

[Sean O'Rourke]

Dan Nicolaescu <dann@ics.uci.edu> writes:
> I run by mistake vc-print-log at the top of the tree of a Linux
> git tree.  After 15 minutes of running I gave up and killed it.

How much of that is fontification?  I've had this[1] in my .emacs
since forever for when I make a similar mistake in comint, and
while it's still slow, it's much less so.

Sean

[1]
(defun my-comint-stfu-toss (x) "")
(defun my-comint-stfu ()
  "Start (or stop) tossing comint output."
  (interactive)
  (if (and (boundp 'comint-preoutput-filter-functions)
	   (member 'my-comint-stfu-toss comint-preoutput-filter-functions))
      (setq comint-preoutput-filter-functions
	    (remove 'my-comint-stfu-toss comint-preoutput-filter-functions))
      (add-hook 'comint-preoutput-filter-functions
		'my-comint-stfu-toss)))

Re: slow processing of process output when it's huge

[Dan Nicolaescu]

Sean O'Rourke <seano@cs.ucla.edu> writes:

  > Dan Nicolaescu <dann@ics.uci.edu> writes:
  > > I run by mistake vc-print-log at the top of the tree of a Linux
  > > git tree.  After 15 minutes of running I gave up and killed it.
  > 
  > How much of that is fontification?

Probably nothing, no fontification appears until done.

Re: slow processing of process output when it's huge

[Dan Nicolaescu]

One more data point that shows that this problem can actually occur for
much smaller process output sizes (less than 10k lines).

emacs -Q -nw
M-: (setq-default case-fold-search nil) RET
M-x rgrep RET emacs RET PATH_TO_EMACS_SOURCE_TREE/lisp RET

this takes a few minutes.

Running the correspondind command:

find . \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o  -type f \( -name \* \) -print0 | xargs -0 -e grep -i -nH -e emacs 

from a shell (and redirecting the output to a file) takes less than one second.