all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* slow processing of process output when it's huge
@ 2008-07-27 22:36 Dan Nicolaescu
  2008-07-27 22:52 ` Lennart Borgman (gmail)
                   ` (3 more replies)
  0 siblings, 4 replies; 7+ messages in thread
From: Dan Nicolaescu @ 2008-07-27 22:36 UTC (permalink / raw)
  To: emacs-devel


I run by mistake vc-print-log at the top of the tree of a Linux git
tree.  After 15 minutes of running I gave up and killed it.  The command
does finish after a while, I left it running overnight.

For that particular tree, the "git log" output is 44MB, 1146946 lines.
The time to run the "git log" command is about 3.5 seconds.
The interesting thing is that when saving the "git log" output to
FILE.gitlog and doing C-x C-f FILE.gitlog takes only a 3-4 seconds, even
when enabling font-locking.

So the problem seems to occur when running "git log" from vc-print-log.

Not sure this is a very interesting test case, but maybe someone wants
to take a look at what could be wrong...




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: slow processing of process output when it's huge
  2008-07-27 22:36 slow processing of process output when it's huge Dan Nicolaescu
@ 2008-07-27 22:52 ` Lennart Borgman (gmail)
  2008-07-28  1:49 ` Stefan Monnier
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 7+ messages in thread
From: Lennart Borgman (gmail) @ 2008-07-27 22:52 UTC (permalink / raw)
  To: Dan Nicolaescu; +Cc: emacs-devel

Dan Nicolaescu wrote:
> I run by mistake vc-print-log at the top of the tree of a Linux git
> tree.  After 15 minutes of running I gave up and killed it.  The command
> does finish after a while, I left it running overnight.
> 
> For that particular tree, the "git log" output is 44MB, 1146946 lines.

Maybe it is 2 * 1 146 946 process switches?

> The time to run the "git log" command is about 3.5 seconds.
> The interesting thing is that when saving the "git log" output to
> FILE.gitlog and doing C-x C-f FILE.gitlog takes only a 3-4 seconds, even
> when enabling font-locking.
> 
> So the problem seems to occur when running "git log" from vc-print-log.
> 
> Not sure this is a very interesting test case, but maybe someone wants
> to take a look at what could be wrong...
> 
> 
> 




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: slow processing of process output when it's huge
  2008-07-27 22:36 slow processing of process output when it's huge Dan Nicolaescu
  2008-07-27 22:52 ` Lennart Borgman (gmail)
@ 2008-07-28  1:49 ` Stefan Monnier
  2008-07-28  2:20   ` Dan Nicolaescu
  2008-07-28  2:53 ` Sean O'Rourke
  2008-07-29 15:48 ` Dan Nicolaescu
  3 siblings, 1 reply; 7+ messages in thread
From: Stefan Monnier @ 2008-07-28  1:49 UTC (permalink / raw)
  To: Dan Nicolaescu; +Cc: emacs-devel

> I run by mistake vc-print-log at the top of the tree of a Linux git
> tree.  After 15 minutes of running I gave up and killed it.  The command
> does finish after a while, I left it running overnight.

Does vc-git run its print-log command asynchronously or synchronously?


        Stefan




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: slow processing of process output when it's huge
  2008-07-28  1:49 ` Stefan Monnier
@ 2008-07-28  2:20   ` Dan Nicolaescu
  0 siblings, 0 replies; 7+ messages in thread
From: Dan Nicolaescu @ 2008-07-28  2:20 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-devel

Stefan Monnier <monnier@iro.umontreal.ca> writes:

  > > I run by mistake vc-print-log at the top of the tree of a Linux git
  > > tree.  After 15 minutes of running I gave up and killed it.  The command
  > > does finish after a while, I left it running overnight.
  > 
  > Does vc-git run its print-log command asynchronously or synchronously?

asynchronously




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: slow processing of process output when it's huge
  2008-07-27 22:36 slow processing of process output when it's huge Dan Nicolaescu
  2008-07-27 22:52 ` Lennart Borgman (gmail)
  2008-07-28  1:49 ` Stefan Monnier
@ 2008-07-28  2:53 ` Sean O'Rourke
  2008-07-28  3:08   ` Dan Nicolaescu
  2008-07-29 15:48 ` Dan Nicolaescu
  3 siblings, 1 reply; 7+ messages in thread
From: Sean O'Rourke @ 2008-07-28  2:53 UTC (permalink / raw)
  To: emacs-devel

Dan Nicolaescu <dann@ics.uci.edu> writes:
> I run by mistake vc-print-log at the top of the tree of a Linux
> git tree.  After 15 minutes of running I gave up and killed it.

How much of that is fontification?  I've had this[1] in my .emacs
since forever for when I make a similar mistake in comint, and
while it's still slow, it's much less so.

Sean

[1]
(defun my-comint-stfu-toss (x) "")
(defun my-comint-stfu ()
  "Start (or stop) tossing comint output."
  (interactive)
  (if (and (boundp 'comint-preoutput-filter-functions)
	   (member 'my-comint-stfu-toss comint-preoutput-filter-functions))
      (setq comint-preoutput-filter-functions
	    (remove 'my-comint-stfu-toss comint-preoutput-filter-functions))
      (add-hook 'comint-preoutput-filter-functions
		'my-comint-stfu-toss)))





^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: slow processing of process output when it's huge
  2008-07-28  2:53 ` Sean O'Rourke
@ 2008-07-28  3:08   ` Dan Nicolaescu
  0 siblings, 0 replies; 7+ messages in thread
From: Dan Nicolaescu @ 2008-07-28  3:08 UTC (permalink / raw)
  To: Sean O'Rourke; +Cc: emacs-devel

Sean O'Rourke <seano@cs.ucla.edu> writes:

  > Dan Nicolaescu <dann@ics.uci.edu> writes:
  > > I run by mistake vc-print-log at the top of the tree of a Linux
  > > git tree.  After 15 minutes of running I gave up and killed it.
  > 
  > How much of that is fontification?

Probably nothing, no fontification appears until done.




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: slow processing of process output when it's huge
  2008-07-27 22:36 slow processing of process output when it's huge Dan Nicolaescu
                   ` (2 preceding siblings ...)
  2008-07-28  2:53 ` Sean O'Rourke
@ 2008-07-29 15:48 ` Dan Nicolaescu
  3 siblings, 0 replies; 7+ messages in thread
From: Dan Nicolaescu @ 2008-07-29 15:48 UTC (permalink / raw)
  To: emacs-devel


One more data point that shows that this problem can actually occur for
much smaller process output sizes (less than 10k lines).

emacs -Q -nw
M-: (setq-default case-fold-search nil) RET
M-x rgrep RET emacs RET PATH_TO_EMACS_SOURCE_TREE/lisp RET

this takes a few minutes.

Running the correspondind command:

find . \( -path \*/SCCS -o -path \*/RCS -o -path \*/CVS -o -path \*/MCVS -o -path \*/.svn -o -path \*/.git -o -path \*/.hg -o -path \*/.bzr -o -path \*/_MTN -o -path \*/_darcs -o -path \*/\{arch\} \) -prune -o  -type f \( -name \* \) -print0 | xargs -0 -e grep -i -nH -e emacs 

from a shell (and redirecting the output to a file) takes less than one second.




^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2008-07-29 15:48 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-07-27 22:36 slow processing of process output when it's huge Dan Nicolaescu
2008-07-27 22:52 ` Lennart Borgman (gmail)
2008-07-28  1:49 ` Stefan Monnier
2008-07-28  2:20   ` Dan Nicolaescu
2008-07-28  2:53 ` Sean O'Rourke
2008-07-28  3:08   ` Dan Nicolaescu
2008-07-29 15:48 ` Dan Nicolaescu

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.