Re: M-x grep is too slow

Re: M-x grep is too slow

[Richard Stallman]

    The time is used in `compilation-filter', i.e. by simply inserting the
    strings in the buffer.

That is quite strange.  I suspect this is due to some sort of hook,
because just inserting strings at the end of the buffer will not get
slower.  Is it due to the font-lock mechanism perhaps?

If you run under GDB, you can stop it with C-z from time to time
and make C and Lisp backtraces.  That should give more info about
what is taking the time.

Can you determine precisely which change made this problem appear?

Can someone else join in looking at the problem?
Here's the way to make it fail

       M-x grep -nH -e while emacs/lisp/gnus/*.el

    had not finished in over 20 minutes.

Re: M-x grep is too slow

[Kenichi Handa]

In article <E1BkAgT-0000zL-9a@fencepost.gnu.org>, Richard Stallman <rms@gnu.org> writes:

>     The time is used in `compilation-filter', i.e. by simply inserting the
>     strings in the buffer.

> That is quite strange.  I suspect this is due to some sort of hook,
> because just inserting strings at the end of the buffer will not get
> slower.  Is it due to the font-lock mechanism perhaps?

> If you run under GDB, you can stop it with C-z from time to time
> and make C and Lisp backtraces.  That should give more info about
> what is taking the time.

> Can you determine precisely which change made this problem appear?

> Can someone else join in looking at the problem?
> Here's the way to make it fail

>        M-x grep -nH -e while emacs/lisp/gnus/*.el

>     had not finished in over 20 minutes.

I tried the above but it finished quite soon.   But, when I tried:

        M-x grep -nH -e while emacs/lisp/*/*.el

it took long time.  So, I turned off global-font-lock-mode
and tried again.  But, *grep* buffer is fontified.   Isn't
it a bug?  Anyway, I tried this:

(add-hook 'grep-mode-hook #'(lambda () (font-lock-mode -1)))

Then the above grep also finished quite soon.  So, yes,
font-lock seems to be the culprit.

By the way, when I did C-h v grep-mode-hook RET and clicked
`grep' of "Defined in `grep'." in *Help* buffer, I got this
message:

	Cannot find definition of `grep-mode-hook' in library `grep'

Actually, this variable is defined not in grep.el but by
defined-dirived-mode.

---
Ken'ichi HANDA
handa@m17n.org

Re: M-x grep is too slow

[Eli Zaretskii]

> Date: Tue, 13 Jul 2004 09:36:47 +0900 (JST)
> From: Kenichi Handa <handa@m17n.org>
> 
> I tried the above but it finished quite soon.   But, when I tried:
> 
>         M-x grep -nH -e while emacs/lisp/*/*.el
> 
> it took long time.  So, I turned off global-font-lock-mode
> and tried again.  But, *grep* buffer is fontified.   Isn't
> it a bug?  Anyway, I tried this:
> 
> (add-hook 'grep-mode-hook #'(lambda () (font-lock-mode -1)))
> 
> Then the above grep also finished quite soon.  So, yes,
> font-lock seems to be the culprit.

Perhaps instrumenting font-lock with elp and running the same test
will show what part(s) of font-lock cause this.

Re: M-x grep is too slow

[Stefan Monnier]

> I tried the above but it finished quite soon.   But, when I tried:
>         M-x grep -nH -e while emacs/lisp/*/*.el
> it took long time.

I see it also.  Grep's output at first comes in fast and then at some point
it abruptly slows down and then it only comes in a couple chars at a time
(with enough time inbetween to see the steps).  Later on it switched again
to "fast" mode, and then switched again to "slow" mode, etc...

The problem is not the time taken in font-lock, but the fact that the text
gets inserted one char at a time (or maybe 2 or 3 chars, but it should not
happen that way; and as a matter of fact it does not happen that way when
grep's output comes in fast).

It seems related to the problem David had with AUCTeX on uniprocessors
running the Linux kernel with the "interactive" scheduler.  Kim installed
a patch to fix it, but maybe the patch isn't doing its job here for
some reason.


        Stefan "running on a uniprocessor Linux kernel"

Re: M-x grep is too slow

[Kim F. Storm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > I tried the above but it finished quite soon.   But, when I tried:
> >         M-x grep -nH -e while emacs/lisp/*/*.el
> > it took long time.
> 
> I see it also.  Grep's output at first comes in fast and then at some point
> it abruptly slows down and then it only comes in a couple chars at a time
> (with enough time inbetween to see the steps).  Later on it switched again
> to "fast" mode, and then switched again to "slow" mode, etc...

I don't see it  (GNU/Linux (redhat9), 1.4 GHz uni-processor)

But it does run SLOOOOW...  In my case, approx 50-70 lines are added per screen update.


> 
> The problem is not the time taken in font-lock, but the fact that the text
> gets inserted one char at a time (or maybe 2 or 3 chars, but it should not
> happen that way; and as a matter of fact it does not happen that way when
> grep's output comes in fast).

Well, font-lock definitely makes this A LOT slower...


Looking with a debugger, a lot of time is spent in
previous-single-property-change.

Maybe caching the last position of 'directory and 'message properties
would make a difference ?


Try this little patch:

*** compile.el	20 Jun 2004 20:59:24 +0200	1.322
--- compile.el	16 Jul 2004 16:14:19 +0200	
***************
*** 1200,1206 ****
  	      (turn-on-font-lock))
  	  (turn-on-font-lock))
        ;; maybe defer font-lock till after derived mode is set up
!       (run-mode-hooks 'compilation-turn-on-font-lock))))
  
  ;;;###autoload
  (define-minor-mode compilation-shell-minor-mode
--- 1200,1207 ----
  	      (turn-on-font-lock))
  	  (turn-on-font-lock))
        ;; maybe defer font-lock till after derived mode is set up
!       ;(run-mode-hooks 'compilation-turn-on-font-lock)
!       )))
  
  ;;;###autoload
  (define-minor-mode compilation-shell-minor-mode

With that, grep runs FAST.


> 
> It seems related to the problem David had with AUCTeX on uniprocessors
> running the Linux kernel with the "interactive" scheduler.  Kim installed
> a patch to fix it, but maybe the patch isn't doing its job here for
> some reason.

If I set process-adaptive-read-buffering to nil this runs even slower for
me, but it still finishes in a couple of minutes...

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: M-x grep is too slow

[Richard Stallman]

    Looking with a debugger, a lot of time is spent in
    previous-single-property-change.

    Maybe caching the last position of `directory' and `message' properties
    would make a difference ?

Do you mean caching at the Lisp level inside Font-Lock mode?
That might be easy to do.

Where is previous-single-property-change being called from?
Maybe a change there is what we need.


Aside from that, I have a feeling it is wrong for Compilation
mode to turn on Font-Lock mode by default.  This nonuniformity
in the control of Font-Lock mode seems wrong.

Re: M-x grep is too slow

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

>     Looking with a debugger, a lot of time is spent in
>     previous-single-property-change.
> 
>     Maybe caching the last position of `directory' and `message' properties
>     would make a difference ?
> 
> Do you mean caching at the Lisp level inside Font-Lock mode?
> That might be easy to do.

I was thinking about doing it explicitly in compile.el...

> 
> Where is previous-single-property-change being called from?
> Maybe a change there is what we need.
> 

There are only a few calls in compile.el, so it cannot be too hard
to see what's needed.  However, there seem to be a call hidden in
a macro so it may not be trivial.  I'll leave it to the experts.

> 
> Aside from that, I have a feeling it is wrong for Compilation
> mode to turn on Font-Lock mode by default.  This nonuniformity
> in the control of Font-Lock mode seems wrong.

I think so too.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: M-x grep is too slow

[Richard Stallman]

    > Do you mean caching at the Lisp level inside Font-Lock mode?
    > That might be easy to do.

    I was thinking about doing it explicitly in compile.el...

Is this slowness in font-lock or is it in compile.el?

Re: M-x grep is too slow

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

>     > Do you mean caching at the Lisp level inside Font-Lock mode?
>     > That might be easy to do.
> 
>     I was thinking about doing it explicitly in compile.el...
> 
> Is this slowness in font-lock or is it in compile.el?

I don't really know.

The code that calls previous-single-property-change is in compile.el,
but it could be that the code is actually called by font-lock.

In any case, the 'message and 'directory properties are local to
compile.el, so I would suppose that the caching (if possible)
would have to be done in compile.el.

Maybe someone who better understands the new compile.el could comment
on this.

Re: M-x grep is too slow

[Richard Stallman]

    The code that calls previous-single-property-change is in compile.el,
    but it could be that the code is actually called by font-lock.

Could you make a backtrace?  Then we could see.

Re: M-x grep is too slow

[Richard Stallman]

    By the way, when I did C-h v grep-mode-hook RET and clicked
    `grep' of "Defined in `grep'." in *Help* buffer, I got this
    message:

	    Cannot find definition of `grep-mode-hook' in library `grep'

    Actually, this variable is defined not in grep.el but by
    defined-dirived-mode.

I think the code in help-mode ought to recognize variable names ending
in ...-mode-hook, and if it can't find a defvar in the file, it should
search for the suitable call to defined-derived-mode in the same file.

Would someone like to implement that?

Re: M-x grep is too slow

[Richard Stallman]

I am not surprised that font-lock is responsible, but we need to
find out what it's doing that takes so long.

If you follow my suggestion to run under GDB, type C-z, and then type
xbacktrace, you'll find out which functions of font-lock mode
are running most of the time.  If you put a call to (debug) in
one of them, you can step through it and see whether it takes
a long time, and what it is doing.

It would be useful to check whether the font lock keywords
used for compile mode have changed in the past few months.
If so, that change might be at fault.