emacs hangs in jit-lock

emacs hangs in jit-lock

[Sam Steingold]

GNU Emacs 22.0.91.1 (i686-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
  of 2006-11-27 on loiso

I am suffering from an Emacs performance problem: when I open "too many
bad files" (say, total 10,000 lines of C code without a single #\{ or
#\( in the first column), Emacs becomes unresponsive.
Every now and then it hangs (taking up 100% of the CPU).
Some typical backtraces:

Lisp Backtrace:
"line-end-position" (0x29e02)
"back-to-indentation" (0x2e0)
"c-beginning-of-macro" (0x27bda8)
"byte-code" (0xa5c7eab)
"c-parse-state" (0x27bda8)
"c-font-lock-complex-decl-prepare" (0x27d0b8)
"font-lock-fontify-keywords-region" (0x27bda8)
"font-lock-default-fontify-region" (0x27bda8)
"font-lock-fontify-region" (0x27bda8)
"run-hook-with-args" (0x840af31)
"byte-code" (0x822f25b)
"jit-lock-fontify-now" (0x27bed8)
"jit-lock-stealth-fontify" (0x83178f9)
"apply" (0x85053d9)
"byte-code" (0x8235b5b)
"timer-event-handler" (0x9867c1c)

Lisp Backtrace:
"byte-code" (0xa5c7eab)
"c-parse-state" (0x285560)
"c-font-lock-complex-decl-prepare" (0x2867f0)
"font-lock-fontify-keywords-region" (0x285560)
"font-lock-default-fontify-region" (0x285560)
"font-lock-fontify-region" (0x285560)
"run-hook-with-args" (0x840af31)
"byte-code" (0x822f25b)
"jit-lock-fontify-now" (0x285648)
"jit-lock-stealth-fontify" (0x83178f9)
"apply" (0x85053d9)
"byte-code" (0x8235b5b)
"timer-event-handler" (0x9867c1c)

Lisp Backtrace:
"re-search-backward" (0xa8972db)
"d-mode-beg-of-defun" (0x8)
"beginning-of-defun-raw" (0x83178c9)
"beginning-of-defun" (0x10)
"c-get-fallback-start-pos" (0x2dbbd0)
"c-parse-state" (0x2dbbd0)
"c-font-lock-complex-decl-prepare" (0x2dcb98)
"font-lock-fontify-keywords-region" (0x2dbbd0)
"font-lock-default-fontify-region" (0x2dbbd0)
"font-lock-fontify-region" (0x2dbbd0)
"run-hook-with-args" (0x840af31)
"byte-code" (0x822f25b)
"jit-lock-fontify-now" (0x2dbbd0)
"jit-lock-stealth-fontify" (0x83178f9)
"apply" (0x85053d9)
"byte-code" (0x8235b5b)
"timer-event-handler" (0x9867c1c)
"read-event" (0x95b932c)
"byte-code" (0x8231eb3)
"mouse-show-mark" (0x1f690)
"mouse-drag-track" (0x9e34fc5)
"mouse-drag-region" (0x9e34fc5)
"call-interactively" (0x8539e89)

Lisp Backtrace:
"read-event" (0x95b932c)
"byte-code" (0x8231eb3)
"mouse-show-mark" (0x1f690)
"mouse-drag-track" (0x9e34fc5)
"mouse-drag-region" (0x9e34fc5)
"call-interactively" (0x8539e89)

Lisp Backtrace:
"c-beginning-of-macro" (0x28b770)
"byte-code" (0xa5c7eab)
"c-parse-state" (0x28b870)
"c-font-lock-complex-decl-prepare" (0x28c9e8)
"font-lock-fontify-keywords-region" (0x28b870)
"font-lock-default-fontify-region" (0x28b870)
"font-lock-fontify-region" (0x28b870)
"run-hook-with-args" (0x840af31)
"byte-code" (0x822f25b)
"jit-lock-fontify-now" (0x28b958)
"jit-lock-stealth-fontify" (0x83178f9)
"apply" (0x85053d9)
"byte-code" (0x8235b5b)
"timer-event-handler" (0x9867c1c)

the problem is that this unresponsiveness can last for MINUTES (I am not
making this up!) and the periods of responsiveness can last for mere
SECONDS.
I understand what Emacs is trying to do, but, really, this "background"
refontification should not take that much time (I have a fairly modern
desktop, in time Emacs takes to parse the files I think I could have
done it by hand with a pencil).
Even if it should, any input from me should immediately interrupt it -
and I see my C-g "delayed" for minutes.
This is a very serious problem for me.
Thanks.

Sam.

Re: emacs hangs in jit-lock

[Chong Yidong]

Sam Steingold <sds@gnu.org> writes:

> "line-end-position" (0x29e02)
> "back-to-indentation" (0x2e0)
> "c-beginning-of-macro" (0x27bda8)
> "byte-code" (0xa5c7eab)
> "c-parse-state" (0x27bda8)
> "c-font-lock-complex-decl-prepare" (0x27d0b8)
> "font-lock-fontify-keywords-region" (0x27bda8)
> "font-lock-default-fontify-region" (0x27bda8)
> "font-lock-fontify-region" (0x27bda8)
> "run-hook-with-args" (0x840af31)
> "byte-code" (0x822f25b)
> "jit-lock-fontify-now" (0x27bed8)
> "jit-lock-stealth-fontify" (0x83178f9)
>
> the problem is that this unresponsiveness can last for MINUTES (I am not
> making this up!) and the periods of responsiveness can last for mere
> SECONDS.
>
> Even if it should, any input from me should immediately interrupt it -
> and I see my C-g "delayed" for minutes.

Since jit-lock-stealth fontify is run as an idle timer, any input
*does* interrupt it.  The only way this could fail is if the
c-font-lock functions interfere with this---either by somehow
resetting the idleness state, or (more likely) taking a very long
time.

If you provided a simple test case, I or someone else could take a
look at this problem.

Re: emacs hangs in jit-lock

[Sam Steingold]

Chong Yidong wrote:
> Sam Steingold <sds@gnu.org> writes:
> 
>> "line-end-position" (0x29e02)
>> "back-to-indentation" (0x2e0)
>> "c-beginning-of-macro" (0x27bda8)
>> "byte-code" (0xa5c7eab)
>> "c-parse-state" (0x27bda8)
>> "c-font-lock-complex-decl-prepare" (0x27d0b8)
>> "font-lock-fontify-keywords-region" (0x27bda8)
>> "font-lock-default-fontify-region" (0x27bda8)
>> "font-lock-fontify-region" (0x27bda8)
>> "run-hook-with-args" (0x840af31)
>> "byte-code" (0x822f25b)
>> "jit-lock-fontify-now" (0x27bed8)
>> "jit-lock-stealth-fontify" (0x83178f9)
>>
>> the problem is that this unresponsiveness can last for MINUTES (I am not
>> making this up!) and the periods of responsiveness can last for mere
>> SECONDS.
>>
>> Even if it should, any input from me should immediately interrupt it -
>> and I see my C-g "delayed" for minutes.
> 
> Since jit-lock-stealth fontify is run as an idle timer, any input
> *does* interrupt it.  The only way this could fail is if the
> c-font-lock functions interfere with this---either by somehow
> resetting the idleness state, or (more likely) taking a very long
> time.

whatever the reason, C-g does not interrupt this multi-minute process.

> If you provided a simple test case, I or someone else could take a
> look at this problem.

I don't think this problem has a "simple test case".
wget 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/lispbibl.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/stream.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/io.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/eval.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/pathname.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/sequence.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/array.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/foreign.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/spvw.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/charstrg.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/hashtabl.d 
http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/emacs/d-mode.el

then load d-mode.el, associate d-mode with *.d, open all the *.d files 
and try to edit, say, lispbibl.d close to the end of file.

Thanks.
Sam.

Re: emacs hangs in jit-lock

[Henrik Enberg]

Sam Steingold <sds@gnu.org> writes:
> Chong Yidong wrote:
> >If you provided a simple test case, I or someone else could take a
> >look at this problem.
> 
> I don't think this problem has a "simple test case".
> wget 
> http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/src/lispbibl.d 

[...]

> http://clisp.cvs.sourceforge.net/*checkout*/clisp/clisp/emacs/d-mode.el
> 
> then load d-mode.el, associate d-mode with *.d, open all the *.d files 
> and try to edit, say, lispbibl.d close to the end of file.

I see the same thing.  The header file below is a simple way to test
it.  Just open it and hit C-v, and things will get very slow.

<http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/usr.bin/mail/extern.h>

Setting jit-lock-stealth-time to nil makes the problem go away.

Re: emacs hangs in jit-lock

[Stefan Monnier]

> Setting jit-lock-stealth-time to nil makes the problem go away.

I motion to change the default of jit-lock-stealth-time to nil.
A non-nil setting may be useful for some situations, but nil is safer and
more desirable for laptop use which is becoming the more prevalent
machines nowadays.


        Stefan

Re: emacs hangs in jit-lock

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Setting jit-lock-stealth-time to nil makes the problem go away.
>
> I motion to change the default of jit-lock-stealth-time to nil.
> A non-nil setting may be useful for some situations, but nil is safer and
> more desirable for laptop use which is becoming the more prevalent
> machines nowadays.

I agree.  Also, modern processors are fast enough that on-the-fly
fontification works perfectly well; the advantage of fontifying in the
background in negligible.

Re: emacs hangs in jit-lock

[martin rudalics]

 > I motion to change the default of jit-lock-stealth-time to nil.
 > A non-nil setting may be useful for some situations, but nil is safer and
 > more desirable for laptop use which is becoming the more prevalent
 > machines nowadays.

I doubt that setting `jit-lock-stealth-time' to nil would solve problems
reported as

 >>then load d-mode.el, associate d-mode with *.d, open all the *.d files
 >>and try to edit, say, lispbibl.d close to the end of file.

and

 > I see the same thing.  The header file below is a simple way to test
 > it.  Just open it and hit C-v, and things will get very slow.
 >
 > <http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/usr.bin/mail/extern.h>
 >
 > Setting jit-lock-stealth-time to nil makes the problem go away.

These are not necessarily the "same things".  Suppose both problems were
caused by stealth fontification proper.  There would not be any need to
edit or scroll buffers to observe a slowdown.  However, the first author
tried to "edit the buffer close to the of file" and the second author
did open it and "hit C-v".  I can see two common points: (1) The major
mode involved was C mode and (2) there was some movement toward the end
of a buffer.

To check whether a problem is really caused by stealth fontification one
has to compare performance with stealth fontification turned off/on but
_without_ displaying the involved buffers in a window.  To detect
whether the slowdown is caused by parsing, it would suffice to open the
relative buffers, go to their ends, and stealthily fontify them without
displaying them.  Finally, to detect whether stealth fontification
simply "hangs" set `jit-lock-stealth-verbose' to t.

FWIW, I think that Alan's recent change to `beginning-of-defun-raw' is
the main culprit for the observed slowdown and it would be very easy to
spot it in the indicated way.  I don't use stealth fontification, hence
defaulting `jit-lock-stealth-time' to nil would just save me one line of
my .emacs.  On the other hand, it would mean throwing out the child with
the bathwater and not cure the real problem.

Re: emacs hangs in jit-lock

[Stefan Monnier]

>> I motion to change the default of jit-lock-stealth-time to nil.
>> A non-nil setting may be useful for some situations, but nil is safer and
>> more desirable for laptop use which is becoming the more prevalent
>> machines nowadays.

> I doubt that setting `jit-lock-stealth-time' to nil would solve problems
> reported as

It will fix a few problems (typically of the sort "Emacs takes 100% cpu
time"), but I agree that it won't really fix the mentioned problems.
Instead it will make them appear less often and more deterministically.
That's still good in my book.


        Stefan

Re: emacs hangs in jit-lock

[martin rudalics]

 > It will fix a few problems (typically of the sort "Emacs takes 100% cpu
 > time"), but I agree that it won't really fix the mentioned problems.

If I open say xdisp.c with Alan's `beginning-of-defun-raw' and jump to
the end of the buffer Emacs takes more than 98% of my CPU with stealth
fontification turned off.  Subsequent M-v's will do the same.  Without
Alan's change Emacs takes less than 5%.

 > Instead it will make them appear less often and more deterministically.
 > That's still good in my book.

We violently agree.

Re: emacs hangs in jit-lock

[Stefan Monnier]

>> It will fix a few problems (typically of the sort "Emacs takes 100% cpu
>> time"), but I agree that it won't really fix the mentioned problems.

> If I open say xdisp.c with Alan's `beginning-of-defun-raw' and jump to
> the end of the buffer Emacs takes more than 98% of my CPU with stealth
> fontification turned off.  Subsequent M-v's will do the same.  Without
> Alan's change Emacs takes less than 5%.

Why am I not surprised ;-) ?


        Stefan

Re: emacs hangs in jit-lock

[martin rudalics]

> Why am I not surprised ;-) ?

I can't debug that :-( maybe you're hanging in some sort of jit-lock.

Re: emacs hangs in jit-lock

[Richard Stallman]

    > If I open say xdisp.c with Alan's `beginning-of-defun-raw' and jump to
    > the end of the buffer Emacs takes more than 98% of my CPU with stealth
    > fontification turned off.

Which change is this?  We were discussing more than one change in that.

Re: emacs hangs in jit-lock

[martin rudalics]

> Which change is this?  We were discussing more than one change in that.

2006-11-08  Alan Mackenzie  <acm@muc.de>

	* emacs-lisp/lisp.el (beginning-of-defun-raw): Code up the
	case (eq open-paren-in-column-0-is-defun-start nil) by searching
	for least nested open-paren.

Re: emacs hangs in jit-lock

[Richard Stallman]

    2006-11-08  Alan Mackenzie  <acm@muc.de>

	    * emacs-lisp/lisp.el (beginning-of-defun-raw): Code up the
	    case (eq open-paren-in-column-0-is-defun-start nil) by searching
	    for least nested open-paren.

I thought that the reason CC mode sets
open-paren-in-column-0-is-defun-start to nil is that it provided a
beginning-of-defun-function, which would have prevented this from
causing a big slowdown.  But I see it does not.

Should we add something to set beginning-of-defun-function?

Re: emacs hangs in jit-lock

[Richard Stallman]

    A non-nil setting may be useful for some situations, but nil is safer and
    more desirable for laptop use which is becoming the more prevalent
    machines nowadays.

Have people found that a non-nil value causes a significant increase
in power drain?

Re: emacs hangs in jit-lock

[Chong Yidong]

Richard Stallman <rms@gnu.org> writes:

>     A non-nil setting may be useful for some situations, but nil is safer and
>     more desirable for laptop use which is becoming the more prevalent
>     machines nowadays.
>
> Have people found that a non-nil value causes a significant increase
> in power drain?

It's not trivial to quantify how much power drain stealth
fontification produces.  But several people have reported that it
causes their laptop fans are turned on, which means the consumption is
not negligible.

Re: emacs hangs in jit-lock

[Richard Stallman]

    It's not trivial to quantify how much power drain stealth
    fontification produces.

On laptops with acpi, you can see the amount of power drain.

However, maybe it is true that stealth fontification is not needed
any more with jit-lock.

Re: emacs hangs in jit-lock

[martin rudalics]

 > I understand what Emacs is trying to do, but, really, this "background"
 > refontification should not take that much time (I have a fairly modern
 > desktop, in time Emacs takes to parse the files I think I could have
 > done it by hand with a pencil).

"Background refontification" is not needed at all to keep Emacs
functional.  Please switch off stealth fontification by setting
`jit-lock-stealth-time' to nil and tell us whether the problem
persists.

Re: emacs hangs in jit-lock

[Richard Stallman]

    I am suffering from an Emacs performance problem: when I open "too many
    bad files" (say, total 10,000 lines of C code without a single #\{ or
    #\( in the first column), Emacs becomes unresponsive.

If you turn off stealth fontification, does the problem go away?