How can I debug a problem triggered from jit-lock's background fontification?

How can I debug a problem triggered from jit-lock's background fontification?

[Alan Mackenzie]

Hi, Emacs!

I have a bug in CC Mode which is triggered by jit-lock inside its
background fontifying.  The problem doesn't occur when jit-lock is
disabled.

I have instrumented the CC Mode function for Edebug, yet something seems
to be inhibiting the invocation of Edebug from inside jit.  No doubt
there's a good reason for this, but it has been getting on my nerves for
years.

Would somebody please suggest a way I can debug the actions of
jit-locking from Edebug, or possibly some other way of making progress.

Many thanks in advance!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How can I debug a problem triggered from jit-lock's background fontification?

[Stefan Monnier]

> I have instrumented the CC Mode function for Edebug, yet something seems
> to be inhibiting the invocation of Edebug from inside jit.  No doubt
> there's a good reason for this, but it has been getting on my nerves for
> years.

Yes, the problem is that jit-lock is called when redisplay is needed so
it's tricky to let Edebug work at this time (it's easy to make it work
with redisplay inhibited, but then you'll need to use it blindly ;-).

> Would somebody please suggest a way I can debug the actions of
> jit-locking from Edebug, or possibly some other way of making progress.

I've had to deal with this problem, as you can imagine, and I feel
your pain.  The way I generally handle this problem is as follows:
- first, try to turn off jit-lock and to reproduce the problem.  This is
  often all it takes, but admittedly, it doesn't always cut it.
- if that fails, then turn jit-lock back on, do M-x trace-function-background
  RET font-lock-fontify-region RET, and then try and reproduce the
  problem (try and be careful to make sure font-lock is only invoked in
  that one buffer during this time).
- if the trace itself doesn't give you enough of a hint (that's usually
  the case, but sometimes, just seeing where jit-lock placed the
  boundaries of its chunks is enough to see the problem), then turn off
  jit-lock again, and call font-lock-fontify-region by hand with the
  args recorded in the trace.  You can then use Edebug to your heart's
  content during this replay.
  I haven't had to do that often enough and with enough calls to
  font-lock-fontify-region to get me to automate this replay, but if you
  end up writing some code to automate it, I'd be happy to add it to
  jit-lock.
- if the above replay does not reproduce the problem, then you may want
  to try meditation.


        Stefan

Re: How can I debug a problem triggered from jit-lock's background fontification?

[Alan Mackenzie]

Hi Stefan,

thanks for this most helpful reply!  Guess what?  I didn't know about
`trace-function-background' until today.  One never stops learning about
Emacs.

I've actually inserted a call to `ignore' into the critical place, with
all the things I want to see as parameters to ignore.  Then I trace
`ignore'.  I _think_ I've got the problem tied down, but that will have
to wait till tomorrow.

Thanks once again.

On Mon, Feb 07, 2011 at 10:40:48AM -0500, Stefan Monnier wrote:

> I've had to deal with this problem, as you can imagine, and I feel
> your pain.  The way I generally handle this problem is as follows:
> - first, try to turn off jit-lock and to reproduce the problem.  This is
>   often all it takes, but admittedly, it doesn't always cut it.
> - if that fails, then turn jit-lock back on, do M-x trace-function-background
>   RET font-lock-fontify-region RET, and then try and reproduce the
>   problem (try and be careful to make sure font-lock is only invoked in
>   that one buffer during this time).
> - if the trace itself doesn't give you enough of a hint (that's usually
>   the case, but sometimes, just seeing where jit-lock placed the
>   boundaries of its chunks is enough to see the problem), then turn off
>   jit-lock again, and call font-lock-fontify-region by hand with the
>   args recorded in the trace.  You can then use Edebug to your heart's
>   content during this replay.
>   I haven't had to do that often enough and with enough calls to
>   font-lock-fontify-region to get me to automate this replay, but if you
>   end up writing some code to automate it, I'd be happy to add it to
>   jit-lock.
> - if the above replay does not reproduce the problem, then you may want
>   to try meditation.


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: How can I debug a problem triggered from jit-lock's background fontification?

[Lennart Borgman]

On Mon, Feb 7, 2011 at 11:19 PM, Alan Mackenzie <acm@muc.de> wrote:
>
> I've actually inserted a call to `ignore' into the critical place, with
> all the things I want to see as parameters to ignore.  Then I trace
> `ignore'.  I _think_ I've got the problem tied down, but that will have
> to wait till tomorrow.

Thanks for that trick, I never thought about using ignore like that.

Re: How can I debug a problem triggered from jit-lock's background fontification?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

On 2/7/11 7:40 AM, Stefan Monnier wrote:
>> I have instrumented the CC Mode function for Edebug, yet something seems
>> to be inhibiting the invocation of Edebug from inside jit.  No doubt
>> there's a good reason for this, but it has been getting on my nerves for
>> years.

I've run into this problem as well.  I actually use lazy-lock for C and
C++ (because the deferred fontification seems to work better), and the
failure mode there even stranger: when breaking into edebug, emacs will
split the current window as many times as it can until something dies
from stack exhaustion.  I can usually recover from this situation by
switching to another buffer, killing the original, and killing the
excess windows, but it's quite annoying.

> Yes, the problem is that jit-lock is called when redisplay is needed so
> it's tricky to let Edebug work at this time (it's easy to make it work
> with redisplay inhibited, but then you'll need to use it blindly ;-).

What about redisplay to a tty frame?

>> Would somebody please suggest a way I can debug the actions of
>> jit-locking from Edebug, or possibly some other way of making progress.
> 
> I've had to deal with this problem, as you can imagine, and I feel
> your pain.  The way I generally handle this problem is as follows:

Speaking of which, quitting Emacs in the middle of a hard lisp loop
during redisplay on X11 (GTK and no-toolkit), nor OS X (AppKit) doesn't
seem to have any effect.  I've discovered that running under gdb,
breaking into Emacs with C-z, and setting debug_on_next_call to 1 works,
but it's cumbersome.  Is quitting during redisplay *supposed* to work?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: How can I debug a problem triggered from jit-lock's background fontification?

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 955 bytes --]

On 2/7/11 2:22 PM, Lennart Borgman wrote:
> On Mon, Feb 7, 2011 at 11:19 PM, Alan Mackenzie <acm@muc.de> wrote:
>>
>> I've actually inserted a call to `ignore' into the critical place, with
>> all the things I want to see as parameters to ignore.  Then I trace
>> `ignore'.  I _think_ I've got the problem tied down, but that will have
>> to wait till tomorrow.
> 
> Thanks for that trick, I never thought about using ignore like that.
> 

That is a neat trick.  One thing I like to do is use something like this
as my "debug probe":

(progn
  (message "foo: %s" foo)
  (redisplay t)
  (sit-for 1)
  t ; or nil, as appropriate
)


This way, I can visualize the progress a function is making as it
travels up and the down the buffer and detect patterns more easily than
I can by looking at numeric output.  If I have debug-on-quit turned on,
I can stop the process when I notice something amiss and figure out what
went wrong.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: How can I debug a problem triggered from jit-lock's background fontification?

[Stefan Monnier]

> I've run into this problem as well.  I actually use lazy-lock for C and
> C++ (because the deferred fontification seems to work better), and the

Note that lazy-lock is slated for removal (probably before Emacs-25.1).

>> Yes, the problem is that jit-lock is called when redisplay is needed so
>> it's tricky to let Edebug work at this time (it's easy to make it work
>> with redisplay inhibited, but then you'll need to use it blindly ;-).
> What about redisplay to a tty frame?

Same difference, since they also support faces.

> Speaking of which, quitting Emacs in the middle of a hard lisp loop
> during redisplay on X11 (GTK and no-toolkit), nor OS X (AppKit) doesn't
> seem to have any effect.  I've discovered that running under gdb,
> breaking into Emacs with C-z, and setting debug_on_next_call to 1 works,
> but it's cumbersome.  Is quitting during redisplay *supposed* to work?

The Elisp code that is not run in direct response to a user event
(pre/post-command-hook, redisplay, process filters, timers, ...) is run
with inhibit-quit set to t, since the user may hit C-g without knowing
that this code is running, i.e. without intending to interrupt
this code.  There's a pending suggestion to let a C-g C-g C-g override
this, but I haven't looked at the patch yet :-(


        Stefan