What is the _essential_ difference between lazy-lock and jit-lock?

What is the _essential_ difference between lazy-lock and jit-lock?

[Alan Mackenzie]

lazy-lock and jit-lock are both font-lock support modes, things which
reduce irritating delays when files are fontified as they are visited and
(to a lesser extent) as they get edited.

However, they both appear to offer the same essential facilities:  lazy
fontification (only the currently displayed part of the buffer is
fontified with any urgency), stealth fontification (undisplayed parts get
fontified whilst the user isn't typing), deferred fontification (whilst
typing, keypresses are given priority over instant fontification).  

jit-lock was written some years after lazy-lock, and its defaults are
different (for example, by default jit-lock starts stealth fontification
after 3 seconds inactivity, lazy-lock after 30 seconds).

[lazy-lock and jit-lock thus stand in sharp contrast to fast-lock, which
merely writes the current fontifications to an auxiliary file so as to
speed up fontification the next time the main file is loaded.]

Question:  Why jit-lock?  What does jit-lock offer that lazy-lock
doesn't?  Why was jit-lock written in the first place?  What was
unsatisfactory about lazy-lock?

I've tried reading the info pages (there seems to be no information on
these support modes) and I've been reading the modes' lisp source files.
Unfortunately, I get lost in the implementation details without seeing
the essentials.

Why jit-lock?

Thanks in advance!

-- 
Alan Mackenzie (Munich, Germany)
Email: aacm@muuc.dee; to decode, wherever there is a repeated letter
(like "aa"), remove half of them (leaving, say, "a").

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Stefan Monnier]

> Why jit-lock?

Lazy-lock was a neat idea to only do fontification on-the-fly: why fontify
1MB of code before displaying the first 40 lines ?

But it was difficult from elisp to do it right: the hooks available to
lazy-lock to detect when the displayed part of the screen changes and which
parts of the buffer are currently displayed are not very satisfactory:
difficult to use (leading to corner cases where fontification is not done
when it should) and inefficient (lazy-lock is triggered much more often than
necessary).  This is even more true on XEmacs and old Emacs where one of the
hooks used by lazy-lock is (or at least was) missing.

jit-lock grew out of this: create one new hook designed specifically for
this job.  It's simpler, more reliable, faster.

The only tradeoff I know of (other than the fact that some of the features
of lazy-lock have not been ported yet to jit-lock) is that lazy-lock has
some more knowledge about the intention behind a movement because it can
tell whether it's been called via window-scroll-functions or via
buffer-switching, ...

lazy-lock should be declared obsolete.


        Stefan

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Alan Mackenzie]

Stefan Monnier <monnier@iro.umontreal.ca> wrote on Thu, 22 Jan 2004
18:57:08 GMT:
>> Why jit-lock?

> Lazy-lock was a neat idea to only do fontification on-the-fly: why
> fontify 1MB of code before displaying the first 40 lines ?

> But it was difficult from elisp to do it right: the hooks available to
> lazy-lock to detect when the displayed part of the screen changes and which
> parts of the buffer are currently displayed are not very satisfactory:
> difficult to use (leading to corner cases where fontification is not done
> when it should) and inefficient (lazy-lock is triggered much more often than
> necessary).  This is even more true on XEmacs and old Emacs where one of the
> hooks used by lazy-lock is (or at least was) missing.

Yes.  The commentry at the start of lazy-lock.el details the anguish
experienced by its maintainer (Simon Marshall, perhaps?) in trying to
formulate the Right hooks.  (Who said that Elisp programming isn't like
boxing?  :-)

OK.  So jit-lock was a second design of the same concept, taking
advantage of the experience gained from lazy-lock.

> jit-lock grew out of this: create one new hook designed specifically
> for this job.

fontification-functions?

> It's simpler, more reliable, faster.

I switched to lazy-lock because jit-lock was causing me problems.
[Admitted, though, that's the Emacs 21.1 version of jit-lock, not the CVS
version.]  When I had lots of buffers loaded (by desktop), jit-lock often
caused my Emacs to hang at 100% CPU usage.  [I reported this, but never
managed to debug it.]  When I was editing a large buffer, continuous
background fontification under jit- gobbled so much CPU power that I
sometimes C-z'ed Emacs to allow (say) a fetchnews run (part of leafnode,
a simple Usenet server) to finish faster.  This is on a 166MHz CPU.
Also, sometimes the refontification done after 3 seconds delay
(jit-lock-stealth-time) fouled up an already correctly fontified line.
That is because it yanks that line out of context, not taking account of
the fact it is not "at the top level".  [To see this, allow Auto Fill
Mode to split the innards of an @code{...} construct in Texinfo mode.
This problem still exists on 21.3.]

> The only tradeoff I know of (other than the fact that some of the
> features of lazy-lock have not been ported yet to jit-lock) is that
> lazy-lock has some more knowledge about the intention behind a movement
> because it can tell whether it's been called via
> window-scroll-functions or via buffer-switching, ...

> lazy-lock should be declared obsolete.

I'll need to check out an up to date jit-lock.  I have a vague feeling
that lazy- still might work better than jit- on machines with modest
CPUs, particularly when Emacs has to share the processor with other
processes.  But I'd need to check out an up to date jit-lock to test
this.

Thanks a lot for the elucidation.

>         Stefan

-- 
Alan Mackenzie (Munich, Germany)
Email: aacm@muuc.dee; to decode, wherever there is a repeated letter
(like "aa"), remove half of them (leaving, say, "a").

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Stefan Monnier]

>> jit-lock grew out of this: create one new hook designed specifically
>> for this job.
> fontification-functions?

Yes.

>> It's simpler, more reliable, faster.

> I switched to lazy-lock because jit-lock was causing me problems.
> [Admitted, though, that's the Emacs 21.1 version of jit-lock, not the CVS
> version.]  When I had lots of buffers loaded (by desktop), jit-lock often
> caused my Emacs to hang at 100% CPU usage.

These have no relationship with the underlying difference.  It only has to
do with details of how the stealth fontification takes place.  On my
laptop, I just turn off stealth-fontification with
(setq jit-lock-stealth-time nil).

> [I reported this, but never managed to debug it.]

There's nothing to debug: it's normal, well-understood behavior.
Better yet: it's done on purpose.

Please switch back to jit-lock and send bug reports so it can be improved.
But only after taking steps like (setq jit-lock-stealth-time nil) which
are needed for situations where power is not free.
Mahybe we can provide better explanations of what can be done for laptops
(to save energy) or for machines where the CPU is also needed for other
tasks (where stealth fontification might still be OK but should be nicer to
other processes: this should be taken care of with jit-lock-stealth-load,
but it might not be good enough).

> When I was editing a large buffer, continuous
> background fontification under jit- gobbled so much CPU power that I
> sometimes C-z'ed Emacs to allow (say) a fetchnews run (part of leafnode,
> a simple Usenet server) to finish faster.  This is on a 166MHz CPU.

C-h v jit-lock-stealth- TAB should list a bunch more variables you can
tweak to change the amount of CPU used in the background.

> Also, sometimes the refontification done after 3 seconds delay
> (jit-lock-stealth-time) fouled up an already correctly fontified line.
> That is because it yanks that line out of context, not taking account of
> the fact it is not "at the top level".  [To see this, allow Auto Fill
> Mode to split the innards of an @code{...} construct in Texinfo mode.
> This problem still exists on 21.3.]

font-lock simply does not know how to fontify a @code{...} construct spread
on several lines.  jit-lock is not to blame for it and lazy-lock will fail
similarly in various circumstances because it too simply inherits the
shortcomings of font-lock.

But maybe jit-lock suffers more often from it, so please send precise
bug-reports if you see cases where lazy-lock gets it consistently right
whereas jit-lock gets it wrong.

As for your problem above: I can't reproduce it with Emacs-CVS, although
I can reproduce similar problems if I keep typing after the line was cut by
auto-fill-mode.  But the problems appears as part of the on-the-fly
highlighting, not as part of the refontification that happens after 3s.
As for how to fix this problem: (setq font-lock-multiline t) might help, at
the cost of a bit slower processing (some have measured a factor
2 difference).

>> lazy-lock should be declared obsolete.

> I'll need to check out an up to date jit-lock.  I have a vague feeling
> that lazy- still might work better than jit- on machines with modest
> CPUs, particularly when Emacs has to share the processor with other
> processes.  But I'd need to check out an up to date jit-lock to test
> this.

That's quite possible, because jit-lock was tuned differently, but this can
be changed fairly easily.  The difference is not due to the fundamental
difference (i.e. which hooks are used) but to different choices of tuning
parameters: you can tune lazy-lock to use just as much CPU as jit-lock and
vice-versa in most cases.


        Stefan

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Alan Mackenzie]

Stefan Monnier <monnier@iro.umontreal.ca> wrote on Thu, 22 Jan 2004
23:37:38 GMT:

>> I switched to lazy-lock because jit-lock was causing me problems.
>> [Admitted, though, that's the Emacs 21.1 version of jit-lock, not the
>> CVS version.]  When I had lots of buffers loaded (by desktop),
>> jit-lock often caused my Emacs to hang at 100% CPU usage.

> These have no relationship with the underlying difference.  It only has to
> do with details of how the stealth fontification takes place.  On my
> laptop, I just turn off stealth-fontification with
> (setq jit-lock-stealth-time nil).

OK.

>> [I reported this, but never managed to debug it.]

> There's nothing to debug: it's normal, well-understood behavior.
> Better yet: it's done on purpose.

Surely not - Emacs _HUNG_ with stealth fontification, accepting no keypresses
(apart from a double C-g), and remained in this state overnight (> 8
hours).  As I said, I reported this [to bug-gnu-emacs, 24 Oct 2002,
"jit-lock hangs emacs in certain circumstances"], but I failed to
follow-up RMS's request for a more precise test-case.  Perhaps I should
look into this again.  Looking at this bug report again, I also said that
lazy-lock sometimes hung this way, too.  I'm sure you're right that it's
the tuning of the stealth fontification that's pertinent here, not the
difference between lazy- and jit-.

> Please switch back to jit-lock and send bug reports so it can be improved.
> But only after taking steps like (setq jit-lock-stealth-time nil) which
> are needed for situations where power is not free.
> Mahybe we can provide better explanations of what can be done for laptops
> (to save energy) or for machines where the CPU is also needed for other
> tasks (where stealth fontification might still be OK but should be nicer to
> other processes: this should be taken care of with jit-lock-stealth-load,
> but it might not be good enough).

The font lock support modes are not documented.  In fact, the reason I
posted my question last night was because I was considering contributing
this documentation myself.

Until your last post, I wasn't really aware that I could tune jit-lock to
my system.  I sort of knew, but wasn't aware.  

Tuning jit-lock is very difficult.  Using customize, even after finding
the Jit Lock customize group (difficult in itself), there is a
bewildering collection of options to set.  (I mean here bewildering to
_me_, and I think I have a better grasp of font lock than typical Emacs
users.)  It would seem that a detailed understanding of how jit-lock
works is a prerequisite for setting these options intelligently.  This is
surely not a good thing.

Perhaps what is wanted is a command like `(font-lock-tune-font-lock
'low-powered-workstation)' by which users could select from amongst
several pre-defined tunings, somewhat analogous to CC Mode's
`c-set-style'.

>> When I was editing a large buffer, continuous background fontification
>> under jit- gobbled so much CPU power that I sometimes C-z'ed Emacs to
>> allow (say) a fetchnews run (part of leafnode, a simple Usenet server)
>> to finish faster.  This is on a 166MHz CPU.

> C-h v jit-lock-stealth- TAB should list a bunch more variables you can
> tweak to change the amount of CPU used in the background.

OK.  Perhaps for me, simply disabling stealth fontification is the right
thing.

>> Also, sometimes the refontification done after 3 seconds delay
>> (jit-lock-stealth-time) fouled up an already correctly fontified line.
>> That is because it yanks that line out of context, not taking account
>> of the fact it is not "at the top level".  [To see this, allow Auto
>> Fill Mode to split the innards of an @code{...} construct in Texinfo
>> mode.  This problem still exists on 21.3.]

> font-lock simply does not know how to fontify a @code{...} construct
> spread on several lines.  jit-lock is not to blame for it and lazy-lock
> will fail similarly in various circumstances because it too simply
> inherits the shortcomings of font-lock.

> But maybe jit-lock suffers more often from it, so please send precise
> bug-reports if you see cases where lazy-lock gets it consistently right
> whereas jit-lock gets it wrong.

You're right there.  jit-lock mis-fontifies by stealth, immediately after
the line is split by Auto Fill Mode, whereas lazy-lock doesn't (even
after 30 seconds).  However, as soon as one types a further character on
the newly split line, the fontification goes wrong (in all support modes,
or none).  This isn't an important difference between jit- and lazy-.
Sorry.

> As for your problem above: I can't reproduce it with Emacs-CVS, although
> I can reproduce similar problems if I keep typing after the line was cut by
> auto-fill-mode.  But the problems appears as part of the on-the-fly
> highlighting, not as part of the refontification that happens after 3s.
> As for how to fix this problem: (setq font-lock-multiline t) might help, at
> the cost of a bit slower processing (some have measured a factor
> 2 difference).

YES, YES YES!!!!  font-lock-multiline does the trick!  [Emacs 21.3].  I
don't understand how, or why, but it works.  Thanks!

[ .... ]

>         Stefan

-- 
Alan Mackenzie (Munich, Germany)
Email: aacm@muuc.dee; to decode, wherever there is a repeated letter
(like "aa"), remove half of them (leaving, say, "a").

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Eli Zaretskii]

> From: Alan Mackenzie<none@example.invalid>
> Newsgroups: gnu.emacs.help
> Date: Fri, 23 Jan 2004 10:36:15 +0000
> 
> > There's nothing to debug: it's normal, well-understood behavior.
> > Better yet: it's done on purpose.
> 
> Surely not - Emacs _HUNG_ with stealth fontification, accepting no keypresses
> (apart from a double C-g), and remained in this state overnight (> 8
> hours).

I think these problems were solved in the meantime (IIRC, they were
due to bad interaction between JIT lock and font-lock definitions for
some modes).  Please try again and see if you still have such
problems.

> Tuning jit-lock is very difficult.  Using customize, even after finding
> the Jit Lock customize group (difficult in itself), there is a
> bewildering collection of options to set.  (I mean here bewildering to
> _me_, and I think I have a better grasp of font lock than typical Emacs
> users.)  It would seem that a detailed understanding of how jit-lock
> works is a prerequisite for setting these options intelligently.  This is
> surely not a good thing.
> 
> Perhaps what is wanted is a command like `(font-lock-tune-font-lock
> 'low-powered-workstation)' by which users could select from amongst
> several pre-defined tunings, somewhat analogous to CC Mode's
> `c-set-style'.

I have a better idea of a user interface:

        (font-lock-tune-font-lock 'my-specific-preferences)
or
	(font-lock-tune-font-lock 'my-favorite-computer)

Seriously, though: that's the first time I hear an Emacs veteran
complain about too many customization options.  Surely, you should
know that they all exist for a good purpose.  Perhaps you want to say
that their documentation needs some work--that is something I could
understand.

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Stefan Monnier]

> Surely not - Emacs _HUNG_ with stealth fontification, accepting no keypresses
> (apart from a double C-g), and remained in this state overnight (> 8
> hours).  As I said, I reported this [to bug-gnu-emacs, 24 Oct 2002,
> "jit-lock hangs emacs in certain circumstances"], but I failed to
> follow-up RMS's request for a more precise test-case.  Perhaps I should
> look into this again.  Looking at this bug report again, I also said that
> lazy-lock sometimes hung this way, too.  I'm sure you're right that it's
> the tuning of the stealth fontification that's pertinent here, not the
> difference between lazy- and jit-.

I'm pretty sure I've seen this report (although I can't remember it),
but it sounds like a different problem than just tuning, more like a real
bug, probably a bad-regexp in the font-lock settings of a major-mode (the
potentially exponential behavior of the current regexp-engine is a common
problem).  This was probably compounded by the fact that inhibit-quit is
set during stealth fontification (input is polled instead).

This inhibit-quit issue is probably the one we should fix, although it's
a bit tricky to do right (after all, you don't want to leave wrong
highlighting just because the user happened to hit C-g for some unrelated
reason).  I think we need a "sledge-hammer" variant of C-g.

> The font lock support modes are not documented.  In fact, the reason I
> posted my question last night was because I was considering contributing
> this documentation myself.

That would be extremely welcome.

> Tuning jit-lock is very difficult.  Using customize, even after finding
> the Jit Lock customize group (difficult in itself), there is a
> bewildering collection of options to set.  (I mean here bewildering to
> _me_, and I think I have a better grasp of font lock than typical Emacs
> users.)  It would seem that a detailed understanding of how jit-lock
> works is a prerequisite for setting these options intelligently.  This is
> surely not a good thing.

> Perhaps what is wanted is a command like `(font-lock-tune-font-lock
> 'low-powered-workstation)' by which users could select from amongst
> several pre-defined tunings, somewhat analogous to CC Mode's
> `c-set-style'.

I think the defaults can be improved to make manual tuning rare.  In most
cases, it should be possible for Emacs to automatically detect it's using
too much CPU time and back-off.  The only manual setting that seems
necessary is "save energy" (or is there some way for Emacs to detect
whether the computer is running on battery?) which should boild down to
turning off jit-lock-stealth-time, although it should be done in a more
obvious way (maybe a global `save-energy' variable which lazy-lock and
jit-lock and others can use).

>>> When I was editing a large buffer, continuous background fontification
>>> under jit- gobbled so much CPU power that I sometimes C-z'ed Emacs to
>>> allow (say) a fetchnews run (part of leafnode, a simple Usenet server)
>>> to finish faster.  This is on a 166MHz CPU.

>> C-h v jit-lock-stealth- TAB should list a bunch more variables you can
>> tweak to change the amount of CPU used in the background.

> OK.  Perhaps for me, simply disabling stealth fontification is the right
> thing.

If you're trying to save power, yes.  If not (i.e. if it's to yield the CPU
to other tasks) then you shouldn't need to tune anything (i.e. the defaults
should be improved).


        Stefan

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Kevin Rodgers]

Stefan Monnier wrote:

> Alan Mackenzie wrote:
> > Perhaps what is wanted is a command like `(font-lock-tune-font-lock
> > 'low-powered-workstation)' by which users could select from amongst
> > several pre-defined tunings, somewhat analogous to CC Mode's
> > `c-set-style'.
> 
> I think the defaults can be improved to make manual tuning rare.  In most
> cases, it should be possible for Emacs to automatically detect it's using
> too much CPU time and back-off.  The only manual setting that seems
> necessary is "save energy" (or is there some way for Emacs to detect
> whether the computer is running on battery?) which should boild down to
> turning off jit-lock-stealth-time, although it should be done in a more
> obvious way (maybe a global `save-energy' variable which lazy-lock and
> jit-lock and others can use).

There are a couple of add-on battery packages for Emacs:


http://groups.google.com/groups?selm=87ptqrx0ex.fsf%40pine.kuee.kyoto-u.ac.jp
http://groups.google.com/groups?selm=87ae45lhb7.fsf_-_%40tod.codefactory.se

-- 
Kevin Rodgers

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Alan Mackenzie]

Eli Zaretskii <eliz@elta.co.il> wrote on Fri, 23 Jan 2004 13:42:53 +0200:
>> From: Alan Mackenzie<none@example.invalid>

>> Surely not - Emacs _HUNG_ with stealth fontification, accepting no
>> keypresses (apart from a double C-g), and remained in this state
>> overnight (> 8 hours).

> I think these problems were solved in the meantime (IIRC, they were due
> to bad interaction between JIT lock and font-lock definitions for some
> modes).

Any chance of a revision number (presumably of jit-lock.el) where this
was fixed?

> Please try again and see if you still have such problems.

I've set font-lock-support-mode back to jit-, and I've been trying all
evening ;-)  With Emacs 21.1, I've had my Emacs lock up just once.
Loading jit-lock.elc V1.31 (the latest from savannah), I've not triggered
the problem.  Not yet.  I'll try loading a few more buffers to see if
that makes things worse.

[ .... ]

>> Perhaps what is wanted is a command like `(font-lock-tune-font-lock
>> 'low-powered-workstation)' by which users could select from amongst
>> several pre-defined tunings, somewhat analogous to CC Mode's
>> `c-set-style'.

> I have a better idea of a user interface:

>         (font-lock-tune-font-lock 'my-specific-preferences)
> or
> 	(font-lock-tune-font-lock 'my-favorite-computer)

> Seriously, though: that's the first time I hear an Emacs veteran
> complain about too many customization options.  Surely, you should
> know that they all exist for a good purpose.

I wasn't meaning to suggest that they should be removed.  Merely that
they should be supplemented with a more handy higher level tool, leaving
the individual options available for specific needs/wants.

> Perhaps you want to say that their documentation needs some work--that
> is something I could understand.

That too.

-- 
Alan Mackenzie (Munich, Germany)
Email: aacm@muuc.dee; to decode, wherever there is a repeated letter
(like "aa"), remove half of them (leaving, say, "a").

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Eli Zaretskii]

> From: Alan Mackenzie<none@example.invalid>
> Newsgroups: gnu.emacs.help
> Date: Fri, 23 Jan 2004 22:34:59 +0000
> 
> > I think these problems were solved in the meantime (IIRC, they were due
> > to bad interaction between JIT lock and font-lock definitions for some
> > modes).
> 
> Any chance of a revision number (presumably of jit-lock.el) where this
> was fixed?

Sorry, no.  And the changes were not in jit-lock.el per se, IIRC, but
rather in the font-lock definitions of several modes.

> > I have a better idea of a user interface:
> 
> >         (font-lock-tune-font-lock 'my-specific-preferences)
> > or
> > 	(font-lock-tune-font-lock 'my-favorite-computer)
> 
> > Seriously, though: that's the first time I hear an Emacs veteran
> > complain about too many customization options.  Surely, you should
> > know that they all exist for a good purpose.
> 
> I wasn't meaning to suggest that they should be removed.  Merely that
> they should be supplemented with a more handy higher level tool, leaving
> the individual options available for specific needs/wants.

I was only semi-humorous when I suggested that Emacs should itself
know about my own preferences.  If we could have that, it would be a
great UI.

However, in practice, this is next to impossible, and so, IMHO, is
your suggestion.  In the specific case in point, that of tuning JIT
lock, too many factors are specific to the peculiarities of the
individual setup of the machine on which it runs.

Instead of wasting effort to invent a tool that will certainly fail in
a large proportion of cases, Emacs lets you tune the feature with a
set of settings.  The documentation of those settings should make it
clear which ones to use in each particular case.  Perhaps if you could
describe the process that you went through when you tried to figure
out what options to change, we could identify what needs to be
improved in the documentation, or perhaps how to change the set of the
options itself.

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Alan Mackenzie]

Eli Zaretskii <eliz@elta.co.il> wrote on Sat, 24 Jan 2004 12:41:32 +0200:
>> From: Alan Mackenzie<none@example.invalid>
>> Newsgroups: gnu.emacs.help
>> Date: Fri, 23 Jan 2004 22:34:59 +0000

>> Any chance of a revision number (presumably of jit-lock.el) where this
>> was fixed?

> Sorry, no.  And the changes were not in jit-lock.el per se, IIRC, but
> rather in the font-lock definitions of several modes.

Hmmm.  Font lock definitions are (usually) just data.  It seems a bit
strange to blame data for hanging a program, rather than the program for
not handling the data.  Were you thinking of "ambiguous" regexps, whose
runtime can explode exponentially w.r.t. the length of the string they're
examining?

For the record, the modes I had loaded when I reported the bug in October
2002 were:  C and AWK (both CC Mode 5.28, a released version), Emacs Lisp
(and Lisp Interaction), Info, Diff, Shell Script, and Fundamental.  Since
then, C and AWK font lock settings have been entirely rewritten.  Of the
other modes in this list, were any of them amongst those whose font lock
definitions were rewritten?

[ .... ]

> Instead of wasting effort to invent a tool that will certainly fail in
> a large proportion of cases, Emacs lets you tune the feature with a set
> of settings [for jit-lock-mode].  The documentation of those settings
> should make it clear which ones to use in each particular case.
> Perhaps if you could describe the process that you went through when
> you tried to figure out what options to change, we could identify what
> needs to be improved in the documentation, or perhaps how to change the
> set of the options itself.

It was more that when I found the settings (on the customization page) I
recoiled in horror.  It was a feeling that I could get lost for days if I
wandered into it, possibly that I might really foul something up.  "Just
leave it alone, it seems to work sort-of OK anyhow."  That was what lead
me to suggesting a set of pre-cooked settings.

Again, the fact that there isn't any documentation for the support modes
in the Emacs manual doesn't help.  I'm going to have a go at fixing this.

-- 
Alan Mackenzie (Munich, Germany)
Email: aacm@muuc.dee; to decode, wherever there is a repeated letter
(like "aa"), remove half of them (leaving, say, "a").

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Alan Mackenzie]

Stefan Monnier <monnier@iro.umontreal.ca> wrote on Fri, 23 Jan 2004
16:29:57 GMT:
>> Surely not - Emacs _HUNG_ with stealth fontification, accepting no
>> keypresses (apart from a double C-g), and remained in this state
>> overnight (> 8 hours).  As I said, I reported this [to bug-gnu-emacs,
>> 24 Oct 2002, "jit-lock hangs emacs in certain circumstances"], but I
>> failed to follow-up RMS's request for a more precise test-case.
>> Perhaps I should look into this again.  Looking at this bug report
>> again, I also said that lazy-lock sometimes hung this way, too.  I'm
>> sure you're right that it's the tuning of the stealth fontification
>> that's pertinent here, not the difference between lazy- and jit-.

> I'm pretty sure I've seen this report (although I can't remember it),
> but it sounds like a different problem than just tuning, more like a
> real bug, probably a bad-regexp in the font-lock settings of a
> major-mode (the potentially exponential behavior of the current
> regexp-engine is a common problem).  This was probably compounded by
> the fact that inhibit-quit is set during stealth fontification (input
> is polled instead).

More info:  When Emacs hangs in this manner, a double C-g followed by
`fg' often (but not always) brings forth this:

: Garbage collection in progress; cannot auto-save now
: but will instead do a real quit after garbage collection ends
: Abort (and dump core)? (y or n)

On replying `n', Emacs frees up again.  Left undisturbed it frequently
goes back into the hung state.

I conjecture that complicated regexps in font-lock settings are somehow
creating garbage at a fabulous rate, and jit-lock and the GC are somehow
thrashing.  Just as soon as jit-lock can be halted and the GC left in
peace, things return to OK.

I've just noticed something: jit-lock-stealth-load seems to be set by
default to 200%.  This seems strange.  Should this perhaps be 20%.  ;-)  

> This inhibit-quit issue is probably the one we should fix, although
> it's a bit tricky to do right (after all, you don't want to leave wrong
> highlighting just because the user happened to hit C-g for some
> unrelated reason).  I think we need a "sledge-hammer" variant of C-g.

[ .... ]

>> OK.  Perhaps for me, simply disabling stealth fontification is the
>> right thing.

> If you're trying to save power, yes.  If not (i.e. if it's to yield the
> CPU to other tasks) then you shouldn't need to tune anything (i.e. the
> defaults should be improved).

OK.

>         Stefan

-- 
Alan Mackenzie (Munich, Germany)
Email: aacm@muuc.dee; to decode, wherever there is a repeated letter
(like "aa"), remove half of them (leaving, say, "a").

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Stefan Monnier]

The following message is a courtesy copy of an article
that has been posted to gnu.emacs.help as well.

[ I suggest we move this to emacs-devel, which I Cc'd. ]

>> I'm pretty sure I've seen this report (although I can't remember it),
>> but it sounds like a different problem than just tuning, more like a
>> real bug, probably a bad-regexp in the font-lock settings of a
>> major-mode (the potentially exponential behavior of the current
>> regexp-engine is a common problem).  This was probably compounded by
>> the fact that inhibit-quit is set during stealth fontification (input
>> is polled instead).

> More info:  When Emacs hangs in this manner, a double C-g followed by
> `fg' often (but not always) brings forth this:

> : Garbage collection in progress; cannot auto-save now
> : but will instead do a real quit after garbage collection ends
> : Abort (and dump core)? (y or n)

> On replying `n', Emacs frees up again.  Left undisturbed it frequently
> goes back into the hung state.

> I conjecture that complicated regexps in font-lock settings are somehow
> creating garbage at a fabulous rate, and jit-lock and the GC are somehow
> thrashing.  Just as soon as jit-lock can be halted and the GC left in
> peace, things return to OK.

Hmm... the regexp-engine does not generate any garbage.  If you're inside
the GC, that means you're not inside the regexp-matcher.  So clearly the
bug is different.  A backtrace (and an xbacktrace) would be helpful.

> I've just noticed something: jit-lock-stealth-load seems to be set by
> default to 200%.  This seems strange.  Should this perhaps be 20%.  ;-)

Have you tried to change it?  200% indeed sounds too high.  But on my
single-user Opteron workstation, with no particular background task
I currently have 16% of CPU use, so a limit of 20% might be a bit low.
Maybe 50% ?

Playing with jit-lock-stealth-nice is also a good idea.  The only problem
with it is that if your machine is very fast and takes 0.001s per chunk,
waiting 0.125s between each chunk leads to ridiculously low CPU usage,
whereas if each chunk takes 1s, then waiting 0.125s between chunks still
leads to more than 80% CPU usage.  So maybe jit-lock-stealth should measure
the time it takes for it to process one chunk and dynamically adapt
jit-lock-stealth-nice so as to produce a given CPU usage.


        Stefan

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Stefan Monnier]

[ I suggest we move this to emacs-devel, which I Cc'd. ]

>> I'm pretty sure I've seen this report (although I can't remember it),
>> but it sounds like a different problem than just tuning, more like a
>> real bug, probably a bad-regexp in the font-lock settings of a
>> major-mode (the potentially exponential behavior of the current
>> regexp-engine is a common problem).  This was probably compounded by
>> the fact that inhibit-quit is set during stealth fontification (input
>> is polled instead).

> More info:  When Emacs hangs in this manner, a double C-g followed by
> `fg' often (but not always) brings forth this:

> : Garbage collection in progress; cannot auto-save now
> : but will instead do a real quit after garbage collection ends
> : Abort (and dump core)? (y or n)

> On replying `n', Emacs frees up again.  Left undisturbed it frequently
> goes back into the hung state.

> I conjecture that complicated regexps in font-lock settings are somehow
> creating garbage at a fabulous rate, and jit-lock and the GC are somehow
> thrashing.  Just as soon as jit-lock can be halted and the GC left in
> peace, things return to OK.

Hmm... the regexp-engine does not generate any garbage.  If you're inside
the GC, that means you're not inside the regexp-matcher.  So clearly the
bug is different.  A backtrace (and an xbacktrace) would be helpful.

> I've just noticed something: jit-lock-stealth-load seems to be set by
> default to 200%.  This seems strange.  Should this perhaps be 20%.  ;-)

Have you tried to change it?  200% indeed sounds too high.  But on my
single-user Opteron workstation, with no particular background task
I currently have 16% of CPU use, so a limit of 20% might be a bit low.
Maybe 50% ?

Playing with jit-lock-stealth-nice is also a good idea.  The only problem
with it is that if your machine is very fast and takes 0.001s per chunk,
waiting 0.125s between each chunk leads to ridiculously low CPU usage,
whereas if each chunk takes 1s, then waiting 0.125s between chunks still
leads to more than 80% CPU usage.  So maybe jit-lock-stealth should measure
the time it takes for it to process one chunk and dynamically adapt
jit-lock-stealth-nice so as to produce a given CPU usage.


        Stefan

Re: What is the _essential_ difference between lazy-lock and jit-lock?

[Eli Zaretskii]

> From: Alan Mackenzie<none@example.invalid>
> Newsgroups: gnu.emacs.help
> Date: Mon, 26 Jan 2004 17:53:46 +0000
> 
> > Sorry, no.  And the changes were not in jit-lock.el per se, IIRC, but
> > rather in the font-lock definitions of several modes.
> 
> Hmmm.  Font lock definitions are (usually) just data.  It seems a bit
> strange to blame data for hanging a program, rather than the program for
> not handling the data.

The program does handle the data, it just takes a long time and lots
of CPU power to do it.

> Were you thinking of "ambiguous" regexps, whose
> runtime can explode exponentially w.r.t. the length of the string they're
> examining?

IIRC, one problem was that font-lock definitions forced Emacs to scan
large portions of the buffer in order to decide how to fontify.  When
such scans need to be done backwards for every word, it could take a
lot of CPU cycles.

> For the record, the modes I had loaded when I reported the bug in October
> 2002 were:  C and AWK (both CC Mode 5.28, a released version), Emacs Lisp
> (and Lisp Interaction), Info, Diff, Shell Script, and Fundamental.  Since
> then, C and AWK font lock settings have been entirely rewritten.  Of the
> other modes in this list, were any of them amongst those whose font lock
> definitions were rewritten?

I don't really know ("cvs annotate" should tell you, right?), but
Shell Script mode sounds like a good candidate.