Stealth fontification GCs a lot

Stealth fontification GCs a lot

[Richard Stallman]

I have been trying Font Lock mode, which I never used to use before.
I notice that when stealth fontification is occurring, Emacs spends
about 1/3 of its time GCing.  The frequent GCs make for annoying
latency when I type another command.

Could someone investigate why so much consing goes on during
fontification, and try to make it more efficient?

Re: Stealth fontification GCs a lot

[Stefan Monnier]

> I have been trying Font Lock mode, which I never used to use before.
> I notice that when stealth fontification is occurring, Emacs spends
> about 1/3 of its time GCing.  The frequent GCs make for annoying
> latency when I type another command.

I haven't looked at the consing behavior, but I'll just mention that I find
stealth fontification annoying and have turned it off a while ago.
Many people have reported problems with it (typically they complain that
Emacs uses a lot of resources even when it's not used, which sucks when
you're running on batteries).
I haven't heard of any good reason why it should be enabled by default
(the argument that it forces us to fix it doesn't hold water since it
still hasn't been fixed after all these years).


        Stefan

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 10 Apr 2005 23:40:14 -0400
> Cc: emacs-devel@gnu.org
> 
> I haven't looked at the consing behavior, but I'll just mention that I find
> stealth fontification annoying and have turned it off a while ago.
> Many people have reported problems with it (typically they complain that
> Emacs uses a lot of resources even when it's not used, which sucks when
> you're running on batteries).

FWIW, I didn't see any such adverse effects of stealth fontification
on any of the machines I work on, although I do remember reading
complaints like you mention above.  For the record, I have in my
~/.emacs these customizations related to stealth fontification:

  (setq jit-lock-stealth-time 16)
  (setq jit-lock-stealth-nice 0.5)
  (setq jit-lock-stealth-verbose t)
  (setq jit-lock-defer-contextually t)

Re: Stealth fontification GCs a lot

[Richard Stallman]

    I haven't looked at the consing behavior, but I'll just mention that I find
    stealth fontification annoying and have turned it off a while ago.

    I haven't heard of any good reason why it should be enabled by default
    (the argument that it forces us to fix it doesn't hold water since it
    still hasn't been fixed after all these years).

I will try turning it off.  But I wonder if just making it wait
longer would make it a desirable feature.  What do you think?

Re: Stealth fontification GCs a lot

[Ralf Angeli]

* Richard Stallman (2005-04-12) writes:

> I will try turning it off.  But I wonder if just making it wait
> longer would make it a desirable feature.  What do you think?

Turning of stealth font locking will result in some drawbacks for the
folding feature of AUCTeX.  Folding in this context means that certain
LaTeX constructs or just their markup is hidden.  Instead a
placeholder or the content of the construct is shown.  The original
content is revealed and available for editing, as soon as point is
moved into the folded area.

For example the text

  This is some \emph{important} text.\footnote{Confer other examples.}

will be shown as

  This is some /important/ text.[f]

(where /.../ means that the enclosing text is displayed in italics).

A standard usage pattern is to open a LaTeX file and perform folding
for the whole buffer.

Now how is this related to stealth font locking?  In order for the
"important" string in the example above to be displayed in italics,
the original string inside the \emph construct has to be fontified
already because folding will pick up the original string with all its
text properties and place it into an overlay.  We currently don't have
a mechanism similar to jit-lock, so folding a whole buffer is in fact
done immediately.  In order to get a correct fontification of the
generated overlays we currently advise users to wait a few seconds
after opening a LaTeX file before doing this operation in order to let
stealth font locking fontify the whole buffer.

For me personally this workaround is working quite well.  But if
stealth font looking was disabled we'd have to find a different way of
achieving this.  It is planned to hook somehow into jit-lock for this,
but I haven't found a good and reliable way for it yet.  Another
possibility would be to re-enable stealth font locking for buffers
under AUCTeX's control (in case folding mode is activated).

-- 
Ralf

Re: Stealth fontification GCs a lot

[Richard Stallman]

I will try your settings.
Maybe they should be the defaults.

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Tue, 12 Apr 2005 13:25:00 -0400
> 
> I will try your settings.

You initially complained about too much consing; I don't claim that
these settings alleviate that (I simply never checked this aspect).
All I'm saying is that with those settings my CPU meter never goes
above a few percents when the system is idle except for Emacs stealth
fontification.

> Maybe they should be the defaults.

Since Stefan says the default settings annoy him, I'd like Stefan to
try mine, compare to what he sees with the defaults, and report CPU
usage for both cases.  If he says the effect is good, that could
indeed be a reason for changing the defaults.

Re: Stealth fontification GCs a lot

[Stefan Monnier]

> Since Stefan says the default settings annoy him, I'd like Stefan to
> try mine, compare to what he sees with the defaults, and report CPU
> usage for both cases.  If he says the effect is good, that could
> indeed be a reason for changing the defaults.

The problem is that I've never noticed any positive effect of stealth
fontification, whereas any setting, no matter how low will eat up my battery
for no good reason.


        Stefan

Re: Stealth fontification GCs a lot

[Richard Stallman]

    Turning of stealth font locking will result in some drawbacks for the
    folding feature of AUCTeX.  Folding in this context means that certain
    LaTeX constructs or just their markup is hidden.

Since stealth font locking proceeds unpredictably, a feature
that depends on it cannot be reliable.

      In order to get a correct fontification of the
    generated overlays we currently advise users to wait a few seconds
    after opening a LaTeX file before doing this operation in order to let
    stealth font locking fontify the whole buffer.

That's terrible!  If fontification of the whole file is fast enough
that users can stand to wait for it, what you really should do is
force font-locking to completion immediately.

Re: Stealth fontification GCs a lot

[Ralf Angeli]

* Richard Stallman (2005-04-13) writes:

>       In order to get a correct fontification of the
>     generated overlays we currently advise users to wait a few seconds
>     after opening a LaTeX file before doing this operation in order to let
>     stealth font locking fontify the whole buffer.
>
> That's terrible!

Well, I didn't write that I like it like this.  It is merely a
workaround until a proper implementation dealing with this problem is
done.

> If fontification of the whole file is fast enough
> that users can stand to wait for it, what you really should do is
> force font-locking to completion immediately.

Currently users can start editing the document and stealth
fontification will fontify the buffer during editing pauses.  So I
don't think this usually is like opening the file and staring at the
screen till fontification finishes.

But I see your point and I'll check if forcing the whole buffer to be
fontified before folding leads to bearable execution times.  An option
could also be to make this configurable; maybe based on processor
speed and buffer sizes.

-- 
Ralf

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 12 Apr 2005 15:52:47 -0400
> Cc: rms@gnu.org, emacs-devel@gnu.org
> 
> The problem is that I've never noticed any positive effect of stealth
> fontification

It decreases the time it takes to page through a file.

> whereas any setting, no matter how low will eat up my battery for no
> good reason.

Perhaps we should have a primitive that checks for a laptop running on
batteries, so that customizations and background ops could test for it
and behave accordingly.

Re: Stealth fontification GCs a lot

[Stefan Monnier]

>> The problem is that I've never noticed any positive effect of stealth
>> fontification
> It decreases the time it takes to page through a file.

That's the theory, I'm talking about practice.

>> whereas any setting, no matter how low will eat up my battery for no
>> good reason.

> Perhaps we should have a primitive that checks for a laptop running on
> batteries, so that customizations and background ops could test for it
> and behave accordingly.

and now we also have power-control on servers, not to prolong battery time
but to reduce heat.  I really don't think it's worth the trouble to try and
recognize "those cases where we can do it without harm".


        Stefan

Re: Stealth fontification GCs a lot

[Richard Stallman]

    The problem is that I've never noticed any positive effect of stealth
    fontification, whereas any setting, no matter how low will eat up my battery
    for no good reason.

Aside from the peculiar AUCTeX issue, does anyone see a benefit in
stealth fontification?  Is the idea that it avoids delays when you move
to other parts of the buffer, because they've been fontified while
you were on the phone?

Re: Stealth fontification GCs a lot

[Richard Stallman]

Your stealth fontification settings seem much better to me, too.
Whether or not stealth fontification should be on by defaults, they
seem to be better parameters for it.

Re: Stealth fontification GCs a lot

[Aidan Kehoe]

 Ar an triú lá déag de mí Aibréan, scríobh Richard Stallman: 

 >     The problem is that I've never noticed any positive effect of stealth
 >     fontification, whereas any setting, no matter how low will eat up my
 >     battery for no good reason.
 > 
 > Aside from the peculiar AUCTeX issue, does anyone see a benefit in
 > stealth fontification?  Is the idea that it avoids delays when you move
 > to other parts of the buffer, because they've been fontified while
 > you were on the phone?

I use and appreciate stealth fontification, because while I care about
different portions of my source files appearing with different colours based
on their syntactic functions, I care *much more* about seeing those source
files *immediately,* when I open the file in question, because there’s a
very good chance I opened the file just to see its contents. As such,
stealth fontification allows me to have both fontification and immediate
text display.

-- 
“I, for instance, am gung-ho about open source because my family is being
held hostage in Rob Malda’s basement. But who fact-checks me, or Enderle,
when we say something in public? No-one!” -- Danny O’Brien

Re: Stealth fontification GCs a lot

[Stefan Monnier]

> I use and appreciate stealth fontification, because while I care about
> different portions of my source files appearing with different colours based
> on their syntactic functions, I care *much more* about seeing those source
> files *immediately,* when I open the file in question, because there’s a
> very good chance I opened the file just to see its contents. As such,
> stealth fontification allows me to have both fontification and immediate
> text display.

I don't understand.  It seems that what you describe is provided by the
just-in-time part of jit-lock, not by the stealth part.


        Stefan

Re: Stealth fontification GCs a lot

[Miles Bader]

On 4/14/05, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> >> The problem is that I've never noticed any positive effect of stealth
> >> fontification
> > It decreases the time it takes to page through a file.
> 
> That's the theory, I'm talking about practice.

FWIW, I've been annoyed in the past by sluggish response caused by
fontification upon display; not recently though, i'm not sure if
that's because of stealth fontification or not.  I'm not referring to
"total paging time" or any such thing, btw, just a sort feeling of
"lag" upon entering a new screen (often when I've used i-search to
skip way ahead in a big file).

I don't have a good feel for the tradeoffs though.  I think the
problem is for most people one of perception more than anything else
-- yeah the laptop case is an exception to this.

-Miles
-- 
Do not taunt Happy Fun Ball.

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: emacs-devel@gnu.org
> Date: Wed, 13 Apr 2005 14:31:38 -0400
> 
> Your stealth fontification settings seem much better to me, too.
> Whether or not stealth fontification should be on by defaults, they
> seem to be better parameters for it.

If no one objects, I can install them as default values.

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Wed, 13 Apr 2005 14:30:36 -0400
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> 
> Is the idea that it avoids delays when you move to other parts of
> the buffer, because they've been fontified while you were on the
> phone?

Yes.

Re: Stealth fontification GCs a lot

[Jonathan Yavner]

(Please pardon the delurk.)

In HAPE mode, I locally set `jit-lock-stealth-nice' to 999, with the comment
  ;;Stealth fontification on a large HAPE database consumes vast
  ;;quantities of RAM and is unlikely to be useful.

I love JIT font-lock and depend on it for proper operation of HAPE, but 
stealth-fontification is pointless when working on a 2MB file that's heavily 
fontified but probably much less than 1% of it will be looked at during any 
one editing session.

Maybe stealth font-lock is pointless all the time.
-- 
   ___
  (   >                 _/_  /
   __/________.   ___.  /   /_   __.   _.
  / /  (_)  / )__<__(__/___/ /__<__(__/ )__
 <_/   jyavner@member.fsf.org

Re: Stealth fontification GCs a lot

[Miles Bader]

On 4/14/05, Jonathan Yavner <jyavner@member.fsf.org> wrote:
> Maybe stealth font-lock is pointless all the time.

Well that's clearly not true.  I think it is pretty hard to tune
because you'd have to guess what the user's going to do...

Maybe it should only try to stealth-fontify some amount of text before
and after the current point (e.g., 100 lines before and after; if the
user moves, then just give up on the old position...), which would
provide a bit of buffering to make page-movement fast, but not spend
ages fontifying the entirety of giant source files.

-Miles
-- 
Do not taunt Happy Fun Ball.

Re: Stealth fontification GCs a lot

[David Kastrup]

Miles Bader <snogglethorpe@gmail.com> writes:

> On 4/14/05, Jonathan Yavner <jyavner@member.fsf.org> wrote:
>> Maybe stealth font-lock is pointless all the time.
>
> Well that's clearly not true.  I think it is pretty hard to tune
> because you'd have to guess what the user's going to do...
>
> Maybe it should only try to stealth-fontify some amount of text
> before and after the current point (e.g., 100 lines before and
> after; if the user moves, then just give up on the old position...),
> which would provide a bit of buffering to make page-movement fast,
> but not spend ages fontifying the entirety of giant source files.

Sounds reasonable.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: Stealth fontification GCs a lot

[Ralf Angeli]

* Miles Bader (2005-04-14) writes:

> Maybe it should only try to stealth-fontify some amount of text before
> and after the current point (e.g., 100 lines before and after; if the
> user moves, then just give up on the old position...), which would
> provide a bit of buffering to make page-movement fast, but not spend
> ages fontifying the entirety of giant source files.

I don't really mind an initial fontification of the whole buffer.  A
worse annoyance of stealth fontification is that this happens again
and again after every modification of the buffer.

-- 
Ralf

Re: Stealth fontification GCs a lot

[Richard Stallman]

    But I see your point and I'll check if forcing the whole buffer to be
    fontified before folding leads to bearable execution times.

If it is too slow, here's another possible idea: explicitly handle
precisely and only the part of fontification that is needed for the
other features to work.  That might be substantially faster.

Re: Stealth fontification GCs a lot

[Richard Stallman]

    Perhaps we should have a primitive that checks for a laptop running on
    batteries, so that customizations and background ops could test for it
    and behave accordingly.

I suppose we could try running the `apm' program to determine this.
If the machine is not a laptop, it usually won't be installed.

Is there any better way?

    and now we also have power-control on servers, not to prolong battery time
    but to reduce heat.

That's a less pressing issue; stealth fontification at a suitable
low level, such as we get with Eli's settings, might well be desirable
anyway on such a server.

Re: Stealth fontification GCs a lot

[Ralf Angeli]

* Richard Stallman (2005-04-14) writes:

>     But I see your point and I'll check if forcing the whole buffer to be
>     fontified before folding leads to bearable execution times.
>
> If it is too slow, here's another possible idea: explicitly handle
> precisely and only the part of fontification that is needed for the
> other features to work.  That might be substantially faster.

Hm, I don't see an easy way to determine such a subset.  A possibility
could be to call `font-lock-fontify-region' only on the regions to be
folded.  But this may lead to wrong results in case there are
constructs enclosing such regions which would influence fontification.

I already commited a patch to AUCTeX's CVS.  The function responsible
for folding a buffer now calls `jit-lock-fontify-now' in case
jit-lock-mode is active and the function is available.  In the tests I
did, the increase of execution time for folding a buffer was barely
noticeable.  In case people will have problems with it, there is a
switch to inhibit forced fontification.

Thanks for your suggestions.

-- 
Ralf

Re: Stealth fontification GCs a lot

[Richard Stallman]

    > Is the idea that it avoids delays when you move to other parts of
    > the buffer, because they've been fontified while you were on the
    > phone?

    Yes.

Does anyone observe actual benefit from this?

Re: Stealth fontification GCs a lot

[Richard Stallman]

    > Your stealth fontification settings seem much better to me, too.
    > Whether or not stealth fontification should be on by defaults, they
    > seem to be better parameters for it.

    If no one objects, I can install them as default values.

Please wait 3 days for objections, then do so.

Re: Stealth fontification GCs a lot

[Richard Stallman]

    I love JIT font-lock and depend on it for proper operation of HAPE, but 
    stealth-fontification is pointless when working on a 2MB file that's heavily 
    fontified but probably much less than 1% of it will be looked at during any 
    one editing session.

Perhaps stealth fontification should be turned off for files larger
than a certain size.

Or perhaps with Eli's revised parameters it is so harmless that you
may as well have it enabled for these files too.  After all, suppose
you visit the file and then spend half an hour on the phone.  It could
have fontified the whole thing for you in that much time--so why
not do so?

    Maybe it should only try to stealth-fontify some amount of text before
    and after the current point (e.g., 100 lines before and after; if the
    user moves, then just give up on the old position...),

That might be a good idea, but 100 lines is clearly too little.
500 or 1000 might be better.

Re: Stealth fontification GCs a lot

[Richard Stallman]

    I don't really mind an initial fontification of the whole buffer.  A
    worse annoyance of stealth fontification is that this happens again
    and again after every modification of the buffer.

Are you saying that stealth fontification rechecks the whole buffer
after any change in it?

Re: Stealth fontification GCs a lot

[Ralf Angeli]

* Richard Stallman (2005-04-15) writes:

>     I don't really mind an initial fontification of the whole buffer.  A
>     worse annoyance of stealth fontification is that this happens again
>     and again after every modification of the buffer.
>
> Are you saying that stealth fontification rechecks the whole buffer
> after any change in it?

I often have that impression.  Now I actually looked at the code and
it does not refontify the whole buffer but the buffer downwards from
the position a change happened.  The doc string of
`jit-lock-after-change' explains that:

(defun jit-lock-after-change (start end old-len)
  "Mark the rest of the buffer as not fontified after a change.
Installed on `after-change-functions'.
START and END are the start and end of the changed text.  OLD-LEN
is the pre-change length.
This function ensures that lines following the change will be refontified
in case the syntax of those lines has changed.  Refontification
will take place when text is fontified stealthily."

-- 
Ralf

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Ralf Angeli <angeli@iwi.uni-sb.de>
> Date: Thu, 14 Apr 2005 13:25:27 +0200
> 
> I don't really mind an initial fontification of the whole buffer.

??? Really?  For large buffers this takes an annoyingly large amount
of time.

> A worse annoyance of stealth fontification is that this happens
> again and again after every modification of the buffer.

In practice, such re-fontification ends very quickly.  So I don't
understand why it is an annoyance for you; can you give a specific
example?

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: emacs-devel@gnu.org
> Date: Thu, 14 Apr 2005 15:03:57 -0400
> 
>     > Is the idea that it avoids delays when you move to other parts of
>     > the buffer, because they've been fontified while you were on the
>     > phone?
> 
>     Yes.
> 
> Does anyone observe actual benefit from this?

I do.  After stealth fontification finishes, paging through the file
is much faster.

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Jonathan Yavner <jyavner@member.fsf.org>
> Date: Thu, 14 Apr 2005 00:17:50 -0400
> 
> I love JIT font-lock and depend on it for proper operation of HAPE, but 
> stealth-fontification is pointless when working on a 2MB file that's heavily 
> fontified but probably much less than 1% of it will be looked at during any 
> one editing session.
> 
> Maybe stealth font-lock is pointless all the time.

Or maybe HAPE files need not be fontified at all, since you say they
are hardly looked at.

Re: Stealth fontification GCs a lot

[Ralf Angeli]

* Eli Zaretskii (2005-04-15) writes:

>> From: Ralf Angeli <angeli@iwi.uni-sb.de>
>> Date: Thu, 14 Apr 2005 13:25:27 +0200
>> 
>> A worse annoyance of stealth fontification is that this happens
>> again and again after every modification of the buffer.
>
> In practice, such re-fontification ends very quickly.  So I don't
> understand why it is an annoyance for you; can you give a specific
> example?

I mostly seeing this in large LaTeX files.  As an example, create a
new .tex file and copy the following lines repeatedly without blank
lines in between into the buffer until you have, say, 10.000 lines:

\newcommand{\mycontent}{\ensuremath{\lambda_{1}}}%
\begin{itemize}%
\item
  \begin{align*}%
    \left\{
      \begin{array}{l}
        b_{c} \text{some long line with $\mycontent$}
      \end{array}
    \right\}
  \end{align*}%
\end{itemize}%

If you want to test this in clean room conditions, save the file,
start an Emacs instance with `emacs -Q', open the file, type `M-x
font-lock-mode RET' and wait.  It's good to have something handy for
monitoring the CPU usage, e.g. top or the CPU meter of GKrellM.  Now
wait for a few seconds until the initial stealth fontification starts.
You should see that CPU usage is growing until the whole buffer is
fontified.  In order to see the effect of re-fontification after a
change (which is done only from the point of change downwards and not
for the whole buffer as already mentioned in the reply to Richard),
insert a random character somewhere at the start of the buffer and
wait again until stealth fontification starts.  This second
fontification pass takes nearly as long as the first because of the
change in the buffer happening near its top.

-- 
Ralf

Re: Stealth fontification GCs a lot

[Richard Stallman]

    I often have that impression.  Now I actually looked at the code and
    it does not refontify the whole buffer but the buffer downwards from
    the position a change happened.

That's half the buffer, on the average.

I think the other changes being considered would keep
the problem down to an acceptable level.

Re: Stealth fontification GCs a lot

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: emacs-devel@gnu.org
> Date: Thu, 14 Apr 2005 15:03:58 -0400
> 
>     > Your stealth fontification settings seem much better to me, too.
>     > Whether or not stealth fontification should be on by defaults, they
>     > seem to be better parameters for it.
> 
>     If no one objects, I can install them as default values.
> 
> Please wait 3 days for objections, then do so.

Done.