Re: Advising in cc-awk.el and namespace

Re: Advising in cc-awk.el and namespace

[Alan Mackenzie]

[emacs-devel added to the To:]

On Wed, 9 Jul 2003, Stefan Monnier wrote:


>Regarding c-awk-advise-fl-for-awk-region:

A quick summary for others:  c-awk-advise-fl-for-awk-region is a macro in
cc-awk.el which advises four font-lock functions (e.g.
font-lock-after-change-function) which get called as
after-change-functions.  The advice changes the BEGIN and END arguments
to encompass the complete region which needs refontifying.  This is done
by syntactically analysing the AWK mode buffer.

>It would really be helpful if package maintainers could tell us when
>they need an `advice'.  It's generally a sign of a shortcoming in the
>basic functionality and so we need to know about it.

I posted  "Subject: Font-lock: Major mode should be able to specify
region to fontify." in gnu.emacs.bug on 10 May 2002 21:31:03 +0200.  This
post contained tentative patches for font-lock.el, jit-lock.el and
lazy-lock.el.   It didn't spark off much discussion.

Personally, I dislike the way font-locking is done at the moment.
jit-lock-mode, though it speeds up the response of Emacs when first
loading a file, is very wasteful of processor cycles.  On my 166MHz
machine (OK - but _somebody_'s got to keep pace with Dan J.  :-), if I
load a large file (say, ..../emacs/src/keyboard.c) wait for jit-lock to
finish fontifying it in the background, then make a small change near the
beginning of this file, needlessly refontifying ~320kBytes consumes ~55%
of the CPU's power for 3½ minutes.

I have jit-lock-defer-contextually set to t, which is what causes the
above.  However, if the major mode were to specify the region to
font-lock, jit-lock deferred "contextual" fontification would never be
needed.

Also, jit-lock sometimes MIS-fontifies code (for example, in Texinfo
mode, if Auto Fill mode breaks up an @code{...} thing, for example, the
new line gets the proper face to begin with, but loses it 3 seconds
later.  This happens because jit-lock yanks that line from its syntactic
context before refontifying it.  I'm talking about the jit-lock that was
shipped with Emacs 21.1.  Sorry if these things have since been fixed.

[ .... ]

>	Stefan

-- 
Alan Mackenzie (Munich, Germany)

Re: Advising in cc-awk.el and namespace

[Stefan Monnier]

> >Regarding c-awk-advise-fl-for-awk-region:
> 
> A quick summary for others:  c-awk-advise-fl-for-awk-region is a macro in
> cc-awk.el which advises four font-lock functions (e.g.
> font-lock-after-change-function) which get called as
> after-change-functions.  The advice changes the BEGIN and END arguments
> to encompass the complete region which needs refontifying.  This is done
> by syntactically analysing the AWK mode buffer.

In Emacs-21, you should be able to use font-lock-multiline (the variable
or the text-property) to get this effect.

> >It would really be helpful if package maintainers could tell us when
> >they need an `advice'.  It's generally a sign of a shortcoming in the
> >basic functionality and so we need to know about it.
> 
> I posted  "Subject: Font-lock: Major mode should be able to specify
> region to fontify." in gnu.emacs.bug on 10 May 2002 21:31:03 +0200.  This
> post contained tentative patches for font-lock.el, jit-lock.el and
> lazy-lock.el.   It didn't spark off much discussion.

I hereby apologize for complaining when the fault was mine: I completely
missed it (I normally read gnu.emacs.bug carefully, especially when there's
`font-lock' in the subject, so I hope I was victim of nntp's unreliability).

> Personally, I dislike the way font-locking is done at the moment.
> jit-lock-mode, though it speeds up the response of Emacs when first
> loading a file, is very wasteful of processor cycles.  On my 166MHz
> machine (OK - but _somebody_'s got to keep pace with Dan J.  :-), if I
> load a large file (say, ..../emacs/src/keyboard.c) wait for jit-lock to
> finish fontifying it in the background, then make a small change near the
> beginning of this file, needlessly refontifying ~320kBytes consumes ~55%
> of the CPU's power for 3½ minutes.

You might want to set jit-lock-stealth-time to nil.

> I have jit-lock-defer-contextually set to t, which is what causes the
> above.  However, if the major mode were to specify the region to
> font-lock, jit-lock deferred "contextual" fontification would never be
> needed.

It is necessary to keep some state (currently it's limited to the
state returned by syntax-ppss) so that you can refontify one
line of a string without having to refontify the whole string
(which might span many lines).  So specifying the region to refontify
(as do font-lock-multiline and your suggested patch) is not sufficient
because it makes Emacs unbearably slow when typing into such a large
region that gets completely refontified each time (the most glaring
example I know of is with smerge-mode when typing into a large
conflict region).

OTOH, there should be a way for the major mode to add some state info
to font-lock when that state cannot be encoded in the output of
`syntax-ppss' (or better yet, a way to add it to syntax-ppss).
In the case of awk, I believe the state can be (and is) made available
in the output of syntax-ppss, so you could use that in your font-lock
settings to know how to fontify part of a string.

It's poorly documented and there's no hard and fast rule of how to do it
(there are several approaches, each with tradeoffs and limitations), so
I'd need to investigate a bit more before telling you how it could work.

> Also, jit-lock sometimes MIS-fontifies code (for example, in Texinfo
> mode, if Auto Fill mode breaks up an @code{...} thing, for example, the
> new line gets the proper face to begin with, but loses it 3 seconds
> later.

I believe it works correctly if you set font-lock-multiline to t (although
there might still be corner cases where it will fail).

> This happens because jit-lock yanks that line from its syntactic
> context before refontifying it.  I'm talking about the jit-lock that was
> shipped with Emacs 21.1.  Sorry if these things have since been fixed.

These are all related to "patterns that span more than one line" which
is generally recognized as a hard problem.


	Stefan


PS: By the way, the comment in cc-fonts.el that says:

  ;; `c-invalid-face-name' is used.  Since font-lock in Emacs expands
  ;; all face names in `font-lock-keywords' as variables we need to have
  ;; a variable for it that resolves to its own name.

is wrong.  It's just that font-lock-keywords specify expressions
(which are passed to `eval') so if you want to specify a face rather
than a variable containing a face you just need to quote the face name.



-------------------------------------------------------
This SF.Net email sponsored by: Parasoft
Error proof Web apps, automate testing & more.
Download & eval WebKing and get a free book.
www.parasoft.com/bulletproofapps1

Re: Advising in cc-awk.el and namespace

[Martin Stjernholm]

"Stefan Monnier" <monnier+gnu/emacs/pretest@cs.yale.edu> wrote:

> PS: By the way, the comment in cc-fonts.el that says:
>
>   ;; `c-invalid-face-name' is used.  Since font-lock in Emacs expands
>   ;; all face names in `font-lock-keywords' as variables we need to have
>   ;; a variable for it that resolves to its own name.
>
> is wrong.  It's just that font-lock-keywords specify expressions
> (which are passed to `eval') so if you want to specify a face rather
> than a variable containing a face you just need to quote the face name.

Thanks for the clarification. The reason for that was to cope with
XEmacs which doesn't evaluate it the same way. But I now noticed that
it too can handle an extra quotation correctly.


-------------------------------------------------------
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0

Re: Advising in cc-awk.el and namespace

[Richard Stallman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

    load a large file (say, ..../emacs/src/keyboard.c) wait for jit-lock to
    finish fontifying it in the background, then make a small change near the
    beginning of this file, needlessly refontifying ~320kBytes consumes ~55%
    of the CPU's power for 3½ minutes.

Why does a small change near the beginning refontify the whole file?
I suppose there are some small changes that might legitimately do so,
but I would not expect most small changes to do so.  What's up?

      I'm talking about the jit-lock that was
    shipped with Emacs 21.1.

Can you please try the latest version?  It is the only way to give
useful feedback.


-------------------------------------------------------
This SF.Net email sponsored by: Parasoft
Error proof Web apps, automate testing & more.
Download & eval WebKing and get a free book.
www.parasoft.com/bulletproofapps1

Re: Advising in cc-awk.el and namespace

[Stefan Monnier]

>     load a large file (say, ..../emacs/src/keyboard.c) wait for jit-lock to
>     finish fontifying it in the background, then make a small change near the
>     beginning of this file, needlessly refontifying ~320kBytes consumes ~55%
>     of the CPU's power for 3½ minutes.
> 
> Why does a small change near the beginning refontify the whole file?
> I suppose there are some small changes that might legitimately do so,
> but I would not expect most small changes to do so.  What's up?

Because in general, it might be necessary and font-lock does
not try to be clever here.

Basically syntactic-fontification (fontification of comments and string)
depends on the output of `syntax-ppss' and because font-lock-keyword
rules are order-dependent (they typically are only applied if there isn't
already some fontification at point, so that keywords aren't highlighted
inside strings), after redoing syntactic-fontification, keywords
fontification needs to be redone as well.

A change at pos N potentially changes the output of syntax-ppss for
all pos>N, so jit-lock just refontifies everything (in the background
with jit-lock-stealth-*).  Of course, in practice the output of
syntax-ppss is generally not changed for all pos>N but only for N<pos<M
(or sometimes it is changed for all pos>N but the change is marginal or
irrelevant and will not influence the result of font-locking), so we could
be more clever, but we would need to cache the parse-state-before-change
so we can compare it to the parse-state-after-change so we can stop when
they are equal again.  It might be worth the effort, but nobody has
bothered coding it up yet.  Turning off jit-lock-stealth-* is a simpler
workaround for the CPU-power-wasting problem.


	Stefan



-------------------------------------------------------
This SF.Net email sponsored by: Parasoft
Error proof Web apps, automate testing & more.
Download & eval WebKing and get a free book.
www.parasoft.com/bulletproofapps1

Re: Advising in cc-awk.el and namespace

[Richard Stallman]

    A change at pos N potentially changes the output of syntax-ppss for
    all pos>N, so jit-lock just refontifies everything (in the background
    with jit-lock-stealth-*).  Of course, in practice the output of
    syntax-ppss is generally not changed for all pos>N but only for N<pos<M
    (or sometimes it is changed for all pos>N but the change is marginal or
    irrelevant and will not influence the result of font-locking), so we could
    be more clever, but we would need to cache the parse-state-before-change
    so we can compare it to the parse-state-after-change so we can stop when
    they are equal again.

It would not be hard to cache the syntactic state every N lines in a
text property, and that way it would be possible to notice when
things get back in sync and end refontification.

However, another solution is to stop refontifying at the next
start-of-defun after the last change.  That should be really simple.
Could someone give that a try?



-------------------------------------------------------
This SF.Net email sponsored by: Parasoft
Error proof Web apps, automate testing & more.
Download & eval WebKing and get a free book.
www.parasoft.com/bulletproofapps1

Re: Advising in cc-awk.el and namespace

[Stefan Monnier]

>     A change at pos N potentially changes the output of syntax-ppss for
>     all pos>N, so jit-lock just refontifies everything (in the background
>     with jit-lock-stealth-*).  Of course, in practice the output of
>     syntax-ppss is generally not changed for all pos>N but only for N<pos<M
>     (or sometimes it is changed for all pos>N but the change is marginal or
>     irrelevant and will not influence the result of font-locking), so we could
>     be more clever, but we would need to cache the parse-state-before-change
>     so we can compare it to the parse-state-after-change so we can stop when
>     they are equal again.
> 
> It would not be hard to cache the syntactic state every N lines in a
> text property, and that way it would be possible to notice when
> things get back in sync and end refontification.
> 
> However, another solution is to stop refontifying at the next
> start-of-defun after the last change.  That should be really simple.
> Could someone give that a try?

That's a very good idea.
The only problem for is that we don't really have that info right now.
The only thing that jit-lock knows is whether to do the "contextual
refontification" or not.  Font-lock and syntax.el know a little bit more,
in that they know how to find the previous start-of-defun, but not the next
one (although depending on the value of font-lock-beginning-of-syntax-function
and/or syntax-begin-function, the next-start-of-defun can be inferred).


	Stefan



-------------------------------------------------------
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0

Re: Advising in cc-awk.el and namespace

[Richard Stallman]

    The only problem for is that we don't really have that info right now.
    The only thing that jit-lock knows is whether to do the "contextual
    refontification" or not.  Font-lock and syntax.el know a little bit more,
    in that they know how to find the previous start-of-defun, but not the next
    one (although depending on the value of font-lock-beginning-of-syntax-function
    and/or syntax-begin-function, the next-start-of-defun can be inferred).

It won't be hard to add a function for them to call to find the next
start-of-defun.  In most cases, pretty much the same code that finds
te previous start-of-defun can be used to find the next one.

If that's enough to make jit-lock much more efficient in most cases,
how about doing it?


-------------------------------------------------------
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0