command to fontify whole buffer?

command to fontify whole buffer?

[Drew Adams]

Sometimes I want to fontify a whole buffer (really).  One use case is to be able
to then use the fontification for other (e.g. code) purposes than visual
highlighting.  But it really doesn't matter why - I just want to. ;-)

With the current lazy/jit fontification, I see no command to do that.
`font-lock-fontify-buffer' doesn't do it.  Marking the whole buffer and using
`(font-lock-fontify-region BEG END)' doesn't do it.

Digging more than a bit, I finally found non-interactive function
`jit-lock-fontify-now', which seems to do the trick.  Why not provide a command
that does it?

(defun fontify-buffer-completely-no-but-really ()
  "Yes, really, completely, right now."
  (interactive)
  (jit-lock-fontify-now))

Oddly enough, the `jit-lock-fontify-now' doc is no different from the doc of all
the other `fontify' functions that do _not_ really, necessarily fontify
everything they apply to.  They all say only that they fontify.  The others do
not say that they don't really, necessarily fontify all they apply to.  And this
function doesn't say that it is any different from the others.  But it seems to
be.

Another question is whether there is some condition that can be tested to know
whether a buffer has really been completely fontified (whether or not that
fontification is completely up-to-date because of subsequent edits).  Text
property `fontified' does not indicate this - is there something that does?

Re: command to fontify whole buffer?

[Alan Mackenzie]

Hi, Drew.

On Thu, Sep 22, 2011 at 06:32:49AM -0700, Drew Adams wrote:
> Sometimes I want to fontify a whole buffer (really).  One use case is to be able
> to then use the fontification for other (e.g. code) purposes than visual
> highlighting.  But it really doesn't matter why - I just want to. ;-)

> With the current lazy/jit fontification, I see no command to do that.
> `font-lock-fontify-buffer' doesn't do it.

Are you sure?  I thought it did, and looking again at the fine source, it
does appear to.

> Marking the whole buffer and using `(font-lock-fontify-region BEG END)'
> doesn't do it.

Ditto.

> Digging more than a bit, I finally found non-interactive function
> `jit-lock-fontify-now', which seems to do the trick.  Why not provide a
> command that does it?

jit-lock-fontify-now                 just calls the function
font-lock-fontify-region             (via a hook) which calls
font-lock-default-fontify-region     (via another hook).

> (defun fontify-buffer-completely-no-but-really ()
>   "Yes, really, completely, right now."
>   (interactive)
>   (jit-lock-fontify-now))

> Oddly enough, the `jit-lock-fontify-now' doc is no different from the doc of all
> the other `fontify' functions that do _not_ really, necessarily fontify
> everything they apply to.  They all say only that they fontify.  The others do
> not say that they don't really, necessarily fontify all they apply to.  And this
> function doesn't say that it is any different from the others.  But it seems to
> be.

A lot of these functions set the 'fontified text property to nil (or
'deferred) with the intention that they get properly fontified later.
Most of the time this actually happens.

> Another question is whether there is some condition that can be tested to know
> whether a buffer has really been completely fontified (whether or not that
> fontification is completely up-to-date because of subsequent edits).  Text
> property `fontified' does not indicate this - is there something that does?

Surely 'fontified DOES do it.  If it has the value t throughout the
buffer, that buffer is fully fontified, otherwise not.  Or have I
misunderstood something?

By the way, I drew a diagram of how jit-lock works about 5 years ago,
tracing the functions which call each other, and the hooks out of which
things get hoiked.  Would you be interested in a copy of it?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: command to fontify whole buffer?

[Lennart Borgman]

On Thu, Sep 22, 2011 at 16:33, Alan Mackenzie <acm@muc.de> wrote:
>
> By the way, I drew a diagram of how jit-lock works about 5 years ago,
> tracing the functions which call each other, and the hooks out of which
> things get hoiked.  Would you be interested in a copy of it?

Add it to Emacs documentation?

Re: command to fontify whole buffer?

[Alan Mackenzie]

Hi, Lennart.

On Thu, Sep 22, 2011 at 04:46:09PM +0200, Lennart Borgman wrote:
> On Thu, Sep 22, 2011 at 16:33, Alan Mackenzie <acm@muc.de> wrote:

> > By the way, I drew a diagram of how jit-lock works about 5 years ago,
> > tracing the functions which call each other, and the hooks out of which
> > things get hoiked.  Would you be interested in a copy of it?

> Add it to Emacs documentation?

Where?  Also, who's going to maintain it?  ;-)  I think it's useful in
its informal state.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: command to fontify whole buffer?

[Dave Abrahams]

on Thu Sep 22 2011, Alan Mackenzie <acm-AT-muc.de> wrote:

> Hi, Lennart.
>
> On Thu, Sep 22, 2011 at 04:46:09PM +0200, Lennart Borgman wrote:
>> On Thu, Sep 22, 2011 at 16:33, Alan Mackenzie <acm@muc.de> wrote:
>
>> > By the way, I drew a diagram of how jit-lock works about 5 years ago,
>> > tracing the functions which call each other, and the hooks out of which
>> > things get hoiked.  Would you be interested in a copy of it?
>
>> Add it to Emacs documentation?
>
> Where?  Also, who's going to maintain it?  ;-)  I think it's useful in
> its informal state.

Maybe a comment in the font locking code would be appropriate.  Anyone
attempting a redesign would be tempted to read it, and thus be aware of
it.  That gives it a fighting chance of staying up-to-date.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: command to fontify whole buffer?

[Lennart Borgman]

On Thu, Sep 22, 2011 at 19:30, Dave Abrahams <dave@boostpro.com> wrote:
>
> on Thu Sep 22 2011, Alan Mackenzie <acm-AT-muc.de> wrote:
>
>> Hi, Lennart.
>>
>> On Thu, Sep 22, 2011 at 04:46:09PM +0200, Lennart Borgman wrote:
>>> On Thu, Sep 22, 2011 at 16:33, Alan Mackenzie <acm@muc.de> wrote:
>>
>>> > By the way, I drew a diagram of how jit-lock works about 5 years ago,
>>> > tracing the functions which call each other, and the hooks out of which
>>> > things get hoiked.  Would you be interested in a copy of it?
>>
>>> Add it to Emacs documentation?
>>
>> Where?  Also, who's going to maintain it?  ;-)  I think it's useful in
>> its informal state.
>
> Maybe a comment in the font locking code would be appropriate.  Anyone
> attempting a redesign would be tempted to read it, and thus be aware of
> it.  That gives it a fighting chance of staying up-to-date.

I agree with Dave!

RE: command to fontify whole buffer?

[Drew Adams]

> > `font-lock-fontify-buffer' doesn't do it.
> 
> Are you sure?  I thought it did, and looking again at the 
> fine source, it does appear to.

No, I only think I'm sure. ;-)

After calling `font-lock-fontify-buffer', I can search for zones with face
`font-lock-function-name-face', for example, and I can see that there are no
such zones after some minimal portion of the buffer.

> > Marking the whole buffer and using 
> `(font-lock-fontify-region BEG END)'
> > doesn't do it.
> 
> Ditto.

ditto

> > Digging more than a bit, I finally found non-interactive function
> > `jit-lock-fontify-now', which seems to do the trick.  Why 
> > not provide a command that does it?
> 
> jit-lock-fontify-now                 just calls the function
> font-lock-fontify-region             (via a hook) which calls
> font-lock-default-fontify-region     (via another hook).

I believe you, but I'm not seeing `font-lock-default-fontify-region' fontify
everything.  I'm no expert on this at all.  Just going by what I think I see.

> Surely 'fontified DOES do it.  If it has the value t throughout the
> buffer, that buffer is fully fontified, otherwise not.

Perhaps you're right that if I checked that property throughout the entire
buffer it would be sufficient.  Dunno.  I probably didn't try testing it
everywhere.  But if that's the best test for full-buffer fontification then I
would still prefer something simpler/quicker.

> Or have I misunderstood something?

No, no doubt I have.  No doubt I still do. ;-)

> By the way, I drew a diagram of how jit-lock works about 5 years ago,
> tracing the functions which call each other, and the hooks 
> out of which things get hoiked.  Would you be interested in a copy of it?

I guess so; thanks.  But I would mostly be interested in confirmation that you
are right etc.  I do not see (e.g. by searching for property zones) that the
entire buffer gets fontified by `font-lock-fontify-buffer', but you say it
does/should.

Re: command to fontify whole buffer?

[Alan Mackenzie]

Hi, Drew.

On Thu, Sep 22, 2011 at 07:56:52AM -0700, Drew Adams wrote:
> > > `font-lock-fontify-buffer' doesn't do it.

> > Are you sure?  I thought it did, and looking again at the 
> > fine source, it does appear to.

> No, I only think I'm sure. ;-)

> After calling `font-lock-fontify-buffer', I can search for zones with face
> `font-lock-function-name-face', for example, and I can see that there are no
> such zones after some minimal portion of the buffer.

Ah.  OK.

> > > Marking the whole buffer and using 
> > `(font-lock-fontify-region BEG END)'
> > > doesn't do it.

> > Ditto.

> ditto

ditto.

> > > Digging more than a bit, I finally found non-interactive function
> > > `jit-lock-fontify-now', which seems to do the trick.  Why 
> > > not provide a command that does it?

> > jit-lock-fontify-now                 just calls the function
> > font-lock-fontify-region             (via a hook) which calls
> > font-lock-default-fontify-region     (via another hook).

> I believe you, but I'm not seeing `font-lock-default-fontify-region' fontify
> everything.  I'm no expert on this at all.  Just going by what I think I see.

> > Surely 'fontified DOES do it.  If it has the value t throughout the
> > buffer, that buffer is fully fontified, otherwise not.

> Perhaps you're right that if I checked that property throughout the entire
> buffer it would be sufficient.  Dunno.  I probably didn't try testing it
> everywhere.  But if that's the best test for full-buffer fontification then I
> would still prefer something simpler/quicker.

Surely `next-property-change' is going to be fast enough.

> > Or have I misunderstood something?

> No, no doubt I have.  No doubt I still do. ;-)

> > By the way, I drew a diagram of how jit-lock works about 5 years ago,
> > tracing the functions which call each other, and the hooks 
> > out of which things get hoiked.  Would you be interested in a copy of it?

> I guess so; thanks.  But I would mostly be interested in confirmation that you
> are right etc.  I do not see (e.g. by searching for property zones) that the
> entire buffer gets fontified by `font-lock-fontify-buffer', but you say it
> does/should.

OK, here goes.  Bear in mind it's five years old.

#########################################################################

(i) DEMAND.  This is a call from the display engine to fontify a bit of the
  buffer.  JIT either fontifies immediately, or sets 'fontified to 'defer,
  invoking deferred fontification ((iii)).
(ii) STEALTH.  This is fontification which happens during idleness.
(iii) DEFERRED.  After a (short, ~0.25 second) idleness period, deferred bits
  (from demand ((i))) get 'fontified set to nil.  WHOOPS!  This leads to an

(iv) CONTEXT.  After a change, 'fontified is marked nil on the buffer from
  that point on.  Intended for unmatched string openers, etc.
(v) AFTER CHANGE.  On a buffer change, the 'fontified property on the
  pertinent text is set to nil.


"*" means that the routine adjusts the region to whole lines.

DEMAND (called by display code)  <----------------------------|
      [fontification-functions] (hook)                        |
      jit-lock-function ->-|                                  |
|-------------<------------+ <is defer-font enabled>?         |
|                          |                                  |
|                          v                                  ^
| |-----------<------------|                                  |
| |                                                           |
| |->* jit-lock-fontify-now  <----------------------------|   |
|      jit-lock-functions (hook)                          |   |
v      font-lock-fontify-region  <------------------------+---+----|
|      [font-lock-fontify-region-function] (hook)         |   |    |
|    * font-lock-default-fontify-region                   |   |    |
|                                                         |   |    |
|DEFERRED (invoked by jit-lock-defer-timer)               |   |    |
|-->   jit-lock-deferred-fontify  ------------------>-----+---|    |
         This sets 'fontified to nil, and calls sit-for,  |   |    |
         causing immediate display (by DEMAND).           |   |    |
                                                          |   |    |
STEALTH (invoked by jit-lock-stealth-timer                |   |    |
      jit-lock-stealth-fontify                            |   ^    |
      jit-lock-fontify-now ----------------->-------------|   |    |
                                                              |    |
CONTEXT (invoked by jit-lock-context-timer)                   |    |
      jit-lock-context-fontify (from timer) -------->---------+    |
        This sets 'fontified to nil on the (extended)         |    |
        region.                                               |    |
                                                              |    |
JIT AFTER CHANGE (called from the after-change hook)          |    ^
    * jit-lock-after-change --------------->------------------|    |
      This sets 'fontified to nil, and relies on DEMAND            |
      to refontify the changed bit during display.                 |
                                                                   |
ORDINARY AFTER CHANGE (without jit)                                |
    * font-lock-after-change-function                              |
      font-lock-fontify-region ---------------->-------------------|
                                                                   |
COMMANDS                                                           |
      font-lock-fontify-block                                      |
      font-lock-fontify-region ---------------->-------------------|
                                                                   |
      font-lock-fontify-buffer                                     |
      [font-lock-fontify-buffer-function] (hook)                   |
      font-lock-default-fontify-buffer                             |
      font-lock-fontify-region ---------------->-------------------|

#########################################################################

Interpretation: Of the five possible triggers (e.g. DEMAND) for
fontification, this shows the sequence of functions triggered by each of
them.  [Symbols] in brackets are hook variables.

As you surmise above, jit-lock-fontify-now is the central function which
actually gets the fontification done.

-- 
Alan Mackenzie (Nuremberg, Germany).

RE: command to fontify whole buffer?

[Drew Adams]

> OK, here goes.  Bear in mind it's five years old.

Thank you.

Re: command to fontify whole buffer?

[Lennart Borgman]

On Thu, Sep 22, 2011 at 17:27, Alan Mackenzie <acm@muc.de> wrote:
> Hi, Drew.
>
> On Thu, Sep 22, 2011 at 07:56:52AM -0700, Drew Adams wrote:
>> > > `font-lock-fontify-buffer' doesn't do it.
>
>> > Are you sure?  I thought it did, and looking again at the
>> > fine source, it does appear to.
>
>> No, I only think I'm sure. ;-)

Looking at the diagram I do not understand why the buffer does not get
fontified by font-lock-fontify-region (when it gets fontified by
jit-lock-fontify-now).

> #########################################################################
...
> | |->* jit-lock-fontify-now  <----------------------------|   |
> |      jit-lock-functions (hook)                          |   |
> v      font-lock-fontify-region  <------------------------+---+----|
> |      [font-lock-fontify-region-function] (hook)         |   |    |
> |    * font-lock-default-fontify-region                   |   |    |
> |                                                         |   |    |
...

Re: command to fontify whole buffer?

[Nix]

On 22 Sep 2011, Alan Mackenzie uttered the following:
> (iii) DEFERRED.  After a (short, ~0.25 second) idleness period, deferred bits
>   (from demand ((i))) get 'fontified set to nil.  WHOOPS!  This leads to an
>
> (iv) CONTEXT.  After a change, 'fontified is marked nil on the buffer from
>   that point on.  Intended for unmatched string openers, etc.

This bit seems to have been severely mangled by something, which is a
shame 'cos it looks like it would be terribly useful if I could figure
out what it meant. :)

-- 
NULL && (void)

Re: command to fontify whole buffer?

[Stefan Monnier]

> Sometimes I want to fontify a whole buffer (really).  One use case is
> to be able to then use the fontification for other (e.g. code)
> purposes than visual highlighting.  But it really doesn't matter why -
> I just want to. ;-)

Agreed.  We need two functions:
- font-lock-flush-region (which marks the region for re-fontification).
- font-lock-ensure-region (which makes sure the region is fontified).
The functions we have were designed before lazy/jit lock and were
adjusted in ad-hoc ways when jit/lazy was added.
For font-lock-ensure-region, you should be able to find sample code for
an implementation in various packages such as ps-print.el and
replace.el.


        Stefan

Re: command to fontify whole buffer?

[Jambunathan K]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Sometimes I want to fontify a whole buffer (really).  One use case is
>> to be able to then use the fontification for other (e.g. code)
>> purposes than visual highlighting.  But it really doesn't matter why -
>> I just want to. ;-)
>
> Agreed.  We need two functions:
> - font-lock-flush-region (which marks the region for re-fontification).
> - font-lock-ensure-region (which makes sure the region is fontified).
> The functions we have were designed before lazy/jit lock and were
> adjusted in ad-hoc ways when jit/lazy was added.
> For font-lock-ensure-region, you should be able to find sample code for
> an implementation in various packages such as ps-print.el and
> replace.el.

In htmlfontify.el (htmlfontify-buffer), hfy-force-fontification calls
either font-lock-fontify-region (in interactive mode) or
font-lock-fontify-buffer (in non-interactive mode).

The latter funcion in turn calls font-lock-fontify-buffer-function which
apparently points to jit-lock-refontify.

I hope either the htmlfontify-buffer DTRT or it may fail to re-fontify
under certain circumstances.

>         Stefan
>
>

--