font lock woes

font lock woes

[Joost Kremers]

EN:SiS(9)
Hi all,

I'm writing a minor mode that (among other things) adds some font lock
patterns to the buffer's font-lock-keywords, using
font-lock-add-keywords, but I'm running into some problems.

First, one of the things I'm trying to do is to add some additional text
properties beside a face spec. (info "(elisp) Search-based Fontification")
has the following to say on this:

,----
| `(MATCHER . FACESPEC)'
| [...]
|      However, FACESPEC can also evaluate to a list of this form:
| 
|           (face FACE PROP1 VAL1 PROP2 VAL2...)
| 
|      to specify the face FACE and various additional text properties to
|      put on the text that matches.  If you do this, be sure to add the
|      other text property names that you set in this way to the value of
|      `font-lock-extra-managed-props' so that the properties will also
|      be cleared out when they are no longer appropriate.
`----

This suggests to me that I should be able to do:

,----
| (font-lock-add-keywords nil `((,(regexp-quote "{++") . '(face cm-addition-face read-only t))) t)
`----

but for some reason, the only thing that works is:

,----
| (font-lock-add-keywords nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))
`----

Am I misunderstanding the info node, or is Info wrong? (I'm running GNU
Emacs 24.3.50.1, BTW).

A second thing I don't understand is how to remove the read-only
property again when the mode is deactivated. I was thinking I should be
able to do the following:

,----
| (font-lock-remove-keywords nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))))
| (let ((inhibit-read-only t))
|   (font-lock-fontify-buffer))
| (setq font-lock-extra-managed-props (delq 'read-only font-lock-extra-managed-props))
`----

I figured if I remove the relevant matcher from font-lock-keywords, then
refontify the buffer and only then remove the read-only property from
font-lock-extra-managed-props, things should work. But they don't: the
text that was made read-only with font-lock remains read-only.

Only if I don't remove 'read-only from font-lock-extra-managed-props at
all, does it work: in that case, the read-only property is removed. But I
don't like the idea that font-lock-extra-managed-props isn't restored to
its original value. So I'm wondering if there's a better way of doing
this.

It's the first time I'm trying to do anything with font-lock. I'd
appreciate any and all comments!


TIA


-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht

Re: font lock woes

[Stefan Monnier]

> This suggests to me that I should be able to do:
> ,----
> | (font-lock-add-keywords nil `((,(regexp-quote "{++") . '(face cm-addition-face read-only t))) t)
> `----
> but for some reason, the only thing that works is:
> ,----
> | (font-lock-add-keywords nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))
> `----

In which way does the other one not work?  Did you check the resulting
font-lock-keywords?

> A second thing I don't understand is how to remove the read-only
> property again when the mode is deactivated.  I was thinking I should be
> able to do the following:

> ,----
> | (font-lock-remove-keywords nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))))
> | (let ((inhibit-read-only t))
> |   (font-lock-fontify-buffer))
> | (setq font-lock-extra-managed-props (delq 'read-only font-lock-extra-managed-props))
> `----

font-lock-fontify-buffer just schedules a refontification, which will be
done later, on-the-fly, piece by piece as text is displayed.

Much simpler is to remove-text-properties by hand.


        Stefan

Re: font lock woes

[Joost Kremers]

Stefan Monnier wrote:
>> This suggests to me that I should be able to do:
>> ,----
>> | (font-lock-add-keywords nil `((,(regexp-quote "{++") . '(face cm-addition-face read-only t))) t)
>> `----
>> but for some reason, the only thing that works is:
>> ,----
>> | (font-lock-add-keywords nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))
>> `----
>
> In which way does the other one not work?  Did you check the resulting
> font-lock-keywords?

Yes, I have, and the matchers are there, but the relevant strings do not
get fontified: cm-addition-face (which is just a foreground color) isn't
applied and matching text is not read-only.

>> A second thing I don't understand is how to remove the read-only
>> property again when the mode is deactivated.  I was thinking I should be
>> able to do the following:
>
>> ,----
>> | (font-lock-remove-keywords nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))))
>> | (let ((inhibit-read-only t))
>> |   (font-lock-fontify-buffer))
>> | (setq font-lock-extra-managed-props (delq 'read-only font-lock-extra-managed-props))
>> `----
>
> font-lock-fontify-buffer just schedules a refontification, which will be
> done later, on-the-fly, piece by piece as text is displayed.

But then why do the colors get removed? I mean, after disabling the mode
with the code above, the font lock colors that my mode adds are removed,
just the read-only property stays.

I see now that that's only in the visible part of the buffer, so either
way it's not going to work the way I'd like, but still, even in the
visible part of the buffer, the call to font-lock-fontify-buffer above
removes the font-lock-face property but not the read-only property.
Which suggests that the properties in font-lock-extra-managed-props are
handled differently from font-lock-face...

> Much simpler is to remove-text-properties by hand.

Yeah, look like...

Thanks for your answer. I'll have to give some thought to how best do do
what I'm after.


-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht
EN:SiS(9)

Re: font lock woes

[Stefan Monnier]

>>> This suggests to me that I should be able to do:
>>> ,----
>>> | (font-lock-add-keywords nil `((,(regexp-quote "{++") . '(face
>>> | cm-addition-face read-only t))) t)
>>> `----
>>> but for some reason, the only thing that works is:
>>> ,----
>>> | (font-lock-add-keywords nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))
>>> `----
>> In which way does the other one not work?  Did you check the resulting
>> font-lock-keywords?
> Yes, I have, and the matchers are there, but the relevant strings do not
> get fontified: cm-addition-face (which is just a foreground color) isn't
> applied and matching text is not read-only.

Could it be that the relevant strings already have some other face?
If so, you'll want to use an OVERRIDE flag (see C-h v font-lock-keywords).

>>> A second thing I don't understand is how to remove the read-only
>>> property again when the mode is deactivated.  I was thinking I should be
>>> able to do the following:
>> 
>>> ,----
>>> | (font-lock-remove-keywords nil `((,(regexp-quote "{++") 0 '(face
>>> | cm-addition-face read-only t))))
>>> | (let ((inhibit-read-only t))
>>> |   (font-lock-fontify-buffer))
>>> | (setq font-lock-extra-managed-props (delq 'read-only font-lock-extra-managed-props))
>>> `----
>> 
>> font-lock-fontify-buffer just schedules a refontification, which will be
>> done later, on-the-fly, piece by piece as text is displayed.

> But then why do the colors get removed? I mean, after disabling the mode
> with the code above, the font lock colors that my mode adds are removed,
> just the read-only property stays.

Because you've removed `read-only' from font-lock-extra-managed-props,
so read-only is not removed any more.

> visible part of the buffer, the call to font-lock-fontify-buffer above
> removes the font-lock-face property but not the read-only property.

No, the call itself does not, it only schedules the buffer for
refontification, i.e. the font-lock-face is only removed (in the visible
part) during the next redisplay, at which point you've already removed
`read-only' from font-lock-extra-managed-props.


        Stefan

Re: font lock woes

[Joost Kremers]

Stefan Monnier wrote:
>> Yes, I have, and the matchers are there, but the relevant strings do not
>> get fontified: cm-addition-face (which is just a foreground color) isn't
>> applied and matching text is not read-only.
>
> Could it be that the relevant strings already have some other face?

No, the relevant strings don't have another face. Before I tried adding
the read-only property, I just had:

,----
| (font-lock-add-keywords nil `((,(regexp-quote "{++") . cm-addition-face)) t)
`----

and that worked fine. Changing that line to 

,----
| (font-lock-add-keywords nil `((,(regexp-quote "{++") . '(face cm-addition-face read-only t))) t)
`----

made fontification of the relevant strings stop working. After changing it to:

,----
| (font-lock-add-keywords nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))
`----

fontification worked again.

>> visible part of the buffer, the call to font-lock-fontify-buffer above
>> removes the font-lock-face property but not the read-only property.
>
> No, the call itself does not, it only schedules the buffer for
> refontification, i.e. the font-lock-face is only removed (in the visible
> part) during the next redisplay, at which point you've already removed
> `read-only' from font-lock-extra-managed-props.

Ah, ok, it makes sense now. Thanks for the explanation.


-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht
EN:SiS(9)

Re: font lock woes

[Stefan Monnier]

> ,----
> | (font-lock-add-keywords
> |  nil `((,(regexp-quote "{++") . '(face cm-addition-face read-only t))) t)
> `----
> made fontification of the relevant strings stop working.
> After changing it to:
> ,----
> | (font-lock-add-keywords
> |  nil `((,(regexp-quote "{++") 0 '(face cm-addition-face read-only t))
> `----
> fontification worked again.

Hmm... I can't find your previous email, but I now notice there are
2 differences in the above lines.  I focused on the `how' argument to
font-lock-add-keywords, but the other important one is that

   (font-lock-add-keywords
    nil `((,(regexp-quote "{++") . '(face cm-addition-face read-only t))) t)

should be

   (font-lock-add-keywords
    nil `((,(regexp-quote "{++") (0 '(face cm-addition-face read-only t)))) t)

because the (REGEXP . FACE) form only works if FACE is a non-list sexp.


        Stefan

Re: font lock woes

[Joost Kremers]

Stefan Monnier wrote:
> Hmm... I can't find your previous email, but I now notice there are
> 2 differences in the above lines.  I focused on the `how' argument to
> font-lock-add-keywords, but the other important one is that
>
>    (font-lock-add-keywords
>     nil `((,(regexp-quote "{++") . '(face cm-addition-face read-only t))) t)
>
> should be
>
>    (font-lock-add-keywords
>     nil `((,(regexp-quote "{++") (0 '(face cm-addition-face read-only t)))) t)
>
> because the (REGEXP . FACE) form only works if FACE is a non-list sexp.

Or what the manual calls a SUBEXP-HIGHLIGHTER, i.e., a list of the form:

,----
| (0 '(face cm-addition-face read-only t)))) t)
`----

However, what confused me was that the manual says:

(info "(elisp) Search-based Fontification")
,----
| `(MATCHER . FACESPEC)'
|      [...]
|      However, FACESPEC can also evaluate to a list of this form:
| 
|           (face FACE PROP1 VAL1 PROP2 VAL2...)
| 
|      to specify the face FACE and various additional text properties to
|      put on the text that matches.
`----

which to me means that it should be possible to have:

(MATCHER . (face FACE PROP1 VAL1))

(save for the quote, but I originally tried both with and without).

which, as you point out, isn't possible. So either I'm reading the
manual wrong, or I should report a bug in the manual. :-)


-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht
EN:SiS(9)

Re: font lock woes

[Stefan Monnier]

> However, what confused me was that the manual says:
> (info "(elisp) Search-based Fontification")
> ,----
> | `(MATCHER . FACESPEC)'
> |      [...]
> |      However, FACESPEC can also evaluate to a list of this form:
> | 
> |           (face FACE PROP1 VAL1 PROP2 VAL2...)
> | 
> |      to specify the face FACE and various additional text properties to
> |      put on the text that matches.
> `----
> which to me means that it should be possible to have:
> (MATCHER . (face FACE PROP1 VAL1))
> (save for the quote, but I originally tried both with and without).
> which, as you point out, isn't possible. So either I'm reading the
> manual wrong, or I should report a bug in the manual. :-)

Looks like a bug in the manual.  Please report it,


        Stefan

Re: font lock woes

[Joost Kremers]

Stefan Monnier wrote:
> Looks like a bug in the manual.  Please report it,

Done.

-- 
Joost Kremers                                   joostkremers@fastmail.fm
Selbst in die Unterwelt dringt durch Spalten Licht
EN:SiS(9)