how to prevent font-lock from messing with a portion of text?

how to prevent font-lock from messing with a portion of text?

[Drew Adams]

How can one prevent font-lock from changing the text properties of a given
portion of text? I tried (put-text-property start end 'fontified t), but C-x
C-s just re-font-lock fontifies the text in question. I also tried binding
`inhibit-modification-hooks' to t, with the same result.

Re: how to prevent font-lock from messing with a portion of text?

[Stefan Monnier]

> How can one prevent font-lock from changing the text properties of a given
> portion of text? I tried (put-text-property start end 'fontified t), but C-x
> C-s just re-font-lock fontifies the text in question.  I also tried binding
> `inhibit-modification-hooks' to t, with the same result.

There are many different ways, each one with advantages and quirks,
I can't recommend any without knowing more of what you're trying to do,


        Stefan

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> > How can one prevent font-lock from changing the text properties
> > of a given portion of text? I tried (put-text-property start end
> > 'fontified t), but C-x C-s just re-font-lock fontifies the text
> > in question.  I also tried binding `inhibit-modification-hooks'
> > to t, with the same result.
>
> There are many different ways, each one with advantages and quirks,
> I can't recommend any without knowing more of what you're trying to do,

Use put-text-property to add a face to some text using the `face' property.
Prevent subsequent "erasure" of that highlighting by font-lock.

I would like to be able to, say, set a particular text property on the same
text, which would cause font-lock to ignore that text (skip over it, as if
it weren't there).

I thought that's what property `fontified' (= t) would do, but that seems
not to be the case. The Elisp manual says this about `fontified':

     This property says whether the character has a face assigned to it
     by font locking.  The display engine tests it to decide whether a
     buffer portion needs refontifying before display.  *Note Auto
     Faces::.  It takes one of three values:

    `nil'
          Font locking is disabled, or the character's `face' property,
          if any, is invalid.

    `defer'
          This value is only used when "just in time" font locking is
          enabled and it means that the character's `face' property is
          invalid and needs deferred fontification.

    `t'
          The character's `face' property, or absence of one, is valid.

Re: how to prevent font-lock from messing with a portion of text?

[Lennart Borgman (gmail)]

Drew Adams wrote:
>>> How can one prevent font-lock from changing the text properties
>>> of a given portion of text? I tried (put-text-property start end
>>> 'fontified t), but C-x C-s just re-font-lock fontifies the text
>>> in question.  I also tried binding `inhibit-modification-hooks'
>>> to t, with the same result.
>> There are many different ways, each one with advantages and quirks,
>> I can't recommend any without knowing more of what you're trying to do,
> 
> Use put-text-property to add a face to some text using the `face' property.
> Prevent subsequent "erasure" of that highlighting by font-lock.
> 
> I would like to be able to, say, set a particular text property on the same
> text, which would cause font-lock to ignore that text (skip over it, as if
> it weren't there).
> 
> I thought that's what property `fontified' (= t) would do, but that seems
> not to be the case. The Elisp manual says this about `fontified':

Did you try using something like the defmacro `save-buffer-state' that 
you can find in font-lock.el?

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> Did you try using something like the defmacro `save-buffer-state' that 
> you can find in font-lock.el?

I don't see the connection. 

Re: how to prevent font-lock from messing with a portion of text?

[Lennart Borgman (gmail)]

Drew Adams wrote:
>> Did you try using something like the defmacro `save-buffer-state' that 
>> you can find in font-lock.el?
> 
> I don't see the connection. 


It was merely a guess from my side. I thought that something in the 
fontification framework was started by the changes you made when you set 
the face.

Maybe I am also misunderstanding what you are trying to do. However I 
tried this:

- emacs -Q
- I opened a html file.
- M-x html-mode
- Did some small changes to modify the buffer
- Then I called the function below

  (defun test-mode-face()
   (interactive)
   (my-save-buffer-state nil
       (put-text-property 5 15 'face 'highlight)))

Now the buffer had the face above from 5 - 15. Then I did C-x C-s. That 
face remained.

If I remove my-save-buffer-state above I never see that highlight face.

Re: how to prevent font-lock from messing with a portion of text?

[Miles Bader]

"Drew Adams" <drew.adams@oracle.com> writes:
> Use put-text-property to add a face to some text using the `face' property.
> Prevent subsequent "erasure" of that highlighting by font-lock.

Did you try using the `font-lock-face' property instead of `face'?

-Miles

-- 
Occam's razor split hairs so well, I bought the whole argument!

Re: how to prevent font-lock from messing with a portion of text?

[Stefan Monnier]

> Use put-text-property to add a face to some text using the `face' property.
> Prevent subsequent "erasure" of that highlighting by font-lock.

The `font-lock-face' property was designed specifically for this kind of case.

> I would like to be able to, say, set a particular text property on the same
> text, which would cause font-lock to ignore that text (skip over it, as if
> it weren't there).

Again, there's no such general facility, so give us more info.

> I thought that's what property `fontified' (= t) would do, but that seems
> not to be the case.

`fontified' is a jit-lock thing, so don't expect to be able to use it to
solve font-lock problems.

> The Elisp manual says this about `fontified':

>      This property says whether the character has a face assigned to it
>      by font locking.  The display engine tests it to decide whether a
>      buffer portion needs refontifying before display.  *Note Auto
>      Faces::.  It takes one of three values:

>     `nil'
>           Font locking is disabled, or the character's `face' property,
>           if any, is invalid.

Huh?  I don't agree with this part of the manual.  `nil' means that the
fontification is not uptodate (so it will be refontified next time the block
is made visible, for example).

So I think the term "invalid" is a bit strong here.  Also it's not "font
locking is disabled" but "jit-lock is disabled".


        Stefan

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> > Use put-text-property to add a face to some text using the
> > `face' property.
> > Prevent subsequent "erasure" of that highlighting by font-lock.
>
> The `font-lock-face' property was designed specifically for this
> kind of case.

1. That's only for Emacs 22.

2. It doesn't prevent font-lock from clobbering the highlighted text when
doing syntactic font lock (e.g. in comments).

3. The highlighting is removed when font-lock-mode is turned off.

4. It won't help with other code (e.g. 3rd library) that happens to use the
`face' property.

IOW, no, `font-lock-face' property was not at all designed specifically for
this
kind of case. Highlight some text, and have it stay highlighted whether or
not font-lock-mode is on. Have font-lock ignore that text when it does its
own highlighting.

Wrt Lennart's suggestion, BTW: I want to be able to also remove the
highlighting and then have font-lock treat that text normally again. So,
nothing that permanently prevents font-lock from fontifying the text would
be useful here.

As a simple example, imagine that you want to be able to use `M-o o'
(facemenu-set-face) on some text in a font-locked buffer. Currently, `M-o o'
just tells you that "Font-lock mode will override any faces you set in this
buffer" - IOW, you're SOL. If you want to see your highlighting, you must
turn off font-lock. Instead of punting this way, `M-o o' could do what's
needed to prevent font-lock from interfering - that is, from changing the
facemenu highlighting.

Yes, that would mean that font-lock highlighting could be thrown off,
depending on what `M-o o' highlighted, but that would be the chance the user
accepts by highlighting text with `M-o o'. The current design makes users
forego use of `M-o o' in font-locked buffers altogether.

That's just an example, to indicate the kind of thing I'm talking about:
using property `face' outside of font-lock to highlight some text, and not
have font-lock step all over it.

BTW, the doc does not explain anywhere (that I've found) just how the
activation of `font-lock-face' is controlled by `font-lock-mode'. It says
only that the mode toggles the activation of the property.

 - Node Precalculated Fontification: "activation [of `font-lock-face']
   is toggled when the user calls `M-x font-lock-mode'."

 - Node Special Properties: "state of activation [of `font-lock-face']
   is controlled by `font-lock-mode'."

IOW, the doc never says which is on and which is off. Please fix the doc to
make the toggle behavior explicit. It should say that property
`font-lock-face' is activated only when `font-lock-mode' is on.

> > I would like to be able to, say, set a particular text property
> > on the same text, which would cause font-lock to ignore that text (skip
> > over it, as if it weren't there).
>
> Again, there's no such general facility

That's too bad. How about adding that, after the release? It seems like a
simple way to prevent font-lock (not just jit-lock or whatever) from
touching certain characters. It's easy to add and remove, and it gives you
as fine-grained control as you like (individual characters).

Use: put text property `font-lock-ignore' on some text, and font-lock would
just skip over that text, ignoring it. Take property `font-lock-ignore' away
from that text, and font-lock would see the text again, taking it into
account when fontifying.

I think this would be a reasonable general mechanism, and I imagine that the
implementation would be straightforward, even trivial.

Unlike property `font-lock-face', `font-lock-ignore' would not be under the
control of `font-lock-mode'. Code could use `font-lock-ignore' to control
the visibility (on/off) of text portions to font-locking.

> > I thought that's what property `fontified' (= t) would do, but
> > that seems not to be the case.
>
> `fontified' is a jit-lock thing, so don't expect to be able to use it to
> solve font-lock problems.

OK. That's not clear to me from the doc, however.

> > The Elisp manual says this about `fontified':
> >
> >      This property says whether the character has a face assigned to it
> >      by font locking.  The display engine tests it to decide whether a
> >      buffer portion needs refontifying before display.  *Note Auto
> >      Faces::.  It takes one of three values:
> >
> >     `nil'
> >           Font locking is disabled, or the character's `face' property,
> >           if any, is invalid.
>
> Huh?  I don't agree with this part of the manual.  `nil' means that the
> fontification is not uptodate (so it will be refontified next
> time the block is made visible, for example).
>
> So I think the term "invalid" is a bit strong here.  Also it's not "font
> locking is disabled" but "jit-lock is disabled".

>From what you say, it sounds as if that whole passage about that property
should be stated in terms of jit-lock only. And there should be a cross
reference to the place where jit-lock is introduced.

Re: how to prevent font-lock from messing with a portion of text?

[Lennart Borgman (gmail)]

Drew Adams wrote:

> Wrt Lennart's suggestion, BTW: I want to be able to also remove the
> highlighting and then have font-lock treat that text normally again. So,
> nothing that permanently prevents font-lock from fontifying the text would
> be useful here.

I understand that my suggestion does not fit your needs now, but it is 
quite easy to let font-lock take over the fontifying again.


> As a simple example, imagine that you want to be able to use `M-o o'
> (facemenu-set-face) on some text in a font-locked buffer. Currently, `M-o o'
> just tells you that "Font-lock mode will override any faces you set in this
> buffer" - IOW, you're SOL. If you want to see your highlighting, you must
> turn off font-lock. Instead of punting this way, `M-o o' could do what's
> needed to prevent font-lock from interfering - that is, from changing the
> facemenu highlighting.

To do that kind of thing you want you have to change 
font-lock-fontify-region function. However that will not work for all 
modes, only those that uses this. So at the moment you really can not do it.

To temporary set a face like you want to do I think you need to use 
overlays.


> BTW, the doc does not explain anywhere (that I've found) just how the
> activation of `font-lock-face' is controlled by `font-lock-mode'. It says
> only that the mode toggles the activation of the property.

It seems to me that there are indeed things missing in the manual.

Re: how to prevent font-lock from messing with a portion of text?

[Stefan Monnier]

>> > Use put-text-property to add a face to some text using the
>> > `face' property.
>> > Prevent subsequent "erasure" of that highlighting by font-lock.
>> 
>> The `font-lock-face' property was designed specifically for this
>> kind of case.

> 1. That's only for Emacs 22.

Yup.

> 2. It doesn't prevent font-lock from clobbering the highlighted text when
> doing syntactic font lock (e.g. in comments).

It's true that font-lock's own faces take precedence.  But the
font-lock-face property is not clobbered.

> 3. The highlighting is removed when font-lock-mode is turned off.

Again, the lont-lock-face properties are not removed.  But yes, it is made
inactive.

> 4. It won't help with other code (e.g. 3rd library) that happens to use the
> `face' property.

Yup.

> IOW, no, `font-lock-face' property was not at all designed specifically for
> this
> kind of case. Highlight some text, and have it stay highlighted whether or
> not font-lock-mode is on. Have font-lock ignore that text when it does its
> own highlighting.

Well, with those extra conditions, then indeed, no it doesn't quite fit
your bill.

How 'bout:
- use font-lock-mode-hook to move `face' properties to/from a new property
  `persistent-face' when turning font-lock-mode on/off.
- add a font-lock-fontify-region-function so that after fontifying
  the region normally, you scan the region for `persistent-face' properties
  and copy those to the `face' property.

> That's too bad.  How about adding that, after the release?

I'm not interested in coding it up because I don't see a need for it, but if
someone provides a clean patch, I don't see any reason why I'd oppose it.

> From what you say, it sounds as if that whole passage about that property
> should be stated in terms of jit-lock only. And there should be a cross
> reference to the place where jit-lock is introduced.

Yes, that's probably right.


        Stefan

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> > 2. It doesn't prevent font-lock from clobbering the highlighted
> >    text when doing syntactic font lock (e.g. in comments).
>
> It's true that font-lock's own faces take precedence.  But the
> font-lock-face property is not clobbered.

I said that the "highlighted text", that is, the _highlighting_, is
clobbered.

> > 3. The highlighting is removed when font-lock-mode is turned off.
>
> Again, the lont-lock-face properties are not removed.  But yes, it is made
> inactive.

Again, I didn't say anything about removal of the `font-lock-face' property.
I said that the "highlighting" is removed. Highlighting is a visual effect.
The highlighting that a user adds is wiped out. The text property might
still be there, but the highlighting that was its purpose is gone.

> > IOW, no, `font-lock-face' property was not at all designed
> > specifically for this kind of case. Highlight some text, and have
> > it stay highlighted whether or not font-lock-mode is on. Have
> > font-lock ignore that text when it does its own highlighting.
>
> Well, with those extra conditions, then indeed, no it doesn't quite fit
> your bill.

Not much "extra" about that. Do you think that users who want to highlight
portions of a buffer (e.g. matching regexps) only want to do so in buffers
that are not font-locked?

For example, facemenu stuff such as `facemenu-set-face' is useless for
font-locked buffers, only because of this limitation. We give users a lame
message telling them, in effect, Fuggedabowdit!, only because we are unable
to tell font-lock "Hands off!" of the highlighting that users add using
facemenu (or other highlighting packages).

I have a highlighting library, for instance
(http://www.emacswiki.org/cgi-bin/wiki/HighLight), that lets you use either
overlays or text properties. If you choose to use text properties, font-lock
clobbers the highlighting. One use of text properties is to do essentially
what `facemenu-set-face' does, including entering new text in some face.
Useless in a font-locked buffer, only because of this silly limitation.

Now, if font-lock didn't itself use the `face' property, things might be
OK...

Or, as I said, if one could at least set some other property (e.g.
`font-lock-ignore') that told font-lock "Hands off!" this text...

> How 'bout:
> - use font-lock-mode-hook to move `face' properties to/from a new property
>   `persistent-face' when turning font-lock-mode on/off.

It's a joke, right? Why don't you make font-lock use such a property itself,
all the time? Make it independent of `face' altogether.

> - add a font-lock-fontify-region-function so that after fontifying
>   the region normally, you scan the region for `persistent-face'
>   properties and copy those to the `face' property.

Right (ridiculous). Move all of `face' to `persistent-face' and then back
again. And besides such a convoluted, heavy workaround?

What about all of those "many different ways, each one with advantages and
quirks" that you promised? How about _one_ simple way to tell font-lock
"Hands off!" this text?

> > That's too bad.  How about adding that, after the release?
>
> I'm not interested in coding it up because I don't see a need for
> it, but if someone provides a clean patch, I don't see any reason
> why I'd oppose it.

I hope someone does. I would myself, if FFS could accept it without employer
papers. I can't believe it would be hard for font-lock to check if text it
wants to work with has an ignore-me property.

Or some other mechanism that allows the same feature: be able to apply
property `face' to some text and not have that highlighting overridden by
what font-lock does. I'm not wedded to any particular way to implement that
feature, but I think it's a needed feature.

> > From what you say, it sounds as if that whole passage about
> > that property should be stated in terms of jit-lock only. And there
> > should be a cross reference to the place where jit-lock is introduced.
>
> Yes, that's probably right.

Glad we could agree about something.

I'd add that the Elisp manual is pretty poor on explaining font lock,
generally. Start with `i font-lock' and see what you get. Keep hitting `,'
and see if you get a good view of what font-lock is all about.

Or start with node _Font Lock Mode_ (the best place to start, and in the
main menu, but not easy to stumble upon with `i'). There's not much in that
first node.  That's fine, but what happens when you go to the next node,
_Font Lock Basics_, which has the menu-item description "Overview of
customizing Font Lock"?

1. That node is simply a description of variable `font-lock-defaults'.

2. That node has nothing to do with "customizing Font Lock", in spite of
what was advertised.

3. That node is _not_ an overview of font lock or a description of "font
lock basics". It's almost the farthest thing from the basics.

Node _Search-based Fontification_ is closer to an overview of font-lock
customization ("the most important variable for customizing..."), but it is
still not the place to start. There should be a real overview of font lock
at the beginning: what it is, what it's for (why), how to customize it
(although that really belongs in the Emacs manual), and how it works. It
should include a getting-started example or two, showing how to use font
lock in your own code (including how to get around it).

I submit that the font-lock doc is not easy to read. You pointed out one
area where it is also incorrect. It would be good if someone who is familiar
with the purpose, use, and implementation of font lock would reread the doc
and try to improve it.

Re: how to prevent font-lock from messing with a portion of text?

[Lennart Borgman (gmail)]

Drew Adams wrote:

>> - add a font-lock-fontify-region-function so that after fontifying
>>   the region normally, you scan the region for `persistent-face'
>>   properties and copy those to the `face' property.
> 
> Right (ridiculous). Move all of `face' to `persistent-face' and then back
> again. And besides such a convoluted, heavy workaround?


Maybe just add the wanted face as 'persistent-face, remove 'fontified 
and then use a function like above? (But I am not sure how to 
"refontify". Is not turning font-lock-mode off/on a bit heavy? Though 
that new font-lock-fontify-region-function could be called directly of 
course. With some twistes then to avoid making the buffer modified.)

A problem with this solution is however that fontification must use 
font-lock-fontify-region-function etc. All major modes does not do that.

Re: how to prevent font-lock from messing with a portion of text?

[Stefan Monnier]

> A problem with this solution is however that fontification must use
> font-lock-fontify-region-function etc.  All major modes does not do that.

Huh?  Which mode doesn't obey font-lock-fontify-region-function?


        Stefan

Re: how to prevent font-lock from messing with a portion of text?

[Lennart Borgman (gmail)]

Stefan Monnier wrote:
>> A problem with this solution is however that fontification must use
>> font-lock-fontify-region-function etc.  All major modes does not do that.
> 
> Huh?  Which mode doesn't obey font-lock-fontify-region-function?

nxml-mode. Though it is not part of Emacs many people think it is the 
best mode for editing XML.

Re: how to prevent font-lock from messing with a portion of text?

[Stefan Monnier]

>>> A problem with this solution is however that fontification must use
>>> font-lock-fontify-region-function etc.  All major modes does not do that.
>> Huh?  Which mode doesn't obey font-lock-fontify-region-function?
> nxml-mode. Though it is not part of Emacs many people think it is the best
> mode for editing XML.

But it doesn't use font-lock either, so no amount of fixing font-lock will
help it, right?


        Stefan

Re: how to prevent font-lock from messing with a portion of text?

[Lennart Borgman (gmail)]

Stefan Monnier wrote:
>>>> A problem with this solution is however that fontification must use
>>>> font-lock-fontify-region-function etc.  All major modes does not do that.
>>> Huh?  Which mode doesn't obey font-lock-fontify-region-function?
>> nxml-mode. Though it is not part of Emacs many people think it is the best
>> mode for editing XML.
> 
> But it doesn't use font-lock either, so no amount of fixing font-lock will
> help it, right?


Yes, I believe so. I does its own fontification and I do not belive it 
has something in the fontification routines it uses that resembles 
font-lock-fontify-region-function. Unfortunately.

Re: how to prevent font-lock from messing with a portion of text?

[Stefan Monnier]

> It's a joke, right? Why don't you make font-lock use such a property
> itself, all the time?  Make it independent of `face' altogether.

Sorry, I thought you were asking for some help.

>> - add a font-lock-fontify-region-function so that after fontifying
>> the region normally, you scan the region for `persistent-face'
>> properties and copy those to the `face' property.

> Right (ridiculous).

If you don't like it, change it.  It's Free Software.

> I would myself, if FSF could accept it without employer papers.

I'm sure you could get papers from your employer if you spent enough time
pestering them.

> I can't believe it would be hard for font-lock to check if text it
> wants to work with has an ignore-me property.

Nobody asks you to believe it would be hard.


        Stefan

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> > I can't believe it would be hard for font-lock to check if text it
> > wants to work with has an ignore-me property.
>
> Nobody asks you to believe it would be hard.

To show what I meant, here's a naive implementation that seems to work.

1. In font-lock.el, use this everywhere that `put-text-property' is used
now:

(defun put-text-property-unless-ignore (start end property value &optional
object)
  "`put-text-property', but ignore text with property `font-lock-ignore'."
  (let ((here (min start end))
        (end1 (max start end)))
    (while (< here end1)
      (unless (get-text-property here 'font-lock-ignore object)
        (put-text-property here (1+ here) property value object))
      (setq here (1+ here)))))

2. In font-lock.el, use this definition of
`font-lock-default-unfontify-region':

(defun font-lock-default-unfontify-region (beg end)
  "Unfontify from BEG to END, unless text with property `font-lock-ignore'."
  (let ((here (min beg end))
        (end1 (max beg end)))
    (while (< here end1)
      (unless (get-text-property here 'font-lock-ignore)
        (remove-list-of-text-properties
         here (1+ here) (append font-lock-extra-managed-props
                                (if font-lock-syntactic-keywords
                                    '(syntax-table face font-lock-multiline)
                                  '(face font-lock-multiline)))))
      (setq here (1+ here)))))

That is enough to get the behavior I described: putting a `font-lock-ignore'
property on selected portions of text tells font-lock "hands off". Try it,
for instance, with highlighting from `facemenu-set-face' (`M-o o') - see end
of mail.

I'm not suggesting these definitions should be used as is. They should be
enough, however, to show what I meant. They should point out that the task
is not that difficult, at least in terms of functionality. Or perhaps they
will point out what I'm misunderstanding or missing - please let me know in
that case. So far, this works OK for me.

Caveats:

a. These are obviously naive implementations, advancing only a character at
a time. The functionality seems to be OK, but the performance could be
improved. Perhaps `text-property-any' (or `-not-all') or
`next-single-property-change' could be used to let
`put-text-property-unless-ignore' process larger stretches of text at a
time.

b. So far, I haven't actually seen fontification being noticeably slower,
but the modified version of `font-lock-default-unfontify-region' is
unacceptably slow, for sure, with a large buffer. It can't simply remove
properties from BEG to END in one fell swoop as the original version does.
To obtain acceptable performance here, it would perhaps be appropriate to
write another built-in (C) analogous to `remove-list-of-text-properties',
but which allows for a similar test.

c. This doesn't take care of any MATCHER functions that might be used in
font-lock specs. Unless coded explicitly to be sensitive to property
`font-lock-ignore', they would act as they do now. I think that's
acceptable, but users would need to be made aware of that.

d. There might be some boundary cases that would need to be tweaked. For
instance, I imagine that (put-text-property pos pos ...) puts the property
at position pos, but (put-text-property-unless-ignore pos pos ...) does
nothing. I don't know whether the current uses of `put-text-property' in
font-lock.el really need to treat the start=end case.

If an Emacs developer were to implement something like this in a performant
way and add it to Emacs, it would mean that facemenu and hilock highlighting
could be used in font-locked buffers, which I think many users would
appreciate. facemenu.el would also need a minor change to
`facemenu-add-face', to put property `font-lock-ignore' on the text it
highlights. hilock.el would similarly need tweaking.

If you want to try this to see what it's like (including performance
penalties), add this line to `facemenu-add-face':

                                ;; Specify the selected frame
                                ;; because nil would mean to use
                                ;; the new-frame default settings,
                                ;; and those are usually nil.
                                (selected-frame))))
++        (put-text-property part-start part-end 'font-lock-ignore t))
        (setq part-start part-end)))

Then load library library `font-lock+.el', which is essentially the code
shown at the top. Use `M-o o' to highlight portions of text, and then turn
font-locking off and on to see the effect and the performance.

Alternatively, you can try it using library `highlight.el' and library
`font-lock+.el'. Do (define-key ctl-x-map [(control ?y)] 'highlight), and
then use `C-x C-y' to highlight (and unhighlight) text. Oh, and you will
need to set `highlight-use-overlays-flag' to nil - the default value uses
overlays, which font lock of course doesn't interfere with.

The code is here:

http://www.emacswiki.org/cgi-bin/wiki/font-lock%2b.el

http://www.emacswiki.org/cgi-bin/wiki/highlight.el

Re: how to prevent font-lock from messing with a portion of text?

[Lennart Borgman (gmail)]

Drew Adams wrote:
>>> I can't believe it would be hard for font-lock to check if text it
>>> wants to work with has an ignore-me property.
>> Nobody asks you to believe it would be hard.
> 
> To show what I meant, here's a naive implementation that seems to work.
> 
> 1. In font-lock.el, use this everywhere that `put-text-property' is used
> now:
> 
> (defun put-text-property-unless-ignore (start end property value &optional
> object)
>   "`put-text-property', but ignore text with property `font-lock-ignore'."
>   (let ((here (min start end))
>         (end1 (max start end)))
>     (while (< here end1)
>       (unless (get-text-property here 'font-lock-ignore object)
>         (put-text-property here (1+ here) property value object))
>       (setq here (1+ here)))))
> 
> 2. In font-lock.el, use this definition of
> `font-lock-default-unfontify-region':
> 
> (defun font-lock-default-unfontify-region (beg end)
>   "Unfontify from BEG to END, unless text with property `font-lock-ignore'."
>   (let ((here (min beg end))
>         (end1 (max beg end)))
>     (while (< here end1)
>       (unless (get-text-property here 'font-lock-ignore)
>         (remove-list-of-text-properties
>          here (1+ here) (append font-lock-extra-managed-props
>                                 (if font-lock-syntactic-keywords
>                                     '(syntax-table face font-lock-multiline)
>                                   '(face font-lock-multiline)))))
>       (setq here (1+ here)))))
> 
> That is enough to get the behavior I described: putting a `font-lock-ignore'
> property on selected portions of text tells font-lock "hands off". Try it,
> for instance, with highlighting from `facemenu-set-face' (`M-o o') - see end
> of mail.

Don't you have to take care of font-lock-default-fontify-region too?

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> Don't you have to take care of font-lock-default-fontify-region too?

Not that I can see. Did you try it? It takes only a minute to try.

`font-lock-default-fontify-region' does its job through other functions,
such as `font-lock-unfontify-region',
`font-lock-fontify-syntactic-keywords-region',
`font-lock-fontify-syntactically-region', and
`font-lock-fontify-keywords-region'.

`font-lock-default-fontify-region' never adds any text properties itself. By
substituting `put-text-property-unless-ignore' for `put-text-property' in
those other functions, `font-lock-default-fontify-region' is "taken care of"
also.

Let me know if you see a problem with it.

Re: how to prevent font-lock from messing with a portion of text?

[Lennart Borgman (gmail)]

Drew Adams wrote:
>> Don't you have to take care of font-lock-default-fontify-region too?
> 
> Not that I can see. Did you try it? It takes only a minute to try.


Sorry, I was a bit unclear. You can not be sure 
font-lock-default-fontify-region is called. That depends on the setting 
of font-lock-fontify-region-function. (And if this frame-work is used 
for the fontifying.)

I guess if it is then your method might work.

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> Sorry, I was a bit unclear. You can not be sure
> font-lock-default-fontify-region is called. That depends on the setting
> of font-lock-fontify-region-function. (And if this frame-work is used
> for the fontifying.)
> I guess if it is then your method might work.

If the default fontify function is not used, then someone wants a
non-default behavior. AFAIK, there is no way to know whether that desired
behavior should respect `font-lock-ignore'. Mind-reading is not scheduled
for addition before Emacs 27, AFAIK ;-).

This seems similar to the MATCHER question. If `font-lock-ignore' or
equivalent is added to Emacs, then this would need to be documented, so
users understand it. IOW, `font-lock-ignore' is respected only by default
fontifying and default unfontifying.

Re: how to prevent font-lock from messing with a portion of text?

[Stefan Monnier]

> (defun put-text-property-unless-ignore (start end property value &optional
> object)
>   "`put-text-property', but ignore text with property `font-lock-ignore'."
>   (let ((here (min start end))
>         (end1 (max start end)))
>     (while (< here end1)
>       (unless (get-text-property here 'font-lock-ignore object)
>         (put-text-property here (1+ here) property value object))
>       (setq here (1+ here)))))

This will still override a face on a word inside a string, because
(get-text-property <string-start> 'font-lock-ignore object) will be nil.
You need to check for the presence of the property over the whole
start..end region.

> 2. In font-lock.el, use this definition of
> `font-lock-default-unfontify-region':

> (defun font-lock-default-unfontify-region (beg end)
>   "Unfontify from BEG to END, unless text with property `font-lock-ignore'."
>   (let ((here (min beg end))
>         (end1 (max beg end)))
>     (while (< here end1)
>       (unless (get-text-property here 'font-lock-ignore)
>         (remove-list-of-text-properties
>          here (1+ here) (append font-lock-extra-managed-props
>                                 (if font-lock-syntactic-keywords
>                                     '(syntax-table face font-lock-multiline)
>                                   '(face font-lock-multiline)))))
>       (setq here (1+ here)))))

Same thing here, except even more so.

Then the problem becomes that doing all those extra checks costs time, all
the time, for a feature which will be almost never used.

OK, here's another option: don't change anything to font-lock, don't fiddle
with it at all, just don't use text-properties to add your special faces.
Use overlays instead.
Problem solved,


        Stefan

Re: how to prevent font-lock from messing with a portion of text?

[Miles Bader]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> OK, here's another option: don't change anything to font-lock, don't fiddle
> with it at all, just don't use text-properties to add your special faces.
> Use overlays instead.

Typical "highlighter" applications seem a bit more suited to overlays
anyway (e.g., if I copy the text, I don't really expect the highlighting
to come along...).

I suppose if there's some reason to use text-properties, "hi-lock.el"
(in the distribution) could be used as an example of one way to do that.
[It actually _uses_ font-lock to do the highlighting which seems fairly
clever -- e.g., if you type more text after having highlighted a phrase,
the phrase gets automagically highlighted in the new text too!]

-Miles
-- 
Yo mama's so fat when she gets on an elevator it HAS to go down.

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> > OK, here's another option: don't change anything to font-lock,
> > don't fiddle with it at all, just don't use text-properties to
> > add your special faces. Use overlays instead.
>
> Typical "highlighter" applications seem a bit more suited to overlays
> anyway (e.g., if I copy the text, I don't really expect the highlighting
> to come along...).

You know, you need not read my posts, but if you don't read them, then you
need not reply to them either.

I explicitly stated that my own highlighting library lets you use either
overlays or text properties, and that using overlays is the default
behavior.

Text properties and overlays act differently, and there can be advantages to
each in different contexts. One is not a replacement for the other, whether
for "typical" highlighting or anything else.

> I suppose if there's some reason to use text-properties, "hi-lock.el"
> (in the distribution) could be used as an example of one way to do that.
> [It actually _uses_ font-lock to do the highlighting which seems fairly
> clever -- e.g., if you type more text after having highlighted a phrase,
> the phrase gets automagically highlighted in the new text too!]

I'm well aware of hi-lock, and I use it. There are advantages and
disadvantage to tying user highlighting to the font-lock mechanism. With it,
you cannot view the highlighting without font-lock-mode being on. Here is
one comparison of some different highlighting approaches:
http://www.emacswiki.org/cgi-bin/wiki/HighlightTemporarily.

Nothing wrong with hi-lock. Nothing wrong with facemenu. Different
approaches have different advantages. Why not let highlighting approaches,
such as facemenu, that use the `face' text property independently of whether
font-lock highlighting is on, be able to prevent font lock from erasing
their highlighting?

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> > (defun put-text-property-unless-ignore (start end property
> value &optional
> > object)
> >   "`put-text-property', but ignore text with property
> `font-lock-ignore'."
> >   (let ((here (min start end))
> >         (end1 (max start end)))
> >     (while (< here end1)
> >       (unless (get-text-property here 'font-lock-ignore object)
> >         (put-text-property here (1+ here) property value object))
> >       (setq here (1+ here)))))
>
> This will still override a face on a word inside a string, because
> (get-text-property <string-start> 'font-lock-ignore object) will be nil.
> You need to check for the presence of the property over the whole
> start..end region.

I have a string in an emacs-lisp-mode buffer. I highlight a word in the
string using property `face' and I put property `font-lock-ignore' on it
also. I turn font-lock on and off - the word highlighting stays. I modify
the file and save it - the word highlighting stays. I highlight the whole
string, or more than a word, with the highlighting starting or ending on or
off a word boundary - the highlighting stays. I don't see the problem you
mention.

Could you explain the problem differently (e.g. recipe to test)? I no doubt
misunderstand.

> > 2. In font-lock.el, use this definition of
> > `font-lock-default-unfontify-region':
>
> > (defun font-lock-default-unfontify-region (beg end)
> >   "Unfontify from BEG to END, unless text with property
> `font-lock-ignore'."
> >   (let ((here (min beg end))
> >         (end1 (max beg end)))
> >     (while (< here end1)
> >       (unless (get-text-property here 'font-lock-ignore)
> >         (remove-list-of-text-properties
> >          here (1+ here) (append font-lock-extra-managed-props
> >                                 (if font-lock-syntactic-keywords
> >                                     '(syntax-table face
> font-lock-multiline)
> >                                   '(face font-lock-multiline)))))
> >       (setq here (1+ here)))))
>
> Same thing here, except even more so.

Please give me a recipe to test. I don't say I don't believe you or that the
code above is foolproof, but I'd like to see what the problem is.

> Then the problem becomes that doing all those extra checks costs time, all
> the time, for a feature which will be almost never used.

I already spoke to performance. I don't claim that the above code is
optimized. I intended it to show in general terms what I meant, so there was
no misunderstanding about that.

> OK, here's another option: don't change anything to font-lock,
> don't fiddle
> with it at all, just don't use text-properties to add your special faces.
> Use overlays instead. Problem solved

As I said, my own highlighting library uses overlays by default. But
sometimes text properties are useful. Facemenu is one example. The
differences that generally make overlays more useful for highlighting also
mean that they are less useful in some contexts. The fact that copying and
pasting text doesn't pick up overlays is one example: sometimes you might
want the faces pasted.

Every difference implies advantages and disadvantages. Text properties are
associated with text; overlays are associated with buffer positions, and so
on. This request is about letting a user-applied `face' property play well
with font-locking. That's all.

As I said, I'm not glued to any particular mechanism or implementation for
allowing that. If it can be proven that any reasonable implementation would
slow things down unacceptably, then an option could be used to turn on the
more costly behavior.

As it stands now, any application that uses the `face' text property gets
that highlighting trampled on by font lock - there is no simple way for an
application to prevent that. That, to me, is undesirable.

Re: how to prevent font-lock from messing with a portion of text?

[Stefan Monnier]

>> >     (while (< here end1)
>> >       (unless (get-text-property here 'font-lock-ignore object)
>> >         (put-text-property here (1+ here) property value object))
>> >       (setq here (1+ here)))))
>> 
>> This will still override a face on a word inside a string, because
>> (get-text-property <string-start> 'font-lock-ignore object) will be nil.
>> You need to check for the presence of the property over the whole
>> start..end region.

> I have a string in an emacs-lisp-mode buffer. I highlight a word in the
> string using property `face' and I put property `font-lock-ignore' on it
> also. I turn font-lock on and off - the word highlighting stays. I modify
> the file and save it - the word highlighting stays. I highlight the whole
> string, or more than a word, with the highlighting starting or ending on or
> off a word boundary - the highlighting stays. I don't see the problem you
> mention.

Sorry, I misread your code.  I didn't pay attention to your loop.
Boy, this is way too inefficient.  You can make it a bit better using
next-property-change, at least.
But still, the performance impact is a problem, because the benefit is
so small.

> I already spoke to performance. I don't claim that the above code is
> optimized. I intended it to show in general terms what I meant, so there
> was no misunderstanding about that.

There's no misunderstanding.

>> OK, here's another option: don't change anything to font-lock, don't
>> fiddle with it at all, just don't use text-properties to add your special
>> faces.  Use overlays instead. Problem solved

> As I said, my own highlighting library uses overlays by default. But
> sometimes text properties are useful. Facemenu is one example.

I don't see any reason why facemenu should use text-properties rather than
overlays.

> As it stands now, any application that uses the `face' text property gets
> that highlighting trampled on by font lock - there is no simple way for an
> application to prevent that. That, to me, is undesirable.

We all agree.  The differnece is whether we consider it sufficiently serious
to warrant action and performance penalties.


        Stefan

RE: how to prevent font-lock from messing with a portion of text?

[Drew Adams]

> Sorry, I misread your code.  I didn't pay attention to your loop.
> Boy, this is way too inefficient.  You can make it a bit better using
> next-property-change, at least.

Yes, I mentioned that (I mentioned `next-single-property-change', actually).

> But still, the performance impact is
> a problem, because the benefit is so small.

It depends on what is observed in practice. I don't claim that others will
observe the same in all contexts, but I don't notice any slowdown during
fontification. There surely is a slowdown, because of the naive
implementation, but I don't notice it.

The question is, is the slowdown a problem in practice? I find that even
buffers that were large (~500K chars) present no problem. And very, very,
large buffers are not completely font-locked by default anyway. IOW, it's
not clear that this would really affect what users perceive.

As I mentioned, unfontifying is a different matter - there, the slowdown is
noticeable and unacceptable, and a C implementation would be needed to
compare with the current C implementation
(`remove-list-of-text-properties').

> >> OK, here's another option: don't change anything to font-lock, don't
> >> fiddle with it at all, just don't use text-properties to add
> >> your special faces.  Use overlays instead. Problem solved
>
> > As I said, my own highlighting library uses overlays by default. But
> > sometimes text properties are useful. Facemenu is one example.
>
> I don't see any reason why facemenu should use text-properties rather than
> overlays.

Facemenu is an example. I suspect that there are good reasons that it uses
the `face' text property (e.g. input of new text?), but I won't argue for or
against that use.

Applications other than facemenu, for whatever reasons, might well use
property `face' for things other than font-locking. There's nothing that
discourages them from doing that, and nothing should. Overlays are not a
general replacement for text properties.

As it stands now, applications cannot really do that and expect font lock to
play well with them. That was my point in suggesting that we turn your
(`persistant-face') suggestion on its head, and use a dedicated property for
font lock - IOW, not have font lock use property `face' at all. That would
be a solution that should have no performance penalty.

The point is that font-lock today uses property `face', and so do/will other
applications, and font lock is not a good neighbor, effectively erasing any
`face' property (erase: the visual effect, not the property itself) that it
does not "own".

The solution, as I see it, is to either 1) move font lock to its own
non-`face' property or 2) have some other mechanism to tell font lock "hands
off!" a given portion of text. I'd prefer the former, for a variety of
reasons; the latter is an ugly little hack, and entails some performance
hits.

> > As it stands now, any application that uses the `face' text
> > property gets that highlighting trampled on by font lock -
> > there is no simple way for an application to prevent that.
> > That, to me, is undesirable.
>
> We all agree.  The differnece is whether we consider it
> sufficiently serious to warrant action and performance penalties.

Cantoning font lock to its own text property (or properties) should not
imply a performance penalty. Doing a `font-lock-ignore' workaround such as I
suggested (but with an optimized implementation) is second best, AFAICT. I
hope some solution to font lock's bad-neighborliness is found (after the
release).