feature request: text property to prevent font-locking

feature request: text property to prevent font-locking

[Drew Adams]

Excuse me if I'm missing something that is already available.
And in that case, please let me know what it is, so I can use it.

I want to apply some ad-hoc highlighting with text property `face'
in a font-locked buffer (whether font-locking occurs before or after
I apply `face' for this highlighting).

I do not want font-lock to override this highlighting.  I do not want
to use `font-lock-keywords' (e.g. with KEEP) in order to create this
highlighting.  I just want to prevent font-lock from overriding it.

I want to apply some other text property to the highlighted text, to
tell font-lock not to fiddle with it ("hands off this text").

1. Does this possibility already exist in Emacs?  I looked but didn't
   find anything.

2. If not, can we please add it?

Wrt #2:

Years ago I added this feature to code that I use.  It works for me,
but I don't claim that the way I implemented it is the way to go.

The code I use is here:
http://www.emacswiki.org/emacs-en/download/font-lock%2b.el

Essentially I did this:

1. Defined function `put-text-property-unless-ignore', which is the
   same as `put-text-property' but ignores text that has property
   `font-lock-ignore'.

2. Redefined function `font-lock-default-unfontify-region' to not
   unfontify text that has property `font-lock-ignore',
   
3. Redefined these functions to use `put-text-property-unless-ignore'
   instead of `put-text-property':

   font-lock-prepend-text-property
   font-lock-append-text-property
   font-lock-fillin-text-property
   font-lock-apply-syntactic-highlight
   font-lock-fontify-syntactically-region
   font-lock-apply-highlight
   font-lock-fontify-anchored-keywords
   font-lock-fontify-keywords-region

   That is, the change to each of these functions is ONLY to use
   `put-text-property-unless-ignore' instead of `put-text-property'.

My implementation of `put-text-property-unless-ignore':

(defun put-text-property-unless-ignore (start end property value
                                        &optional object)
  "`put-text-property', but ignore text with property `font-lock-ignore'."
  (let ((here  (min start end))
        (end1  (max start end))
        chg)
    (while (< here end1)
      (setq chg  (next-single-property-change here 'font-lock-ignore
                                              object end1))
      (unless (get-text-property here 'font-lock-ignore object)
        (put-text-property here chg property value object))
      (setq here  chg))))

Perhaps there is a better way to do this.  Dunno.

In any case, my request is just for Emacs to have such a feature,
however it might be implemented.  Even better would be an answer that
Emacs already has something like this that I'm unaware of.

I use this feature for various highlighting (and unhighlighting)
commands, so that the highlighting works for font-locked text too.
This includes `facemenu-add-face' (my version).

Re: feature request: text property to prevent font-locking

[Alan Mackenzie]

Hi, Drew.

On Thu, Aug 28, 2014 at 02:18:28PM -0700, Drew Adams wrote:
> Excuse me if I'm missing something that is already available.
> And in that case, please let me know what it is, so I can use it.

> I want to apply some ad-hoc highlighting with text property `face'
> in a font-locked buffer (whether font-locking occurs before or after
> I apply `face' for this highlighting).

> I do not want font-lock to override this highlighting.  I do not want
> to use `font-lock-keywords' (e.g. with KEEP) in order to create this
> highlighting.  I just want to prevent font-lock from overriding it.

> I want to apply some other text property to the highlighted text, to
> tell font-lock not to fiddle with it ("hands off this text").

> 1. Does this possibility already exist in Emacs?  I looked but didn't
>    find anything.

> 2. If not, can we please add it?

[ .... ]

For this particular application of fontifying, you might be able to use
overlays, which take priority over text properties (see page "Overlay
Properties" in the Elisp manual).  You might have to define a lot of
attributes for the overlay face to prevent attributes from font-locking
"showing through".

That feels like a bit of a kludge, though.  The facility you've sketched
out (and implemented) might well be a useful one.

-- 
Alan Mackenzie (Nuremberg, Germany).

RE: feature request: text property to prevent font-locking

[Drew Adams]

> For this particular application of fontifying, you might be able to
> use overlays, which take priority over text properties (see page
> "Overlay Properties" in the Elisp manual).  You might have to define
> a lot of attributes for the overlay face to prevent attributes from
> font-locking "showing through".
> 
> That feels like a bit of a kludge, though.  The facility you've
> sketched out (and implemented) might well be a useful one.

Just in case I was not clear enough -

I do use overlays, when they are appropriate.  They are essentially
about buffer positions, not the text in the buffer.  But yes, one
can fiddle with overlays to work around the effective removal, in a
font-locked buffer, of the possibility of using `face' as a text
property for other than font-locking.  Like others, I have done that.

I suspect that people have gotten used to this behavior by font-lock
and act as if it were normal.

But font-lock does not (should not) own text-property `face', even
in font-locked buffers.  You should not need to resort to using
`face' with overlays, just to get around font-lock's effective
confiscation of it for buffer text.  You should be able to apply
`face' to any buffer text and have a way to prevent it being taken
over by font-lock there.

And yes, people sometimes try other ways to work around the
expropriation of `face' by font-lock.  They try to show links on
top of font-locked text by using property `display', for example.

And yes, people sometimes end up using font-lock itself to add
ad-hoc highlighting, not because it is particularly appropriate for
the particular use case, but because there is little choice to do
otherwise.

AFAICT, this is just a missing Emacs feature: be able to exclude
zones of text from being taken over by font-lock.  Stop font-lock
from changing property `face' from a value set by non font-lock
highlighting.

This feature is essentially what font-lock itself already provides for
given font-lock rules (keywords), via keyword `keep'.  The problem is
that Font-lock does not recognize any such protection outside
`font-lock-keywords'.

So again, I am not looking to solve a particular highlighting problem
by a workaround (overlays, other text properties such as `display',...).

I'm arguing that there should be a simple way to tell font-lock "Hands
off!" particular text.  It should be simple to indicate that property
`face' on this or that text is not open to being altered by font-lock.

And I do have a solution - a trivial patch, as I indicated.  And for
users a simple way to tell font-lock "Hands off!": just put text
property `font-lock-ignore' where you do not want font-lock to change
text properties.  I cannot see a downside to providing this.

But since your reply is the only one so far, Alan, I conclude so far
that there is no special interest in this -the problem or the
proposed solution.  That's OK by me - I'll continue to use the code
I have.  And others can continue to add a `display' property or
whatever to get around the roughshod behavior of font-lock.  Been
there, done that.

And just in case, I'll send the patch, with `report-emacs-bug'.

As I said:
 I use this feature for various highlighting (and unhighlighting)
 commands, so that the highlighting works for font-locked text too.
 This includes `facemenu-add-face' (my version).

I use it for links (`link' face), `facemenu-add-face' (any face),
`hlt-highlighter' & `hlt-eraser' (mouse-drag highlighter/eraser),
`hlt-region' (arbitrary (un)highlighting), `hlt-yank-props' (paste
copied text properties), and more.

Re: feature request: text property to prevent font-locking

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

() Drew Adams <drew.adams@oracle.com>
() Sat, 30 Aug 2014 13:15:46 -0700 (PDT)

   But since your reply is the only one so far, Alan, I conclude
   so far that there is no special interest in this -the problem
   or the proposed solution.

Maybe it takes a while for answers to arrive.

As you point out, font-lock must self-gate in some manner,
especially to avoid doing useless re-work.  Have you looked at
the source?  A brief dive shows me that

 ‘font-lock-mode’ docstring mentions
 ‘font-lock-fontify-buffer’ which calls
 ‘font-lock-fontify-buffer-function’ which has the value
 ‘jit-lock-refontify’ (for me), which calls
 ‘put-text-property’ w/
 property name ‘fontified’ and
 property value ‘nil’

suggesting that property ‘fontified’ w/ non-‘nil’ value is what
jit-lock must do as testimony of its work.  Now, that took all of
two minutes.  The next twenty or two-hundred (if i were lucky
enough to have them available) would be to understand what
weirdness must mar this simple model.  Maybe you can do that?

Regardless, i like the straightforward ‘font-lock-ignore’ method
you demo.  It would be nice to have that level of friendliness in
Emacs (and documented!).

-- 
Thien-Thi Nguyen
   GPG key: 4C807502
   (if you're human and you know it)
      read my lisp: (responsep (questions 'technical)
                               (not (via 'mailing-list)))
                     => nil

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

RE: feature request: text property to prevent font-locking

[Drew Adams]

>    But since your reply is the only one so far, Alan, I conclude
>    so far that there is no special interest in this -the problem
>    or the proposed solution.
> 
> Maybe it takes a while for answers to arrive.

Yes, maybe. That's why I called the conclusion tentative ("so far").

> As you point out, font-lock must self-gate in some manner,
> especially to avoid doing useless re-work.  Have you looked at
> the source?  A brief dive shows me that
> 
>  ‘font-lock-mode’ docstring mentions
>  ‘font-lock-fontify-buffer’ which calls
>  ‘font-lock-fontify-buffer-function’ which has the value
>  ‘jit-lock-refontify’ (for me), which calls
>  ‘put-text-property’ w/
>  property name ‘fontified’ and
>  property value ‘nil’
> 
> suggesting that property ‘fontified’ w/ non-‘nil’ value is what
> jit-lock must do as testimony of its work.  Now, that took all of
> two minutes.  The next twenty or two-hundred (if i were lucky
> enough to have them available) would be to understand what
> weirdness must mar this simple model.  Maybe you can do that?

I did look into that in the past (and again before my OP message).
And I did try simply setting `fontified' to non-nil, with no luck.

It's quite possible I am missing something simple, which is why
I asked about that.  I still would like to hear that that is the
case, and how.

> Regardless, i like the straightforward ‘font-lock-ignore’ method
> you demo.  It would be nice to have that level of friendliness in
> Emacs (and documented!).

Seems simple to me too.  But maybe there is something simpler
and already available.  After all, font lock was not added
yesterday.  And surely people have tried to use ad hoc
highlighting in a font-locked buffer.

If nothing else, they must have tried `facemenu-set-face'
(`M-o o') and `facemenu-add-face', which are about as old as
font-lock.  But maybe they tried and just gave up when they saw
the unfriendly message "Font-lock mode will override any faces
you set in this buffer", as if that were the final (discouraging)
word.

In any case, it is not the final word.  With the patch I sent
(bug #18367), font-lock no longer prevents you from using `M-o o'
etc.  (If the patch is accepted then that message should of
course be removed.)

That message's presence is another thing that gives me the
impression that I am maybe not missing something: if there were
already a simple way to stop font-lock from overriding facemenu
highlighting then why would we warn people that it does override
it?  Why wouldn't the `facemenu.el' code just DTRT and stop
font-lock's king-of-the-sandbox bullying ;-) for the space of
facemenu's ad hoc highlighting?

Re: feature request: text property to prevent font-locking

[Stefan Monnier]

By, the way, these kinds of problems are the reason why I suggested
adding "planes" to text-properties.

It should be relatively easy to implement.


        Stefan

Re: feature request: text property to prevent font-locking

[Wolfgang Jenkner]

On Thu, Aug 28 2014, Drew Adams wrote:

> I want to apply some ad-hoc highlighting with text property `face'
> in a font-locked buffer (whether font-locking occurs before or after
> I apply `face' for this highlighting).

> I do not want font-lock to override this highlighting.  I do not want
> to use `font-lock-keywords' (e.g. with KEEP) in order to create this
> highlighting.  I just want to prevent font-lock from overriding it.

So, let me try to understand the extent of the problem.

font-lock-default-unfontify-region leaves alone any "alternative"
(defined in char-property-alias-alist) of the `face' text property.

In particular, this is how the font-lock-face text property works.

Now, on the one hand, keyword fontification does not override
"alternatives" (except if the highlight pattern has an `override' flag).
This is because text-property-not-all recognizes them and thus prevents
font-lock-apply-highlight from adding a value for the `face' property.

On the other hand, some experimenting suggests that syntactic
fontification does override those "alternatives".

> I want to apply some other text property to the highlighted text,
> to tell font-lock not to fiddle with it ("hands off this text").

Remove it from the value of `font-lock-extra-managed-props'.

tl; dr: Existing facilities reduce the problem to clashes with syntactic
fontification.

Wolfgang

RE: feature request: text property to prevent font-locking

[Drew Adams]

> By, the way, these kinds of problems are the reason why I suggested
> adding "planes" to text-properties.
> 
> It should be relatively easy to implement.

Sorry, that's too vague for me.  But I guess it further confirms that
I am not missing something simple.

RE: feature request: text property to prevent font-locking

[Drew Adams]

> font-lock-default-unfontify-region leaves alone any "alternative"
> (defined in char-property-alias-alist) of the `face' text property.

"Leaves alone?"  It think it does the opposite: it removes its
highlighting: it treats the alternative property like it treats
`face' (and `font-lock-face').  But perhaps I misunderstand you.

> In particular, this is how the font-lock-face text property works.

Yes, and thanks for pointing that out, as it is not obvious how
`font-lock-face' is handled by `font-lock-default-unfontify-region'.

But again, IIUC, it does not "leave alone" `font-lock-face'd text.
It does just the opposite: it unfontifies it.

That's even worse wrt unhighlighting than not doing anything.
If you do nothing, then text that you have applied `face' to
will at least appear highlighted when you turn off `font-lock-mode'.

By using your suggested "leave alone" approach, you give font-lock
control over not only highlighting but unhighlighting, so your
ad hoc highlighting will never show, whether font-lock is on or off.

> Now, on the one hand, keyword fontification does not override
> "alternatives" (except if the highlight pattern has an `override'
> flag). This is because text-property-not-all recognizes them and
> thus prevents font-lock-apply-highlight from adding a value for
> the `face' property.

Yes.  That is the point, AFAICT, of `font-lock-face': to let you
give font-lock control over some ad hoc highlighting.  And yes,
you can use an "alternative" property for the same thing.

> On the other hand, some experimenting suggests that syntactic
> fontification does override those "alternatives".

Yes, I noticed that too.

But beyond that: What you are describing is just the control that
font-lock has over highlighting, and how one can give it control
over additional, non-`font-lock-keywords' highlighting by using
`font-lock-face' (or another property besides `face', designated
as a `face' "alternative").

What I am talking about is the opposite: Not giving font-lock
control over additional, ad hoc highlighting, but taking font-lock
control away, for given ad hoc highlighting.  I don't want turning
font-lock on or off to affect the given highlighting at all.

That's the point.  It's not that I'm looking for a way to let
font-lock control some non-`font-lock-keywords' highlighting.
That we can do already, using property `font-lock-face'.

(And my guess is that the fact that this is the only feature for
ad hoc highlighting that is compatible with font-lock is a reason
that we have seen an expansion of the use of `font-lock-face'.
IOW, if you want some ad hoc highlighting, you are currently
pretty much obliged to give font-lock control over it, by using
that approach.)

> > I want to apply some other text property to the highlighted text,
> > to tell font-lock not to fiddle with it ("hands off this text").
> 
> Remove it from the value of `font-lock-extra-managed-props'.

Remove what?  The highlighting property?  It's `face', which is
already not in `font-lock-extra-managed-props'.  The default value
of `font-lock-extra-managed-props' is nil (or it should be; and
hopefully it will be again, after bug #18343 gets fixed).

That's the point: font-lock should not "own" `face'.  And it
pretty much does currently.  Whether it did in the beginning I
don't know, but I kinda doubt it.  Property `face' was not
invented for font-lock, AFAIK.  Font-lock is one application of
the feature of faces; nothing more.  But it does not play well
with other uses of that feature - it is a sandbox bully.

> tl; dr: Existing facilities reduce the problem to clashes with
> syntactic fontification.

That's your point, not mine.  But yes, the feature I proposed
lets you protect given text also from overriding by syntactic
fontification.

RE: feature request: text property to prevent font-locking

[Drew Adams]

> With the patch I sent (bug #18367), font-lock no longer prevents
> you from using `M-o o' etc.  (If the patch is accepted then that
> message ["Font-lock mode will override any faces you set in this
> buffer"] should of course be removed.)

That statement is not clear; sorry.  The patch I sent is only
for font-lock.el.  It makes it possible to allow facemenu.el
to protect its highlighting from font-locking.  It does not, by
itself, protect facemenu highlighting.

For that, another patch is needed, which I will be glad to send
if this feature is accepted.  If you want to try it now to see,
just load file facemenu+.el, in addition to file font-lock+.el.

http://www.emacswiki.org/emacs-en/download/facemenu%2b.el
http://www.emacswiki.org/emacs-en/download/font-lock%2b.el

Re: feature request: text property to prevent font-locking

[Stefan Monnier]

>> By, the way, these kinds of problems are the reason why I suggested
>> adding "planes" to text-properties.
>> It should be relatively easy to implement.
> Sorry, that's too vague for me.  But I guess it further confirms that
> I am not missing something simple.

I suggested that there be several planes of properties, so font-lock
would place its properties in the plane `font-lock' while other packages
can use their own plane.  Then you'd separately specify rules for how to
merge the various planes (with rules that can be distinct for each kind
of property).
The merge of properties would not take place during redisplay but
instead would take place when the properties are added/removed.


        Stefan

RE: feature request: text property to prevent font-locking

[Drew Adams]

> >> By, the way, these kinds of problems are the reason why I
> >> suggested adding "planes" to text-properties.
> >> It should be relatively easy to implement.
> >
> > Sorry, that's too vague for me.  But I guess it further confirms
> > that I am not missing something simple.
> 
> I suggested that there be several planes of properties, so font-lock
> would place its properties in the plane `font-lock' while other
> packages can use their own plane.  Then you'd separately specify
> rules for how to merge the various planes (with rules that can
> be distinct for each kind of property). The merge of properties
> would not take place during redisplay but instead would take place
> when the properties are added/removed.

Doesn't sound particularly simple to use (can you say "usine a
gaz"?), even if it is "relatively easy to implement".  But I admit
that I have only a vague notion of what you have in mind.

While waiting for your implementation of that multifaceted feature,
perhaps we can adopt the simple change I suggested? (A bug fix, IMO.)

But again, I'm OK with just continuing to use the fix in my code,
if Emacs doesn't want it.

Re: feature request: text property to prevent font-locking

[Alan Mackenzie]

Hi, All.

On Sun, Aug 31, 2014 at 03:57:14PM -0400, Stefan Monnier wrote:
> >> By, the way, these kinds of problems are the reason why I suggested
> >> adding "planes" to text-properties.
> >> It should be relatively easy to implement.
> > Sorry, that's too vague for me.  But I guess it further confirms that
> > I am not missing something simple.
> 
> I suggested that there be several planes of properties, so font-lock
> would place its properties in the plane `font-lock' while other packages
> can use their own plane.  Then you'd separately specify rules for how to
> merge the various planes (with rules that can be distinct for each kind
> of property).
> The merge of properties would not take place during redisplay but
> instead would take place when the properties are added/removed.
> 
> 
>         Stefan
> 

I think you mean this:

#########################################################################
From: Stefan Monnier <monnier@iro.umontreal.ca>
To: emacs-devel@gnu.org
Date: Mon, 04 Oct 2010 01:37:26 +0200
Subject: Re: Combining face and map stuff

> Which reminds me of another thing.  I would like to define some
> commands
> local to a region, which can be done with 'local-map in overlays or
> 'keymap in text properties.  But can I combine them, too, in possibly
> overlapping regions, and get the aggregate keymap?

Here's an idea with which I've beeing toying:

- rather than have just one set of text-properties per char, make it
  possible to have several.  E.g. font-lock would use its own set of
  text properties.  One way to do that is that a `property' can now be
  a cons whose car is a "plane" and whose cdr is a property.
  So for example, the special font-lock-face (which currently gets
  "mapped" to face' via char-property-alias-alist) would just be `face'
  because font-lock would use (font-lock . face).
  A property like `face' would really be equivalent to (nil . face).

- then you add property-specific merge functions.  I.e. when looking up
  `face' you'd get the merge of all the `face' properties of the various
  text-property planes.  You'd probably want those merge functions to be
  written in Elisp and customizable.  So merging `keymap' properties
  becomes easy (well, it may benefit from multiple-keymap inheritance,
  which is a completely different topic).

- now font-lock can erase all the properties of the `font-lock' plane
  without worrying about erasing properties installed by other packages.

- to avoid calling Elisp code to merge things during text-property
  lookups, the merge would take place in put-text-property (i.e. no need
  to change the redisplay code at all).

- modifying a plane other than the default (nil) one could be considered
  as "not modifying the buffer" (just like adding/removing overlays).

- we could obsolete char-property-alias-alist which is only ever used
  by/for font-lock-face.

- maybe outline-minor-mode could use text-properties rather than
  overlays to make text invisible (tho it would make it impossible(?) to
  use outline-minor-mode in an indirect buffer without affecting the
  base buffer and the other indirect buffers).  So it might reduce the
  need for overlays (which have the disadvantage of being
  algorithmically slow/costly).


        Stefan
#########################################################################

This might be "relatively simple" to implement, but relative to what,
I'm not sure.  ;-)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: feature request: text property to prevent font-locking

[Stephen J. Turnbull]

Alan Mackenzie writes:

 > This might be "relatively simple" to implement, but relative to what,
 > I'm not sure.  ;-)

You're well on your way to reinventing "extents", you know. :-(

Re: feature request: text property to prevent font-locking

[Stefan Monnier]

>> This might be "relatively simple" to implement, but relative to what,
>> I'm not sure.  ;-)
> You're well on your way to reinventing "extents", you know. :-(

No, we did that already (except we called them overlays ;-)


        Stefan

Re: feature request: text property to prevent font-locking

[Stephen J. Turnbull]

Stefan Monnier writes:
 > >> This might be "relatively simple" to implement, but relative to what,
 > >> I'm not sure.  ;-)
 > > You're well on your way to reinventing "extents", you know. :-(
 > 
 > No, we did that already (except we called them overlays ;-)

Since you apparently need reminding, let me point out that Extents are
capable of providing both text property and overlay behavior, and
several useful combinations of both as well.  "Planes" of text
properties is just one example of behavior that is trivial to
implement with extents.  Drew's desired behavior is trivial to
implement given XEmacs's implementation of text properties (that seems
somewhat accidental to me).

Sure, there are few places where XEmacs implementations vary from
Emacs semantics, but IIRC that's by deliberate choice, with one
exception where XEmacs chose consistency of abstraction over 100%
Emacs compatibility.  99.44% of the time the implementations are not
distinguishable by end users.

Re: feature request: text property to prevent font-locking

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Overlays are not the same as extents.  Extents try to be both
text properties and overlays, and I think that mixture can't
be implemented without incoherent behaviors.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: feature request: text property to prevent font-locking

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> Overlays are not the same as extents.  Extents try to be both
> text properties and overlays, and I think that mixture can't
> be implemented without incoherent behaviors.

I'm not complete sure whether extents can or cannot cover all use cases
of overlays/properties: it was some time since I last looked.  The main
problem I had when working with them is that the purported generality is
elusive since part of the difference in Emacs' semantics of overlays and
text properties is encoded in extent properties while other parts are
encoded in the parameters you pass to the subroutines for accessing
extents.  The result is that the complexity for a particular application
tends to bleed all over the place whenever you don't restrict yourself
to the Emacs abstractions, most easily done by using compatibility
functions.

However, there is no point in Emacs not supporting overlay and text
properties identically whenever a particular property is not inherently
tied to the difference between overlay and text properties.
I definitely remember being annoyed in the past by some properties
requiring the use of one and not the other.  But I don't remember the
details right now.

Having the same underlying implementation for overlays and text
properties is of course a reliable way to have the same supported set of
properties by default.

-- 
David Kastrup