Conflicting overlays and the 'priority property

Conflicting overlays and the 'priority property

[Tassilo Horn]

Hi all,

I use Nicolaj Schumacher's highlight-symbol.el, which highlights the
symbol at point or some selected symbol by using hi-lock to add a
colorized background.  Now the problem is that this background change is
not visible on the current line, iff hl-line-mode is enabled.

Both use overlays, the former via hi-lock, the latter manually.  So
there is an overlay for the whole line, and in that line there's another
overlay for one symbol.  Both specify a face property with some
background color.  Ok, so there is a conflict.  It seems, that hl-line
always wins.

Reading the elisp manual, there is:

,----[ (info "(elisp)Overlay Properties") ]
| `priority'
|      This property's value (which should be a nonnegative integer
|      number) determines the priority of the overlay.  No priority, or
|      `nil', means zero.
| 
|      The priority matters when two or more overlays cover the same
|      character and both specify the same property; the one whose
|      `priority' value is larger overrides the other.  For the `face'
|      property, the higher priority overlay's value does not completely
|      override the other value; instead, its face attributes override
|      the face attributes of the lower priority `face' property.
`----

So I tried use that.  I changed hi-lock like this:

--8<---------------cut here---------------start------------->8---
diff -u -L /usr/share/emacs/23.1.91/lisp/hi-lock.el.gz -L /tmp/buffer-content-5991ty0 /tmp/jka-com5991f8D /tmp/buffer-content-5991ty0
--- /usr/share/emacs/23.1.91/lisp/hi-lock.el.gz
+++ /tmp/buffer-content-5991ty0
@@ -568,7 +568,7 @@
   "Inhibit the action of `hi-lock-font-lock-hook'.
 This is used by `hi-lock-set-pattern'.")
 
-(defun hi-lock-set-pattern (regexp face)
+(defun hi-lock-set-pattern (regexp face &optional priority)
   "Highlight REGEXP with face FACE."
   (let ((pattern (list regexp (list 0 (list 'quote face) t)))
 	;; The call to `font-lock-add-keywords' below might disable
@@ -596,7 +596,8 @@
               (let ((overlay (make-overlay (match-beginning 0) (match-end 0))))
                 (overlay-put overlay 'hi-lock-overlay t)
                 (overlay-put overlay 'hi-lock-overlay-regexp serial)
-                (overlay-put overlay 'face face))
+                (overlay-put overlay 'face face)
+		         (overlay-put overlay 'priority priority))
               (goto-char (match-end 0)))))))))
 
 (defun hi-lock-set-file-patterns (patterns)

Diff finished.  Sun Jan 17 18:55:02 2010
--8<---------------cut here---------------end--------------->8---

Then I changed highlight-symbol.el to call that function with a priority
value of 9999.

Unfortunately, that doesn't change anything.  Still hl-line-mode's
overlay takes precedence over the symbol overlay from highlight-symbol /
hi-lock.  But hl-line mode does not use the priority property, so I'm
pretty sure that the background face property applied by
highlight-symbol should take precedence over that of hl-line, at least
that would comply with the docs.

Am I doing something wrong, or is that a bug?  

Bye,
Tassilo

hi-lock and overlays (was: Conflicting overlays and the 'priority property)

[Tassilo Horn]

Hi all,

I was wrong with my previous mail.  The problem that hi-lock pattern on
the current line are not highlighted, when `hl-line-mode' is enabled is
not due to conflicting overlays.  The problem is that hi-lock uses
overlays only when the undocumented variable `font-lock-fontified' is
nil (see `hi-lock-set-pattern').  Else, it uses normal fontification,
and overlays (like the one for the current line created by
`hl-line-mode') take precedence over that.

So now I have some questions:

  - Why does hi-lock use two different styles of highlighting?  IMO, if
    a user does M-x highlight-regexp RET foo RET hi-yellow RET, he
    expects to get that highlighting even if there are other overlays,
    so creating an overlay (and maybe even setting a priority) is the
    only safe bet.

  - Why is the call to `font-lock-add-keywords' done unconditionally,
    although it's possible that overlays might be used.  Isn't that
    superfluous in the overlay case?

I've changed the definition of `hi-lock-set-pattern' like this, which
looks more correct to me.

--8<---------------cut here---------------start------------->8---
diff -u /home/horn/repos/el/emacs/trunk/lisp/hi-lock.el /tmp/buffer-content-6420a_A
--- /home/horn/repos/el/emacs/trunk/lisp/hi-lock.el	2010-01-17 19:26:05.248012589 +0100
+++ /tmp/buffer-content-6420a_A	2010-01-18 22:24:26.839291297 +0100
@@ -568,19 +568,24 @@
   "Inhibit the action of `hi-lock-font-lock-hook'.
 This is used by `hi-lock-set-pattern'.")
 
-(defun hi-lock-set-pattern (regexp face)
-  "Highlight REGEXP with face FACE."
-  (let ((pattern (list regexp (list 0 (list 'quote face) t)))
-	;; The call to `font-lock-add-keywords' below might disable
-	;; and re-enable font-lock mode.  If so, we don't want
-	;; `hi-lock-font-lock-hook' to run.  This can be removed once
-	;; Bug#635 is fixed. -- cyd
-	(hi-lock--inhibit-font-lock-hook t))
+(defun hi-lock-set-pattern (regexp face &optional permit-font-lock)
+  "Highlight REGEXP with face FACE.
+If PERMIT-FONT-LOCK is non-nil, allow normal font-lock
+fontification instead of creating overlays.  In that case,
+matches of REGEXP won't be highlighted, if they occur inside an
+overlay, because overlays take precedence over normal
+fontification."
+  (let ((pattern (list regexp (list 0 (list 'quote face) t))))
     (unless (member pattern hi-lock-interactive-patterns)
-      (font-lock-add-keywords nil (list pattern) t)
       (push pattern hi-lock-interactive-patterns)
-      (if font-lock-fontified
-          (font-lock-fontify-buffer)
+      (if (and font-lock-fontified permit-font-lock)
+          (let (;; The call to `font-lock-add-keywords' below might disable
+		;; and re-enable font-lock mode.  If so, we don't want
+		;; `hi-lock-font-lock-hook' to run.  This can be removed once
+		;; Bug#635 is fixed. -- cyd
+		(hi-lock--inhibit-font-lock-hook t))
+	    (font-lock-add-keywords nil (list pattern) t)
+	    (font-lock-fontify-buffer))
         (let* ((serial (hi-lock-string-serialize regexp))
                (range-min (- (point) (/ hi-lock-highlight-range 2)))
                (range-max (+ (point) (/ hi-lock-highlight-range 2)))
--8<---------------cut here---------------end--------------->8---

I've tested the function with PERMIT-FONT-LOCK set to t, and then it
acts exactly as it does right now, e.g. it highlights every occurence of
REGEXP that is not inside an overlay.

When called with PERMIT-FONT-LOCK set to nil (or omitted), it always
creates overlays and the highlighting is done even if the REGEXP matches
inside other overlays.

Do you think my change is reasonable and should be committed?  Or is
there a rationale for the current way that I didn't grasp till now?

Bye,
Tassilo

Re: hi-lock and overlays (was: Conflicting overlays and the 'priority property)

[Lennart Borgman]

On Mon, Jan 18, 2010 at 10:33 PM, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Hi all,
>
> I was wrong with my previous mail.  The problem that hi-lock pattern on
> the current line are not highlighted, when `hl-line-mode' is enabled is
> not due to conflicting overlays.  The problem is that hi-lock uses
> overlays only when the undocumented variable `font-lock-fontified' is
> nil (see `hi-lock-set-pattern').  Else, it uses normal fontification,
> and overlays (like the one for the current line created by
> `hl-line-mode') take precedence over that.


I would rather turn the question this way:

- Why can't an overlay have a priority that is lower than text?

Re: hi-lock and overlays

[Tassilo Horn]

Lennart Borgman <lennart.borgman@gmail.com> writes:

Hi Lennart,

>> I was wrong with my previous mail.  The problem that hi-lock pattern
>> on the current line are not highlighted, when `hl-line-mode' is
>> enabled is not due to conflicting overlays.  The problem is that
>> hi-lock uses overlays only when the undocumented variable
>> `font-lock-fontified' is nil (see `hi-lock-set-pattern').  Else, it
>> uses normal fontification, and overlays (like the one for the current
>> line created by `hl-line-mode') take precedence over that.
>
>
> I would rather turn the question this way:
>
> - Why can't an overlay have a priority that is lower than text?

,----[ (info "(elisp)Overlay Properties") ]
| `priority'
|      This property's value (which should be a nonnegative integer
|      number) determines the priority of the overlay.  No priority, or
|      `nil', means zero.
| 
|      The priority matters when two or more overlays cover the same
|      character and both specify the same property; the one whose
|      `priority' value is larger overrides the other.  For the `face'
|      property, the higher priority overlay's value does not completely
|      override the other value; instead, its face attributes override
|      the face attributes of the lower priority `face' property.
| 
|      Currently, all overlays take priority over text properties.  Please
|      avoid using negative priority values, as we have not yet decided
|      just what they should mean.
`----

Hm, reading the last paragraph, this would be an option.  A negative
priority could be used to make text properties precede overlay
properties.

Anyway, the two other questions of my previous mail still stand.

Bye,
Tassilo

Re: hi-lock and overlays

[Stefan Monnier]

>   - Why does hi-lock use two different styles of highlighting?  IMO, if
>     a user does M-x highlight-regexp RET foo RET hi-yellow RET, he
>     expects to get that highlighting even if there are other overlays,
>     so creating an overlay (and maybe even setting a priority) is the
>     only safe bet.

overlays are implemented inefficiently, so if you have many of them,
redisplay, buffer modifications, and other operations can become
noticeably slower.


        Stefan

Re: hi-lock and overlays

[Tassilo Horn]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>>   - Why does hi-lock use two different styles of highlighting?  IMO, if
>>     a user does M-x highlight-regexp RET foo RET hi-yellow RET, he
>>     expects to get that highlighting even if there are other overlays,
>>     so creating an overlay (and maybe even setting a priority) is the
>>     only safe bet.
>
> overlays are implemented inefficiently, so if you have many of them,
> redisplay, buffer modifications, and other operations can become
> noticeably slower.

I see, but still I think that deciding between font-lock and overlays by
the value of `font-lock-fontified' is weird, and I'd vote for an
optional parameter that lets one specify the method to be used
explicitly.  What do yau think?

Ok, back to the original problem that hl-line-mode's overlay supresses
the font-lock highlighting added by hi-lock.  The cleanest solution
seems to be to decide on the usage of negative overlay priorities to
make font-lock take precedence here, and hl-line-mode's overlay would
use one.  But I don't know how feasible that is.

Bye,
Tassilo