jit-lock doesn't honor font-lock-lines-before

jit-lock doesn't honor font-lock-lines-before

[martin rudalics]

The customizable variable `font-lock-lines-before' is not honored by
jit-lock mode.  That's irritating since jit-lock is turned on by default
and other font-lock variables _are_ respected when jit-lock is active.

Below find two proposals:

(1) Preserve the current state of affairs and provide the appropriate
     hints in the doc-string:

(defcustom font-lock-lines-before 1
   "*Number of lines before the changed text to include in refontification.

This variable is not considered by `jit-lock-mode' and other Font lock
support modes."
   :type 'integer
   :group 'font-lock
   :version "22.1")

     and the Elisp manual (which doesn't mention jit-lock so far):

  -- Variable: font-lock-lines-before
      This variable specifies the number of extra lines to consider when
      refontifying the buffer after each text change.  Font lock begins
      refontifying from that number of lines before the changed region.
      The default is 1, but using a larger value can be useful for
      coping with multi-line patterns.  This variable is not considered
      by `jit-lock-mode' and other Font lock support modes.

(2) Modify `jit-lock-after-change' appropriately by, for example, inserting

        (forward-line (- font-lock-lines-before))

     before

        (setq start (line-beginning-position))

I prefer proposal (1).

Re: jit-lock doesn't honor font-lock-lines-before

[Richard M. Stallman]

    (2) Modify `jit-lock-after-change' appropriately by, for example, inserting

	    (forward-line (- font-lock-lines-before))

	 before

	    (setq start (line-beginning-position))

That seems like the right fix.  Why do you think it isn't?

Re: jit-lock doesn't honor font-lock-lines-before

[Stefan Monnier]

> The customizable variable `font-lock-lines-before' is not honored by
> jit-lock mode.

I know what that means in therms of what the code does, but I'm wondering in
which circumstance it makes a visible difference to the end user.
Do you have a test case?


        Stefan

Re: jit-lock doesn't honor font-lock-lines-before

[martin rudalics]

 >     (2) Modify `jit-lock-after-change' appropriately by, for example, inserting
 >
 > 	    (forward-line (- font-lock-lines-before))
 >
 > 	 before
 >
 > 	    (setq start (line-beginning-position))
 >
 > That seems like the right fix.  Why do you think it isn't?
 >
 >

Because this means that every time I insert one character, redisplay
would refontify `font-lock-lines-before' in addition to the current
line.  I suppose that the impact of this variable has not been noticed
so far because jit-lock doesn't consider it.

On the other hand, jit-lock wouldn't be able to handle this variable in
the same way as font-lock does: font-lock refontifies immediately in the
after-change hook.  jit-lock's after-change hook would only reset the
fontified property to nil.  Whether and when these lines are refontified
would be _also_ decided by the redisplay engine.  And the latter doesn't
know about `font-lock-lines-before'.  For example, editing the first
line in a window would not "include the line before in refontification".

Re: jit-lock doesn't honor font-lock-lines-before

[martin rudalics]

Stefan Monnier wrote:
 >>The customizable variable `font-lock-lines-before' is not honored by
 >>jit-lock mode.
 >
 >
 > I know what that means in therms of what the code does, but I'm wondering in
 > which circumstance it makes a visible difference to the end user.
 > Do you have a test case?
 >
 >

Suppose not: Wouldn't that mean `font-lock-lines-before' is useless?

It's not very difficult to contrive test cases for this.  By default
`font-lock-multiline' is nil. `font-lock-fontify-anchored-keywords'
won't alter it - the appropriate lines have been commented out.  Write
an arbitrary multiline pattern.  Now font-lock won't give it the
`font-lock-multiline' text property and jit-lock not necessarily reset
its `fontified' text property after a change.  `font-lock-after-change',
on the other hand, may refontify it provided `font-lock-lines-before' is
large enough.

You might argue now that the user should customize `font-lock-multiline'
appropriately and major modes set it as necessary.  (By the way, is the
value `undecided' useful - I don't see tests like

`(eq font-lock-multiline t)' or `(eq font-lock-multiline 'undecided)'

anywhere?)  Let's therefore suppose `font-lock-multiline' is t in the
following example: I display a buffer in a new window, and jump
immediately to a line within a multiline pattern.  More precisely, the
first line of the pattern is before window-start and the last line of
the pattern appears within the window.  In this case, the pattern might
not be fontified properly - fontification triggered by redisplay does
not fontify text before window-start.  Modifying the first line of the
window will cause redisplay to consider only that line (and possibly
subsequent ones).  The pattern will not be fontified properly unless I
scroll to its beginning.  Plain font-lock, on the other hand, could
handle this case when `font-lock-lines-before' is set appropriately.

However, I believe that `font-lock-lines-before' is a brute force
approach to handle such cases and could remarkably slow down editing if
it were honored by jit-lock mode.  Multiline patterns are too delicate
and should be treated in a completely different way.

Re: jit-lock doesn't honor font-lock-lines-before

[Richard M. Stallman]

    Because this means that every time I insert one character, redisplay
    would refontify `font-lock-lines-before' in addition to the current
    line.

Would that be bad?

      Whether and when these lines are refontified
    would be _also_ decided by the redisplay engine.  And the latter doesn't
    know about `font-lock-lines-before'.  For example, editing the first
    line in a window would not "include the line before in refontification".

This is a good point.  However, it can probably be fixed by a change in
the fontification function--when it is called, it should extend the starting
point for refontification upward by the specified number of lines.
(This might be the only fix needed.)

Re: jit-lock doesn't honor font-lock-lines-before

[Stefan Monnier]

>>> The customizable variable `font-lock-lines-before' is not honored by
>>> jit-lock mode.
>> 
>> I know what that means in therms of what the code does, but I'm wondering in
>> which circumstance it makes a visible difference to the end user.
>> Do you have a test case?

> Suppose not: Wouldn't that mean `font-lock-lines-before' is useless?

Not at all.  font-lock-lines-before has not been introduced to force
refontification of the previous lines but because the N previous lines are
needed as context in order to properly refontify the current line.
Now the code does in fact refontify the previous lines, but it is
a side-effect rather than one of the original goals.

> It's not very difficult to contrive test cases for this.  By default
> `font-lock-multiline' is nil. `font-lock-fontify-anchored-keywords'
> won't alter it - the appropriate lines have been commented out.  Write
> an arbitrary multiline pattern.  Now font-lock won't give it the
> `font-lock-multiline' text property and jit-lock not necessarily reset
> its `fontified' text property after a change.  `font-lock-after-change',
> on the other hand, may refontify it provided `font-lock-lines-before' is
> large enough.

Please show me a test case.

> However, I believe that `font-lock-lines-before' is a brute force
> approach to handle such cases and could remarkably slow down editing if
> it were honored by jit-lock mode.  Multiline patterns are too delicate
> and should be treated in a completely different way.

Indeed, but nobody has put the work needed to handle them properly.
font-lock-multiline is one hack, font-lock-lines-before is another,
font-lock-fontification-face-function is yet another.


        Stefan

Re: jit-lock doesn't honor font-lock-lines-before

[Richard M. Stallman]

    Because this means that every time I insert one character, redisplay
    would refontify `font-lock-lines-before' in addition to the current
    line.

That might be somewhat slower--but how much?  Would it even be
noticeable?

      jit-lock's after-change hook would only reset the
    fontified property to nil.  Whether and when these lines are refontified
    would be _also_ decided by the redisplay engine.

It's not decided by the redisplay engine alone.  When calls for
a certain region to be fontified, jit-lock could be changed to look
at the previous line too.

Re: jit-lock doesn't honor font-lock-lines-before

[martin rudalics]

 >     Because this means that every time I insert one character, redisplay
 >     would refontify `font-lock-lines-before' in addition to the current
 >     line.
 >
 > Would that be bad?

It would slow down editing long lines.  jit-lock is supposed to make
editing faster.

 >
 >       Whether and when these lines are refontified
 >     would be _also_ decided by the redisplay engine.  And the latter doesn't
 >     know about `font-lock-lines-before'.  For example, editing the first
 >     line in a window would not "include the line before in refontification".
 >
 > This is a good point.  However, it can probably be fixed by a change in
 > the fontification function--when it is called, it should extend the starting
 > point for refontification upward by the specified number of lines.
 > (This might be the only fix needed.)
 >
 >

In `font-lock-default-fontify-region', immediately before the comment

           ;; check to see if we should expand the beg/end area for
           ;; proper multiline matches

insert either

           (setq beg (line-beginning-position (1- font-lock-lines-before)))

or - but this would have to be mentioned in the doc-string of
`font-lock-lines-before' -

           (when font-lock-multiline
             (setq beg (line-beginning-position (1- font-lock-lines-before))))

remove the corresponding line from `font-lock-after-change-function' and
wait what happens?  I'm still sceptical, though ... fontifications would
always overlap by `font-lock-lines-before' lines.

Re: jit-lock doesn't honor font-lock-lines-before

[martin rudalics]

 >>Suppose not: Wouldn't that mean `font-lock-lines-before' is useless?
 >
 >
 > Not at all.

Then you mean that `font-lock-lines-before' is useless with jit-lock?

 > ... font-lock-lines-before has not been introduced to force
 > refontification of the previous lines but because the N previous lines are
 > needed as context in order to properly refontify the current line.
 > Now the code does in fact refontify the previous lines, but it is
 > a side-effect rather than one of the original goals.
 >
 >

When the "N previous lines" have not been fontified yet because they are
before `window-start' you do have to fontify them anyway.  Otherwise any
- syntactical, for example - context infomation needed to fontify the
current line correctly might be invalid.

 >>It's not very difficult to contrive test cases for this.  By default
 >>`font-lock-multiline' is nil. `font-lock-fontify-anchored-keywords'
 >>won't alter it - the appropriate lines have been commented out.  Write
 >>an arbitrary multiline pattern.  Now font-lock won't give it the
 >>`font-lock-multiline' text property and jit-lock not necessarily reset
 >>its `fontified' text property after a change.  `font-lock-after-change',
 >>on the other hand, may refontify it provided `font-lock-lines-before' is
 >>large enough.
 >
 >
 > Please show me a test case.
 >
 >

Try

(defun foobar (bound)
   (while (re-search-forward "\\<foo\\>" bound 'bound)
     (save-excursion
       (when (re-search-forward "\\<bar\\>" (line-end-position 2) 'lep)
         (font-lock-prepend-text-property
          (match-beginning 0) (match-end 0) 'face 'font-lock-warning-face)))))

(defun foobar-add ()
   (interactive)
   (font-lock-add-keywords nil '(foobar . nil) t))

In some buffer do `foobar-add', write "foo" on one line and "bar" on the
next.  Plain font-lock will handle it, jit-lock won't.

 >>However, I believe that `font-lock-lines-before' is a brute force
 >>approach to handle such cases and could remarkably slow down editing if
 >>it were honored by jit-lock mode.  Multiline patterns are too delicate
 >>and should be treated in a completely different way.
 >
 >
 > Indeed, but nobody has put the work needed to handle them properly.
 > font-lock-multiline is one hack, font-lock-lines-before is another,
 > font-lock-fontification-face-function is yet another.

I don't know about font-lock-fontification-face-function.
`font-lock-multiline' is not a hack, at least not a priori.  It simply
has not been developed yet.  `font-lock-lines-before' is a brute hack.
However, until `font-lock-multiline' is implemented correctly, it might
be better than nothing.

Re: jit-lock doesn't honor font-lock-lines-before

[Stefan Monnier]

>>> Suppose not: Wouldn't that mean `font-lock-lines-before' is useless?
>> Not at all.
> Then you mean that `font-lock-lines-before' is useless with jit-lock?

....fumbling about...

Hmm.... now I see what you mean.  I simply assumed that the code for
font-lock-lines-before had been added at the same place as the code for
font-lock-multiline, so it would work just as well for jit-lock as for
lazy-lock or anything else.
As it currently stands, it indeed is broken.  Sorry for being so stubborn.

I suggest the patch below, which also sets the default value to 0: except
for some special modes which rely on font-lock-lines-before (and which
should thus set it explicitly to an appropriate value), setting it to
something more than 0 will just waste CPU.

> I don't know about font-lock-fontification-face-function.
> `font-lock-multiline' is not a hack, at least not a priori.  It simply
> has not been developed yet.  `font-lock-lines-before' is a brute hack.
> However, until `font-lock-multiline' is implemented correctly, it might
> be better than nothing.

I designed and implemented font-lock-multiline.  It's a hack.  It can't be
fixed.  If you want reliable and efficient multiline keywords, you have to
specify them differently (read: change/extend the format of
font-lock-keywords).  Although you are right that you could get it to work
more or less right for the special case of anchored keywords.


        Stefan


--- font-lock.el	22 aoû 2005 10:22:15 -0400	1.271
+++ font-lock.el	01 sep 2005 09:43:30 -0400	
@@ -293,7 +282,7 @@
 		 (integer :tag "size"))
   :group 'font-lock)
 
-(defcustom font-lock-lines-before 1
+(defcustom font-lock-lines-before 0
   "*Number of lines before the changed text to include in refontification."
   :type 'integer
   :group 'font-lock
@@ -1049,6 +1038,8 @@
 	  ;; Use the fontification syntax table, if any.
 	  (when font-lock-syntax-table
 	    (set-syntax-table font-lock-syntax-table))
+          (goto-char beg)
+          (setq beg (line-beginning-position (- 1 font-lock-lines-before)))
 	  ;; check to see if we should expand the beg/end area for
 	  ;; proper multiline matches
 	  (when (and font-lock-multiline
@@ -1105,8 +1096,7 @@
       (save-match-data
 	;; Rescan between start of lines enclosing the region.
 	(font-lock-fontify-region
-	 (progn (goto-char beg)
-		(forward-line (- font-lock-lines-before)) (point))
+	 (progn (goto-char beg) (forward-line 0) (point))
 	 (progn (goto-char end) (forward-line 1) (point)))))))
 
 (defun font-lock-fontify-block (&optional arg)

Re: jit-lock doesn't honor font-lock-lines-before

[Stefan Monnier]

> In `font-lock-default-fontify-region', immediately before the comment

>            ;; check to see if we should expand the beg/end area for
>            ;; proper multiline matches

> insert either

>            (setq beg (line-beginning-position (1- font-lock-lines-before)))

Oh, I see we violently agree.  I've committed the change.

> or - but this would have to be mentioned in the doc-string of
> `font-lock-lines-before' -

>            (when font-lock-multiline
>              (setq beg (line-beginning-position (1- font-lock-lines-before))))

I'd rather not force people to use font-lock-multiline at the same time as
font-lock-lines-before since they interact in non-trivial ways.
I'd actually rather recommend to avoid using the two together.

> remove the corresponding line from `font-lock-after-change-function' and
> wait what happens?  I'm still sceptical, though ... fontifications would
> always overlap by `font-lock-lines-before' lines.

The default setting of font-lock-lines-before makes it a nop (as it
should).  Major modes which rely on it should set it to the
appropriate value.


        Stefan

Re: jit-lock doesn't honor font-lock-lines-before

[martin rudalics]

 > The default setting of font-lock-lines-before makes it a nop (as it
 > should).  Major modes which rely on it should set it to the
 > appropriate value.

You should modify the corresponding entry in the Elisp manual too.

Re: jit-lock doesn't honor font-lock-lines-before

[Richard M. Stallman]

     > Would that be bad?

    It would slow down editing long lines.  jit-lock is supposed to make
    editing faster.

That would be bad if the slowdown is substantial.  But we need
to see if it is substantial in order to see if it is bad.

Re: jit-lock doesn't honor font-lock-lines-before

[Richard M. Stallman]

    I suggest the patch below, which also sets the default value to 0: except
    for some special modes which rely on font-lock-lines-before (and which
    should thus set it explicitly to an appropriate value), setting it to
    something more than 0 will just waste CPU.

If you change the default to 0, would you please modify C mode
to set it to 1?

Re: jit-lock doesn't honor font-lock-lines-before

[Stefan Monnier]

>> Would that be bad?
>     It would slow down editing long lines.  jit-lock is supposed to make
>     editing faster.

> That would be bad if the slowdown is substantial.  But we need
> to see if it is substantial in order to see if it is bad.

It's probably insignificant, and it's only for those modes which set
font-lock-lines-before to a non-0 value.


        Stefan

Re: jit-lock doesn't honor font-lock-lines-before

[Stefan Monnier]

>     I suggest the patch below, which also sets the default value to 0: except
>     for some special modes which rely on font-lock-lines-before (and which
>     should thus set it explicitly to an appropriate value), setting it to
>     something more than 0 will just waste CPU.

> If you change the default to 0, would you please modify C mode
> to set it to 1?

I don't know where it's needed and neither do I know what's the correct
place to put such a thing in CC-mode.
Alan or Martin?


        Stefan

Re: jit-lock doesn't honor font-lock-lines-before

[Richard M. Stallman]

    > If you change the default to 0, would you please modify C mode
    > to set it to 1?

    I don't know where it's needed and neither do I know what's the correct
    place to put such a thing in CC-mode.
    Alan or Martin?

Who does it is not crucial, but I don't want it to be overlooked.
Would whoever does this please respond to this message?


-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf