Need help with search based font-locking

Need help with search based font-locking

[Tassilo Horn]

Hi all,

in my greql-mode [1], I setup font-locking like that:

--8<---------------cut here---------------start------------->8---
(define-derived-mode greql-mode text-mode "GReQL"
  "A major mode for GReQL2."
  [...]
  (setq font-lock-defaults
        '((greql-fontlock-keywords-1
           greql-fontlock-keywords-2
           greql-fontlock-keywords-3)))
  (add-hook 'after-save-hook 'greql-set-fontlock-keywords-3 t t)
  [...]
  (define-key greql-mode-map (kbd "C-c C-f") 'greql-format))
--8<---------------cut here---------------end--------------->8---

`greql-fontlock-keywords-3's value is not static, it frequently changes.
So I update it after saving (as you can see), and at several other
places.  The updating of the variable works, but how do I tell font-lock
that it should use the new value and re-fontify the current buffer?

As a brute-force workaround, I can do

   (set (make-local-variable 'font-lock-keywords)
        greql-fontlock-keywords-3)
   (redisplay t)

but I guess this is very bad style.  And even then some things that
should be highlighted are not.  For those, I need to delete a char and
add it back to apply the new fontification.

I really tried to dig into the elisp manual, but I couldn't find any
help with font-lock-defaults KEYWORDS that aren't fixed.

Thanks for help!
Tassilo
__________
[1] GReQL is a graph query language developed at our institute.

Re: Need help with search based font-locking

[Lennart Borgman]

On Tue, Dec 22, 2009 at 10:45 PM, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Hi all,
>
> in my greql-mode [1], I setup font-locking like that:
>
> --8<---------------cut here---------------start------------->8---
> (define-derived-mode greql-mode text-mode "GReQL"
>  "A major mode for GReQL2."
>  [...]
>  (setq font-lock-defaults
>        '((greql-fontlock-keywords-1
>           greql-fontlock-keywords-2
>           greql-fontlock-keywords-3)))
>  (add-hook 'after-save-hook 'greql-set-fontlock-keywords-3 t t)
>  [...]
>  (define-key greql-mode-map (kbd "C-c C-f") 'greql-format))
> --8<---------------cut here---------------end--------------->8---
>
> `greql-fontlock-keywords-3's value is not static, it frequently changes.
> So I update it after saving (as you can see), and at several other
> places.  The updating of the variable works, but how do I tell font-lock
> that it should use the new value and re-fontify the current buffer?
>
> As a brute-force workaround, I can do
>
>   (set (make-local-variable 'font-lock-keywords)
>        greql-fontlock-keywords-3)
>   (redisplay t)
>
> but I guess this is very bad style.  And even then some things that
> should be highlighted are not.  For those, I need to delete a char and
> add it back to apply the new fontification.
>
> I really tried to dig into the elisp manual, but I couldn't find any
> help with font-lock-defaults KEYWORDS that aren't fixed.


Maybe you can use jit-lock-refontify?

Re: Need help with search based font-locking

[Tassilo Horn]

Lennart Borgman <lennart.borgman@gmail.com> writes:

Hey Lennart,

>> I really tried to dig into the elisp manual, but I couldn't find any
>> help with font-lock-defaults KEYWORDS that aren't fixed.
>
> Maybe you can use jit-lock-refontify?

Hm, I have

  (setq font-lock-defaults
        '((greql-fontlock-keywords-1
           greql-fontlock-keywords-2
           greql-fontlock-keywords-3)))

in my mode setup.  Then I change the value of greql-fontlock-keywords-3
and call `jit-lock-refontify'.  But it doesn't apply the new
fontification.  And the reason is, that font-lock-keywords isn't
updated.

Then I digged a bit into the font-lock code, and at least it looks like
`font-lock-set-defaults' is the function I'm looking for.  So now I use
this:

  (let (font-lock-set-defaults) (font-lock-set-defaults))
  (jit-lock-refontify (point-min) (point-max))
  (redisplay t)

When greql-fontlock-keywords-3 changes in a way, that elements that were
highlighted before shouldn't be anymore, it works instantly.  But the
other way round doesn't.  I'll demonstrate that with an example:

In the GReQL language, you can import schema elements to formulate
queries using shorted names.  The font-locking in level 3 should
highlight existing, valid element names.

For example, the "Class" in V{Class} should be highlighted, because
frontend.java.Class is imported:

--8<---------------cut here---------------start------------->8---
import frontend.java.Class;
sort(
  from pe : V{Class}
  with count(pe <--{frontend.java.PackageContainsElement}) = 0
  reportSet
    from inner : pe -->{frontend.java.PackageContainsElement}
    reportSet describe(inner)
    end
  end
)
--8<---------------cut here---------------end--------------->8---

When writing the import statement and saving, the fontification will
eventually appear, but I need to insert a line break, so that V{Class}
is on another line.  Does font-lock skip unchanged lines somehow?  If
so, how do I change that?

Bye,
Tassilo

Re: Need help with search based font-locking

[Tassilo Horn]

Tassilo Horn <tassilo@member.fsf.org> writes:

Hi all,

> When greql-fontlock-keywords-3 changes in a way, that elements that were
> highlighted before shouldn't be anymore, it works instantly.  But the
> other way round doesn't.

Ups, that was a error by me.  So this topic is solved. ;-)

Bye,
Tassilo

Re: Need help with search based font-locking

[Stefan Monnier]

>   (let (font-lock-set-defaults) (font-lock-set-defaults))
>   (jit-lock-refontify (point-min) (point-max))
>   (redisplay t)

This assumes lots of things about font-lock's implementation (including
the use of jit-lock).
A slightly less invasive implementation would be:

  (font-lock-mode -1)
  (kill-local-variable 'font-lock-set-defaults)
  (font-lock-mode 1)

tho the `font-lock-set-defaults' bit is still ugly.

So, I'd recommend you submit a patch which adds a new function that does
the above 3 steps (call it `font-lock-refresh-all' or something), and
then use that one.

In your case, tho, a better option might be to change your
greql-font-lock-keywords-3 so it doesn't change.
See sh-font-lock-here-doc for an example of how you might be able to do
that.  Admittedly, this may not always work because it may depend on the
order the hilighting is done (and this order is not necessarily
sequential for font-lock-keywords).  So you may need to move some of the
work (the one that modifies the match regexp) to
font-lock-syntactic-keywords (which is guaranteed to be applied
sequentially).


        Stefan

Re: Need help with search based font-locking

[Lennart Borgman]

On Thu, Dec 24, 2009 at 4:07 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>>   (let (font-lock-set-defaults) (font-lock-set-defaults))
>>   (jit-lock-refontify (point-min) (point-max))
>>   (redisplay t)
>
> This assumes lots of things about font-lock's implementation (including
> the use of jit-lock).
> A slightly less invasive implementation would be:
>
>  (font-lock-mode -1)
>  (kill-local-variable 'font-lock-set-defaults)
>  (font-lock-mode 1)
>
> tho the `font-lock-set-defaults' bit is still ugly.
>
> So, I'd recommend you submit a patch which adds a new function that does
> the above 3 steps (call it `font-lock-refresh-all' or something), and
> then use that one.

I think you meant Tassilo should do that but I might as well right the
patch so here it is:

Index: font-lock.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/font-lock.el,v
retrieving revision 1.354
diff -c -r1.354 font-lock.el
*** font-lock.el	2 Oct 2009 03:48:41 -0000	1.354
--- font-lock.el	24 Dec 2009 03:44:49 -0000
***************
*** 1767,1772 ****
--- 1767,1780 ----

  (defvar font-lock-set-defaults nil)	; Whether we have set up defaults.

+ (defun lock-font-refresh-defaults ()
+   "Refresh defaults and restart fontification.
+ Set defaults again as if function `font-lock-defaults' had not
+ been called and then restart fontification."
+   (before-lock-mode -1)
+   (kill-local-variable 'font-lock-set-defaults)
+   (font-lock-mode 1))
+
  (defvar font-lock-mode-major-mode)
  (defun font-lock-set-defaults ()
    "Set fontification defaults appropriately for this mode.

Re: Need help with search based font-locking

[Stefan Monnier]

> + (defun lock-font-refresh-defaults ()
> +   "Refresh defaults and restart fontification.
> + Set defaults again as if function `font-lock-defaults' had not
> + been called and then restart fontification."
> +   (before-lock-mode -1)
> +   (kill-local-variable 'font-lock-set-defaults)
> +   (font-lock-mode 1))

I thought I was the only one who could muster so many typos in so few
lines.  BTW, the docstring should be more radical without reference to
internal functions like font-lock-set-defaults, as in "Restart,
recomputing everything from scratch" and then explain that it's
typically used to let font-lock react to external changes in variables
like font-lock-defaults and keywords.


        Stefan

Re: Need help with search based font-locking

[Lennart Borgman]

On Thu, Dec 24, 2009 at 5:46 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
>> + (defun lock-font-refresh-defaults ()
>> +   "Refresh defaults and restart fontification.
>> + Set defaults again as if function `font-lock-defaults' had not
>> + been called and then restart fontification."
>> +   (before-lock-mode -1)
>> +   (kill-local-variable 'font-lock-set-defaults)
>> +   (font-lock-mode 1))
>
> I thought I was the only one who could muster so many typos in so few
> lines.  BTW, the docstring should be more radical without reference to
> internal functions like font-lock-set-defaults, as in "Restart,
> recomputing everything from scratch" and then explain that it's
> typically used to let font-lock react to external changes in variables
> like font-lock-defaults and keywords.


Ehum ;-)

This is slightly better:

Index: font-lock.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/font-lock.el,v
retrieving revision 1.354
diff -c -r1.354 font-lock.el
*** font-lock.el	2 Oct 2009 03:48:41 -0000	1.354
--- font-lock.el	24 Dec 2009 13:47:41 -0000
***************
*** 1767,1772 ****
--- 1767,1790 ----

  (defvar font-lock-set-defaults nil)	; Whether we have set up defaults.

+ (defun font-lock-refresh-defaults ()
+   "Restart fontification in current buffer after recomputing from defaults.
+ Recompute fontification variables using `font-lock-defaults' (or,
+ if nil, using `font-lock-defaults-alist') and
+ `font-lock-maximum-decoration'.  Then restart fontification.
+
+ Use this function when you have changed any of the above
+ variables directly.
+
+ Note: This function will erase modifications done by
+ `font-lock-add-keywords' or `font-lock-remove-keywords', but will
+ preserve `hi-lock-mode' highlighting patterns \(and any other
+ setting set up in `fontlock-mode-hook')."
+   (let (font-lock-mode-hook)
+     (font-lock-mode -1))
+   (kill-local-variable 'font-lock-set-defaults)
+   (font-lock-mode 1))
+
  (defvar font-lock-mode-major-mode)
  (defun font-lock-set-defaults ()
    "Set fontification defaults appropriately for this mode.

Re: Need help with search based font-locking

[Tassilo Horn]

Lennart Borgman <lennart.borgman@gmail.com> writes:

Hi Lennart,

> This is slightly better:
>
> Index: font-lock.el
> ===================================================================
> RCS file: /sources/emacs/emacs/lisp/font-lock.el,v
> retrieving revision 1.354
> diff -c -r1.354 font-lock.el
> *** font-lock.el	2 Oct 2009 03:48:41 -0000	1.354
> --- font-lock.el	24 Dec 2009 13:47:41 -0000
> ***************
> *** 1767,1772 ****
> --- 1767,1790 ----
>
>   (defvar font-lock-set-defaults nil)	; Whether we have set up defaults.
>
> + (defun font-lock-refresh-defaults ()
> +   "Restart fontification in current buffer after recomputing from defaults.
> + Recompute fontification variables using `font-lock-defaults' (or,
> + if nil, using `font-lock-defaults-alist') and
> + `font-lock-maximum-decoration'.  Then restart fontification.
> +
> + Use this function when you have changed any of the above
> + variables directly.
> +
> + Note: This function will erase modifications done by
> + `font-lock-add-keywords' or `font-lock-remove-keywords', but will
> + preserve `hi-lock-mode' highlighting patterns \(and any other
> + setting set up in `fontlock-mode-hook')."
> +   (let (font-lock-mode-hook)
> +     (font-lock-mode -1))
> +   (kill-local-variable 'font-lock-set-defaults)
> +   (font-lock-mode 1))
> +
>   (defvar font-lock-mode-major-mode)
>   (defun font-lock-set-defaults ()
>     "Set fontification defaults appropriately for this mode.

What's the reason for disabling `font-lock-mode-hook' while deactivating
`font-lock-mode', but not while enabling it again?

Bye,
Tassilo

Re: Need help with search based font-locking

[Lennart Borgman]

On Mon, Dec 28, 2009 at 2:20 PM, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Lennart Borgman <lennart.borgman@gmail.com> writes:
>
> Hi Lennart,


Hi Tassilo,


>> This is slightly better:
>>
>> Index: font-lock.el
>> ===================================================================
>> RCS file: /sources/emacs/emacs/lisp/font-lock.el,v
>> retrieving revision 1.354
>> diff -c -r1.354 font-lock.el
>> *** font-lock.el      2 Oct 2009 03:48:41 -0000       1.354
>> --- font-lock.el      24 Dec 2009 13:47:41 -0000
>> ***************
>> *** 1767,1772 ****
>> --- 1767,1790 ----
>>
>>   (defvar font-lock-set-defaults nil) ; Whether we have set up defaults.
>>
>> + (defun font-lock-refresh-defaults ()
>> +   "Restart fontification in current buffer after recomputing from defaults.
>> + Recompute fontification variables using `font-lock-defaults' (or,
>> + if nil, using `font-lock-defaults-alist') and
>> + `font-lock-maximum-decoration'.  Then restart fontification.
>> +
>> + Use this function when you have changed any of the above
>> + variables directly.
>> +
>> + Note: This function will erase modifications done by
>> + `font-lock-add-keywords' or `font-lock-remove-keywords', but will
>> + preserve `hi-lock-mode' highlighting patterns \(and any other
>> + setting set up in `fontlock-mode-hook')."
>> +   (let (font-lock-mode-hook)
>> +     (font-lock-mode -1))
>> +   (kill-local-variable 'font-lock-set-defaults)
>> +   (font-lock-mode 1))
>> +
>>   (defvar font-lock-mode-major-mode)
>>   (defun font-lock-set-defaults ()
>>     "Set fontification defaults appropriately for this mode.
>
> What's the reason for disabling `font-lock-mode-hook' while deactivating
> `font-lock-mode', but not while enabling it again?


Oh, thanks. I should have commented on that in the code.

It is just because hi-lock is turned off otherwise. But there is a variable

  hi-lock--inhibit-font-lock-hook

I could have used too. However I thought there might be other things
behaving similar to hi-lock in this respect.



> Bye,
> Tassilo
>
>
>

Re: Need help with search based font-locking

[Tassilo Horn]

Lennart Borgman <lennart.borgman@gmail.com> writes:

Hi Lennart,

>> What's the reason for disabling `font-lock-mode-hook' while
>> deactivating `font-lock-mode', but not while enabling it again?
>
> Oh, thanks. I should have commented on that in the code.
>
> It is just because hi-lock is turned off otherwise. But there is a
> variable
>
>   hi-lock--inhibit-font-lock-hook
>
> I could have used too. However I thought there might be other things
> behaving similar to hi-lock in this respect.

Hm, I think, in that case let-binding `hi-lock--inhibit-font-lock-hook'
would be more appropriate, because it's a special case to circumvent the
asymmetry of the function `hi-lock-font-lock-hook', which switches off
`hi-lock-mode' when `font-lock-mode' is disabled, but doesn't switch it
on in the other case.

If nobody objects, I'll commit version modified this way.

Bye,
Tassilo

Re: Need help with search based font-locking

[Tassilo Horn]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

Hi Stefan,

> In your case, tho, a better option might be to change your
> greql-font-lock-keywords-3 so it doesn't change.  See
> sh-font-lock-here-doc for an example of how you might be able to do
> that.

Now my greql-font-lock-keywords-3 are static, and contain two functions
that drive the search and are steered by another variable that may
change.  This given me much more flexibility (now I not only highlight
valid schema elements, but also put a warning face on unknown types,
thus 2 functions).

Bye,
Tassilo

Re: Need help with search based font-locking

[Stefan Monnier]

>> In your case, tho, a better option might be to change your
>> greql-font-lock-keywords-3 so it doesn't change.  See
>> sh-font-lock-here-doc for an example of how you might be able to do
>> that.

> Now my greql-font-lock-keywords-3 are static, and contain two functions
> that drive the search and are steered by another variable that may
> change.  This given me much more flexibility (now I not only highlight
> valid schema elements, but also put a warning face on unknown types,
> thus 2 functions).

Great to hear, it,


        Stefan