font-lock function matcher sample

font-lock function matcher sample

[Arjan Bos]

Hi,

Could anyone of you please post a sample for a font-lock function matcher?

I'm looking for a way to fontify words accoriding to their scrabble 
score. So if a word would score 6 points in american scrabble, it should 
get a certain font-locking.

Currently, I have the following elisp, but it does not work. I've been 
looking at some samples from the emacs lisp directories, but failed to 
work out the pattern. Also the info node on font-locking and the various 
font-locking doc-strings were helpful, but not helpful enough. My elisp 
swings between locking emacs (presumably on font-locking) and no 
font-locking at all. (this of course when I twiddle around with it)


(defvar scrabble-font-lock-keywords
   (list '(scrabble-6-matcher (1 font-lock-warning-face))
))

(defun scrabble-6-matcher (limit)
   "returns t when the scrabble score of a word is 6."
   (if (and (re-search-forward "\\([a-z]\\)*" limit t)
        (< (scrabble-last-word-score) 5))
       (progn
	(set-match-data
	 (list
	  (match-beginning 1) (match-end 1)
	  (match-beginning 1) (match-end 1)
	  nil nil)))
     ;; else
     (set-match-data
      (list
       (match-beginning 1) (match-end 1)
       nil nil
       (match-beginning 1) (match-end 1)))
     t))



(defun scrabble-last-word-score ()
   ""
   5)

TIA,

Arjan
-- 
--
If you really want to contact me, then replace the "I see you" text by 
its three letter accronym, hetnet.

Fabricate Diem PVNC, Motto of the Night Watch -- Terry Pratchett

Re: font-lock function matcher sample

[Alan Mackenzie]

Arjan Bos <Arjan.Bos@nospam.iseeyou.nl> wrote on Sat, 17 Jul 2004
23:08:59 +0200:
> Hi,

> Could anyone of you please post a sample for a font-lock function matcher?

The font-lock function matcher wasn't (and possibly still isn't) fully
documented in the elisp manual.  The following may be helpful:

     When FUNCTION is called, it receives one argument, the limit of the
     search.  Its search should start at point and not extend beyond the
     limit.  FUNCTION should return non-`nil' if it succeeds, and set the
     match data to describe the match that was found.  FUNCTION will be
     called repeatedly with the same limit, and with point where the
     previous invocation left it, until it fails.  It need not reset
     point to a sensible value on failure.

> I'm looking for a way to fontify words accoriding to their scrabble
> score. So if a word would score 6 points in american scrabble, it
> should get a certain font-locking.

> Currently, I have the following elisp, but it does not work.

Have you discovered the joys of edebug, yet?  If not, I thoroughly
recommend you to invest a few hours learning it.  For example, put point
inside `scrabble-6-matcher' and do C-u C-M-x to instrument it for edebug.
Then call the function directly with M-: (scrabble-6-matcher 200).
Enjoy!

A hint: if you're going to be using edebug within "live" font-lock
routines, do this first:  M-: (setq font-lock-support-mode nil).
Otherwise the jit-lock timer routine will kick in 3 seconds after you've
starting looking at your function, destroying your concentration and
peace of mind.

> I've been looking at some samples from the emacs lisp directories, but
> failed to work out the pattern. Also the info node on font-locking and
> the various font-locking doc-strings were helpful, but not helpful
> enough.  My elisp swings between locking emacs (presumably on
> font-locking) and no font-locking at all. (this of course when I
> twiddle around with it)

> (defvar scrabble-font-lock-keywords
>    (list '(scrabble-6-matcher (1 font-lock-warning-face))
> ))

> (defun scrabble-6-matcher (limit)
>    "returns t when the scrabble score of a word is 6."
>    (if (and (re-search-forward "\\([a-z]\\)*" limit t)
>         (< (scrabble-last-word-score) 5))
>        (progn
> 	(set-match-data
> 	 (list
> 	  (match-beginning 1) (match-end 1)
> 	  (match-beginning 1) (match-end 1)
> 	  nil nil)))
>      ;; else
>      (set-match-data
>       (list
>        (match-beginning 1) (match-end 1)
>        nil nil
>        (match-beginning 1) (match-end 1)))
>      t))

> (defun scrabble-last-word-score ()
>    ""
>    5)

The "no font-locking at all" is probably because you've caused an error
inside an after-change-functions thing, namely the font-locking.  In this
case, Emacs Lisp _removes_ the guilty hook from after-change-functions,
since the only alternative would be totally hanging the system.

As for locking the system, I suspect you've got into an infinite loop
with your regular expression: (re-search-forward "\\([a-z]\\)*" limit t).
This regexp will match any sequence of lower case letters, including an
empty one.  ;-(.  Probably you really want something more like
"\\([a-z]\\)+".  Then ask yourself whether you really want the "+"
_outside_ the grouping parentheses.  ;-)

Have fun!

> Arjan

-- 
Alan Mackenzie (Munich, Germany)
Email: aacm@muuc.dee; to decode, wherever there is a repeated letter
(like "aa"), remove half of them (leaving, say, "a").

Re: font-lock function matcher sample

[Arjan Bos]

Alan Mackenzie wrote:
> Arjan Bos <Arjan.Bos@nospam.iseeyou.nl> wrote on Sat, 17 Jul 2004
> 23:08:59 +0200:
> 
> The font-lock function matcher wasn't (and possibly still isn't) fully
> documented in the elisp manual.  The following may be helpful:
> 
>      When FUNCTION is called, it receives one argument, the limit of the
>      search.  Its search should start at point and not extend beyond the
>      limit.  FUNCTION should return non-`nil' if it succeeds, and set the
>      match data to describe the match that was found.  FUNCTION will be
>      called repeatedly with the same limit, and with point where the
>      previous invocation left it, until it fails.  It need not reset
>      point to a sensible value on failure.
> 
Yeah, I found that passage. It was useful, up to a point. What I have 
now, works slightly better. I found out that I need to find words and 
calculate their scrabble-score.
Is it true then that I need to keep on searching forward until I find a 
word whose scrabble score matches 6 (or until I reach limit)? If so, I'm 
having trouble when point is at (- limit 1). When that's the case, the 
matcher function is called repeatedly without stopping.

> 
>>I'm looking for a way to fontify words accoriding to their scrabble
>>score. So if a word would score 6 points in american scrabble, it
>>should get a certain font-locking.
> 
> 
>>Currently, I have the following elisp, but it does not work.
> 
> 
> Have you discovered the joys of edebug, yet?  If not, I thoroughly
> recommend you to invest a few hours learning it.  For example, put point
> inside `scrabble-6-matcher' and do C-u C-M-x to instrument it for edebug.
Normally I call M-x edebug-defun. How is this different?

> Then call the function directly with M-: (scrabble-6-matcher 200).
> Enjoy!
> 
> A hint: if you're going to be using edebug within "live" font-lock
> routines, do this first:  M-: (setq font-lock-support-mode nil).
> Otherwise the jit-lock timer routine will kick in 3 seconds after you've
> starting looking at your function, destroying your concentration and
> peace of mind.
I tried it without the (setq ...) It didn't destroy anything. In fact, 
it didn't do anything at all. Although, come to think of it, I got a few 
malloc error messages from emacs in my console. But maybe that is 
because I'm using a cvs emacs from a week ago.

> 
> As for locking the system, I suspect you've got into an infinite loop
> with your regular expression: (re-search-forward "\\([a-z]\\)*" limit t).
> This regexp will match any sequence of lower case letters, including an
> empty one.  ;-(.  Probably you really want something more like
> "\\([a-z]\\)+".  Then ask yourself whether you really want the "+"
> _outside_ the grouping parentheses.  ;-)

Thanks, currently I'm having a bit more success with "\\<\\(.+?\\)\\>

> 
> Have fun!
Thanks! This /is/ fun! It's my summer holiday project. Done when I do 
not have to decorate my house (you know how spouses are ;-) )

Arjan

-- 
--
If you really want to contact me, then replace the "I see you" text by 
its three letter accronym, ICU.

Fabricate Diem PVNC, Motto of the Night Watch -- Terry Pratchett

Re: font-lock function matcher sample

[Stefan Monnier]

> trouble when point is at (- limit 1). When that's the case, the matcher
> function is called repeatedly without stopping.

That should only be the case if your function returns non-nil without
moving point.


        Stefan

Re: font-lock function matcher sample

[Stefan Monnier]

Basically calling FUNCTION as in

   (FUNCTION lim)

should behave similarly (as far as mach-data, point, and
return value go) to

   (re-search-forward RE lim t)


-- Stefan

Re: font-lock function matcher sample

[Arjan Bos]

Stefan Monnier wrote:
> Basically calling FUNCTION as in
> 
>    (FUNCTION lim)
> 
> should behave similarly (as far as mach-data, point, and
> return value go) to
> 
>    (re-search-forward RE lim t)
> 
>

Thanks, that matches my ideas.

What I have currently is working correctly. First I find a word. If 
word, limit and point are the same as the previous time the function was 
called, I return `nil'.
Otherwise I calculate the scrabble score for it and if that matches the 
asked for scrabble score (say 6) then I return `t'. In all other cases, 
I return `nil'

like in:

(defun beatnik-=-matcher-p (limit score-match)
   "Tries to find a word whose scrable score matches score-match.
If such a word was found, t is returned."
   (if (and (re-search-forward beatnik-font-matcher-regexp limit t)
	   (= (beatnik-last-word-score) score-match))
       (progn
	(setq beatnik-last-match (list (match-string-no-properties 1)
				       limit
				       score-match
				       (point)))
	t)
     ;; else
     (if (equal (list (match-string-no-properties 1)
		     limit
		     score-match
		     (point))
	       beatnik-last-match)
	;; we already found it, return nil
	  nil
       ;; else
       (while (and (re-search-forward beatnik-font-matcher-regexp limit t)
		  (not (= (beatnik-last-word-score) score-match))))
       (if (= (beatnik-last-word-score) score-match)
	  (progn
	    (setq beatnik-last-match (list (match-string-no-properties 1)
					   limit
					   score-match
					   (point)))
	    t)
	;; else
	nil))))


Stephan and Alan, many thanks for helping me out. Maybe your comments 
should find their way to the doc-string or the info pages?

Thanks!

Arjan

-- 
--
If you really want to contact me, then replace the "I see you" text by 
its three letter accronym, ICU.

Fabricate Diem PVNC, Motto of the Night Watch -- Terry Pratchett

Re: font-lock function matcher sample

[Stefan Monnier]

> What I have currently is working correctly.  First I find a word.  If word,
> limit and point are the same as the previous time the function was called,
> I return `nil'.

The fact that you need to compare with last search indicates that there's
something fishy with your code.

>        (while (and (re-search-forward beatnik-font-matcher-regexp limit t)
> 		  (not (= (beatnik-last-word-score) score-match))))

Here you lose the information about whether the search succeeded or failed
which the source of your problems.  I.e. if the search fails immediately
(typically you've hit LIMIT) then (= (beatnik-last-word-score) score-match)
will be true and your code will screw up.

How about the code below instead:

(defun beatnik-=-matcher-p (limit score-match)
   "Tries to find a word whose scrable score matches score-match.
If such a word was found, t is returned."
   (let (found)
     (while (and (setq found (re-search-forward
                              beatnik-font-matcher-regexp limit t))
                 (/= (beatnik-last-word-score) score-match)))
     found))

> Stephan and Alan, many thanks for helping me out.  Maybe your comments
> should find their way to the doc-string or the info pages?

>From what I can see, your problems had nothing to do understanding how
font-lock works.  But if you have a concrete suggestion for how to change
the docstring, patches are welcome.


        Stefan

Re: font-lock function matcher sample

[Arjan Bos]

Stefan Monnier wrote:
>>What I have currently is working correctly.  First I find a word.  If word,
>>limit and point are the same as the previous time the function was called,
>>I return `nil'.
> 
> 
> The fact that you need to compare with last search indicates that there's
> something fishy with your code.
> 
> 
>>       (while (and (re-search-forward beatnik-font-matcher-regexp limit t)
>>		  (not (= (beatnik-last-word-score) score-match))))
> 
> 
> Here you lose the information about whether the search succeeded or failed
> which the source of your problems.  I.e. if the search fails immediately
> (typically you've hit LIMIT) then (= (beatnik-last-word-score) score-match)
> will be true and your code will screw up.
> 
> How about the code below instead:
> 
> (defun beatnik-=-matcher-p (limit score-match)
>    "Tries to find a word whose scrable score matches score-match.
> If such a word was found, t is returned."
>    (let (found)
>      (while (and (setq found (re-search-forward
>                               beatnik-font-matcher-regexp limit t))
>                  (/= (beatnik-last-word-score) score-match)))
>      found))
> 

This is almost what I intended. This sets `found' to `t' even when the 
asked for scrabble score via (beatnik-last-word-score) isn't found. But 
when I move the `and' into the `setq' it doesn't work, meaning that it 
locks itself fontifying.

(defun beatnik-=-matcher-p (limit score-match)
   "Tries to find a word whose scrable score matches score-match.
If such a word was found, t is returned."
   (let (found)
     ;; doesn't work:
     (while (and (setq found (re-search-forward
			     beatnik-font-matcher-regexp limit t))
		      (/= (beatnik-last-word-score) score-match)))
     found))

So, unfortunately, I'm back to my kludge with the global variable that 
keeps track of the last matched match.

> 
>>Stephan and Alan, many thanks for helping me out.  Maybe your comments
>>should find their way to the doc-string or the info pages?
> 
> 
> From what I can see, your problems had nothing to do understanding how
> font-lock works.  But if you have a concrete suggestion for how to change
> the docstring, patches are welcome.
> 

Well, what I really missed was a sample function. I think the one you 
posted could well serve as a template. The info page 
((elisp)Search-based Fontification.) had the needed input and output for 
the defun, but the doc-string for `font-lock-defaults' had neither. 
Maybe it could point to the info page?
Although the info page contains all the needed info, I can understand it 
now in hindsight, but the thing about `match-data' /was/ quite confusing 
for me.


And the biggest nuisance in figuring this out was that emacs crashed a 
lot when getting things wrong in the defun. It gave lots of `malloc' 
errors and `double free' errors. Which is, of course, not a pretty way 
to cancel fontifying ;-) .

-- 
--
If you really want to contact me, then replace the "I see you" text by 
its three letter accronym, hetnet.

Fabricate Diem PVNC, Motto of the Night Watch -- Terry Pratchett

Re: font-lock function matcher sample

[Stefan Monnier]

> > How about the code below instead:
> > (defun beatnik-=-matcher-p (limit score-match)
> >    "Tries to find a word whose scrable score matches score-match.
> > If such a word was found, t is returned."
> >    (let (found)
> >      (while (and (setq found (re-search-forward
> >                               beatnik-font-matcher-regexp limit t))
> >                  (/= (beatnik-last-word-score) score-match)))
> >      found))

> This is almost what I intended. This sets `found' to `t' even when the asked
> for scrabble score via (beatnik-last-word-score) isn't found.

Huh?  The while loop is only exited if found is nil or if
(beatnik-last-word-score) is equal to score-match, AFAICT.

I don't understand the problem you're referring to.

> But when I move the `and' into the `setq' it doesn't work, meaning that it
> locks itself fontifying.

> (defun beatnik-=-matcher-p (limit score-match)
>    "Tries to find a word whose scrable score matches score-match.
> If such a word was found, t is returned."
>    (let (found)
>      ;; doesn't work:
>      (while (and (setq found (re-search-forward
> 			     beatnik-font-matcher-regexp limit t))
> 		      (/= (beatnik-last-word-score) score-match)))
>      found))

This code seems to be exactly equl to the one above.

> Well, what I really missed was a sample function. I think the one you posted
> could well serve as a template.  The info page ((elisp)Search-based
> Fontification.) had the needed input and output for the defun, but the
> doc-string for `font-lock-defaults' had neither.  Maybe it could point to the
> info page?

I've changed the docstring to mention the fact that point is expected to
move, and to mention that it should all behave like re-search-forward.

> And the biggest nuisance in figuring this out was that emacs crashed a lot
> when getting things wrong in the defun.  It gave lots of `malloc' errors
> and `double free' errors.  Which is, of course, not a pretty way to cancel
> fontifying ;-) .

These bugs should be reported and fixed (but there's no point reporting them
unless you can provide a reliable recipe to reproduce them or unless you can
verify that they are still present in the CVS version of Emacs).

But even if they get fixed, I agree that debugging font-lock-keywords
settings is painful.  Turning off jit-lock-mode can help, tho.


        Stefan

Re: font-lock function matcher sample

[Arjan Bos]

Stefan Monnier wrote:
>>>How about the code below instead:
>>>(defun beatnik-=-matcher-p (limit score-match)
>>>   "Tries to find a word whose scrable score matches score-match.
>>>If such a word was found, t is returned."
>>>   (let (found)
>>>     (while (and (setq found (re-search-forward
>>>                              beatnik-font-matcher-regexp limit t))
>>>                 (/= (beatnik-last-word-score) score-match)))
>>>     found))
> 
> 
>>This is almost what I intended. This sets `found' to `t' even when the asked
>>for scrabble score via (beatnik-last-word-score) isn't found.
> 
> 
> Huh?  The while loop is only exited if found is nil or if
> (beatnik-last-word-score) is equal to score-match, AFAICT.
You're right about the while loop, but still you're setting `found' to
`t' even when the word doesn't equal score-match. And it is `found' that
is returned


> 
> I don't understand the problem you're referring to.
> 
> 
>>But when I move the `and' into the `setq' it doesn't work, meaning that it
>>locks itself fontifying.
> 
> This code seems to be exactly equl to the one above.

Oops, I meant:

(defun beatnik-=-matcher-p (limit score-match)
    "Tries to find a word whose scrable score matches score-match.
If such a word was found, t is returned."
    (let (found)
      ;; doesn't work:
      (while (setq found (and (re-search-forward
			        beatnik-font-matcher-regexp limit t))
		             (/= (beatnik-last-word-score) score-match)))
      found))

> 
> 
>>Well, what I really missed was a sample function. I think the one you posted
>>could well serve as a template.  The info page ((elisp)Search-based
>>Fontification.) had the needed input and output for the defun, but the
>>doc-string for `font-lock-defaults' had neither.  Maybe it could point to the
>>info page?
> 
> 
> I've changed the docstring to mention the fact that point is expected to
> move, and to mention that it should all behave like re-search-forward.
Great! The World of Emacs gained at least this from your effort!

> 
>
<snip />

> 
> These bugs should be reported and fixed (but there's no point reporting them
> unless you can provide a reliable recipe to reproduce them or unless you can
> verify that they are still present in the CVS version of Emacs).

I agree that bugs should be reported and fixed. But I do not really have
a reliable testcase to reproduce them. Especially since RMS himself
stated recently that he cannot check errors on MacOS X since he doesn't
have the hardware.

> 
> But even if they get fixed, I agree that debugging font-lock-keywords
> settings is painful.  Turning off jit-lock-mode can help, tho.
I couldn't even get edebug-defun to get started, so I did it the
old-fashioned way by using (message "string").

Arjan

-- 
--
If you really want to contact me, then replace the "I see you" text by
its three letter accronym, HetNet.

Fabricate Diem PVNC, Motto of the Night Watch -- Terry Pratchett

Re: font-lock function matcher sample

[Stefan Monnier]

>>>> How about the code below instead:
>>>> (defun beatnik-=-matcher-p (limit score-match)
>>>> "Tries to find a word whose scrable score matches score-match.
>>>> If such a word was found, t is returned."
>>>> (let (found)
>>>> (while (and (setq found (re-search-forward
>>>> beatnik-font-matcher-regexp limit t))
>>>> (/= (beatnik-last-word-score) score-match)))
>>>> found))
>> 
>>> This is almost what I intended. This sets `found' to `t' even when the asked
>>> for scrabble score via (beatnik-last-word-score) isn't found.
>> Huh?  The while loop is only exited if found is nil or if
>> (beatnik-last-word-score) is equal to score-match, AFAICT.
> You're right about the while loop, but still you're setting `found' to
> `t' even when the word doesn't equal score-match. And it is `found' that
> is returned

But if it is t and score-match is not equal, than you go around the loop
once more where found will be reset to either nil or t depending on whether
there are more words to find.
So I still don't understand the problem.

> (defun beatnik-=-matcher-p (limit score-match)
>     "Tries to find a word whose scrable score matches score-match.
> If such a word was found, t is returned."
>     (let (found)
>       ;; doesn't work:
>       (while (setq found (and (re-search-forward
> 			        beatnik-font-matcher-regexp limit t))
> 		             (/= (beatnik-last-word-score) score-match)))
>       found))

This will always return nil because if found is non-nil, we don't exit
the loop.

> I agree that bugs should be reported and fixed.  But I do not really have
> a reliable testcase to reproduce them.  Especially since RMS himself
> stated recently that he cannot check errors on MacOS X since he doesn't
> have the hardware.

I strongly doubt such problems would be specific to Mac OS X (tho they may
have more serious implication since C-g tends not to work as well).

>> But even if they get fixed, I agree that debugging font-lock-keywords
>> settings is painful.  Turning off jit-lock-mode can help, tho.
> I couldn't even get edebug-defun to get started, so I did it the
> old-fashioned way by using (message "string").

If jit-lock-mode is turned off, edebug-defun should work just fine (at
least that's been my experience).  With jit-lock, I've had it work and not
work at different points in time.


        Stefan

Re: font-lock function matcher sample

[Arjan Bos]

Stefan Monnier wrote:

<snip />
>>You're right about the while loop, but still you're setting `found' to
>>`t' even when the word doesn't equal score-match. And it is `found' that
>>is returned
> 
> 
> But if it is t and score-match is not equal, than you go around the loop
> once more where found will be reset to either nil or t depending on whether
> there are more words to find.
> So I still don't understand the problem.
> 
<snip />
and Stefan wrote :
> 
> This will always return nil because if found is non-nil, we don't exit
> the loop.
Mmm, yes, you're right. But this means that I'm not clever enough to see 
through those nested statements. I need to unroll them before they make 
sense to me. So I'll give it another try.

Well, thanks a lot! And I'll try to make a working crash recipe for the 
font-locking lockup problem. Looking at the crash log indicates that the 
thing that crashes is an Apple component and I did mail it to Apple already:

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_PROTECTION_FAILURE (0x0002) at 0x00000014

Thread 0 Crashed:
0   com.apple.HIToolbox 	0x927ed93c HIObject::Retain() + 0x14
1   <<00000000>> 	0x0039a790 proc_encode_coding_system + 0x904
2   com.apple.HIToolbox 	0x927f2340 RetainMenu(MenuData*) + 0x10
3   com.apple.HIToolbox 	0x9282b4dc NormalizeMenu + 0x24
4   com.apple.HIToolbox 	0x9282b63c HiliteMenuTitle(MenuData*, 
MenuData*, unsigned char) + 0x38
5   com.apple.HIToolbox 	0x9285dad8 MenuSelectCore(Point, double, 
unsigned long, OpaqueMenuRef**, unsigned short*) + 0x40
6   com.apple.HIToolbox 	0x92883ffc MenuSelect + 0x60
7   com.gnu.Emacs       	0x00149d4c x_activate_menubar + 0x48 
(macmenu.c:957)
8   com.gnu.Emacs       	0x00085868 kbd_buffer_get_event + 0x290 
(keyboard.c:4022)


> 
> 

-- 
--
If you really want to contact me, then replace the "I see you" text by 
its three letter accronym, ICU.

Fabricate Diem PVNC, Motto of the Night Watch -- Terry Pratchett

Re: font-lock function matcher sample

[Stefan Monnier]

> Mmm, yes, you're right.  But this means that I'm not clever enough to see
> through those nested statements.  I need to unroll them before they make
> sense to me.

Don't think so operationally.  Instead take a step back and only consider
the things you know for sure: right after the while loop the condition is
nil.


        Stefan

Re: font-lock function matcher sample

[Arjan Bos]

Stefan Monnier wrote:
>>Mmm, yes, you're right.  But this means that I'm not clever enough to see
>>through those nested statements.  I need to unroll them before they make
>>sense to me.
> 
> 
> Don't think so operationally.  Instead take a step back and only consider
> the things you know for sure: right after the while loop the condition is
> nil.
> 
Stefan,

Did they already put up a statue for you? 'cause you deserve one. I took
another stab at it with your prototype function and now it works! I'll
post beatnik.el to gnu.emacs.sources shortly.

Thanks a lot!

Arjan

-- 
--
If you really want to contact me, then replace the "I see you" text by
its three letter accronym, HetNet.

Fabricate Diem PVNC, Motto of the Night Watch -- Terry Pratchett