diary font-lock issue

diary font-lock issue

[Glenn Morris]

I'm trying to fix a bug recently reported by Stephen Berman
("Fontification problem with fancy diary") on the pretest-bug list.
I'm stumped, and would appreciate some help from people knowledgeable
about font-lock.

I can simplify the issue to this:


elisp:
-------
(setq foo-font-lock-keywords
      '(("\\(\\(?:Fri\\|Mon\\|S\\(?:atur\\|un\\)\\|\\(?:T\\(?:hur\\|ue\\)\\|Wedne\\)s\\)day\\), \\(A\\(?:pril\\|ugust\\)\\|December\\|February\\|J\\(?:anuary\\|u\\(?:ly\\|ne\\)\\)\\|Ma\\(?:rch\\|y\\)\\|\\(?:Novem\\|Octo\\|Septem\\)ber\\) [0-9]+, -?[0-9]+\n=+$" . font-lock-keyword-face)))

(define-derived-mode foo-mode fundamental-mode
  "foo"
  (set (make-local-variable 'font-lock-defaults)
       '(foo-font-lock-keywords t))
  (set (make-local-variable 'font-lock-multiline) t))


sample input file:
-------
Friday, May 5, 2006
===================
9.00 Xxxx Xxxxxx Xxxxx xXxxxxxxxxxxxx
15.20-15.50 Xxxx Xxxxxxxxxxxxxxxxx
17.30-19.00 Xxxx Xxxxxxxxxx
18.00-19.30 Xxxx XXX

Monday, May 8, 2006
===================
8.00 Xxxxxxxx xx Xxxxxxxx Xxxxxxxxxx xxx Xxxxxxxxxxxxx
14.30 Xxxxxx XxxxxXXxXxxxxx xxx Xxxx
15.30-17.00 Xxxx XxxxxXX
16.45 Xxxx xxx XxxxxXX xxxxxxx
17.15-18.15 Xxxx XxxxxXx

Tuesday, May 9, 2006
====================
14.50 Xxxx xxx Xxxxxxxxxxxxxxxxxxxxx xxxxxxx
15.30-16.00 Xxxx

Wednesday, May 10, 2006
=======================
20.00-21.15 Xxxxxx Xxxxxx


------

Defining foo-mode as above, and visiting the above example file, the
last "Wednesday, May 10, 2006" does not get font-locked. If the
"15.30-16.00" line in the immediately previous Tuesday section is not
there, it works as it should.

Alternatively, if the "\n=+$" part is removed from the font-lock
keywords, it works as it should. That made me think it might be a
multiline font-lock pattern issue. Hence I added the setting of
font-lock-multiline in the mode definition (the diary does not have
this), but that did not help.


TIA for any help.


PS For clarity, the font-lock definition can be replaced by:

(setq foo-font-lock-keywords
      (list
       (cons
        (format "%s, %s %s, %s"
                (regexp-opt '("Monday" "Tuesday" "Wednesday" "Thursday" "Friday"
                              "Saturday" "Sunday") t)
                (regexp-opt '("January" "February" "March" "April" "May" "June"
                              "July" "August" "September" "October" "November"
                              "December") t)
                "[0-9]+" "[0-9]+\n=+$")
        font-lock-keyword-face)))

Re: diary font-lock issue

[martin rudalics]

[-- Attachment #1: Type: text/plain, Size: 2059 bytes --]

 > elisp:
 > -------
 > (setq foo-font-lock-keywords
 >       '(("\\(\\(?:Fri\\|Mon\\|S\\(?:atur\\|un\\)\\|\\(?:T\\(?:hur\\|ue\\)\\|Wedne\\)s\\)day\\), \\(A\\(?:pril\\|ugust\\)\\|December\\|February\\|J\\(?:anuary\\|u\\(?:ly\\|ne\\)\\)\\|Ma\\(?:rch\\|y\\)\\|\\(?:Novem\\|Octo\\|Septem\\)ber\\) 
[0-9]+, -?[0-9]+\n=+$" . font-lock-keyword-face)))
 >
 > (define-derived-mode foo-mode fundamental-mode
 >   "foo"
 >   (set (make-local-variable 'font-lock-defaults)
 >        '(foo-font-lock-keywords t))
 >   (set (make-local-variable 'font-lock-multiline) t))
 >
 >
 > sample input file:
 > -------
 > Friday, May 5, 2006
 > ===================
 > 9.00 Xxxx Xxxxxx Xxxxx xXxxxxxxxxxxxx
 > 15.20-15.50 Xxxx Xxxxxxxxxxxxxxxxx
 > 17.30-19.00 Xxxx Xxxxxxxxxx
 > 18.00-19.30 Xxxx XXX
 >
 > Monday, May 8, 2006
 > ===================
 > 8.00 Xxxxxxxx xx Xxxxxxxx Xxxxxxxxxx xxx Xxxxxxxxxxxxx
 > 14.30 Xxxxxx XxxxxXXxXxxxxx xxx Xxxx
 > 15.30-17.00 Xxxx XxxxxXX
 > 16.45 Xxxx xxx XxxxxXX xxxxxxx
 > 17.15-18.15 Xxxx XxxxxXx
 >
 > Tuesday, May 9, 2006
 > ====================
 > 14.50 Xxxx xxx Xxxxxxxxxxxxxxxxxxxxx xxxxxxx
 > 15.30-16.00 Xxxx
 >
 > Wednesday, May 10, 2006

<----------------------------------- end of font-lock-fontify-region

 > =======================
 > 20.00-21.15 Xxxxxx Xxxxxx

`foo-font-lock-keywords' will fail to find a match when the end of
`font-lock-fontify-region' is positioned as indicated above.

Earlier it was possible to handle this - in most cases - by setting
`font-lock-lines-before' to 1, but I don't know how this will be done in
the future.  So far you could try the largely untested code I attached.

As soon as `font-lock-extend-region' has undergone its final revision
you should be able to replace the after-change-hook by writing an
appropriate `font-lock-extend-region-function'.  If
`font-lock-extend-region' will be regularly called by say
`font-lock-default-fontify-region', you will have to simply check
whether the first character to be fontified is an "=" and you may use
your original code practically unchanged.


[-- Attachment #2: foo-mode --]
[-- Type: text/plain, Size: 1516 bytes --]

(defun foo-bar (bound)
  (let (beg end)
    (and (re-search-forward
	  "\\(\\(?:\\(?:Fri\\|Mon\\|S\\(?:atur\\|un\\)\\|\\(?:T\\(?:hur\\|ue\\)\\|Wedne\\)s\\)day\\)\
, \\(?:A\\(?:pril\\|ugust\\)\\|December\\|February\\|J\\(?:anuary\\|u\\(?:ly\\|ne\\)\\)\
\\|Ma\\(?:rch\\|y\\)\\|\\(?:Novem\\|Octo\\|Septem\\)ber\\) [0-9]+, -?[0-9]+\n\\)\\(=+\n\\)?" bound t)
	 (or (and (> (match-end 0) (match-end 1))
		  (setq beg (match-beginning 0))
		  (setq end (match-end 0)))
	     (and (= (match-end 1) bound)
		  (setq beg (match-beginning 0))
		  (save-excursion
		    (save-match-data
		      (and (looking-at "=+\n")
			   (setq end (match-end 0)))))))
	 (and beg end
	      (or (put-text-property beg end 'font-lock-multiline t) t)))))

(setq foo-font-lock-keywords
      '((foo-bar 1 'font-lock-keyword-face)))

(defun foo-mode-after-change (start end old-len)
  (save-excursion
    (goto-char start)
    (beginning-of-line)
    (when (eq (char-after) ?\=))
      (let* ((buffer-undo-list t)
	     (inhibit-read-only t)
	     (inhibit-point-motion-hooks t)
	     (inhibit-modification-hooks t)
	     deactivate-mark
	     buffer-file-name
	     buffer-file-truename)
	(put-text-property
	 (line-beginning-position 0) (line-beginning-position) 'fontified nil))))

(define-derived-mode foo-mode fundamental-mode
  "foo"
  (set (make-local-variable 'font-lock-defaults)
       '(foo-font-lock-keywords t))
  (set (make-local-variable 'font-lock-multiline) t)
  (add-hook 'after-change-functions 'foo-mode-after-change nil t))

[-- Attachment #3: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel

Re: diary font-lock issue

[Glenn Morris]

Hi Martin, thanks for the response.


martin rudalics wrote:

> `foo-font-lock-keywords' will fail to find a match when the end of
> `font-lock-fontify-region' is positioned as indicated above.

> As soon as `font-lock-extend-region' has undergone its final revision
> you should be able to replace the after-change-hook by writing an
> appropriate `font-lock-extend-region-function'.  If
> `font-lock-extend-region' will be regularly called by say
> `font-lock-default-fontify-region', you will have to simply check
> whether the first character to be fontified is an "=" and you may use
> your original code practically unchanged.
>

> (defun foo-bar (bound)
[...]
> (setq foo-font-lock-keywords
>       '((foo-bar 1 'font-lock-keyword-face)))

Just looking at this quickly, it looks to be doing the same job as the
regexp I originally quoted, but using a function instead to do the
match. Setting font-lock-multiline to t in the buffer means that it
gets added as a property to all fontified regions, so yours is more
elegant (only adding the property where needed), but the two ought to
be equivalent, right?

> (defun foo-mode-after-change (start end old-len)

> (define-derived-mode foo-mode fundamental-mode
>   "foo"
>   (set (make-local-variable 'font-lock-defaults)
>        '(foo-font-lock-keywords t))
>   (set (make-local-variable 'font-lock-multiline) t)
>   (add-hook 'after-change-functions 'foo-mode-after-change nil t))

This all seems very complex. According to:

<http://lists.gnu.org/archive/html/emacs-devel/2006-04/msg00881.html>

"the font-lock-multiline property should be enough in all cases to
make it unnecessary to use an after-change-function hook."

That's a long thread, but AFAICS, it seems that for simple keyword
fontification, setting font-lock-multiline for the buffer ought to be
enough.

(Though later on I find:

<http://lists.gnu.org/archive/html/emacs-devel/2006-04/msg01130.html>

    > (i) The matching of keywords which span line breaks;
    ... setting the font-lock-multiline variable... won't reliably
    take care of (i)

which confuses me.)


I'm also confused as to why the initial fontification is not working
as it should. There are no "changes" involved in this instance.
(Indeed, the fancy diary buffer is not something one edits, so this is
about as simple as multiline font-lockign can get.)

Re: diary font-lock issue

[martin rudalics]

 >>`foo-font-lock-keywords' will fail to find a match when the end of
 >>`font-lock-fontify-region' is positioned as indicated above.
[...]
 >
 > Just looking at this quickly, it looks to be doing the same job as the
 > regexp I originally quoted, but using a function instead to do the
 > match. Setting font-lock-multiline to t in the buffer means that it
 > gets added as a property to all fontified regions, so yours is more
 > elegant (only adding the property where needed), but the two ought to
 > be equivalent, right?

Not really, I put a question mark at the end of the regexp.  Hence I
also match date lines not followed by a =[=]* line.  To handle that very
case - which is the cause of your problem - I use a function.  You
probably could do that more elegantly with anchored matches.

 > This all seems very complex. According to:
 >
 > <http://lists.gnu.org/archive/html/emacs-devel/2006-04/msg00881.html>
 >
 > "the font-lock-multiline property should be enough in all cases to
 > make it unnecessary to use an after-change-function hook."

Indeed, it *should*, eventually ...

 > That's a long thread, but AFAICS, it seems that for simple keyword
 > fontification, setting font-lock-multiline for the buffer ought to be
 > enough.
 >
 > (Though later on I find:
 >
 > <http://lists.gnu.org/archive/html/emacs-devel/2006-04/msg01130.html>
 >
 >     > (i) The matching of keywords which span line breaks;
 >     ... setting the font-lock-multiline variable... won't reliably
 >     take care of (i)
 >
 > which confuses me.)

You might have to solve the following problem: Suppose you have the
single line

Friday, May 5, 2006

in your buffer and you type a "=" on the next line as in

Friday, May 5, 2006
=

There's no multiline property around before you type the "=" but you
probably want to highlight the date line after.  That's what my ugly
after-change-hook is for.

 > I'm also confused as to why the initial fontification is not working
 > as it should. There are no "changes" involved in this instance.
 > (Indeed, the fancy diary buffer is not something one edits, so this is
 > about as simple as multiline font-lockign can get.)

I explained that at the very beginning of this message.  If you never
edit the buffer you obviously don't need font-lock-multiline and you
don't need the after-change hook either, just use a simplified version
of foo-bar like

(defun foo-bar (bound)
   (and (re-search-forward
	"\\(\\(?:\\(?:Fri\\|Mon\\|S\\(?:atur\\|un\\)\\|\\(?:T\\(?:hur\\|ue\\)\\|Wedne\\)s\\)day\\)\
, \\(?:A\\(?:pril\\|ugust\\)\\|December\\|February\\|J\\(?:anuary\\|u\\(?:ly\\|ne\\)\\)\
\\|Ma\\(?:rch\\|y\\)\\|\\(?:Novem\\|Octo\\|Septem\\)ber\\) [0-9]+, -?[0-9]+\n\\)\\(=+\n\\)?" bound t)
        (or (> (match-end 0) (match-end 1))
	   (and (= (match-end 1) bound)
		(save-match-data
		  (looking-at "=+\n"))))))

But never try to match things that span two or more lines the way you
did.  It will always fail at boundaries of jit-lock chunks.

Re: diary font-lock issue

[Stefan Monnier]

> This all seems very complex. According to:

> <http://lists.gnu.org/archive/html/emacs-devel/2006-04/msg00881.html>

> "the font-lock-multiline property should be enough in all cases to
> make it unnecessary to use an after-change-function hook."

That's true.  His after-change-function is not needed.

> That's a long thread, but AFAICS, it seems that for simple keyword
> fontification, setting font-lock-multiline for the buffer ought to be
> enough.

No.  font-lock-multiline (just like after-change-functions) only help with
the re-fontification of multiline keywords.  The problem you're facing is
the one of initial detection of a multiline keyword.

> (Indeed, the fancy diary buffer is not something one edits, so this is
> about as simple as multiline font-lockign can get.)

Actually the initial detection of multiline keywords is the hard part, not
the re-fontification after a buffer change.

Note that a workaround for your problem is to turn off jit-lock (you can do
so buffer-locally by setting font-lock-support-mode to nil).


        Stefan

Re: diary font-lock issue

[Stefan Monnier]

> (setq foo-font-lock-keywords
>       '(("\\(\\(?:Fri\\|Mon\\|S\\(?:atur\\|un\\)\\|\\(?:T\\(?:hur\\|ue\\)\\|Wedne\\)s\\)day\\), \\(A\\(?:pril\\|ugust\\)\\|December\\|February\\|J\\(?:anuary\\|u\\(?:ly\\|ne\\)\\)\\|Ma\\(?:rch\\|y\\)\\|\\(?:Novem\\|Octo\\|Septem\\)ber\\) [0-9]+, -?[0-9]+\n=+$" . font-lock-keyword-face)))

> (define-derived-mode foo-mode fundamental-mode
>   "foo"
>   (set (make-local-variable 'font-lock-defaults)
>        '(foo-font-lock-keywords t))
>   (set (make-local-variable 'font-lock-multiline) t))

I'd suggest the following 100% untested:

(define-derived-mode foo-mode fundamental-mode
  "foo"
  (set (make-local-variable 'font-lock-defaults)
       '(foo-font-lock-keywords t nil nil nil
         (font-lock-fontify-region-function
          . foo-font-lock-fontify-region-function)
         (font-lock-multiline . t))))

(defun foo-font-lock-fontify-region-function (beg end &optional loudly)
  (goto-char beg)
  (forward-line 0)
  (if (looking-at "=+$") (setq beg (line-beginning-position 0)))
  (goto-char end)
  (unless (bolp) (forward-line 1))
  (if (looking-at "=+$") (setq end (line-beginning-position 2)))
  (font-lock-default-fontify-region beg end loudly))


-- Stefan

Re: diary font-lock issue

[Glenn Morris]

Thank you Stefan and Martin, I see now what the problem is. The
information is basically in the elisp manual, so I should have been
able to figure it out, sorry.

Stefan Monnier wrote:

> (defun foo-font-lock-fontify-region-function (beg end &optional loudly)
>   (goto-char beg)
>   (forward-line 0)
>   (if (looking-at "=+$") (setq beg (line-beginning-position 0)))
>   (goto-char end)
>   (unless (bolp) (forward-line 1))
>   (if (looking-at "=+$") (setq end (line-beginning-position 2)))
>   (font-lock-default-fontify-region beg end loudly))

One thing I don't get about this - it looks like I can't rely on
font-lock regions beginning/ending at the start of lines. Given that,
how does normal jit-font-locking of non-multiline keywords work, since
regions could start/end in the middle of keywords? Is there always one
or more overlap line between neighbouring font-lock regions or
something? I guess this is the reason I don't need to make any region
adjustments on the first line of my multiline pattern.

Re: diary font-lock issue

[Stefan Monnier]

>> (defun foo-font-lock-fontify-region-function (beg end &optional loudly)
>> (goto-char beg)
>> (forward-line 0)
>> (if (looking-at "=+$") (setq beg (line-beginning-position 0)))
>> (goto-char end)
>> (unless (bolp) (forward-line 1))
>> (if (looking-at "=+$") (setq end (line-beginning-position 2)))
>> (font-lock-default-fontify-region beg end loudly))

> One thing I don't get about this - it looks like I can't rely on
> font-lock regions beginning/ending at the start of lines.

You can't rely on it in font-lock-fontify-region-function, because the
round-up-to-whole-lines is done in font-lock-default-fontify-region.


        Stefan

Re: diary font-lock issue

[martin rudalics]

 > I'd suggest the following 100% untested:
 > ...
 > (defun foo-font-lock-fontify-region-function (beg end &optional loudly)
 >   (goto-char beg)
 >   (forward-line 0)
 >   (if (looking-at "=+$") (setq beg (line-beginning-position 0)))
 >   (goto-char end)
 >   (unless (bolp) (forward-line 1))
 >   (if (looking-at "=+$") (setq end (line-beginning-position 2)))
 >   (font-lock-default-fontify-region beg end loudly))

100% untested, maybe.  100% elegant, for sure.  Would it be very
difficult to add this as an example to the "Other Font Lock Variables"
section of the Elisp manual?