font-lock-defaults doesn't work??

font-lock-defaults doesn't work??

[Peter Tury]

Hi,

I am trying to create a new major mode from scratch. I found that
syntax parsing doesn't work properly sometimes. Why? Am I missed
something or this is a bug?

Details:

I have the following defun for start. If I remove the "secretly
must-to-have parts" (see below) then M-: (syntax-ppss) returns lists
as if it would parse a lisp buffer: characters in a line after `;'
reported as in-comment chars, and real comments (= delimited by `/*'
and `*/' doesn't recognized as comments. However, fontification works
nicely -- most of the time, but not always: _sometimes_, if I put `;'
inside a /*-comment and then delete this `;', then comment-color is
removed... More interestingly once I got nil for the value of
font-lock-syntax-table (queried via C-h v font-lock-syntax-table from
the buffer where I activated this new mode via M-x t-mode
previously)!?

So it seems syntax parsing is wrong and fontification is
indeterministic iff I set(??) font-lock-syntax-table via
font-lock-defaults, but everything works well if I directly set it via
set-syntax-table.

Is this normal? Do you know the reason? Should I report it as a bug?

(defun t-mode ()
  "test major mode"
  (interactive)

  (kill-all-local-variables)
  (setq major-mode (quote t-mode))
  (setq mode-name "t-mode")

  ;; secretly must-to-have parts -- start
  (let ((t-syntax-table (make-syntax-table)))
    (modify-syntax-entry ?/ ". 14" t-syntax-table)
    (modify-syntax-entry ?* ". 23" t-syntax-table)
    (set-syntax-table t-syntax-table))
  ;; secretly must-to-have parts -- end

  (set (make-local-variable 'font-lock-defaults)
       '(nil nil t
             ((?/ . ". 14")
              (?* . ". 23"))))

  (run-mode-hooks 't-mode-hook))

Note: emacs' help writes for font-lock-syntax-table: "this is normally
set via `font-lock-defaults'", while elisp manual writes for
make-syntax-table (in 35.3): "most major mode syntax tables are
created in this way" -- however I would think that make-syntax-table
is unusable if I set font-lock-syntax-table via font-lock-
defaults...??

(I use patched EmacsW32 v feb.20.2007, and haven't tested the
situation with -Q...)

Thanks for your help in advance,
P

Re: font-lock-defaults doesn't work??

[Tim X]

"Peter Tury" <tury.peter@gmail.com> writes:

> Hi,
>
> I am trying to create a new major mode from scratch. I found that
> syntax parsing doesn't work properly sometimes. Why? Am I missed
> something or this is a bug?
>

I suspect you have missed something. Many modes are doing what you are trying
to do and successfully, so its unlikely you have found a bug. However, it is
possible, especially since your using a devlopment code base (emacs 22)., but I
think unlikely. 

You are also re-inventing a wheel to some extent. Use define-derived-mode and
let emacs take care of some of the heavy lifting for you and build on work
already done! Also, see the example provided on the emacs wiki site.

,----[ C-h f define-derived-mode RET ]
| define-derived-mode is a Lisp macro in `derived.el'.
| (define-derived-mode CHILD PARENT NAME &optional DOCSTRING &rest BODY)
| 
| Create a new mode as a variant of an existing mode.
| 
| The arguments to this command are as follow:
| 
| CHILD:     the name of the command for the derived mode.
| PARENT:    the name of the command for the parent mode (e.g. `text-mode')
|            or nil if there is no parent.
| NAME:      a string which will appear in the status line (e.g. "Hypertext")
| DOCSTRING: an optional documentation string--if you do not supply one,
|            the function will attempt to invent something useful.
| BODY:      forms to execute just before running the
|            hooks for the new mode.  Do not use `interactive' here.
| 
| BODY can start with a bunch of keyword arguments.  The following keyword
|   arguments are currently understood:
| :group GROUP
| 	Declare the customization group that corresponds to this mode.
| 	The command `customize-mode' uses this.
| :syntax-table TABLE
| 	Use TABLE instead of the default.
| 	A nil value means to simply use the same syntax-table as the parent.
| :abbrev-table TABLE
| 	Use TABLE instead of the default.
| 	A nil value means to simply use the same abbrev-table as the parent.
| 
| Here is how you could define LaTeX-Thesis mode as a variant of LaTeX mode:
| 
|   (define-derived-mode LaTeX-thesis-mode LaTeX-mode "LaTeX-Thesis")
| 
| You could then make new key bindings for `LaTeX-thesis-mode-map'
| without changing regular LaTeX mode.  In this example, BODY is empty,
| and DOCSTRING is generated by default.
| 
| On a more complicated level, the following command uses `sgml-mode' as
| the parent, and then sets the variable `case-fold-search' to nil:
| 
|   (define-derived-mode article-mode sgml-mode "Article"
|     "Major mode for editing technical articles."
|     (setq case-fold-search nil))
| 
| Note that if the documentation string had been left out, it would have
| been generated automatically, with a reference to the keymap.
| 
| The new mode runs the hook constructed by the function
| `derived-mode-hook-name'.
| 
| See Info node `(elisp)Derived Modes' for more details.
| 
| [back]
`----



> Details:
>
> I have the following defun for start. If I remove the "secretly
> must-to-have parts" (see below) then M-: (syntax-ppss) returns lists
> as if it would parse a lisp buffer: characters in a line after `;'
> reported as in-comment chars, and real comments (= delimited by `/*'
> and `*/' doesn't recognized as comments. However, fontification works
> nicely -- most of the time, but not always: _sometimes_, if I put `;'
> inside a /*-comment and then delete this `;', then comment-color is
> removed... More interestingly once I got nil for the value of
> font-lock-syntax-table (queried via C-h v font-lock-syntax-table from
> the buffer where I activated this new mode via M-x t-mode
> previously)!?
>
> So it seems syntax parsing is wrong and fontification is
> indeterministic iff I set(??) font-lock-syntax-table via
> font-lock-defaults, but everything works well if I directly set it via
> set-syntax-table.
>
> Is this normal? Do you know the reason? Should I report it as a bug?
>
> (defun t-mode ()
>   "test major mode"
>   (interactive)
>
>   (kill-all-local-variables)
>   (setq major-mode (quote t-mode))
>   (setq mode-name "t-mode")
>
>   ;; secretly must-to-have parts -- start
>   (let ((t-syntax-table (make-syntax-table)))
>     (modify-syntax-entry ?/ ". 14" t-syntax-table)
>     (modify-syntax-entry ?* ". 23" t-syntax-table)
>     (set-syntax-table t-syntax-table))
>   ;; secretly must-to-have parts -- end
>
>   (set (make-local-variable 'font-lock-defaults)
>        '(nil nil t
>              ((?/ . ". 14")
>               (?* . ". 23"))))
>

from memory, I don't htink this bit is correct and you shouldn't need it. See
the example on the eamcs wiki. Comment font locking is driven by the comment
characters defined in the syntax table. You shouldn't need to also set them
explicitly in the font-lock stuff (at least, I've not needed to in the derived
modes I've been working on in the past). 


>   (run-mode-hooks 't-mode-hook))
>
> Note: emacs' help writes for font-lock-syntax-table: "this is normally
> set via `font-lock-defaults'", while elisp manual writes for
> make-syntax-table (in 35.3): "most major mode syntax tables are
> created in this way" -- however I would think that make-syntax-table
> is unusable if I set font-lock-syntax-table via font-lock-
> defaults...??
>

have a look at a fairly simple mode and see how they do the font-lock stuff. I
found sql.el quite useful (part of emacs). Again, look at the emacs wikki,
there is some really useful ifo there. I think you may be making things more
complicated than necessary. 

Tim

-- 
tcross (at) rapttech dot com dot au

Re: font-lock-defaults doesn't work??

[Peter Tury]

Tim X:
> You are also re-inventing a wheel to some extent. Use define-derived-mode

If I remember well, define-derived-mode is for very simple new major
modes what practically doesn't do more than fontification, new key
bindings and so. However, I plan to add (much?) more staff later. What
I sent above is the minimal extract from the current code what can be
used to check if it works in the same way for you, than for me.

Have you tried it? Does it work well for you even without the
"secretly must-to-have parts"? Anyway it looks strange for me if
fontification works (almost) well, but syntax parsing doesn't -- I
think they should be in sync. in any (normal) cases...

> from memory, I don't htink this bit is correct and you shouldn't need it. See
> the example on the eamcs wiki. Comment font locking is driven by the comment
> characters defined in the syntax table. You shouldn't need to also set them
> explicitly in the font-lock stuff.

I thought the other way (according to emacs help, what I showed
previously): I planned to use _only_ font-lock-defaults and thought it
sets the syntax table for me. I need syntax properties as well anyway,
so I don't plan to deal too much with the syntax-table: only comment
delimiters seems to be in the right place there in my case...

> have a look at a fairly simple mode and see how they do the font-lock stuff.

Yes, I searched for "official" settings in my elisp directory, but I
afraid all of what I checked are old, so maybe they don't use new(?)
possibilities. Thus I tried what I could understand from the actual
emacs documentation...

> I found sql.el quite useful (part of emacs). Again, look at the emacs wikki,
> there is some really useful ifo there.

I'll check sql.el also, thanks for the tip. What emacswiki page you
think?

\bye
P

Re: font-lock-defaults doesn't work??

[Lennart Borgman (gmail)]

Peter Tury wrote:
> Tim X:
>> You are also re-inventing a wheel to some extent. Use define-derived-mode
> 
> If I remember well, define-derived-mode is for very simple new major
> modes what practically doesn't do more than fontification, new key
> bindings and so. 

I do not know this very well, but maybe you are thinking of 
define-generic-mode? The elisp manu says

    "Even if the new mode is not an obvious derivative of any other mode,
it is convenient to use `define-derived-mode' with a `nil' parent
argument, since it automatically enforces the most important coding
conventions for you."

Re: font-lock-defaults doesn't work??

[Tim X]

"Peter Tury" <tury.peter@gmail.com> writes:

> Tim X:
>> You are also re-inventing a wheel to some extent. Use define-derived-mode
>
> If I remember well, define-derived-mode is for very simple new major
> modes what practically doesn't do more than fontification, new key
> bindings and so. However, I plan to add (much?) more staff later. What
> I sent above is the minimal extract from the current code what can be
> used to check if it works in the same way for you, than for me.
>

I think you might be thinking of one of the other facilities, like generic mode
or mmm mode. 

There are a number of conventions and setup requirements for a new mode that
are common to almost all modes. the define-derived-mode sets things up to make
this easy. You define the font-lock keywords, syntax table, indent function and
whatever else you want to do. some people just start with deriving from
fundamental mode and then incrementally build up the functionality they need. 

> Have you tried it? Does it work well for you even without the
> "secretly must-to-have parts"? Anyway it looks strange for me if
> fontification works (almost) well, but syntax parsing doesn't -- I
> think they should be in sync. in any (normal) cases...
>

I'm not an expert in this area, but I have created a couple of my own modes. In
the most recent one I've been working on, comment characters are either /*
....*/ or -- for a single line comment. All I needed to do to get these lines
fontified in the comment font was to modify the syntax table to set the comment
characters. I also modified the syntax table to change some of the definitions
of what are considered to be word characters, symbol characters etc, but this
was done to get syntax parsing to work (I do this to determine whether I'm
inside a procedures argument list and indent the arguments on subsequent lines
to the opening (. ). It has nothing to do with font-locking directly. 

To define the font-lock stuff, I just define variables representing regular
expressions for the different font-lock types (keyword, built-in, warning,
etc). I use regexp-opt to get an optimised regexp for the block of words and
then set them via the facilities provided in define-derived-mode. 

I then define an indent function and a calculate-indent-size function, set the
appropriate local variables and the basics of my mode (font-locking and
indentation) are done. Now I'm enhancing it further by adding align rules to
see if I get nicer alignment of some definitions. After that, I plan to add
some abbrev stuff and possibly some skeleton/tempo templates to create
'electric' operations and add imenu support. 

Nearly all of this has been straight-forward and based on information I got
from the emacs wikki site. The trick I found was to do things incrementally. I
actually started by getting indentation to work the way I wanted. I then did
the font-lock stuff and am now adding other bits.

>> from memory, I don't htink this bit is correct and you shouldn't need it. See
>> the example on the eamcs wiki. Comment font locking is driven by the comment
>> characters defined in the syntax table. You shouldn't need to also set them
>> explicitly in the font-lock stuff.
>
> I thought the other way (according to emacs help, what I showed
> previously): I planned to use _only_ font-lock-defaults and thought it
> sets the syntax table for me. I need syntax properties as well anyway,
> so I don't plan to deal too much with the syntax-table: only comment
> delimiters seems to be in the right place there in my case...
>
>> have a look at a fairly simple mode and see how they do the font-lock stuff.
>
> Yes, I searched for "official" settings in my elisp directory, but I
> afraid all of what I checked are old, so maybe they don't use new(?)
> possibilities. Thus I tried what I could understand from the actual
> emacs documentation...
>
>> I found sql.el quite useful (part of emacs). Again, look at the emacs wikki,
>> there is some really useful ifo there.
>
> I'll check sql.el also, thanks for the tip. What emacswiki page you
> think?
>

See http://www.emacswiki.org/cgi-bin/wiki/CreateNewMajorMode

and in particular, follow the link concerning derived mode.
-- 
tcross (at) rapttech dot com dot au

Re: font-lock-defaults doesn't work??

[Peter Tury]

Hi,

thanks for all of you for the valuable and really helpful answers!!

Now I see I basically confused two things: font-lock-syntax-table and
(major mode's) syntax-table are not necessarily the same --> thus it
can easily happen that fontification (based on font-lock-syntax-table)
and syntax parsing (based on syntax-table) works (in)correctly
independently from each other. In my problematic example I set
font-lock-syntax-table well, but missed the set of syntax-table (since
I thought it is automatic). Now I understand the normal way (in my
case) is the opposite one (what was suggested by you): setting
syntax-table well, and let font-lock-defaults to inherit it (by using
`nil' in its definition). I tried and saw it works perfectly now.

Also thanks for the explanation of the details behind the wrong
functionality of my wrong code! (I was misled by the info sentence (in
35.3 Syntax Table Funtions > make-syntax-table): "This function creates
a new syntax table, with all values initialized to `nil'." The
following sentences were not clear enough for me...)

And yes, it seems I confused define-derived-mode and
define-generic-mode -- so I'll check -derived-mode and the wiki pages
also!

\bye
P

Re: font-lock-defaults doesn't work??

[martin rudalics]

The problem is that for `t-syntax-table' you use `make-syntax-table'
which inherits from `standard-syntax-table' (the one used for
Fundamental mode) since you don't specify an OLDTABLE argument.
`font-lock-set-defaults' first uses `copy-syntax-table' with the value
returned by `syntax-table' as argument (which could be
`emacs-lisp-mode-syntax-table' since you encounter problems with
semicolons) and then applies the modifications you provided.

But when you define t-mode you eventually have to think about other
things as well - moving around in that buffer, parsing, skipping
comments, ...  Hence you eventually need t-syntax-table with the
comment-syntax you define anyway for buffers in t-mode.  What's wrong
with providing that syntax-table as the current one to font-locking?