Derived Mode 101 HOWTO

Derived Mode 101 HOWTO

[Tim Johnson]

This builds on the topic from another thread.
  "Restricting 'add-hook to a specific file extension"
Here's the goal:
  From the simple (and non-working) example using 'add-hook,
  create a derived mode using 'scheme-mode and a supporting the newlisp
  programming language which is based on lisp.
 AND
1)Add two *new* faces, one for user libraries, and another for
  multi-line strings.
2)Override two existing faces and re-populate with newlisp-specific
syntax.
3)Add a menu
4)Add keybinding.
Compatibility:
  Must be usable by both forks. Xemacs *does not* support
  font-lock-add-keywords. Must use a menuing approach compatible to both
  emacs and Xemacs.

HISTORICALLY: Much of these goals have been implemented albeit crudely, using
'font-lock-add-faces and the 'add-hook approach. Compatible to GNU emacs only.

When finished, I'll add in the keyword-based help system I've built, I'm
actually kind of proud of it. Definitely is useful.

Sample code follows. First thing to tackle is syntax highlighting. I'm
big on that 'cuz my vision sucks. 
;; Results thus far: This file is loaded without error, but
;;  The syntax highlight scheme does not work.

(defface font-lock-newlisp-keywords-face
  '((((class color) (background dark)) (:foreground "tan"))
    (((class color) (background light)) (:foreground "green4"))
    (((class grayscale) (background light)) (:foreground "DimGray" :italic t))
    (((class grayscale) (background dark)) (:foreground "LightGray" :italic t))
    (t (:bold t)))
  "Font Lock mode face used to highlight 
   keywords for Newlisp programming language."
  :group 'font-lock-faces)
(defvar font-lock-newlisp-keywords-face 'font-lock-newlisp-keywords-face)
(defconst word-begin "\\b\\(")
(defconst word-end "\\)\\b")
(defconst 
  newlisp-keywords ;; just a few
   (regexp-opt '( 
    "acos" "add" "and" "append" "append" "apply" "args" "array"
    ) ) )
(defvar newlisp-font-lock-keywords nil "List of newlisp keywords and faces")
(add-hook 'scheme-mode-hook
      (lambda ()
       (progn
        (require 'font-lock)
        (setq newlisp-font-lock-keywords
          (list
            '((concat word-begin newlisp-keywords word-end)
              font-lock-newlisp-keywords-face )
            ))
        (setq scheme-font-lock-keywords
          (append scheme-font-lock-keywords newlisp-font-lock-keywords))
        (message "Newlisp extensions added for Xemacs"))))
(provide 'newlisp++) ;; highlight doesn't work...

Looking forward to input.
-- 
Tim Johnson <tim@johnsons-web.com>
      http://www.alaska-internet-solutions.com

Re: Derived Mode 101 HOWTO

[Stefan Monnier]

Place at the beginning of your newlisp.el file:

  (require 'scheme)

> (defface font-lock-newlisp-keywords-face
>   '((((class color) (background dark)) (:foreground "tan"))
>     (((class color) (background light)) (:foreground "green4"))
>     (((class grayscale) (background light)) (:foreground "DimGray" :italic t))
>     (((class grayscale) (background dark)) (:foreground "LightGray" :italic t))
>     (t (:bold t)))
>   "Font Lock mode face used to highlight 
>    keywords for Newlisp programming language."
>   :group 'font-lock-faces)
> (defvar font-lock-newlisp-keywords-face 'font-lock-newlisp-keywords-face)
> (defconst word-begin "\\b\\(")
> (defconst word-end "\\)\\b")

Every global name you define should use the "newlisp-" prefix.

> (defconst 
>   newlisp-keywords ;; just a few
>    (regexp-opt '( 
>     "acos" "add" "and" "append" "append" "apply" "args" "array"
>     ) ) )

> (defvar newlisp-font-lock-keywords nil "List of newlisp keywords and faces")

Initialize it directly:

  (defvar newlisp-font-lock-keywords
    `(,@scheme-font-lock-keywords
      (,(concat "\\<\\(" newlisp-keywords "\\)\\>")
       . font-lock-newlisp-keywords-face))
    "List of newlisp keywords and faces")

> (add-hook 'scheme-mode-hook
>       (lambda ()
>        (progn
>         (require 'font-lock)
>         (setq newlisp-font-lock-keywords
>           (list
>             '((concat word-begin newlisp-keywords word-end)
>               font-lock-newlisp-keywords-face )
>             ))
>         (setq scheme-font-lock-keywords
>           (append scheme-font-lock-keywords newlisp-font-lock-keywords))
>         (message "Newlisp extensions added for Xemacs"))))

Replace by define-derived-mode:

  (define-derived-mode newlisp-mode scheme-mode "Newlisp"
    "A major mode for Newlisp."
    (set (make-local-variable 'font-lock-defaults)
         (cons newlisp-font-lock-keywords
               ;; Copy the rest of font-lock-defaults from
               ;; scheme-mode if available.
               (or (cdr font-lock-defaults)
                   '(nil t (("+-*/.<>=!?$%_&~^:" . "w")))))))

Also check (again?) sample-mode.el.


        Stefan

Re: Derived Mode 101 HOWTO

[Tim Johnson]

I have written a file called newlisp-mode.el, based on Stefan's
instructions, as best as I can follow.
Results are as follows:
When loading a .lsp file with GNU emacs, newlisp-mode is loaded without
complaint.
eval->major-mode gives me "newlisp-mode". 

However syntax highlighting for 'newlisp-font-lock-keywords fails.

eval->font-lock-defaults shows that the dotted list composed from
  newlisp-keywords and font-lock-newlisp-keywords-face
  is present. 

The scheme font-lock-keywords-face and  font-lock-function-name-face
are successfully implemented.

As for Xemacs, loading the file and the mode fails with the following
error message: (may be wrapped and obfuscated by mailer)
 Wrong type argument: symbolp,
   ("\\<\\(a\\(?:cos\\|dd\\|nd\\|pp\\(?:end\\(?:\\)?\\|
   ly\\)\\|r\\(?:gs\\|ray\\)\\)\\)\\>"
   . font-lock-newlisp-keywords-face)
NOTE: This refers to the same dotted list constructed as a component of
      newlisp-font-lock-keywords

;; Code follows:
(require 'scheme)
;; ==========================================================================
(defface font-lock-newlisp-keywords-face
  '((((class color) (background dark)) (:foreground "yellow"))
    (((class color) (background light)) (:foreground "green4"))
    (((class grayscale) (background light)) (:foreground "DimGray" :italic t))
    (((class grayscale) (background dark)) (:foreground "LightGray" :italic t))
    (t (:bold t)))
  "Font Lock mode face used to highlight
   keywords for Newlisp programming language."
  :group 'font-lock-faces)
;; ==========================================================================
(defconst
  newlisp-keywords ;; just a few
   (regexp-opt '(
    "acos" "add" "and" "append" "append" "apply" "args" "array"
    )))
;; ==========================================================================
(defvar newlisp-font-lock-keywords
   `(,@scheme-font-lock-keywords
     (,(concat "\\<\\(" newlisp-keywords "\\)\\>")
      . font-lock-newlisp-keywords-face))
   "List of newlisp keywords and faces")
;; ==========================================================================
(define-derived-mode newlisp-mode scheme-mode "newlisp"
  "A major mode for Newlisp."
  (set (make-local-variable 'font-lock-defaults)
       (cons newlisp-font-lock-keywords
             ;; Copy the rest of font-lock-defaults from
             ;; scheme-mode if available.
             (or (cdr font-lock-defaults)
                 '(nil t (("+-*/.<>=!?$%_&~^:" . "w")))))))
-- 
Tim Johnson <tim@johnsons-web.com>
      http://www.alaska-internet-solutions.com

Re: Derived Mode 101 HOWTO

[Stefan Monnier]

> However syntax highlighting for 'newlisp-font-lock-keywords fails.

In what way, exactly.
If you look at one of the keywords you're trying to highlight (like
"array"), what does it look like?  What does M-x list-text-properties-at say
about the `face' property on those keywords?

>   (set (make-local-variable 'font-lock-defaults)
>        (cons newlisp-font-lock-keywords
              ^^
               '
This missing quote is probably what bothers XEmacs.


        Stefan

Re: Derived Mode 101 HOWTO

[Tim Johnson]

On 2006-03-08, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> However syntax highlighting for 'newlisp-font-lock-keywords fails.
>
> In what way, exactly.
> If you look at one of the keywords you're trying to highlight (like
> "array"), what does it look like?  What does M-x list-text-properties-at say
> about the `face' property on those keywords?

 It is not colorized.

 list-text-properties-at returns:
 "Text property at 2235:  fontified  t"

 Would expect something like:
 "Text properties at 2235:

 face                 font-lock-keyword-face
 fontified            t
 "
 
>>   (set (make-local-variable 'font-lock-defaults)
>>        (cons newlisp-font-lock-keywords
>               ^^
>                '
> This missing quote is probably what bothers XEmacs.

 I believe that you are correct, placing the quote
 "cured" that problem.
 Now I have another error message from Xemacs:
 """
 Range must be t or a character: "+-*/.<>=!?$%_&~^:"
 """
 Xemacs appears to not like the last line of the
 'define-derived-mode

 Thank you.

-- 
Tim Johnson <tim@johnsons-web.com>
      http://www.alaska-internet-solutions.com

Re: Derived Mode 101 HOWTO

[Stefan Monnier]

>>> However syntax highlighting for 'newlisp-font-lock-keywords fails.
>> 
>> In what way, exactly.
>> If you look at one of the keywords you're trying to highlight (like
>> "array"), what does it look like?  What does M-x list-text-properties-at say
>> about the `face' property on those keywords?

>  It is not colorized.

>  list-text-properties-at returns:
>  "Text property at 2235:  fontified  t"

>  Would expect something like:
>  "Text properties at 2235:

>  face                 font-lock-keyword-face
>  fontified            t
>  "

What is the value of `font-lock-keywords' in that buffer?
 
>  Now I have another error message from XEmacs:
>  """
>  Range must be t or a character: "+-*/.<>=!?$%_&~^:"
>  """

Please do (setq debug-on-error t) and then reproduce the problem.
You'll hopefully get a backtrace that will give us more info.


        Stefan

Re: Derived Mode 101 HOWTO

[Tim Johnson]

On 2006-03-11, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>>> However syntax highlighting for 'newlisp-font-lock-keywords fails.
>
> What is the value of `font-lock-keywords' in that buffer?
;; Value begins below (wrapped)
(t
("(\\(define\\*?\\(\\(\\|-public\\|-method\\|-generic\\(-procedure\\)?\\)\\|
\\(-syntax\\|-macro\\)\\|-class\\|-module\\)\\)\\>[
]*(?\\(\\sw+\\)?" 
(1 font-lock-keyword-face) (6 (cond ... ... ...) nil t)) 
("\\<\\(a\\(?:cos\\|dd\\|nd\\|pp\\(?:end\\(?:\\)?\\|ly\\)\\|r\\
(?:gs\\|ray\\)\\)\\)\\>" 
(0 font-lock-newlisp-keywords-face)))
;; End value

>>  Now I have another error message from XEmacs:
>>  """
>>  Range must be t or a character: "+-*/.<>=!?$%_&~^:"
>>  """
>
> Please do (setq debug-on-error t) and then reproduce the problem.
> You'll hopefully get a backtrace that will give us more info.

  ;; Okay. All trace follows:
  Signaling: (error "Range must be t or a character" "+-*/.<>=!?$%_&~^:")
  signal(error ("Range must be t or a character" "+-*/.<>=!?$%_&~^:"))
  signal-error(error ("Range must be t or a character" "+-*/.<>=!?$%_&~^:"))
  normal-top-level()
  
;; below are most recent messages:
Entering debugger...
Loading debug...done
Loading debug...
Loading newlisp-mode...done
Loading newlisp-mode...
Loading /home/tim/.session...
Loading font...done
Loading font...
Paren mode is paren
Loading paren...done
Loading paren...
Loading blink-cursor...done
Loading blink-cursor...

Perhaps time for
(defvar running-xemacs (string-match "XEmacs\\|Lucid" emacs-version))
??
Also:
Googled:
http://www.google.com/search?hl=en&lr=&q=%22Range+must+be+t+or+a+character%22+
xemacs&btnG=Search
Refernces to chartab.c
Thank You
tim (is/was C programmer)
-- 
Tim Johnson <tim@johnsons-web.com>
      http://www.alaska-internet-solutions.com

Re: Derived Mode 101 HOWTO

[Johan Bockgård]

Tim Johnson <tim@johnsons-web.com> writes:

> (defvar newlisp-font-lock-keywords
>    `(,@scheme-font-lock-keywords
>      (,(concat "\\<\\(" newlisp-keywords "\\)\\>")
>       . font-lock-newlisp-keywords-face))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[C-h v font-lock-keywords]

Emacs:

    "FACENAME is an expression whose value is the face name to use."
                    ^^^^^^^^^^^^^^^^^^^^^^

XEmacs:

    "FACE-FORM :== The symbol naming a defined face. | Expression whos
    value is the face name to use. If you want FACE-FORM to be a
    symbol that evaluates to a face, use a form like "(progn sym)"."


> (define-derived-mode newlisp-mode scheme-mode "newlisp"
>   "A major mode for Newlisp."
>   (set (make-local-variable 'font-lock-defaults)
>        (cons newlisp-font-lock-keywords
>              ;; Copy the rest of font-lock-defaults from
>              ;; scheme-mode if available.
>              (or (cdr font-lock-defaults)
>                  '(nil t (("+-*/.<>=!?$%_&~^:" . "w")))))))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[C-h v font-lock-defaults]

Emacs:

  "If SYNTAX-ALIST is non-nil, it should be a list of cons pairs of the form
  (CHAR-OR-STRING . STRING) used to set the local Font Lock syntax table"
   ^^^^^^^^^^^^^^

XEmacs:

  "If SYNTAX-ALIST is non-nil, it should be a list of cons pairs of the form
  (CHAR . STRING) used to set the local Font Lock syntax table"
   ^^^^

-- 
Johan Bockgård

Re: Derived Mode 101 HOWTO

[Stefan Monnier]

> Perhaps time for
> (defvar running-xemacs (string-match "XEmacs\\|Lucid" emacs-version))

Definitely not.  If you have to resort to that (unlikely for now), please
use (featurep 'xemacs).


        Stefan

Re: Derived Mode 101 HOWTO

[Tim Johnson]

On 2006-03-12, Johan Bockgård <bojohan+news@dd.chalmers.se> wrote:

  Hello Johan:

> Emacs:
>
>   "If SYNTAX-ALIST is non-nil, it should be a list of cons pairs of the form
>   (CHAR-OR-STRING . STRING) used to set the local Font Lock syntax table"
>    ^^^^^^^^^^^^^^
>
> XEmacs:
>
>   "If SYNTAX-ALIST is non-nil, it should be a list of cons pairs of the form
>   (CHAR . STRING) used to set the local Font Lock syntax table"
>    ^^^^

  Thanks for the input. Sorry to seem so dense, but are you suggesting
  that to make Xemacs happy, that there should be a list of dotted
  pairs, one for each in "+-*/.<>=!?$%_&~^:" 

  Could you provide an example?
  
  Thanks
  tim

-- 
Tim Johnson <tim@johnsons-web.com>
      http://www.alaska-internet-solutions.com

Re: Derived Mode 101 HOWTO

[Johan Bockgård]

Tim Johnson <tim@johnsons-web.com> writes:

>  Thanks for the input. Sorry to seem so dense, but are you
>  suggesting that to make Xemacs happy, that there should be a list
>  of dotted pairs, one for each in "+-*/.<>=!?$%_&~^:"

Yes.

>  Could you provide an example?

Taken from XEmacs' scheme.el:

    ;; XEmacs change
    ((?+ . "w") (?- . "w") (?* . "w") (?/ . "w")
     (?. . "w") (?< . "w") (?> . "w") (?= . "w")
     (?? . "w") (?$ . "w") (?% . "w") (?_ . "w")
     (?& . "w") (?~ . "w") (?^ . "w") (?: . "w"))

-- 
Johan Bockgård

Re: Derived Mode 101 HOWTO

[Tim Johnson]

On 2006-03-12, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Perhaps time for
>> (defvar running-xemacs (string-match "XEmacs\\|Lucid" emacs-version))
 
  I thought that I had already posted the following, but since I see no
  sign of it, one more time (and thanks to Johan's suggestion) the
  derived-mode form now looks like this:
;; code
 (define-derived-mode newlisp-mode scheme-mode "newlisp"
  "A major mode for Newlisp."
  (set (make-local-variable 'font-lock-defaults)
       (cons 'newlisp-font-lock-keywords
             ;; Copy the rest of font-lock-defaults from
             ;; scheme-mode if available.
             (or (cdr font-lock-defaults)
                 '(nil t 
					   ;(("+-*/.<>=!?$%_&~^:" . "w"))
					   ((?+ . "w") (?- . "w") (?* . "w") (?/ . "w")
					    (?. . "w") (?< . "w") (?> . "w") (?= . "w")
                        (?? . "w") (?$ . "w") (?% . "w") (?_ . "w")
                        (?& . "w") (?~ . "w") (?^ . "w") (?: . "w"))
					   ))))
  (message "Newlisp Derived Mode Loaded!")) 
  The good news:
  The words for the face now appear as highlighted properly in XEmacs, but
  The bad news is:

  there is no fontifaction in GNU Emacs.
  list-text-properties-at returns 
  Text property at 2326:  fontified  t
  And emacs shows the following message:

  Error during redisplay: 
    (void-variable font-lock-newlisp-keywords-face) [2 times]

  Even with debug on, I see no stack trace, this is stored
  in *Messages*

  evaluating font-lock-keywords returns the following:

  (t
  ("(\\(define\\*?\\(\\(\\|-public\\|-method\\|-generic\\(-procedure\\)?\\)\\
  |\\(-syntax\\|-macro\\)\\|-class\\|-module\\)\\)\\>[
  ]*(?\\(\\sw+\\)?" (1 font-lock-keyword-face) (6 (cond ... ... ...) nil
  t))
  ("\\<\\(a\\(?:cos\\|dd\\|nd\\|pp\\(?:end\\(?:\\)?\\|ly\\)\\|r\\(?:gs\\
  |ray\\)\\)\\)\\>"
  (0 font-lock-newlisp-keywords-face)))

  wrapped to accomodate mailer...
-- 
Tim Johnson <tim@johnsons-web.com>
      http://www.alaska-internet-solutions.com

Re: Derived Mode 101 HOWTO

[Johan Bockgård]

Tim Johnson <tim@johnsons-web.com> writes:

>   Error during redisplay: 
>     (void-variable font-lock-newlisp-keywords-face) [2 times]

I already tried to explain this one in the first part of my earlier
post (subtly); look closer. Together with the error message you should
be able to figure it out.

-- 
Johan Bockgård

Re: Derived Mode 101 HOWTO

[Tim Johnson]

In gnu.emacs.help, you wrote:
> Tim Johnson <tim@johnsons-web.com> writes:
>
>>   Error during redisplay:
>>     (void-variable font-lock-newlisp-keywords-face) [2 times]
>
> I already tried to explain this one in the first part of my earlier
> post (subtly); look closer. Together with the error message you should
> be able to figure it out.

  :-) You're too kind..
      In your post, you provided this below:
;;=======================================================================
 [C-h v font-lock-keywords]

Emacs:

    "FACENAME is an expression whose value is the face name to use."
                    ^^^^^^^^^^^^^^^^^^^^^^
XEmacs:

    "FACE-FORM :== The symbol naming a defined face. | Expression whos
    value is the face name to use. If you want FACE-FORM to be a
    symbol that evaluates to a face, use a form like "(progn sym)"."
;;=======================================================================

 And unfortunately I do not entirely understand. Are you saying that
 XEmacs needs a 'progn form for the face?

 By quoting the face:
(defvar newlisp-font-lock-keywords
   `(,@scheme-font-lock-keywords
     (,(concat "\\<\\(" newlisp-keywords "\\)\\>")
      . 'font-lock-newlisp-keywords-face) ;; quoted here
     )
   "List of newlisp keywords and faces")
Highlighting is now 'enabled' in GNU emacs
   *but*
is now 'disabled' in XEmacs.

Don't know how to resolve ...
thanks
tim

-- 
Tim Johnson <tim@johnsons-web.com>
      http://www.alaska-internet-solutions.com

Re: Derived Mode 101 HOWTO

[Johan Bockgård]

Tim Johnson <tim@johnsons-web.com> writes:

>  And unfortunately I do not entirely understand. Are you saying that
>  XEmacs needs a 'progn form for the face?
>
>  By quoting the face:
> (defvar newlisp-font-lock-keywords
>    `(,@scheme-font-lock-keywords
>      (,(concat "\\<\\(" newlisp-keywords "\\)\\>")
>       . 'font-lock-newlisp-keywords-face) ;; quoted here

Well, 'font-lock-newlisp-keywords-face is an "expression whose value
is the face name to use", but apparently you shouldn't trust the
documentation. (That's one of the reasons i don't use XEmacs. :)

Try replacing the last line above with this:

      0 'font-lock-newlisp-keywords-face)

-- 
Johan Bockgård

Re: Derived Mode 101 HOWTO

[Tim Johnson]

Hi Johan:

In gnu.emacs.help, you wrote:
> Well, 'font-lock-newlisp-keywords-face is an "expression whose value
> is the face name to use", but apparently you shouldn't trust the
> documentation. (That's one of the reasons i don't use XEmacs. :)

  I'm a GNU type of person, but we have identified XEmacs as a possible
  platform for Windows-using clients, since our clients think that it is
  more modern-looking (on windows) than GNU emacs.

   We'll see. "modern-looking" is not so important to me.

   Additionally, I have found this newsgroup *far* more helpful than the
   XEmacs newsgroup (no offense meant to the other).

> Try replacing the last line above with this:
>
>       0 'font-lock-newlisp-keywords-face)

  You have put the finishing nail in this piece of the endeavor!
;; code below. See the addition of a second face
(defvar newlisp-font-lock-keywords
   `(,@scheme-font-lock-keywords
     (,(concat "\\<\\(" newlisp-keywords "\\)\\>")
      0 'font-lock-newlisp-keywords-face)
     (,(concat "\\<\\(" newlisp-user-keywords"\\)\\>")
        0 'font-lock-newlisp-user-keywords-face))
   "List of newlisp keywords and faces")

I want to add keybinding and a menu -  I expect that will go more smoothly.
:-)
Look for Derived Mode 102 HOWTO in a week or so.

After I get the mode where I want it, then I will also provide a (hopefully)
easy-follow 'template' to add further derived modes with minimum heartburn.

Next step is to read much documentation in depth.

Thank you Johan.
Thank you Stefan.

Cheers
Tim

-- 
Tim Johnson <tim@johnsons-web.com>
      http://www.alaska-internet-solutions.com