Migrating from font-lock-syntactic-keywords to syntax-propertize-function

Migrating from font-lock-syntactic-keywords to syntax-propertize-function

[Tassilo Horn]

Hi all,

I want to convert some old fontification code from using the now
obsolete `font-lock-syntactic-keywords' variable to using a custom
`syntax-propertize-function'.

Looking at examples inside Emacs itself, it seems the way to go is using
something like

  (defun my-make-syntax-propertize-function ()
    (syntax-propertize-rules
      ;; My rules here.
      ))

and then

  (setq-local syntax-propertize-function
              (my-make-syntax-propertize-function))

when setting up fontification.

Now I have two questions:

1. With `font-lock-syntactic-keywords', a MATCH-HIGHLIGHT could have the
   form (SUBEXP SYNTAX OVERRIDE LAXMATCH) but with
   `syntax-propertize-rules' only (NUMBER SYNTAX) is allowed where
   SUBEXP and NUMBER have the same meaning.  So how does one convert
   entries which use OVERRIDE and/or LAXMATCH to the new mechanism?  (I
   don't use LAXMATCH, so that's not too important.)

2. Some of my syntax rules are not static, so I cannot provide a fixed
   set of rules to `syntax-propertize-rules'.  Is there anything better
   than using `eval' like

     (defun my-make-syntax-propertize-function ()
       (eval
        `(syntax-propertize-rules
          ,@(mapcar
             #'my-convert-legacy-syntactic-keywords
             (my-compute-syntactic-keywords)))))

   or should I prefer writing a manual `syntax-propertize-function'?

Bye,
Tassilo   

Re: Migrating from font-lock-syntactic-keywords to syntax-propertize-function

[Dmitry Gutov]

On 12.05.2020 13:15, Tassilo Horn wrote:
>    (setq-local syntax-propertize-function
>                (my-make-syntax-propertize-function))

This will be

   (setq-local syntax-propertize-function
               #'my-make-syntax-propertize-function)

> 2. Some of my syntax rules are not static, so I cannot provide a fixed
>     set of rules to `syntax-propertize-rules'.  Is there anything better
>     than using `eval' like
> 
>       (defun my-make-syntax-propertize-function ()
>         (eval
>          `(syntax-propertize-rules
>            ,@(mapcar
>               #'my-convert-legacy-syntactic-keywords
>               (my-compute-syntactic-keywords)))))
> 
>     or should I prefer writing a manual `syntax-propertize-function'?

syntax-propertize-rules can have elisp code inside, just like 
font-lock-keywords. See js-syntax-propertize as one example.

Re: Migrating from font-lock-syntactic-keywords to syntax-propertize-function

[Stefan Monnier]

>   (defun my-make-syntax-propertize-function ()
>     (syntax-propertize-rules
>       ;; My rules here.
>       ))

Note that this is a "constant function".  It macroexpands to

    (defun my-make-syntax-propertize-function ()
      (lambda (..) ...))

> 1. With `font-lock-syntactic-keywords', a MATCH-HIGHLIGHT could have the
>    form (SUBEXP SYNTAX OVERRIDE LAXMATCH) but with
>    `syntax-propertize-rules' only (NUMBER SYNTAX) is allowed where
>    SUBEXP and NUMBER have the same meaning.  So how does one convert
>    entries which use OVERRIDE and/or LAXMATCH to the new mechanism?  (I
>    don't use LAXMATCH, so that's not too important.)

font-lock-syntactic-keywords was applied by scanning the whole region
to apply the first rule, then scanning the whole region again to apply
the second rules, etc...
So OVERRIDE was needed to decide what to do when two rules match
on the same chunk of text.

syntax-propertize-rules applies all the rules in a single scan, so only
one rule can match at a given spot, and hence OVERRIDE is not useful.
(as for LAXMATCH, the rules behave the same way as if LAXMATCH was
always set).

> 2. Some of my syntax rules are not static, so I cannot provide a fixed
>    set of rules to `syntax-propertize-rules'.  Is there anything better
>    than using `eval' like
>
>      (defun my-make-syntax-propertize-function ()
>        (eval
>         `(syntax-propertize-rules
>           ,@(mapcar
>              #'my-convert-legacy-syntactic-keywords
>              (my-compute-syntactic-keywords)))))

If your regexps are not static, then indeed you need to resort to `eval`.
This is done in `fortran-make-syntax-propertize-function`, if you want
to see an example.

>    or should I prefer writing a manual `syntax-propertize-function'?

It's up to you.
Of course, another option is `syntax-propertize-via-font-lock`, which
I'd not recommend, but if you're just looking for a quick-fix...


        Stefan

Re: Migrating from font-lock-syntactic-keywords to syntax-propertize-function

[Tassilo Horn]

Hi Dmitry and Stefan,

thank you both, I think I somehow made it.

http://git.savannah.gnu.org/cgit/auctex.git/commit/?h=font-latex-update&id=e6076a4f8cf2f06e956ad1a60728ca3b2eb11a83

Bye,
Tassilo

Re: Migrating from font-lock-syntactic-keywords to syntax-propertize-function

[Tassilo Horn]

Tassilo Horn <tsdh@gnu.org> writes:

> thank you both, I think I somehow made it.
>
> http://git.savannah.gnu.org/cgit/auctex.git/commit/?h=font-latex-update&id=e6076a4f8cf2f06e956ad1a60728ca3b2eb11a83

That was a bit too quick.  It didn't work because we have a lot of
matchers with back-references in order to set syntax properties for
things like \verb|(setq foo t)| where the delimiter (here |) may be any
character, e.g., \verb/(setq foo t)/, \verb-(setq foo t)- would also be
ok.

So for now I this `syntax-propertize-function':

--8<---------------cut here---------------start------------->8---
(defun font-latex-syntax-propertize-function (start end)
  "The `syntax-propertize-function' for (La)TeX documents."
  (with-no-warnings
    (let ((font-lock-syntactic-keywords
           (if (derived-mode-p 'doctex-mode)
               font-latex-doctex-syntactic-keywords
             font-latex-syntactic-keywords)))
      (font-lock-fontify-syntactic-keywords-region start end))))
--8<---------------cut here---------------end--------------->8---

That works good but of course is still the obsolete mechanism, right?
However, I don't feel like implementing the search and syntax-property
setting myself would have a better outcome.

Why does `syntax-propertize-rules' not support back-references?  It
already tracks which regexp group is what, so it could also renumber
back-references, no?  Natively hacking away, it seems like this would do
the trick:

--8<---------------cut here---------------start------------->8---
modified   lisp/emacs-lisp/syntax.el
@@ -140,13 +140,15 @@ syntax-propertize-multiline
   (cons beg end))
 
 (defun syntax-propertize--shift-groups (re n)
-  (replace-regexp-in-string
-   "\\\\(\\?\\([0-9]+\\):"
-   (lambda (s)
-     (replace-match
-      (number-to-string (+ n (string-to-number (match-string 1 s))))
-      t t s 1))
-   re t t))
+  (let ((incr (lambda (s)
+                (replace-match
+                 (number-to-string
+                  (+ n (string-to-number (match-string 1 s))))
+                 t t s 1))))
+    (replace-regexp-in-string
+     "[^\\]\\\\\\([0-9]+\\)" incr
+     (replace-regexp-in-string "\\\\(\\?\\([0-9]+\\):" incr re t t)
+     t t)))
 
 (defmacro syntax-propertize-precompile-rules (&rest rules)
   "Return a precompiled form of RULES to pass to `syntax-propertize-rules'.
--8<---------------cut here---------------end--------------->8---

Judging from the result of

--8<---------------cut here---------------start------------->8---
(syntax-propertize-rules
 ("\\(one\\)\\(two\\)\\(\\1\\)" (1 "|") (2 "."))
 ("\\(three\\)\\(four\\)\\(\\1\\)" (1 "|") (2 ".")))
--8<---------------cut here---------------end--------------->8---

I'd say it works (in this simple case).

Bye,
Tassilo