Help Understanding syntax-propertize-function

Help Understanding syntax-propertize-function

[Reza Nikoopour]

[-- Attachment #1: Type: text/plain, Size: 1720 bytes --]

Hello All,

I'm trying to implement here doc syntax highlighting.  I've reviewed the
implementations in shell-script-mode and hcl-mode but I don't really
understand what is happening. I've read the documentation for
syntax-propertize-function (
https://www.gnu.org/software/emacs/manual/html_node/elisp/Syntax-Properties.html)
but that didn't help me understand what's going on.

Could someone help explain the following code:

(defun hcl--syntax-propertize-heredoc (end)
  (let ((ppss (syntax-ppss)))
    (when (eq t (nth 3 ppss))
      (let ((key (get-text-property (nth 8 ppss) 'hcl-here-doc-marker))
            (case-fold-search nil))
        (when (re-search-forward
               (concat "^\\(?:[ \t]*\\)" (regexp-quote key) "\\(\n\\)")
               end 'move)
          (let ((eol (match-beginning 1)))
            (put-text-property eol (1+ eol)
                               'syntax-table (string-to-syntax "|"))))))))

(defun hcl--font-lock-open-heredoc (start string eol)
  (unless (or (memq (char-before start) '(?< ?>))
     (save-excursion
                (goto-char start)
                (hcl--in-string-or-comment-p)))
    (let ((str (replace-regexp-in-string "['\"]" "" string)))
      (put-text-property eol (1+ eol) 'hcl-here-doc-marker str)
      (prog1 (string-to-syntax "|")
        (goto-char (+ 2 start))))))

(defun hcl--syntax-propertize-function (start end)
  (goto-char start)
  (hcl--syntax-propertize-heredoc end)
  (funcall
   (syntax-propertize-rules
    (hcl--here-doc-beg-re
     (2 (hcl--font-lock-open-heredoc
         (match-beginning 0) (match-string 1) (match-beginning 2))))
    ("\\s|" (0 (prog1 nil (hcl--syntax-propertize-heredoc end)))))
   (point) end))


Cheers,
Reza

[-- Attachment #2: Type: text/html, Size: 3864 bytes --]

Re: Help Understanding syntax-propertize-function

[Stefan Monnier]

> Could someone help explain the following code:

I'm not sure which part you don't understand, so I'll stick to
generic ideas:

> (defun hcl--syntax-propertize-function (start end)
>   (goto-char start)
>   (hcl--syntax-propertize-heredoc end)
>   (funcall
>    (syntax-propertize-rules
>     (hcl--here-doc-beg-re
>      (2 (hcl--font-lock-open-heredoc
>          (match-beginning 0) (match-string 1) (match-beginning 2))))
>     ("\\s|" (0 (prog1 nil (hcl--syntax-propertize-heredoc end)))))
>    (point) end))

Since we don't know where `start` will be, it can be within a heredoc.
So we first call `hcl--syntax-propertize-heredoc` which should detect
when we're inside a heredoc and if so process it until its end and if not
do nothing.

Once that's done, we know we're not in a heredoc, hence we're in
"normal code" and we then use `syntax-propertize-rules` to try and
detect a few interesting conditions.  One of them is when we find
a match for `hcl--here-doc-beg-re`, in that case we call
`hcl--font-lock-open-heredoc` to place some syntactic marker at the
beginning of the heredoc.  The other is when we see a char of
syntactic category `|`, which doesn't occur in the wild but should
presumably be the result of `hcl--font-lock-open-heredoc` having placed
it there, so it announces the beginning of a heredoc, in which case we
can call `hcl--syntax-propertize-heredoc` to process it.


        Stefan

Re: Help Understanding syntax-propertize-function

[Reza Nikoopour]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

> I'm not sure which part you don't understand, so I'll stick to
> generic ideas:

Thanks that helped a lot and enabled me to figure out what to ask!

> So we first call `hcl--syntax-propertize-heredoc` which should detect
> when we're inside a heredoc and if so process it until its end and if not
> do nothing.

I'm struggling to understand what `hcl--syntax-propertize-heredoc` is
doing.
Could you provide a high level explanation like you did with
`hcl--syntax-propertize-function`.
That helped me a lot.

Cheers,
Reza

[-- Attachment #2: Type: text/html, Size: 653 bytes --]

Re: Help Understanding syntax-propertize-function

[Stefan Monnier]

> I'm struggling to understand what `hcl--syntax-propertize-heredoc` is
> doing.
> Could you provide a high level explanation like you did with
> `hcl--syntax-propertize-function`.
> That helped me a lot.

> (defun hcl--syntax-propertize-heredoc (end)

The purpose of the function is to do the "syntax-propertize" of the
inside of a heredoc.  It presumes that point *may* be in a heredoc but
not necessarily.

>   (let ((ppss (syntax-ppss)))
>     (when (eq t (nth 3 ppss))

Here we fetched the syntax state at point and then we checked that we
are indeed inside a heredoc (or at least a "string like thing" that was
opened using the `|` syntax category; in this mode should should only
ever happen if we have placed this syntax because we found a heredoc
marker).

>       (let ((key (get-text-property (nth 8 ppss) 'hcl-here-doc-marker))

This fetches the string that was used in the heredoc opener and which
has to be used as heredoc closer.  It's been placed on the opening char
at position (nth 8 ppss) by the same code that placed the `|` syntax
property on that same character.

>             (case-fold-search nil))

I guess here that the code does that because HCL defines the heredoc
marker to be case-significant.

>         (when (re-search-forward
>                (concat "^\\(?:[ \t]*\\)" (regexp-quote key) "\\(\n\\)")
>                end 'move)

Here we look for the heredoc end marker.  If we can't find one before
`end`, it just means that the heredoc extends further and hence we have
nothing to do (we could actually remove/override any `|` syntax that
might appear before `end` in case such a thing is possible, but
apparently the rest of code is arranged so that this is not needed).

>           (let ((eol (match-beginning 1)))
>             (put-text-property eol (1+ eol)
>                                'syntax-table (string-to-syntax "|"))))))))

If we did find the heredoc end marker before `end`, then we mark it as
being the end by adding the `|` syntax to the ending character.


        Stefan

Re: Help Understanding syntax-propertize-function

[Reza Nikoopour]

[-- Attachment #1: Type: text/plain, Size: 578 bytes --]

Thank you so much for that explanation.  It helped a lot.

> It's been placed on the opening char
> at position (nth 8 ppss) by the same code that placed the `|` syntax
> property on that same character.

I'm a bit confused as to what places the `|` syntax property.  I see
`(put-text-property eol (1+ eol) 'hcl-here-doc-marker str)` in
`hcl--font-lock-open-heredoc`.  But my understanding is that syntactic
phase happens before the fontification.  So I don't understand how
`hcl--font-lock-open-heredoc` is able to place the symbol for
`hcl--syntax-propertize-function`.

Reza

[-- Attachment #2: Type: text/html, Size: 727 bytes --]

Re: Help Understanding syntax-propertize-function

[Stefan Monnier]

>> It's been placed on the opening char
>> at position (nth 8 ppss) by the same code that placed the `|` syntax
>> property on that same character.
>
> I'm a bit confused as to what places the `|` syntax property.  I see
> `(put-text-property eol (1+ eol) 'hcl-here-doc-marker str)` in
> `hcl--font-lock-open-heredoc`.  But my understanding is that syntactic
> phase happens before the fontification.

`hcl--font-lock-open-heredoc` is called by
`hcl--syntax-propertize-function`.
The `font-lock` in the name is just a relic, I expect.


        Stefan

Re: Help Understanding syntax-propertize-function

[Reza Nikoopour]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

> The `font-lock` in the name is just a relic, I expect.

Ah okay, that clears up a lot.  Thank you so much for taking the time to
explain all of this. It helped me get a better understanding of what's
happening.

Cheers,
Reza

[-- Attachment #2: Type: text/html, Size: 354 bytes --]

Re: Help Understanding syntax-propertize-function

[Filipp Gunbin]

There's the prog1 form at the end of hcl--font-lock-open-heredoc:

...
      (prog1 (string-to-syntax "|")
        (goto-char (+ 2 start))))))

It looks like it's meant to be a special "(prog1 EXP . EXPS)" form for
syntax-propertize-rules.  But it's inside of a defun used as SYNTAX:

...
   (syntax-propertize-rules
    (hcl--here-doc-beg-re
     (2 (hcl--font-lock-open-heredoc
         (match-beginning 0) (match-string 1) (match-beginning 2))))

I wonder - how could that work?  Does syntax-propertize-rules inspect
the SYNTAX expression to check for suchlike forms?

Filipp

Re: Help Understanding syntax-propertize-function

[Stefan Monnier]

> There's the prog1 form at the end of hcl--font-lock-open-heredoc:
>
> ...
>       (prog1 (string-to-syntax "|")
>         (goto-char (+ 2 start))))))

Indeed, there is, but I don't know what worries you bout it.

> It looks like it's meant to be a special "(prog1 EXP . EXPS)" form for
> syntax-propertize-rules.

I don't think so.

> I wonder - how could that work?  Does syntax-propertize-rules inspect
> the SYNTAX expression to check for suchlike forms?

No it doesn't.


        Stefan

Re: Help Understanding syntax-propertize-function

[Filipp Gunbin]

On 24/03/2021 18:10 -0400, Stefan Monnier wrote:

>> There's the prog1 form at the end of hcl--font-lock-open-heredoc:
>>
>> ...
>>       (prog1 (string-to-syntax "|")
>>         (goto-char (+ 2 start))))))
>
> Indeed, there is, but I don't know what worries you bout it.
>
>> It looks like it's meant to be a special "(prog1 EXP . EXPS)" form for
>> syntax-propertize-rules.
>
> I don't think so.

Then prog1 is confusing there.  It's not special, as you confirm, and
it's not needed otherwise.  Just

(goto-char (+ 2 start))
(string-to-syntax "|")

will do - without raising suspicions :-)

Filipp