Combining syntax comment sequences (yaml-mode)

Combining syntax comment sequences (yaml-mode)

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 2029 bytes --]

Hello,

I have reached out for maintainership of yaml-mode recently and am
currently fixing a number of bugs that have been reported on its issue
tracker.  One of these bugs is rather subtle as it demonstrates that its
syntax table isn't quite correct:

YAML does use the "#" character to indicate a comment, like many other
popular languages.  You'd typically encode this in a syntax table as
follows:

(modify-syntax-entry ?# "<" yaml-mode-syntax-table)
(modify-syntax-entry ?\n ">" yaml-mode-syntax-table)

While this covers most cases one would run into, it doesn't conform to
the YAML specification[1]:  A comment is either a token separated by
whitespace from other tokens or is on its own line.  This rule allows
you to use something like "foo#bar" as a token without "bar" getting
interpreted as a comment.  However with the code above "bar" will get
fontified as if it were a comment.

To highlight the former kind of comments correctly, the following works:

(modify-syntax-entry ?\s ". 1" yaml-mode-syntax-table)
(modify-syntax-entry ?# ". 2" yaml-mode-syntax-table)
(modify-syntax-entry ?\n ">" yaml-mode-syntax-table)

To do the same for the latter kind of comments:

(modify-syntax-entry ?\n "> 1" yaml-mode-syntax-table)
(modify-syntax-entry ?# ". 2" yaml-mode-syntax-table)

I cannot find a way to combine both code snippets though.  The obvious
approach yields incorrect results:

(modify-syntax-entry ?\s ". 1" yaml-mode-syntax-table)
(modify-syntax-entry ?# ". 2" yaml-mode-syntax-table)
(modify-syntax-entry ?\n "> 1" yaml-mode-syntax-table)

Is there some other way to combine multiple comment syntaxes?  I find
the existing documentation rather confusing and couldn't find any
tooling for debugging syntax table problems (other than staring hard at
them and having an incomplete mental model of how they work).

I've attached a test file for verifying the results.  yaml-mode itself
can be found on Github[2].

[1]: http://www.yaml.org/spec/1.2/spec.html#id2780069
[2]: https://github.com/yoshiki/yaml-mode

[-- Attachment #2: test.yaml --]
[-- Type: text/plain, Size: 33 bytes --]

foo: bar#baz
# qux quux
qux #bar

Re: Combining syntax comment sequences (yaml-mode)

[Andreas Schwab]

Vasilij Schneidermann <v.schneidermann@gmail.com> writes:

> While this covers most cases one would run into, it doesn't conform to
> the YAML specification[1]:  A comment is either a token separated by
> whitespace from other tokens or is on its own line.  This rule allows
> you to use something like "foo#bar" as a token without "bar" getting
> interpreted as a comment.  However with the code above "bar" will get
> fontified as if it were a comment.

Very similar to bourne shell syntax, so look at sh-script for hints,
especially sh-syntax-propertize-function.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Combining syntax comment sequences (yaml-mode)

[Vasilij Schneidermann]

> Very similar to bourne shell syntax, so look at sh-script for hints,
> especially sh-syntax-propertize-function.

Thanks, I've assumed the only alternative was writing extra
fontification and movement functions.  If I understand correctly, this
is a helper function that applies extra syntax-table properties and can
therefore go beyond what a syntax table is able to do?

Re: Combining syntax comment sequences (yaml-mode)

[Dmitry Gutov]

On 12/23/2015 08:28 PM, Vasilij Schneidermann wrote:
> If I understand correctly, this
> is a helper function that applies extra syntax-table properties and can
> therefore go beyond what a syntax table is able to do?

That's right. So in that function, you can check whether each particular 
hash character is preceded by space or bol.

Re: Combining syntax comment sequences (yaml-mode)

[Vasilij Schneidermann]

With the given hint I was able to write up a helper function adding
`syntax-table` properties:

(defvar yaml-mode-syntax-propertize-function
  (syntax-propertize-rules
   ("^#" (0 "<"))
   ("[ \t]+#" (0 "<"))))

While this looks much neater than my previous approach, it doesn't
nearly function as well.  After loading up the file, not a single
comment is highlighted correctly.  The only thing that triggers
refontification is deleting(!) text close to the comment.  Inserting
text(!!) before or after the comment disables the highlighting again.

This is inacceptable behaviour and since I'm not in the mood for
figuring out how font-lock works, I'll wait and see whether someone's
got an idea how to combine the syntax entries presented in the original
post.