Font Locking issue: Need help/advice

Font Locking issue: Need help/advice

[Serghei Iakovlev]

[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]

Hello,

I am creating a new mode called zephir-mode [1].  The main goals of
this mode is to provide syntax highlighting, indentation, movement,
navigation support and Imenu for Zephir [2] language.

Currently I'm working on fontification and faced a strange
behavior.  Please see animated GIF I attached to this email to
demonstrate the issue.  The code responsible for fontification
demonstrated in GIF is located here [3].  I used GNU Emacs 27.0.90
to demonstrate and test this issue.

I'd like to provide the patterns I'm trying to catch:

  protected foo;
  protected static foo;
  public bar = 1;
  private baz = [];
  private baz = [] { set };
  protectd foo { get };
  public static bar = 42;
  private static baz = 13 { get };
  private baz = 13 { get, set, toString };

However, my tests passes without any problem [4]. So I need
help here.

Actually I have few questions related to this issue:

- This is my first experience with complex object fontification and
  I suspect that I have chosen an insufficiently effective or
  concise approach/tool/API.  Could someone take a look over used
  approach and provide a small feedback about the overall strategy?

- What happens on the GIF?  How I can fix this? How to avoid similar
  in the future?


- In fact, I would like to be able to independently receive the
  answer to the question "Why did this happen?" so I'm interesting
  on debugging practices.  How to debug similar issues?

[1]: https://github.com/zephir-lang/zephir-mode
[2]: https://zephir-lang.com
[3]:
https://github.com/zephir-lang/zephir-mode/blob/0.6.0/zephir-mode.el#L850-L901
[4]:
https://github.com/zephir-lang/zephir-mode/blob/0.6.0/test/test-zephir-mode-font-lock.el#L239-L266

-- 
Serghei


[-- Attachment #2: zephir-mode-font-lock-issue.gif --]
[-- Type: image/gif, Size: 500490 bytes --]

Re: Font Locking issue: Need help/advice

[Stephen Leake]

Serghei Iakovlev <egrep@protonmail.ch> writes:

> I am creating a new mode called zephir-mode [1].  The main goals of
> this mode is to provide syntax highlighting, indentation, movement,
> navigation support and Imenu for Zephir [2] language.

The ELPA package wisi can help here (I'm the author).

> Currently I'm working on fontification and faced a strange
> behavior.  Please see animated GIF I attached to this email to
> demonstrate the issue.  The code responsible for fontification
> demonstrated in GIF is located here [3].  I used GNU Emacs 27.0.90
> to demonstrate and test this issue.

This is the kind of thing that made move away from regexp to a true
parser for ada-mode, which eventually led to the current wisi package.

> I'd like to provide the patterns I'm trying to catch:
>
>   protected foo;
>   protected static foo;
>   public bar = 1;
>   private baz = [];
>   private baz = [] { set };
>   protectd foo { get };
>   public static bar = 42;
>   private static baz = 13 { get };
>   private baz = 13 { get, set, toString };

This definitely would be easier with a parser.

> - What happens on the GIF?  How I can fix this? How to avoid similar
>   in the future?
>
>
> - In fact, I would like to be able to independently receive the
>   answer to the question "Why did this happen?" so I'm interesting
>   on debugging practices.  How to debug similar issues?

I did not try to answer these, since my real answer is "use wisi".

> [2]: https://zephir-lang.com

I don't see a grammar there, which is the first step in using wisi. Is
there a grammar available somewhere? or one for a similar language, that
would be easy to modify?

-- 
-- Stephe

Re: Font Locking issue: Need help/advice

[Serghei]

Hello Stephen,

> > I am creating a new mode called zephir-mode [1]. The main goals of
> > this mode is to provide syntax highlighting, indentation, movement,
> > navigation support and Imenu for Zephir 2 language.
>
> The ELPA package wisi can help here (I'm the author).

In this case users will have to run an external process, right?
This is a little worrying for me.  I would prefer to use the
built-in functionality for as long as possible.  Partly due to the
fact that users are reluctant (and not always successful) installs
new software.


> > I'd like to provide the patterns I'm trying to catch:
> > protected foo;
> > protected static foo;
> > public bar = 1;
> > private baz = [];
> > private baz = [] { set };
> > protectd foo { get };
> > public static bar = 42;
> > private static baz = 13 { get };
> > private baz = 13 { get, set, toString };
>
> This definitely would be easier with a parser.


Of course, any LR(k) parser can handle this.  But what about SMIE?
Won't it be more organic?


> I did not try to answer these, since my real answer is "use wisi".

In any case, thanks for the answer, I will definitely try this
because I enjoy studying things like scanners, parsers, grammars,
semantics and so on.

--
Serghei

Re: Font Locking issue: Need help/advice

[David Engster]

> - What happens on the GIF?  How I can fix this? How to avoid similar
>   in the future?

What happens here is that the initial font-lock goes over the whole
currently visible section, but as soon as you edit one line, font-lock
will be called only for the part of the buffer that has changed, and it
seems your font-lock rules work differently in this case because the
context has changed.

> - In fact, I would like to be able to independently receive the
>   answer to the question "Why did this happen?" so I'm interesting
>   on debugging practices.  How to debug similar issues?

For better understanding how font-lock actually works, I highly
recommend using font-lock-studio

https://github.com/Lindydancer/font-lock-studio

Also, take a look at the other linked packages in the description. In
your case, 'highlight-refontification' could be useful to see which
parts of the buffer actually get refontified when you edit it.

-David

Re: Font Locking issue: Need help/advice

[Yuan Fu]

> On Apr 3, 2020, at 6:36 PM, David Engster <deng@randomsample.de> wrote:
> 
>> - What happens on the GIF?  How I can fix this? How to avoid similar
>>  in the future?
> 
> What happens here is that the initial font-lock goes over the whole
> currently visible section, but as soon as you edit one line, font-lock
> will be called only for the part of the buffer that has changed, and it
> seems your font-lock rules work differently in this case because the
> context has changed.
> 
>> - In fact, I would like to be able to independently receive the
>>  answer to the question "Why did this happen?" so I'm interesting
>>  on debugging practices.  How to debug similar issues?
> 
> For better understanding how font-lock actually works, I highly
> recommend using font-lock-studio
> 
> https://github.com/Lindydancer/font-lock-studio
> 
> Also, take a look at the other linked packages in the description. In
> your case, 'highlight-refontification' could be useful to see which
> parts of the buffer actually get refontified when you edit it.
> 
> -David
> 

It is also helpful to look at jit-lock.el since that’s the default on-the-fly fortification facility. Basically it marks all text after the changed region (including the region) as “unfontified” when you edit the buffer. And the redisplay (C) code will call jit-lock functions (which calls font-lock functions) to fontify the unfontified text when it needs to display them. IIUC jit-lock is given chunks of unfontified text (several thousand characters) each time. 

Yuan

Re: Font Locking issue: Need help/advice

[Stephen Leake]

Serghei <egrep@protonmail.ch> writes:

> Hello Stephen,
>
>> > I am creating a new mode called zephir-mode [1]. The main goals of
>> > this mode is to provide syntax highlighting, indentation, movement,
>> > navigation support and Imenu for Zephir 2 language.
>>
>> The ELPA package wisi can help here (I'm the author).
>
> In this case users will have to run an external process, right?

They will have to run ~/.emacs.d/elpa/ada-mode-i.jj/build.sh and
install.sh manually when the package is installed; that requires an Ada
compiler, and manual installation a compiler vendor library (the Debian
packages are _not_ adequate, as of last month). That's fine for people
who are already using Ada; it is a pretty high bar for others.

(now that I've actually said that in an email, I realize I might be able
to improve it some).

Actually running the process is transparent.

> This is a little worrying for me. I would prefer to use the built-in
> functionality for as long as possible. Partly due to the fact that
> users are reluctant (and not always successful) installs new software.

At one time, wisi supported an elisp-only option. But I did not want to
implement error correction in the elisp parser; it was hard enough in
Ada. So the elisp was dropped.

The built-in package semantic provides a parser generator and run-time
in elisp. That parser does not do error correction, so the results are
less than satisfactory.

>> > I'd like to provide the patterns I'm trying to catch:
>> > protected foo;
>> > protected static foo;
>> > public bar = 1;
>> > private baz = [];
>> > private baz = [] { set };
>> > protectd foo { get };
>> > public static bar = 42;
>> > private static baz = 13 { get };
>> > private baz = 13 { get, set, toString };
>>
>> This definitely would be easier with a parser.
>
>
> Of course, any LR(k) parser can handle this.  But what about SMIE?
> Won't it be more organic?

I tried SMIE for Ada; it proved inadequate, since in Ada you always have
to parse the whole file. For just the code you show here, SMIE should be
ok. And since it focusses on local parsing, it should be pretty
error-tolerant.

eglot (https://github.com/joaotavora/eglot) is an option, if there is an
existing LSP server for your language (or for a similar-enough
language). At some point, I'd like to allow wisi to use an LSP server as
the parser.

-- 
-- Stephe

Re: Font Locking issue: Need help/advice

[Serghei]

> > -   What happens on the GIF? How I can fix this? How to avoid similar
> >     in the future?
> >
>
> What happens here is that the initial font-lock goes over the whole
> currently visible section, but as soon as you edit one line, font-lock
> will be called only for the part of the buffer that has changed, and it
> seems your font-lock rules work differently in this case because the
> context has changed.

Although I didn't receive answers to all my questions in this
thread, your suggestion about inconsistent font-lock rules turned
out as a clue.  I found that even if I delete all other font-lock
rules (except this one), the problem will still remain.  And I
decided to refactor it from scratch.  After more careful refactor of
this anchored font-lock matcher, I was able to optimize some of its
parts [1].  Now it works as expected.  Thanks for the tip!

Well, parsing complex context sensitive languages with a bunch of
very weak regexes is not a good idea.  Anchored font-lock matchers
certainly help but still seem like a complicated approach.

> For better understanding how font-lock actually works, I highly
> recommend using font-lock-studio
>
> https://github.com/Lindydancer/font-lock-studio

I have heard about this package several times over the past few
years.  However, I still lack some simple instructions on how to
find font-lock issues using this package.  Perhaps a small guide
with an example of finding font-lock problems would help.


[1]: https://github.com/zephir-lang/zephir-mode/pull/28/commits/52ff40451b565d3f4a10b30e9c71e136b7d60021

--
Serghei