Semantic font-lock for Java

Semantic font-lock for Java

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 1136 bytes --]

I've been working on Semantic font-lock for Java using CEDET Semantic
infrastructure which uses Wisent incremental parsing under the hood. So far
the results look promising. Hence I'm sharing the same.
[image: Post image]
<https://preview.redd.it/poxkw6o7a4c41.png?width=849&format=png&auto=webp&s=a6f6e973f7e555eea1b87edd942c733db2340957>

It uses following face definitions for highlighting.

(defvar semantic-format-face-alist
  `( (function . font-lock-function-name-face)
     (variable . font-lock-variable-name-face)
     (type . font-lock-type-face)
     ;; These are different between Emacsen.
     (include . ,'font-lock-constant-face)
     (package . , 'font-lock-constant-face)
     ;; Not a tag, but instead a feature of output
     (label . font-lock-string-face)
     (comment . font-lock-comment-face)
     (keyword . font-lock-keyword-face)
     (annotation . font-lock-builtin-face)
     (constant . bold)
     ;;(member . font-lock-member-face)
     (abstract . italic)
     (static . italic)
     (documentation . font-lock-doc-face)
     )

*Code:* Emacs customization (https://gitlab.com/atamariya/emacs/tree/dev)

[-- Attachment #2: Type: text/html, Size: 2265 bytes --]

Re: Semantic font-lock for Java

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Tue, 21 Jan 2020 17:09:41 +0530
> 
> I've been working on Semantic font-lock for Java using CEDET Semantic infrastructure which uses Wisent
> incremental parsing under the hood. So far the results look promising. Hence I'm sharing the same.

Thanks, this is something I consider an important development
direction for Emacs.

> Code: Emacs customization (https://gitlab.com/atamariya/emacs/tree/dev)

This seems to be a complete forked repository.  Is it possible to show
a patch relative to the current master?

Re: Semantic font-lock for Java

[Eli Zaretskii]

> Date: Tue, 21 Jan 2020 17:41:26 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Code: Emacs customization (https://gitlab.com/atamariya/emacs/tree/dev)
> 
> This seems to be a complete forked repository.  Is it possible to show
> a patch relative to the current master?

Oh, and btw: did you make any speed comparisons with the default
font-lock?  E.g., you could run a benchmark of scrolling through a
large source file top to bottom, and show the results in terms of time
and number of GCs.

Thanks.

Re: Semantic font-lock for Java

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

> This seems to be a complete forked repository.  Is it possible to show
> a patch relative to the current master?
>
https://gitlab.com/atamariya/emacs/compare/v0.5...dev

Most relevant changes are in lisp/cedet/semantic/java.el . Other minor
changes are for disabling jit-lock.
- Font lock setup : semantic-font-lock-mode()
- Fontify region function: extract()
- Helper function: semantic-fontify-tag()

you could run a benchmark of scrolling through a
> large source file top to bottom, and show the results in terms of time
> and number of GCs.

Not sure how to do this. If you can help me with steps, I can collect the
data.

[-- Attachment #2: Type: text/html, Size: 1156 bytes --]

Re: Semantic font-lock for Java

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

Also want to highlight some noticeable features in the comparison pic:

   1.

   Static variables and functions are highlighted differently.
   2.

   Constant variables are highlighted differently.
   3.

   Not only the declaration, but the usages are highlighted also.
   4.

   Package is highlighted correctly.

[image: Post image]

On Tue, Jan 21, 2020 at 5:09 PM Anand Tamariya <atamariya@gmail.com> wrote:

> I've been working on Semantic font-lock for Java using CEDET Semantic
> infrastructure which uses Wisent incremental parsing under the hood. So far
> the results look promising. Hence I'm sharing the same.
> [image: Post image]
> <https://preview.redd.it/poxkw6o7a4c41.png?width=849&format=png&auto=webp&s=a6f6e973f7e555eea1b87edd942c733db2340957>
>
> It uses following face definitions for highlighting.
>
> (defvar semantic-format-face-alist
>   `( (function . font-lock-function-name-face)
>      (variable . font-lock-variable-name-face)
>      (type . font-lock-type-face)
>      ;; These are different between Emacsen.
>      (include . ,'font-lock-constant-face)
>      (package . , 'font-lock-constant-face)
>      ;; Not a tag, but instead a feature of output
>      (label . font-lock-string-face)
>      (comment . font-lock-comment-face)
>      (keyword . font-lock-keyword-face)
>      (annotation . font-lock-builtin-face)
>      (constant . bold)
>      ;;(member . font-lock-member-face)
>      (abstract . italic)
>      (static . italic)
>      (documentation . font-lock-doc-face)
>      )
>
> *Code:* Emacs customization (https://gitlab.com/atamariya/emacs/tree/dev)
>

[-- Attachment #2: Type: text/html, Size: 2959 bytes --]

Re: Semantic font-lock for Java

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Wed, 22 Jan 2020 10:43:21 +0530
> Cc: emacs-devel@gnu.org
> 
>  This seems to be a complete forked repository.  Is it possible to show
>  a patch relative to the current master?
> 
> https://gitlab.com/atamariya/emacs/compare/v0.5...dev

Thanks.

> Most relevant changes are in lisp/cedet/semantic/java.el . Other minor changes are for disabling jit-lock.
> - Font lock setup : semantic-font-lock-mode()
> - Fontify region function: extract()
> - Helper function: semantic-fontify-tag()

I don't think I understand the reasons for the changes in font-lock.el
and cc-mode.el.  Can you explain?

Also, what do you mean by "disable jit-lock"?  Where does that happen?
And if you disable jit-lock, does that mean Emacs will no longer
fontify only the portions of the window it is about to display?

>  you could run a benchmark of scrolling through a
>  large source file top to bottom, and show the results in terms of time
>  and number of GCs.
> 
> Not sure how to do this. If you can help me with steps, I can collect the data. 

I find this function useful:

  (defun scroll-up-benchmark ()
    (interactive)
    (let ((oldgc gcs-done)
	  (oldtime (float-time)))
      (condition-case nil (while t (scroll-up) (redisplay))
	(error (message "GCs: %d Elapsed time: %f seconds"
			(- gcs-done oldgc) (- (float-time) oldtime))))))

Run it after loading a file, and look at the times with the default
font-lock and with your modifications.

Re: Semantic font-lock for Java

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 1685 bytes --]

> I don't think I understand the reasons for the changes in font-lock.el
> and cc-mode.el.  Can you explain?
>
These changes are there only for easy testing. They are unlikely to be part
of final patch.

Also, what do you mean by "disable jit-lock"?  Where does that happen?
> And if you disable jit-lock, does that mean Emacs will no longer
> fontify only the portions of the window it is about to display?
>
 There are some after change functions and region extend functions which
are not really necessary with my approach. Hence I had to disable those
portions. Following call stack should help you better visualize.
- redisplay_internal (C function)                                   3   2%
 - jit-lock-function                                                2   1%
  - jit-lock-fontify-now                                            2   1%
   - jit-lock--run-functions                                        2   1%
    - run-hook-wrapped                                              2   1%
     - #<compiled 0x158872bab969>                                   2   1%
      - font-lock-fontify-region                                    2   1%
       - *extract*                                                    2   1%

>  you could run a benchmark of scrolling through a
> >  large source file top to bottom, and show the results in terms of time
> >  and number of GCs.
>
BTW I should admit that it is WIP. Right now, my focus is only on accuracy.
I have made no attempts on optimization. So consider the results
accordingly.

   - *Off* GCs: 0 Elapsed time: 0.840946 seconds
   - *Custom* GCs: 0 Elapsed time: 1.172106 seconds
   - *Builin* GCs: 1 Elapsed time: 0.991606 seconds

[-- Attachment #2: Type: text/html, Size: 2547 bytes --]

Re: Semantic font-lock for Java

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Thu, 23 Jan 2020 11:06:57 +0530
> Cc: emacs-devel@gnu.org
> 
>  Also, what do you mean by "disable jit-lock"?  Where does that happen?
>  And if you disable jit-lock, does that mean Emacs will no longer
>  fontify only the portions of the window it is about to display?
> 
>  There are some after change functions and region extend functions which are not really necessary with my
> approach. Hence I had to disable those portions. Following call stack should help you better visualize.
> - redisplay_internal (C function)                                   3   2%
>  - jit-lock-function                                                2   1%
>   - jit-lock-fontify-now                                            2   1%
>    - jit-lock--run-functions                                        2   1%
>     - run-hook-wrapped                                              2   1%
>      - #<compiled 0x158872bab969>                                   2   1%
>       - font-lock-fontify-region                                    2   1%
>        - extract                                                    2   1%

(I suppose you meant "extend", not "extract"?)

Then I'm unsure why you said you needed to "disable jit-lock".
Removing the calls to CC mode's after-change hooks doesn't disable JIT
font lock in any way.  What am I missing?

>  >  you could run a benchmark of scrolling through a
>  >  large source file top to bottom, and show the results in terms of time
>  >  and number of GCs.
> 
> BTW I should admit that it is WIP. Right now, my focus is only on accuracy. I have made no attempts on
> optimization. So consider the results accordingly.
> 
> * Off GCs: 0 Elapsed time: 0.840946 seconds
> * Custom GCs: 0 Elapsed time: 1.172106 seconds
> * Builin GCs: 1 Elapsed time: 0.991606 seconds

I'm not sure I understand these results.  What does each line
represent?  Did you run the same benchmark with the default font-lock
and with your modifications, and if so, which of these lines shows
results for what variant of font-lock?

From my POV, the most important part of this work is to have
infrastructure in place that would allow us to use additional syntax
and semantics analyzers as the engine for fontification.  So I'm
especially interested in coming up with a set of settings and user
options for selecting this or that analyzer, while keeping the overall
design of JIT fontifcations as similar to what we have now as
practically possible.

Thanks.

Re: Semantic font-lock for Java

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

> >       - font-lock-fontify-region                                    2
>  1%
> >        - extract                                                    2
>  1%
>
> (I suppose you meant "extend", not "extract"?)
>
I was referring to font-lock-extend-region-functions variable. "extract"
function name is just a temporary short name. Makes my testing easier. Will
name it something better when I've it working according to my preferences.

Then I'm unsure why you said you needed to "disable jit-lock".
> Removing the calls to CC mode's after-change hooks doesn't disable JIT
> font lock in any way.  What am I missing?
>
I added in clarification that I had to **disable portions** . Look at
lisp/progmodes/cc-mode.el (746-748). Those lines are commented. Similarly
some other places in cc-mode.el and font-lock.el are commented.

> * Off GCs: 0 Elapsed time: 0.840946 seconds
> > * Custom GCs: 0 Elapsed time: 1.172106 seconds
> > * Builin GCs: 1 Elapsed time: 0.991606 seconds
>
> I'm not sure I understand these results.  What does each line
> represent?  Did you run the same benchmark with the default font-lock
> and with your modifications, and if so, which of these lines shows
> results for what variant of font-lock?
>
> Off - Refers to when font-lock is disabled
Custom - Refers to when my "extract" function is doing the fontification
Builtin - Refers to jit-lock mode as that's the default for font-lock mode

Hope that helps.

[-- Attachment #2: Type: text/html, Size: 2224 bytes --]

Re: Semantic font-lock for Java

[Eli Zaretskii]

> From: Anand Tamariya <atamariya@gmail.com>
> Date: Fri, 24 Jan 2020 21:45:55 +0530
> Cc: emacs-devel@gnu.org
> 
>  > * Off GCs: 0 Elapsed time: 0.840946 seconds
>  > * Custom GCs: 0 Elapsed time: 1.172106 seconds
>  > * Builin GCs: 1 Elapsed time: 0.991606 seconds
> 
>  I'm not sure I understand these results.  What does each line
>  represent?  Did you run the same benchmark with the default font-lock
>  and with your modifications, and if so, which of these lines shows
>  results for what variant of font-lock?
> 
> Off - Refers to when font-lock is disabled
> Custom - Refers to when my "extract" function is doing the fontification
> Builtin - Refers to jit-lock mode as that's the default for font-lock mode

So we are talking about a 17% slow-down?  Not bad.

Thanks.

Re: Semantic font-lock for Java

[David Engster]

>> From: Anand Tamariya <atamariya@gmail.com>
>> Date: Fri, 24 Jan 2020 21:45:55 +0530
>> Cc: emacs-devel@gnu.org
>
>> 
>>  > * Off GCs: 0 Elapsed time: 0.840946 seconds
>>  > * Custom GCs: 0 Elapsed time: 1.172106 seconds
>>  > * Builin GCs: 1 Elapsed time: 0.991606 seconds
>> 
>>  I'm not sure I understand these results.  What does each line
>>  represent?  Did you run the same benchmark with the default font-lock
>>  and with your modifications, and if so, which of these lines shows
>>  results for what variant of font-lock?
>> 
>> Off - Refers to when font-lock is disabled
>> Custom - Refers to when my "extract" function is doing the fontification
>> Builtin - Refers to jit-lock mode as that's the default for font-lock mode
>
> So we are talking about a 17% slow-down?  Not bad.

Yes, but to be fair, this does not include the initial parsing by
Semantic/Wisent/Bovine. This is usually done for the whole buffer when
Emacs is idle, and not during scrolling.

-David

Re: Semantic font-lock for Java

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Anand Tamariya <atamariya@gmail.com>,  emacs-devel@gnu.org
> Date: Fri, 24 Jan 2020 18:00:02 +0100
> 
> > So we are talking about a 17% slow-down?  Not bad.
> 
> Yes, but to be fair, this does not include the initial parsing by
> Semantic/Wisent/Bovine. This is usually done for the whole buffer when
> Emacs is idle, and not during scrolling.

So what happens when the file is visited and its first window-full is
displayed?

Re: Semantic font-lock for Java

[David Engster]

>> From: David Engster <deng@randomsample.de>
>> Cc: Anand Tamariya <atamariya@gmail.com>,  emacs-devel@gnu.org
>> Date: Fri, 24 Jan 2020 18:00:02 +0100
>> 
>> > So we are talking about a 17% slow-down?  Not bad.
>> 
>> Yes, but to be fair, this does not include the initial parsing by
>> Semantic/Wisent/Bovine. This is usually done for the whole buffer when
>> Emacs is idle, and not during scrolling.
>
> So what happens when the file is visited and its first window-full is
> displayed?

I can't speak for how Anand has configured his Emacs, but the default is
(was?)  that the file is displayed first and after a short idle time (1
second?)  Semantic will start parsing the current buffer. If you use
Semantic's "decoration mode", overlays will be put on the parsed
structures. So my guess is the scroll benchmark is simply 17% slower
because there are more overlays in the buffer, not because of actual
parsing.

The parsing itself is pretty slow, unsurprsingly, since it is all in
ELisp. Eric has invested a ton of time to make it bearable, mostly
through aggressive caching and by making the parser as "lazy" as
possible (only parse what is currently needed and do the rest when Emacs
is idle).

That being said, I of course welcome Anand's work which shows that we
already have a framework to store and query ASTs in Emacs. If we want to
incorporate something like Treesitter, one should at least take a look
at it.

-David

Re: Semantic font-lock for Java

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

> So my guess is the scroll benchmark is simply 17% slower
>
>> because there are more overlays in the buffer, not because of actual
> parsing.
>
Perfect explanation David. Though I'll add that parsing is not terribly
slow. It's usable. Here's a demo video showcasing the same
https://youtu.be/4UcPIdTJLOM .

[-- Attachment #2: Type: text/html, Size: 842 bytes --]

Re: Semantic font-lock for Java

[David Engster]

>     So my guess is the scroll benchmark is simply 17% slower
>    
>     because there are more overlays in the buffer, not because of
>     actual
>     parsing.
>
> Perfect explanation David. Though I'll add that parsing is not
> terribly slow. It's usable. Here's a demo video showcasing the same 
> https://youtu.be/4UcPIdTJLOM .

Note that Semantic will cache the AST between sessions, so if you've
loaded that file before, it will just load the cache.

The time needed for parsing varies wildly between languages. The Java
parser is one of the faster ones, because it's an LALR parser, which is
more efficient than LL (which Semantic uses for C and C++). It is also
very old, and at that time (around Java 1.4 or so) the language was
still very easy to parse (it doesn't support generics, for instance).

-David

Re: Semantic font-lock for Java

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 28 Jan 2020 18:21:09 +0100
> 
> > Perfect explanation David. Though I'll add that parsing is not
> > terribly slow. It's usable. Here's a demo video showcasing the same 
> > https://youtu.be/4UcPIdTJLOM .
> 
> Note that Semantic will cache the AST between sessions, so if you've
> loaded that file before, it will just load the cache.
> 
> The time needed for parsing varies wildly between languages. The Java
> parser is one of the faster ones, because it's an LALR parser, which is
> more efficient than LL (which Semantic uses for C and C++). It is also
> very old, and at that time (around Java 1.4 or so) the language was
> still very easy to parse (it doesn't support generics, for instance).

Thanks.  I'd be interested to see the times of showing the first
windowful of a Java file visited for the first time, and then the same
file visited again, first unchanged, and then with some small changes
in the first windowful.  Comparisons of these times with the default
font-lock would be useful for assessing the feature.  (If the same can
be done for C/C++, that's be better yet.)

Re: Semantic font-lock for Java

[Alan Mackenzie]

Hello, Anand.

On Thu, Jan 23, 2020 at 11:06:57 +0530, Anand Tamariya wrote:

[ .... ]

> BTW I should admit that it is WIP. Right now, my focus is only on accuracy.
> I have made no attempts on optimization. So consider the results
> accordingly.

Just as a matter of interest, CC Mode has an extensive test suite for
indentation and fontification.  It is available in directory /tests in
the mercurial repository which can be downloaded as detailed on
http://cc-mode.sourceforge.net/hgaccess.php.  You might find the parts
dealing with Java and fontification interesting or useful.  There are
instructions on use near the beginning of the file 000tests.el

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Semantic font-lock for Java

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

>
> > https://youtu.be/4UcPIdTJLOM .
>
> Note that Semantic will cache the AST between sessions, so if you've
> loaded that file before, it will just load the cache.
>
First use parse/build delay is an inescapable and acceptable  situation in
Java world. Even popular IDEs like Eclipse do that.
While Emacs will only parse current and relevant files, Eclipse insists on
"building" the complete project.

It is also
> very old, and at that time (around Java 1.4 or so) the language was
> still very easy to parse (it doesn't support generics, for instance).
>
Additional info - I've added support for annotations and generics. It's
shown in the demo https://youtu.be/4UcPIdTJLOM?t=50 .
Only Enum is pending.

[-- Attachment #2: Type: text/html, Size: 1265 bytes --]

Re: Semantic font-lock for Java

[Anand Tamariya]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

> Just as a matter of interest, CC Mode has an extensive test suite for
> indentation and fontification.  It is available in directory /tests in
> the mercurial repository which can be downloaded as detailed on
> http://cc-mode.sourceforge.net/hgaccess.php.  You might find the parts
> dealing with Java and fontification interesting or useful.  There are
> instructions on use near the beginning of the file 000tests.el
>
> That looks like a comprehensive suite. Thanks.

[-- Attachment #2: Type: text/html, Size: 793 bytes --]

Re: Semantic font-lock for Java

[Eric Ludlam]

On 1/29/20 12:14 AM, Anand Tamariya wrote:
>      > https://youtu.be/4UcPIdTJLOM .
> 
>     Note that Semantic will cache the AST between sessions, so if you've
>     loaded that file before, it will just load the cache.
> 
> First use parse/build delay is an inescapable and acceptable  situation 
> in Java world. Even popular IDEs like Eclipse do that.
> While Emacs will only parse current and relevant files, Eclipse insists 
> on "building" the complete project.
> 
>     It is also
>     very old, and at that time (around Java 1.4 or so) the language was
>     still very easy to parse (it doesn't support generics, for instance).
> 
> Additional info - I've added support for annotations and generics. It's 
> shown in the demo https://youtu.be/4UcPIdTJLOM?t=50 .
> Only Enum is pending.
> 

This is all very cool.  Supporting font-lock had been often requested, 
seemed possible, but I never got to it.  It's great to see the demo and 
that the tools were able to support it.

Thanks for sharing.
Eric