State of the overlay branch (again)?

State of the overlay branch (again)?

[Sebastian Sturm]

Hi all,

I hope I'm not getting on anybody's nerves by repeatedly asking the same 
question, but a while ago I was asking about the state of the Emacs 
overlay branch 
(https://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00565.html) 
because overlays are heavily used by some implementations of semantic 
highlighting (see emacs-ccls, lsp-mode).

Somehow I had thought that the noverlay branch had long since been 
merged with master but when I recently tried to find evidence for that 
merge, I couldn't find any.

So, is the new overlay functionality part of Emacs master (heavily 
renamed perhaps), or if not, is the merge still planned? If so, is there 
anything I could do to help speed up the merge process (as of yet I 
don't have any experience with the overlay mechanism or Emacs' C 
internals though)

thanks,
Sebastian

Re: State of the overlay branch (again)?

[Eli Zaretskii]

> From: Sebastian Sturm <s.sturm@arkona-technologies.de>
> Date: Wed, 25 Sep 2019 09:59:02 +0200
> 
> Somehow I had thought that the noverlay branch had long since been 
> merged with master but when I recently tried to find evidence for that 
> merge, I couldn't find any.
> 
> So, is the new overlay functionality part of Emacs master (heavily 
> renamed perhaps)

No, it wasn't merged.

> or if not, is the merge still planned?

We still hope to merge it at some point.

> If so, is there anything I could do to help speed up the merge
> process (as of yet I don't have any experience with the overlay
> mechanism or Emacs' C internals though)

AFAIK, the work on the branch is not finished, and it is not yet ready
for being merged.  So the problem is not the merging itself.

At this point, I also doubt I'd agree to merge it before the emacs-27
branch is cut, as we already have too many potentially destabilizing
changes on master, and adding one more will run the risk of making the
Emacs 27 pretest even longer than I fear it will be.

Re: State of the overlay branch (again)?

[Sebastian Sturm]

I see, thanks for the quick response!

> No, it wasn't merged.
> 
>> or if not, is the merge still planned?
> 
> We still hope to merge it at some point.
> 
>> If so, is there anything I could do to help speed up the merge
>> process (as of yet I don't have any experience with the overlay
>> mechanism or Emacs' C internals though)
> 
> AFAIK, the work on the branch is not finished, and it is not yet ready
> for being merged.  So the problem is not the merging itself.
> 
> At this point, I also doubt I'd agree to merge it before the emacs-27
> branch is cut, as we already have too many potentially destabilizing
> changes on master, and adding one more will run the risk of making the
> Emacs 27 pretest even longer than I fear it will be.

Re: State of the overlay branch (again)?

[Stefan Monnier]

> (https://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00565.html) 
> because overlays are heavily used by some implementations of semantic
>  highlighting (see emacs-ccls, lsp-mode).
[...]
> If so, is there anything I could do to help speed up the
> merge process?

I think benchmarks comparing the noverlay branch with master
(especially on those problematic cases) could do wonders to motivate
someone to finish the development&merge (assuming the benchmarks show
that it improves performance significantly, of course).


        Stefan

Re: State of the overlay branch (again)?

[Sebastian Sturm]

sure. Opening one of my somewhat larger C++ source files (~2300 lines, 
which produces close to 4000 semantic highlighting overlays), I 
repeatedly created new lines using Spacemacs's default 'o' keybinding (I 
guess vanilla Emacs wouldn't behave differently in this respect though I 
didn't try). Doing that at different positions within the buffer (while 
keeping those positions roughly identical from run to run, though both 
my personal perception and the resulting timing data suggest that 
position doesn't matter), elp-instrument-function told me that a recent 
build of emacs 27 (git commit #52172d23401) takes close to half a second 
to open a single newline (which matches what I'm seeing on screen during 
editing):

           Function Name   Call Count Elapsed Time  Average Time
emacs-27: evil-open-below 7          2.877079901   0.4110114144

Removing all overlays using (remove-overlays) while leaving 
semantic-highlighting enabled (i.e., any new highlights sent by clangd 
would still be processed and converted into overlays during editing) 
resulted in the following profile:

emacs-27 [overlays cleared]: evil-open-below  7 0.0555971120  0.0079424445

By comparison, this is what I get using the noverlay branch (same file, 
same number of overlays, 3774):

emacs-noverlay: evil-open-below  7 0.058601084 0.0083715834

and, for comparison, with all overlays removed:

emacs-noverlay: [overlays cleared]: evil-open-below  9 0.03571364 
0.0039681822

On 9/25/19 2:32 PM, Stefan Monnier wrote:
>> (https://lists.gnu.org/archive/html/emacs-devel/2018-03/msg00565.html)
>> because overlays are heavily used by some implementations of semantic
>>   highlighting (see emacs-ccls, lsp-mode).
> [...]
>> If so, is there anything I could do to help speed up the
>> merge process?
> 
> I think benchmarks comparing the noverlay branch with master
> (especially on those problematic cases) could do wonders to motivate
> someone to finish the development&merge (assuming the benchmarks show
> that it improves performance significantly, of course).
> 
> 
>          Stefan
>