[NS] Emacs 28 (IOSurface) renders much slower than Emacs 27 on macOS Big Sur

[NS] Emacs 28 (IOSurface) renders much slower than Emacs 27 on macOS Big Sur

[Yiming Chen]

[-- Attachment #1: Type: text/html, Size: 8511 bytes --]

[-- Attachment #2: Type: text/plain, Size: 3442 bytes --]

Hi there!
I know that we've switched to a IOSurface based rendering for nsterm in
master branch.
Now master renders more precisely than emacs-27 (i.e. less flickers or
blank screens.)
But I feel that its rendering performance is much worse due to the new
buffering mechanism backed by IOSurface.


Several cases that I can feel this worse performance:
1. moving a maximized emacs frame from a larger screen to a smaller
   screen (I use hammerspoon to do that)
   I can see the emacs-28 frame shrink to the smaller screen size, then
   move to the smaller screen
2. scrolling
   I set `scroll-conservatively' to 101 in order to achieve the
   smooth-scrolling effect.
   When I scroll down a large text buffer constantly (hold `j' in
   evil-mode or `C-n' in emacs-mode),
   emacs-28 hangs more times than emacs-27 (emacs-27 scrolls very
   smoothly except it would flicker from time to time)

   See also
   <https://www.reddit.com/r/emacs/comments/mhdjxb/macos_emacs_with_metal_support/guhbck4/>
   for a previous discussion.
3. child-frame
   child-frame like company-box appears more slowly in emacs-28 than
   emacs-27

Sorry I haven't found a way to measure the rendering performance except
the `scroll-up-benchmark' script from Alan Third in the reddit comment
mentioned above.


I think this poor rendering performance obscures the improvements
brought by native-compile.
So I want to fix this performance issue and make Emacs runs on macOS as
fast as on Linux.

I've spent the last several days trying to understand how nsterm works
and if we can use a more efficient rendering API.
But I failed completely, mostly because I have no Cocoa development
experience before.
Here are the 3 steps I want to take to improve the rendering
performance:
1. Try to restore the nsterm behavior from Emacs27 (which is much
   simpler than master IMHO), and start again from there

   I've reverted the code and made it compile (see
   <https://github.com/emacs-mirror/emacs/compare/master...dsdshcym:revert-to-emacs-27-nsterm>)
   But Emacs failed to start and raised an error `Font ‘Menlo’ is not
   defined'

2. Use layer-based view for rendering

   I found that macvim also faced the similar issue when macOS 10.14
   came out (<https://github.com/macvim-dev/macvim/issues/751>).
   They used NSImage to implement a double rendering solution
   (<https://github.com/macvim-dev/macvim/pull/757/files>), which was
   also slower than before.
   Then they switched from NSImage to layer-based view in
   <https://github.com/macvim-dev/macvim/commit/dba6293677e0633917e3054cfddec1293e5ab3fb>.
   So I wonder if we can do the same.

3. Switch to CALayer (Core Animation) to use GPU (Metal) rendering for
   even better performance than before

   If 2 (layer-based view) is possible, then I hope we can leverage
   CALayer to make nsterm renders even faster.


I was so desperate about this issue, so I also tried other window
systems on macOS:
1. emacs-pgtk (<https://github.com/masm11/emacs/issues/95>):
   technically emacs-pgtk should work on macOS with gtk3, but the
   performance was even worse
2. emacs-ng + webrender
   (<https://github.com/emacs-ng/emacs-ng/issues/207>)
   webrender hasn't been working on macOS neither

If anyone can point a direction for me so I can investigate more, that
would be the best.
Thanks in advance!

Cheers,
Yiming

Re: [NS] Emacs 28 (IOSurface) renders much slower than Emacs 27 on macOS Big Sur

[Alan Third]

On Sun, May 02, 2021 at 09:40:03PM +0800, Yiming Chen wrote:
> Hi there!
> I know that we've switched to a IOSurface based rendering for nsterm in
> master branch.
> Now master renders more precisely than emacs-27 (i.e. less flickers or
> blank screens.)
> But I feel that its rendering performance is much worse due to the new
> buffering mechanism backed by IOSurface.

Yup. That's the trade-off.

> I think this poor rendering performance obscures the improvements
> brought by native-compile.
> So I want to fix this performance issue and make Emacs runs on macOS as
> fast as on Linux.

I've spent over a year on-and-off on this. Good luck.

> 1. Try to restore the nsterm behavior from Emacs27 (which is much
>    simpler than master IMHO), and start again from there
> 
>    I've reverted the code and made it compile (see
>    <https://github.com/emacs-mirror/emacs/compare/master...dsdshcym:revert-to-emacs-27-nsterm>)
>    But Emacs failed to start and raised an error `Font ‘Menlo’ is not
>    defined'

That error must be something unrelated, as the drawing buffer changes
didn't touch the font code... I've got no suggestions on where to
start, though. My knowledge of font handling is non-existent.

Actually, a quick look shows that you've enabled the nsfont backend
which just doesn't work in macOS, so I'd bet that's something to
investigate.

I notice quite a few other problems... If I'm being honest, master has
diverged so much from emacs-27 that if I were to do this I'd probably
do it manually rather than try reverting commits.

And I disagree that master is more complex than emacs-27. The endless
little work-arounds to try and get the emacs-27 rendering working made
it a much more brittle setup.

> 2. Use layer-based view for rendering
> 
>    I found that macvim also faced the similar issue when macOS 10.14
>    came out (<https://github.com/macvim-dev/macvim/issues/751>).
>    They used NSImage to implement a double rendering solution
>    (<https://github.com/macvim-dev/macvim/pull/757/files>), which was
>    also slower than before.
>    Then they switched from NSImage to layer-based view in
>    <https://github.com/macvim-dev/macvim/commit/dba6293677e0633917e3054cfddec1293e5ab3fb>.
>    So I wonder if we can do the same.
>
> 3. Switch to CALayer (Core Animation) to use GPU (Metal) rendering for
>    even better performance than before
> 
>    If 2 (layer-based view) is possible, then I hope we can leverage
>    CALayer to make nsterm renders even faster.

So... yes. The IOSurface system is already using layer-based drawing.
Have a look at updateLayer, for example.

I'm not sure that CALayer really gives you anything in terms of
performance for this sort of thing. CoreAnimation is all about moving
different layers about efficiently, afaict, which isn't something we
need to do in Emacs.

And bear in mind that in order to send an image to the Metal renderer
you still need to transfer it to the GPU, and Apple recommend using
IOSurface as it's the most efficient way to do that.

AND Yamamoto Mitsuharu disabled Metal rendering in the Mac port as he
found the simple IOSurface based layer backing was faster.

AND the Chromium developers decided to go with the simple IOSurface
based layer backing for performance reasons too.

One place that a big improvement could be found is in disabling double
buffering and only drawing to one IOSurface, as you don't need to copy
the contents of the different IOSurfaces to each other. But I found
that produced unacceptable tearing effects.

But all that said, it could be that copying viewWillDraw from master
to emacs-27 could solve some of the flickering. In my experimentation
I don't believe it does, but I've never been very good at inducing the
"flickering".

-- 
Alan Third

Re: [NS] Emacs 28 (IOSurface) renders much slower than Emacs 27 on macOS Big Sur

[Yiming Chen]

Hi Alan,

Thanks for your reply and sorry for my late response.

I've been using 
https://github.com/emacs-mirror/emacs/commit/6f1e1ba633 for a few 
days,
and switched to 
https://github.com/emacs-mirror/emacs/commit/784ec4b752 today.

I feel the drawing performance improves a lot in 
scratch/ns/surface-stuff branch,
and I can finally see the improvements brought by native-compile.
(since drawing is no longer a bottle-neck, maybe?)

Thanks a lot for your work!

Best,
Yiming


Alan Third <alan@idiocy.org> writes:

> On Sun, May 02, 2021 at 09:40:03PM +0800, Yiming Chen wrote:
>> Hi there!
>> I know that we've switched to a IOSurface based rendering for 
>> nsterm in
>> master branch.
>> Now master renders more precisely than emacs-27 (i.e. less 
>> flickers or
>> blank screens.)
>> But I feel that its rendering performance is much worse due to 
>> the new
>> buffering mechanism backed by IOSurface.
>
> Yup. That's the trade-off.
>
>> I think this poor rendering performance obscures the 
>> improvements
>> brought by native-compile.
>> So I want to fix this performance issue and make Emacs runs on 
>> macOS as
>> fast as on Linux.
>
> I've spent over a year on-and-off on this. Good luck.
>
>> 1. Try to restore the nsterm behavior from Emacs27 (which is 
>> much
>>    simpler than master IMHO), and start again from there
>>
>>    I've reverted the code and made it compile (see
>>    <https://github.com/emacs-mirror/emacs/compare/master...dsdshcym:revert-to-emacs-27-nsterm>)
>>    But Emacs failed to start and raised an error `Font ‘Menlo’ 
>>    is not
>>    defined'
>
> That error must be something unrelated, as the drawing buffer 
> changes
> didn't touch the font code... I've got no suggestions on where 
> to
> start, though. My knowledge of font handling is non-existent.
>
> Actually, a quick look shows that you've enabled the nsfont 
> backend
> which just doesn't work in macOS, so I'd bet that's something to
> investigate.
>
> I notice quite a few other problems... If I'm being honest, 
> master has
> diverged so much from emacs-27 that if I were to do this I'd 
> probably
> do it manually rather than try reverting commits.
>
> And I disagree that master is more complex than emacs-27. The 
> endless
> little work-arounds to try and get the emacs-27 rendering 
> working made
> it a much more brittle setup.
>
>> 2. Use layer-based view for rendering
>>
>>    I found that macvim also faced the similar issue when macOS 
>>    10.14
>>    came out 
>>    (<https://github.com/macvim-dev/macvim/issues/751>).
>>    They used NSImage to implement a double rendering solution
>>    (<https://github.com/macvim-dev/macvim/pull/757/files>), 
>>    which was
>>    also slower than before.
>>    Then they switched from NSImage to layer-based view in
>>    <https://github.com/macvim-dev/macvim/commit/dba6293677e0633917e3054cfddec1293e5ab3fb>.
>>    So I wonder if we can do the same.
>>
>> 3. Switch to CALayer (Core Animation) to use GPU (Metal) 
>> rendering for
>>    even better performance than before
>>
>>    If 2 (layer-based view) is possible, then I hope we can 
>>    leverage
>>    CALayer to make nsterm renders even faster.
>
> So... yes. The IOSurface system is already using layer-based 
> drawing.
> Have a look at updateLayer, for example.
>
> I'm not sure that CALayer really gives you anything in terms of
> performance for this sort of thing. CoreAnimation is all about 
> moving
> different layers about efficiently, afaict, which isn't 
> something we
> need to do in Emacs.
>
> And bear in mind that in order to send an image to the Metal 
> renderer
> you still need to transfer it to the GPU, and Apple recommend 
> using
> IOSurface as it's the most efficient way to do that.
>
> AND Yamamoto Mitsuharu disabled Metal rendering in the Mac port 
> as he
> found the simple IOSurface based layer backing was faster.
>
> AND the Chromium developers decided to go with the simple 
> IOSurface
> based layer backing for performance reasons too.
>
> One place that a big improvement could be found is in disabling 
> double
> buffering and only drawing to one IOSurface, as you don't need 
> to copy
> the contents of the different IOSurfaces to each other. But I 
> found
> that produced unacceptable tearing effects.
>
> But all that said, it could be that copying viewWillDraw from 
> master
> to emacs-27 could solve some of the flickering. In my 
> experimentation
> I don't believe it does, but I've never been very good at 
> inducing the
> "flickering".

Re: [NS] Emacs 28 (IOSurface) renders much slower than Emacs 27 on macOS Big Sur

[Alan Third]

On Tue, Jun 01, 2021 at 10:29:01AM +0800, Yiming Chen wrote:
> Hi Alan,
> 
> Thanks for your reply and sorry for my late response.
> 
> I've been using https://github.com/emacs-mirror/emacs/commit/6f1e1ba633 for
> a few days,
> and switched to https://github.com/emacs-mirror/emacs/commit/784ec4b752
> today.
> 
> I feel the drawing performance improves a lot in scratch/ns/surface-stuff
> branch,
> and I can finally see the improvements brought by native-compile.
> (since drawing is no longer a bottle-neck, maybe?)

Thanks, I'm glad it's made a difference.

I've pushed the main bit of the surface-stuff branch into master
(246e107d73) and the last commit I had in there is currently sitting
in a local branch I'm dedicating to a bit of a refactor of some of the
nsterm code.
-- 
Alan Third

Re: [NS] Emacs 28 (IOSurface) renders much slower than Emacs 27 on macOS Big Sur

[Yiming Chen]

> the last commit I had in there is currently sitting in a local 
> branch
> I'm dedicating to a bit of a refactor of some of the nsterm 
> code.

Great! Can't wait to try that out again :)

Alan Third <alan@idiocy.org> writes:

> On Tue, Jun 01, 2021 at 10:29:01AM +0800, Yiming Chen wrote:
>> Hi Alan,
>>
>> Thanks for your reply and sorry for my late response.
>>
>> I've been using 
>> https://github.com/emacs-mirror/emacs/commit/6f1e1ba633 for
>> a few days,
>> and switched to 
>> https://github.com/emacs-mirror/emacs/commit/784ec4b752
>> today.
>>
>> I feel the drawing performance improves a lot in 
>> scratch/ns/surface-stuff
>> branch,
>> and I can finally see the improvements brought by 
>> native-compile.
>> (since drawing is no longer a bottle-neck, maybe?)
>
> Thanks, I'm glad it's made a difference.
>
> I've pushed the main bit of the surface-stuff branch into master
> (246e107d73) and the last commit I had in there is currently 
> sitting
> in a local branch I'm dedicating to a bit of a refactor of some 
> of the
> nsterm code.