emacs rendering comparisson between emacs23 and emacs26.3

emacs rendering comparisson between emacs23 and emacs26.3

[rrandresf]

Hi.

On my TODO list (kind of worn hole). I have several notes.

One of them is comparing the rendering on emacs23 and emacs26.3. Today
Alan Mackenzie share an elisp Snippet:

--8<---------------cut here---------------start------------->8---
(defmacro time-it (&rest forms)
  "Time the running of a sequence of forms using `float-time'.
Call like this: \"M-: (time-it (foo ...) (bar ...) ...)\"."
  `(let ((start (float-time)))
    ,@forms
    (- (float-time) start)))

(defun time-scroll (&optional arg)
  (interactive "P")
  (message "%s"
           (time-it
            (condition-case nil
                (while t
                  (if arg (scroll-down) (scroll-up))
                  (sit-for 0))
              (error nil)))))
--8<---------------cut here---------------end--------------->8---

Alan suggested this to Andrea Corallo (BTW nice work with your branch
Andrea. and I still remember the discussion about numbering and source
code Alan)
--8<---------------cut here---------------start------------->8---
, visit .../emacs/src/xdisp.c, and do M-: (time-scroll)
--8<---------------cut here---------------end--------------->8---

When I saw this. The first thing that come to my mind was (it is
useful on one of the items of my TODO'S). So I did It on xdisp.c from emacs26-3 source
code.

Applying it on emacs23 and on emacs26 (on an opi+2e machine with 1 Ghz
processor)
This is the result:
--8<---------------cut here---------------start------------->8---
emacs-23
162.7052059173584
emacs-26.3
281.4721345901489
--8<---------------cut here---------------end--------------->8---


The difference is very notorious. I realized it from a long time on
another ARM machine with 480 Mhz (which i still use aka my phone). But
I have not got the time and tools for letting you guys know about it. So
thanks Alan for sharing it.

Teoretically. What would be the cause of this difference on rendering
between versions?

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> --8<---------------cut here---------------start------------->8---
> emacs-23
> 162.7052059173584
> emacs-26.3
> 281.4721345901489
> --8<---------------cut here---------------end--------------->8---
[...]
> Teoretically. What would be the cause of this difference on rendering
> between versions?

It's hard to tell without digging deeper, it could be in part due to
changes to the actual redisplay code or to the rest of the C code, but
I suspect it's mostly due to improvements (featurewise) in CC-mode.

I just compared this test's performance using `c-mode` vs using
`sm-c-mode` (a comparatively very bare-bones major mode limited to
"normal" C (no support for any C++ feature nor for old K&R-style C, nor
for very modern C)), and the speed difference was even larger.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: rrandresf@gmail.com
> Date: Sat, 21 Mar 2020 22:19:27 +0000
> 
> Teoretically. What would be the cause of this difference on rendering
> between versions?

Emacs 23 didn't support bidirectional display.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello!

On Sat, Mar 21, 2020 at 22:19:27 +0000, rrandresf@gmail.com wrote:
> Hi.

> On my TODO list (kind of worn hole). I have several notes.

> One of them is comparing the rendering on emacs23 and emacs26.3. Today
> Alan Mackenzie share an elisp Snippet:

[ .... ]

> When I saw this. The first thing that come to my mind was (it is
> useful on one of the items of my TODO'S). So I did It on xdisp.c from
> emacs26-3 source code.

> Applying it on emacs23 and on emacs26 (on an opi+2e machine with 1 Ghz
> processor)
> This is the result:
> --8<---------------cut here---------------start------------->8---
> emacs-23
> 162.7052059173584
> emacs-26.3
> 281.4721345901489
> --8<---------------cut here---------------end--------------->8---


> The difference is very notorious. I realized it from a long time on
> another ARM machine with 480 Mhz (which i still use aka my phone). But
> I have not got the time and tools for letting you guys know about it. So
> thanks Alan for sharing it.

> Teoretically. What would be the cause of this difference on rendering
> between versions?

There have been lots of bug fixes and new features in CC Mode.  Many of
these have needed extra processing.

Still, an increase of ~75% in processing time over the best part of a
decade isn't that bad.  Surely?  Certainly not when compared with the
increase in available computing power.

One workaround is to reduce the fontification level with
font-lock-maximum-decoration.  A value of 3 is full fontification, 2 is
quite a bit less.

> Best Regards

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Amin Bandali]

[-- Attachment #1: Type: text/plain, Size: 726 bytes --]

Hello,

I may be in the minority, but...

Alan Mackenzie <acm@muc.de> writes:

[...]
>
> Still, an increase of ~75% in processing time over the best part of a
> decade isn't that bad.  Surely?  Certainly not when compared with the
> increase in available computing power.
>
[...]

Just as a data point, my main machine is a ThinkPad released in 2008
with a Core 2 Duo processor, which is not nearly as fast as the newer
Core i5/i7 processors found in more recent machines since.

That is not to say I don't enjoy and appreciate useful new features in
GNU Emacs, but rather that I think we should keep a close eye on the
performance penalties accumulated over time and (continue) trying to
minimize them as much as we can. :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Amin Bandali <bandali@gnu.org>
> Date: Sun, 22 Mar 2020 11:08:25 -0400
> 
> That is not to say I don't enjoy and appreciate useful new features in
> GNU Emacs, but rather that I think we should keep a close eye on the
> performance penalties accumulated over time and (continue) trying to
> minimize them as much as we can. :-)

Indeed, we should, and we do.  Several slowdowns in CC mode were
reported during the recent years, and at least some of them were
fixed.  Feel free to report more, preferably with profile data to show
the bottlenecks and test cases to use in making CC mode faster.

Thanks.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Still, an increase of ~75% in processing time over the best part of a
  > > decade isn't that bad.  Surely?  Certainly not when compared with the
  > > increase in available computing power.

There has been no increase in computing power in the Free World.
I expect that Amin chooses these older machines

  > Just as a data point, my main machine is a ThinkPad released in 2008
  > with a Core 2 Duo processor, which is not nearly as fast as the newer
  > Core i5/i7 processors found in more recent machines since.

for the same reason I choose them: the FSF recommends them, because newer
processors have back doors.  Intel processors newer than that contain
the "Management Engine" back door and can't start up without it.

So please, everyone, take care to maintain good performance on these
older machines.

Is it possible, to restore most of the lost peformance by deactivating
some new features?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Richard.

On Sun, Mar 22, 2020 at 22:51:39 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > > Still, an increase of ~75% in processing time over the best part of a
>   > > decade isn't that bad.  Surely?  Certainly not when compared with the
>   > > increase in available computing power.

> There has been no increase in computing power in the Free World.
> I expect that Amin chooses these older machines

>   > Just as a data point, my main machine is a ThinkPad released in 2008
>   > with a Core 2 Duo processor, which is not nearly as fast as the newer
>   > Core i5/i7 processors found in more recent machines since.

> for the same reason I choose them: the FSF recommends them, because newer
> processors have back doors.  Intel processors newer than that contain
> the "Management Engine" back door and can't start up without it.

That's a good point.  Presumably processors without these back doors are
difficult to come by nowadays.

> So please, everyone, take care to maintain good performance on these
> older machines.

> Is it possible, to restore most of the lost peformance by deactivating
> some new features?

The only way at the moment (for CC Mode) is to change
font-lock-maximum-decoration from t to 2.

Most of the increased processing comes from fixing bugs rather than
adding new features.  Keeping up with, in particular, the latest C++
standard (which seems to appear every three years) is a challenge, as
they steadily make shallow syntactic analysis more and more difficult.

> -- 
> Dr Richard Stallman
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> Most of the increased processing comes from fixing bugs rather than
> adding new features.  Keeping up with, in particular, the latest C++
> standard (which seems to appear every three years) is a challenge, as
> they steadily make shallow syntactic analysis more and more difficult.

FWIW: Not just *shallow*!


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[rrandresf]

Hi Alan. Hi. Richard.

>> contain the "Management Engine" back door and can't start up
>> without it.

Alan> That's a good point.  Presumably processors without these back
Alan> doors are difficult to come by nowadays.

>> Is it possible, to restore most of the lost peformance by
>> deactivating some new features?

Alan> The only way at the moment (for CC Mode) is to change
Alan> font-lock-maximum-decoration from t to 2.

BTW. On my comparissons that variable was set no nil
as Doc says:
--8<---------------cut here---------------start------------->8---
If nil, use the default decoration (typically the minimum available).
--8<---------------cut here---------------end--------------->8---
Also xdisp.c. Just load c-mode not c++-mode

Alan> Most of the increased processing comes from fixing bugs rather
Alan> than adding new features.  Keeping up with, in particular, the
Alan> latest C++ standard (which seems to appear every three years) is
Alan> a challenge, as they steadily make shallow syntactic analysis
Alan> more and more difficult.

I have complete the same test on all Emacsen (Sxemacs-git and Xemacs
21.5). Nothing beats GNU/Emacs23 on rendering.

As Eli pointed out on another email on this same thread. It could be
BIDI. I would need to check on configure If it is possible to deactivate
BIDI support and see what difference it Does.

With the help of Michael Albinus I have update the emacs23 builtin tramp
version to the latest one that supports GNU/Emacs23 (wellcome multihops and
adb). I think perhaps in the future I could upgrade org-mode to the last
version that supports GNU/Emacs23 (¿?). And I am going to stay with that setup on
low power machines. BTW. Not having magit on GNU/Emacs23 have made me
discover the niceties about vc-dir (sometimes it hangs on GNU/Emacs23) not
on 26.3.

Also on the same subject. There are several packages that require
GNU/Emacs24 as minimum but they DO work fine with GNU/Emacs23. Yes I Do use
GNU/Emacs23 with package.el. and GNU/Emacs23 coexist with GNU/Emacs26.3

Also GNU/Emacs23. with all the patches from Gentoo (thanks Ulrich
Mueller). compiles fine with latest and greatest tooling {glibc 2.30} on x86 (32 and 64 bits). But does not compile on
ARM (a low power machine like orange pi). It stays on a loop. I have no have the time for attaching gdb to
it. But I plan on  doing it sometime in the future. So for ARM architecture just
CANNOT_DUMP works with a lot of tweaking for compilation (Do not try it).

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: rrandresf@gmail.com
> Date: Thu, 26 Mar 2020 20:38:22 +0000
> Cc: Richard Stallman <rms@gnu.org>, emacs-devel@gnu.org
> 
> As Eli pointed out on another email on this same thread. It could be
> BIDI. I would need to check on configure If it is possible to deactivate
> BIDI support and see what difference it Does.

It's not possible.  Bidirectional editing support in Emacs is
unconditional, it isn't an optional feature.

What I would suggest is to compare with Emacs 24.1, which already
supported bidi.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi Eli.

Eli> It's not possible.  Bidirectional editing support in Emacs is
Eli> unconditional, it isn't an optional feature.
Mmmm.

Eli> What I would suggest is to compare with Emacs 24.1, which already
Eli> supported bidi.

Well. Based on past experience.
I am going to try to compile emacs24.1 on this ARM machine. If not
possible. I am going to move to another machine (compaq n600c an x86
with Ghz processor and 256 Mb memory). On of this two options should work.

Then I am going to try the test with default cc-mode (the one that comes
with 24.1). And after that I
am going to plug cc-mode from 23.4 on 24.1. Just for taking out cc-mode
out of equation and concentrating just on bidi.

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

To tell the people with the strongest commitment to freedom to use an
old version of Emacs is to choose failure.  We need to re-optimize the
Emacs master so it gives good performance on those machines.

Suggesting that we disable certain features is ok.  Perhaps there
could be a command to select several of these modes together.

  > The only way at the moment (for CC Mode) is to change
  > font-lock-maximum-decoration from t to 2.

That is ok as part of the solution.

Regarding C++ indentation, I have two suggestions.

* We could support a faster indentation mode for C code
and for C++ files that use a limited repertoire of C++ constructs.

* Maybe we can come up with additional tricks comparable to
{-in-column-0 to optimize parsing.  Emacs users would put these things
into source files so as to get better peformance.

  > BTW. On my comparissons that variable was set no nil
  > as Doc says:
  > --8<---------------cut here---------------start------------->8---
  > If nil, use the default decoration (typically the minimum available).
  > --8<---------------cut here---------------end--------------->8---
  > Also xdisp.c. Just load c-mode not c++-mode

What were the results of the comparisons you did with this setting?
(I wasn't following this topic until yesterday.)

  > As Eli pointed out on another email on this same thread. It could be
  > BIDI. I would need to check on configure If it is possible to deactivate
  > BIDI support and see what difference it Does.

Would you please try that?  But don't limit yourself to existing
configuration options -- try changing it at the C level if you can.
If you find that this makes a big difference (a "bi di" ?), we can
arrange to turn it on and off at run time.

  > With the help of Michael Albinus I have update the emacs23 builtin tramp
  > version to the latest one that supports GNU/Emacs23

The slash is not called for here.  In "GNU Emacs", "GNU" acts as an adjectival
modifying "Emacs", so it is correct to separate them with just a space.

At least that is one problem that is easy to fix.

							(wellcome multihops and
  > adb).

Has Tramp become a lot slower?  Or is it only that the current Tramp doesn't
work on Emacs 23?

	  I think perhaps in the future I could upgrade org-mode to the last
  > version that supports GNU Emacs 23 (¿?). And I am going to stay with that setup on
  > low power machines.

Has Org mode become a lot slower?  Or is it only that the current Org
mode doesn't work on Emacs 23?


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It's not possible.  Bidirectional editing support in Emacs is
  > unconditional, it isn't an optional feature.

It isn't optional _now_ but maybe there is an opportunity to optimize
redisplay if the userssays "no BIDI".

Maybe some of the pre-bidi code could be brought back and used
when the user says there is no BIDI in the buffer.

  > What I would suggest is to compare with Emacs 24.1, which already
  > supported bidi.

It would be interesting to compare 24 and 23, to see what slowdown
there is.  And compare other subsequent versions to see what slowdown
there is.  These could give hints for what operations are worth trying
to optimize.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: rrandresf@gmail.com, acm@muc.de, emacs-devel@gnu.org
> Date: Fri, 27 Mar 2020 22:48:38 -0400
> 
>   > It's not possible.  Bidirectional editing support in Emacs is
>   > unconditional, it isn't an optional feature.
> 
> It isn't optional _now_ but maybe there is an opportunity to optimize
> redisplay if the userssays "no BIDI".
> 
> Maybe some of the pre-bidi code could be brought back and used
> when the user says there is no BIDI in the buffer.

If you find volunteers to work on such changes, sure, why not.  I
don't volunteer, mind you: the display code is complicated enough even
without that.

> It would be interesting to compare 24 and 23, to see what slowdown
> there is.

I expect an average twofold slowdown due to bidirectional support.
The OP indicates less than that (comparing Emacs 23 with Emacs 26), so
it's reasonable, IMO, and probably means the CC Mode contribution is
relatively minor.

The tricky part is testing the bidirectional part and the CC Mode part
separately, by loading the old Emacs-23 CC Mode files into later
versions of Emacs.  Maybe Alan could help by providing a Lisp file
which, when loaded, would load CC Mode from a given directory -- this
should be run before visiting any C/C++ file.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 10:09:59 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> 
> > It would be interesting to compare 24 and 23, to see what slowdown
> > there is.
> 
> I expect an average twofold slowdown due to bidirectional support.

Actually, I take that back: in benchmarks such as scroll-up-benchmark,
and in a buffer with predominantly ASCII text, the effect of
bidirectional support should be very small, perhaps 10% or 20%.

So the issue of testing old v23.x CC Mode with newer versions of Emacs
becomes more important than I originally thought.  Just as a teaser,
here's the profile I obtained by scrolling in Emacs 26.3 through the
entire xdisp.c, it seems to point to c-find-decl-spots and its
immediate callers and subroutines as the hot spots:

- command-execute                                                1446 100%
 - call-interactively                                            1446 100%
  - funcall-interactively                                        1446 100%
   - execute-extended-command                                    1446 100%
    - command-execute                                            1446 100%
     - call-interactively                                        1446 100%
      - funcall-interactively                                    1446 100%
       - scroll-up-benchmark                                     1446 100%
        - let                                                    1446 100%
         - condition-case                                        1446 100%
          - while                                                1446 100%
           - redisplay                                           1351  93%
            - redisplay_internal (C function)                    1071  74%
             - jit-lock-function                                 1064  73%
              - jit-lock-fontify-now                             1063  73%
               - jit-lock--run-functions                         1063  73%
                - run-hook-wrapped                               1063  73%
                 - #<compiled 0x2000000008356520>                1063  73%
                  - font-lock-fontify-region                     1061  73%
                   - c-font-lock-fontify-region                  1061  73%
                    - font-lock-default-fontify-region            982  67%
                     - font-lock-fontify-keywords-region          969  67%
                      - c-font-lock-declarations                  686  47%
                       - c-find-decl-spots                        674  46%
                        - #<compiled 0x20000000077e07c0>          477  32%
                         - c-get-fontification-context            253  17%
                          - c-looking-at-or-maybe-in-bracelist    203  14%
                           - c-backward-token-2                   196  13%
                            - c-backward-sws                       35   2%
                             - c-beginning-of-macro                12   0%
                                back-to-indentation                 6   0%
                          - c-backward-token-2                     16   1%
                           - c-backward-sws                         7   0%
                              c-beginning-of-current-token          1   0%
                             c-beginning-of-current-token           1   0%
                          - c-back-over-compound-identifier        15   1%
                           - c-backward-sws                         6   0%
                            - c-beginning-of-macro                  3   0%
                               back-to-indentation                  3   0%
                             c-on-identifier                        5   0%
                             c-backward-token-2                     2   0%
                          - c-backward-sws                          6   0%
                             c-beginning-of-macro                   1   0%
                         - c-forward-decl-or-cast-1               125   8%
                          - c-forward-type                         65   4%
                           - c-forward-name                        18   1%
                              c-forward-sws                         4   0%
                           - c-syntactic-content                    4   0%
                              c-forward-sws                         1   0%
                              apply                                 1   0%
                             c-add-type                             3   0%
                           - c-forward-keyword-clause               2   0%
                              c-forward-sws                         1   0%
                          - c-backward-token-2                      9   0%
                             c-backward-sws                         3   0%
                             c-beginning-of-current-token           2   0%
                          - c-directly-in-class-called-p            3   0%
                           - c-looking-at-decl-block                2   0%
                            - c-syntactic-skip-backward             2   0%
                             - c-parse-state                        2   0%
                              - c-parse-state-1                     1   0%
                               - c-parse-state-get-strategy         1   0%
                                - c-beginning-of-macro              1   0%
                                   back-to-indentation              1   0%
                              - c-beginning-of-macro                1   0%
                                 back-to-indentation                1   0%
                           - c-parse-state                          1   0%
                              c-parse-state-1                       1   0%
                          - c-forward-name                          3   0%
                             c-forward-sws                          2   0%
                            c-forward-sws                           3   0%
                            c-forward-annotation                    1   0%
                         - c-forward-label                         33   2%
                          - c-after-conditional                     8   0%
                           - c-backward-token-2                     6   0%
                            - c-backward-sws                        5   0%
                             - c-beginning-of-macro                 1   0%
                                back-to-indentation                 1   0%
                               c-beginning-of-current-token         1   0%
                          - c-syntactic-re-search-forward           7   0%
                           - c-beginning-of-macro                   4   0%
                              back-to-indentation                   1   0%
                            c-forward-sws                           4   0%
                            c-beginning-of-current-token            1   0%
                         - c-backward-sws                          12   0%
                          - c-beginning-of-macro                    2   0%
                             back-to-indentation                    1   0%
                         - c-font-lock-single-decl                 10   0%
                          - c-font-lock-declarators                 9   0%
                           - c-forward-declarator                   8   0%
                            - c-backward-sws                        3   0%
                               c-beginning-of-current-token         2   0%
                            - c-forward-name                        2   0%
                               c-forward-sws                        1   0%
                              c-syntactic-re-search-forward         1   0%
                         - c-backward-token-2                      10   0%
                            c-backward-sws                          4   0%
                            c-beginning-of-current-token            3   0%
                        - c-bs-at-toplevel-p                      141   9%
                         - c-brace-stack-at                       135   9%
                          - c-update-brace-stack                  126   8%
                           - c-syntactic-re-search-forward         87   6%
                            - c-beginning-of-macro                 22   1%
                               back-to-indentation                 15   1%
                             c-beginning-of-current-token          17   1%
                             match-string-no-properties             1   0%
                        - c-beginning-of-macro                     13   0%
                           back-to-indentation                      9   0%
                          c-forward-sws                            11   0%
                          c-backward-sws                            1   0%
                      - c-font-lock-cut-off-declarators           116   8%
                       - c-back-over-member-initializers           35   2%
                        - c-parse-state                             8   0%
                         - c-parse-state-1                          8   0%
                          - c-remove-stale-state-cache              3   0%
                           - c-beginning-of-macro                   3   0%
                              back-to-indentation                   1   0%
                          - c-append-to-state-cache                 2   0%
                           - c-beginning-of-macro                   2   0%
                              back-to-indentation                   2   0%
                          - c-parse-state-get-strategy              1   0%
                             c-beginning-of-macro                   1   0%
                        - c-just-after-func-arglist-p               7   0%
                           c-beginning-of-statement-1               7   0%
                        - c-backward-sws                            5   0%
                         - c-beginning-of-macro                     2   0%
                            back-to-indentation                     2   0%
                        - c-back-over-compound-identifier           2   0%
                           c-on-identifier                          2   0%
                       - c-syntactic-skip-backward                 30   2%
                        - c-parse-state                            16   1%
                         - c-parse-state-1                         16   1%
                          - c-append-lower-brace-pair-to-state-cache   5   0%
                             c-beginning-of-macro                   1   0%
                            c-remove-stale-state-cache              2   0%
                            c-append-to-state-cache                 2   0%
                          - c-remove-stale-state-cache-backwards    2   0%
                             c-state-balance-parens-backwards       1   0%
                           - c-state-literal-at                     1   0%
                              c-state-pp-to-literal                 1   0%
                          - c-parse-state-get-strategy              1   0%
                           - c-beginning-of-macro                   1   0%
                              #<compiled 0x2000000001a3f060>        1   0%
                          c-beginning-of-macro                      1   0%
                       - c-determine-limit                         22   1%
                          c-state-semi-pp-to-literal                8   0%
                        - c-determine-limit                         3   0%
                           c-state-semi-pp-to-literal               2   0%
                       - c-parse-state                              6   0%
                        - c-parse-state-1                           6   0%
                           c-append-lower-brace-pair-to-state-cache 6   0%
                       - c-forward-decl-or-cast-1                   5   0%
                          c-forward-annotation                      1   0%
                        - c-forward-type                            1   0%
                           c-forward-name                           1   0%
                       - c-get-fontification-context                3   0%
                        - c-looking-at-or-maybe-in-bracelist        3   0%
                           c-backward-token-2                       3   0%
                       - c-at-toplevel-p                            3   0%
                        - c-search-uplist-for-classkey              3   0%
                         - c-looking-at-decl-block                  3   0%
                          - c-syntactic-skip-backward               3   0%
                           - c-parse-state                          2   0%
                            - c-parse-state-1                       1   0%
                               c-remove-stale-state-cache-backwards 1   0%
                              c-beginning-of-macro                  1   0%
                         c-forward-sws                              2   0%
                       - c-font-lock-single-decl                    1   0%
                          c-font-lock-declarators                   1   0%
                      - c-font-lock-enclosing-decls                54   3%
                       - c-determine-limit                         25   1%
                        - c-state-semi-pp-to-literal               12   0%
                           #<compiled 0x2000000001a640b8>           1   0%
                        - c-determine-limit                         4   0%
                           c-state-semi-pp-to-literal               3   0%
                       - c-syntactic-skip-backward                 16   1%
                        - c-parse-state                            10   0%
                         - c-parse-state-1                          9   0%
                          - c-parse-state-get-strategy              3   0%
                           - c-beginning-of-macro                   2   0%
                              back-to-indentation                   1   0%
                          - c-remove-stale-state-cache-backwards    3   0%
                             c-state-balance-parens-backwards       1   0%
                           - c-state-literal-at                     1   0%
                              c-state-pp-to-literal                 1   0%
                             c-beginning-of-macro                   1   0%
                          - c-append-to-state-cache                 1   0%
                             c-beginning-of-macro                   1   0%
                           c-beginning-of-macro                     1   0%
                        - c-backward-sws                            2   0%
                         - c-beginning-of-macro                     1   0%
                            back-to-indentation                     1   0%
                          copy-tree                                 1   0%
                          c-beginning-of-macro                      1   0%
                       - c-parse-state                              7   0%
                        - c-parse-state-1                           6   0%
                         - c-append-to-state-cache                  4   0%
                            c-beginning-of-macro                    1   0%
                           c-remove-stale-state-cache               1   0%
                        - c-beginning-of-macro                      1   0%
                           back-to-indentation                      1   0%
                         c-forward-sws                              1   0%
                      - c-font-lock-complex-decl-prepare           25   1%
                       - c-parse-state                             15   1%
                        - c-parse-state-1                          15   1%
                         - c-append-to-state-cache                  8   0%
                            c-append-lower-brace-pair-to-state-cache  1   0%
                         - c-parse-state-get-strategy               3   0%
                          - c-beginning-of-macro                    2   0%
                             back-to-indentation                    2   0%
                          - c-get-fallback-scan-pos                 1   0%
                           - beginning-of-defun                     1   0%
                              beginning-of-defun-raw                1   0%
                         - c-remove-stale-state-cache               2   0%
                          - c-beginning-of-macro                    2   0%
                             back-to-indentation                    1   0%
                       - c-backward-sws                             7   0%
                        - c-beginning-of-macro                      5   0%
                           back-to-indentation                      2   0%
                        #<compiled 0x20000000077e7950>             13   0%
                        #<compiled 0x20000000077e7538>             12   0%
                      - c-font-lock-enum-tail                      11   0%
                       - c-backward-over-enum-header                6   0%
                        - c-backward-token-2                        4   0%
                         - c-backward-sws                           4   0%
                            c-beginning-of-current-token            1   0%
                       - c-parse-state                              5   0%
                        - c-parse-state-1                           4   0%
                         - c-remove-stale-state-cache               2   0%
                            c-beginning-of-macro                    1   0%
                           c-append-to-state-cache                  1   0%
                          c-beginning-of-macro                      1   0%
                        #<compiled 0x20000000077e79d0>             11   0%
                        #<compiled 0x20000000077e7aa0>              7   0%
                      - c-font-lock-invalid-single-quotes           7   0%
                       - c-literal-limits                           3   0%
                          c-state-full-pp-to-literal                3   0%
                        #<compiled 0x20000000077e7678>              7   0%
                      - #<compiled 0x2000000007807748>              3   0%
                       - c-font-lock-doc-comments                   3   0%
                        - c-literal-start                           2   0%
                           c-state-semi-pp-to-literal               2   0%
                        #<compiled 0x20000000077e7808>              2   0%
                        c-font-lock-enum-body                       1   0%
                     - font-lock-fontify-syntactically-region       9   0%
                        syntax-ppss                                 7   0%
                    - c-before-context-fl-expand-region            77   5%
                     - mapc                                        77   5%
                      - #<compiled 0x20000000078876d8>             77   5%
                       - c-context-expand-fl-region                76   5%
                        - c-fl-decl-start                          62   4%
                         - c-syntactic-skip-backward               27   1%
                          - c-parse-state                          11   0%
                           - c-parse-state-1                       11   0%
                            - c-append-to-state-cache               5   0%
                             - c-beginning-of-macro                 1   0%
                                back-to-indentation                 1   0%
                               c-append-lower-brace-pair-to-state-cache   1   0%
                            - c-parse-state-get-strategy            1   0%
                             - c-beginning-of-macro                 1   0%
                                back-to-indentation                 1   0%
                              c-remove-stale-state-cache            1   0%
                          - c-beginning-of-macro                    1   0%
                             back-to-indentation                    1   0%
                         - c-determine-limit                       17   1%
                            c-state-semi-pp-to-literal              9   0%
                          - c-determine-limit                       2   0%
                           - c-determine-limit                      1   0%
                              c-state-semi-pp-to-literal            1   0%
                             c-state-semi-pp-to-literal             1   0%
                         - c-literal-start                          6   0%
                          - c-state-semi-pp-to-literal              6   0%
                             c-parse-ps-state-below                 2   0%
                         - c-looking-at-or-maybe-in-bracelist       5   0%
                            c-backward-token-2                      5   0%
                         - c-forward-type                           3   0%
                            c-forward-name                          2   0%
                           c-backward-sws                           1   0%
                        - c-fl-decl-end                            11   0%
                         - c-literal-start                          7   0%
                          - c-state-semi-pp-to-literal              6   0%
                             c-parse-ps-state-below                 1   0%
                           c-on-identifier                          1   0%
                           c-forward-declarator                     1   0%
                         - c-backward-sws                           1   0%
                            c-beginning-of-macro                    1   0%
                    bug-reference-fontify                           2   0%
               file-remote-p                                        3   0%
             - find-image                                           2   0%
                image-search-load-path                              2   0%
             - eval                                                 1   0%
                if                                                  1   0%
            - timer-event-handler                                   1   0%
             - apply                                                1   0%
                jit-lock-force-redisplay                            1   0%

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

> Has Tramp become a lot slower?  Or is it only that the current Tramp doesn't
> work on Emacs 23?

The latter. Current Tramp (2.5.x) supports Emacs 25 pp.

Best regards, Michael.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 27 Mar 2020 22:40:47 -0400
> Cc: acm@muc.de, emacs-devel@gnu.org
> 
> To tell the people with the strongest commitment to freedom to use an
> old version of Emacs is to choose failure.  We need to re-optimize the
> Emacs master so it gives good performance on those machines.

That is of course the ideal we should always strive to, but it's
unreasonable to expect newer versions of Emacs to stay as fast as the
oder ones: we do add functionality, and some of that eats up CPU
cycles.  We do try to optimize performance, and occasionally do make
newer versions faster (e.g., Emacs 26 is about 30% faster than Emacs
25 on my machine, when scrolling through a large C file), but those
are exceptions rather than the rule, because significant speedups are
only possible by redesign and reimplementation of core functions,
something we do rarely.

> Suggesting that we disable certain features is ok.

Then my suggestion is to disable font-lock: that makes scrolling
through xdisp.c in Emacs 26.3 8 times faster, and almost as fast as
Emacs 23.4.

> Regarding C++ indentation, I have two suggestions.
> 
> * We could support a faster indentation mode for C code
> and for C++ files that use a limited repertoire of C++ constructs.
> 
> * Maybe we can come up with additional tricks comparable to
> {-in-column-0 to optimize parsing.  Emacs users would put these things
> into source files so as to get better peformance.

I think the way to increase performance of C/C++ fontification,
indentation, etc. is to use syntactic analysis in C (or another
similarly fast compiled language).  Several programs/libraries are
available out there for this purpose, and we should encourage
interested individuals to work on making these used by our major modes
that support programming languages.  The time we could do that with
ad-hoc regexps and the likes of syntax-ppss has gone, IMO.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 12:58:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> 
> That is of course the ideal we should always strive to, but it's
> unreasonable to expect newer versions of Emacs to stay as fast as the
> oder ones: we do add functionality, and some of that eats up CPU
  ^^^^
"older", sorry.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Ulrich Mueller]

>>>>> On Fri, 27 Mar 2020, Richard Stallman wrote:

>> It's not possible.  Bidirectional editing support in Emacs is
>> unconditional, it isn't an optional feature.

> It isn't optional _now_ but maybe there is an opportunity to optimize
> redisplay if the userssays "no BIDI".

> Maybe some of the pre-bidi code could be brought back and used
> when the user says there is no BIDI in the buffer.

>> What I would suggest is to compare with Emacs 24.1, which already
>> supported bidi.

> It would be interesting to compare 24 and 23, to see what slowdown
> there is.  And compare other subsequent versions to see what slowdown
> there is.  These could give hints for what operations are worth trying
> to optimize.

I've run the benchmark for several Emacs versions, on a Thinkpad L580
from 2018 (Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz). In all cases, I
used xdisp.c from version 26.3 as the test file.

  18.59        n/a    0.320
  23.4       5.436    1.019
  24.5      25.482    1.636
  25.3      22.370    1.610
  26.3      16.533    1.911
  27.0.90   17.602    2.339

First column is the version, second and third column are the times with
font-lock-mode enabled and disabled, respectively.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Sat, Mar 28, 2020 at 10:09:59 +0300, Eli Zaretskii wrote:
> > From: Richard Stallman <rms@gnu.org>
> > Cc: rrandresf@gmail.com, acm@muc.de, emacs-devel@gnu.org
> > Date: Fri, 27 Mar 2020 22:48:38 -0400

[ .... ]

> I expect an average twofold slowdown due to bidirectional support.
> The OP indicates less than that (comparing Emacs 23 with Emacs 26), so
> it's reasonable, IMO, and probably means the CC Mode contribution is
> relatively minor.

> The tricky part is testing the bidirectional part and the CC Mode part
> separately, by loading the old Emacs-23 CC Mode files into later
> versions of Emacs.  Maybe Alan could help by providing a Lisp file
> which, when loaded, would load CC Mode from a given directory -- this
> should be run before visiting any C/C++ file.

Here you go:

(defvar loaded-cc-version nil)
(defun load-cc-version (dir)
  "Load the version of CC Mode contained in directory DIR."
  (interactive "D")
  (or (string-match "/$" dir)
      (setq dir (concat dir "/")))
  (let (fname)
    ;; `c-fallback-style' gets stuff pushed onto it by loading cc-vars.  So
    ;; clear it out first, after a quick and dirty sanity check.
    (if (file-exists-p (concat dir "cc-vars.elc"))
        (setq c-fallback-style nil))
    (mapc (lambda (f)
            (condition-case err
                (load-file (setq fname (concat dir "cc-" f ".elc")))
              (error (message "Couldn't load %s" fname))))
          '("defs" "vars" "fix" "engine" "cmds" "menus"
            "align" "styles" "awk" "fonts" "mode"
            "subword" "bytecomp" "compat" "guess"))
    (setq loaded-cc-version dir)))

You can actually run load-cc-version repeatedly within an Emacs session
(I frequently do), together with M-x c-mode (etc.) on the pertinent
buffers.  Very occasionally, there're problems with defvars when their
value have changed between versions, since they don't get reinitialised.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Sat, Mar 28, 2020 at 12:08:30 +0300, Eli Zaretskii wrote:
> > Date: Sat, 28 Mar 2020 10:09:59 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org

> > > It would be interesting to compare 24 and 23, to see what slowdown
> > > there is.

> > I expect an average twofold slowdown due to bidirectional support.

> Actually, I take that back: in benchmarks such as scroll-up-benchmark,
> and in a buffer with predominantly ASCII text, the effect of
> bidirectional support should be very small, perhaps 10% or 20%.

> So the issue of testing old v23.x CC Mode with newer versions of Emacs
> becomes more important than I originally thought.  Just as a teaser,
> here's the profile I obtained by scrolling in Emacs 26.3 through the
> entire xdisp.c, it seems to point to c-find-decl-spots and its
> immediate callers and subroutines as the hot spots:

Yes.  They are the functions which search for and analyse declarations.
That they take so much of the processing time is more a reflection on
the complexity of the task rather than an opportunity for optimisation.
(Which isn't to say that further optimisation is impossible, just that
it won't be easy.)

> - command-execute                                                1446 100%
>  - call-interactively                                            1446 100%
>   - funcall-interactively                                        1446 100%
>    - execute-extended-command                                    1446 100%
>     - command-execute                                            1446 100%
>      - call-interactively                                        1446 100%
>       - funcall-interactively                                    1446 100%
>        - scroll-up-benchmark                                     1446 100%
>         - let                                                    1446 100%
>          - condition-case                                        1446 100%
>           - while                                                1446 100%
>            - redisplay                                           1351  93%
>             - redisplay_internal (C function)                    1071  74%
>              - jit-lock-function                                 1064  73%
>               - jit-lock-fontify-now                             1063  73%
>                - jit-lock--run-functions                         1063  73%
>                 - run-hook-wrapped                               1063  73%
>                  - #<compiled 0x2000000008356520>                1063  73%
>                   - font-lock-fontify-region                     1061  73%
>                    - c-font-lock-fontify-region                  1061  73%
>                     - font-lock-default-fontify-region            982  67%
>                      - font-lock-fontify-keywords-region          969  67%
>                       - c-font-lock-declarations                  686  47%
>                        - c-find-decl-spots                        674  46%
>                         - #<compiled 0x20000000077e07c0>          477  32%
>                          - c-get-fontification-context            253  17%
>                           - c-looking-at-or-maybe-in-bracelist    203  14%
>                            - c-backward-token-2                   196  13%
>                             - c-backward-sws                       35   2%
>                              - c-beginning-of-macro                12   0%
>                                 back-to-indentation                 6   0%
>                           - c-backward-token-2                     16   1%

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 11:43:01 +0000
> Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> (defvar loaded-cc-version nil)
> (defun load-cc-version (dir)
>   "Load the version of CC Mode contained in directory DIR."
>   (interactive "D")
>   (or (string-match "/$" dir)
>       (setq dir (concat dir "/")))
>   (let (fname)
>     ;; `c-fallback-style' gets stuff pushed onto it by loading cc-vars.  So
>     ;; clear it out first, after a quick and dirty sanity check.
>     (if (file-exists-p (concat dir "cc-vars.elc"))
>         (setq c-fallback-style nil))
>     (mapc (lambda (f)
>             (condition-case err
>                 (load-file (setq fname (concat dir "cc-" f ".elc")))
>               (error (message "Couldn't load %s" fname))))
>           '("defs" "vars" "fix" "engine" "cmds" "menus"
>             "align" "styles" "awk" "fonts" "mode"
>             "subword" "bytecomp" "compat" "guess"))
>     (setq loaded-cc-version dir)))

Thanks, but when I use this command to load CC Mode from Emacs 23.4,
and then visit xdisp.c, I get an error message:

  File mode specification error: (void-function make-local-hook)

and CC mode is not turned on.  What am I missing?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Sat, Mar 28, 2020 at 15:05:21 +0300, Eli Zaretskii wrote:
> > Date: Sat, 28 Mar 2020 11:43:01 +0000
> > Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > (defvar loaded-cc-version nil)
> > (defun load-cc-version (dir)
> >   "Load the version of CC Mode contained in directory DIR."
> >   (interactive "D")
> >   (or (string-match "/$" dir)
> >       (setq dir (concat dir "/")))
> >   (let (fname)
> >     ;; `c-fallback-style' gets stuff pushed onto it by loading cc-vars.  So
> >     ;; clear it out first, after a quick and dirty sanity check.
> >     (if (file-exists-p (concat dir "cc-vars.elc"))
> >         (setq c-fallback-style nil))
> >     (mapc (lambda (f)
> >             (condition-case err
> >                 (load-file (setq fname (concat dir "cc-" f ".elc")))
> >               (error (message "Couldn't load %s" fname))))
> >           '("defs" "vars" "fix" "engine" "cmds" "menus"
> >             "align" "styles" "awk" "fonts" "mode"
> >             "subword" "bytecomp" "compat" "guess"))
> >     (setq loaded-cc-version dir)))

> Thanks, but when I use this command to load CC Mode from Emacs 23.4,
> and then visit xdisp.c, I get an error message:

>   File mode specification error: (void-function make-local-hook)

> and CC mode is not turned on.  What am I missing?

make-local-hook was removed from Emacs in around 2010.  I think it still
exists in XEmacs (as much as anything can still be said to exist there).

If you can give me some time to eat my lunch, I'll seek out the
workaround I must have introduced some time after Emacs 23.4.  There may
be one or two other obsolete variables/functions which may now need
workarounds to allow timing of the old CC Mode versions.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 12:18:06 +0000
> Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> If you can give me some time to eat my lunch

Sure, not much point in programming on empty stomach ;-)

Thanks.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   File mode specification error: (void-function make-local-hook)
>
> and CC mode is not turned on.  What am I missing?

(defalias 'make-local-hook #'ignore)


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   18.59        n/a    0.320
>   23.4       5.436    1.019
>   24.5      25.482    1.636
>   25.3      22.370    1.610
>   26.3      16.533    1.911
>   27.0.90   17.602    2.339
>
> First column is the version, second and third column are the times with
> font-lock-mode enabled and disabled, respectively.

Thanks.  Some more questions need pondering before we can know what
might need to be done:
- What is the target?  is 5s good enough?  What about 10s?
- The middle column is basically a test of the font-lock performance of
  CC-mode's `c-mode`.  Is `c-mode` really all we care about here?
  [ I believe for the OP's case `c-mode` was just taken as an example
    but is not the only case he cares about, maybe not even the main one.  ]


-- Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Sat, Mar 28, 2020 at 15:28:50 +0300, Eli Zaretskii wrote:
> > Date: Sat, 28 Mar 2020 12:18:06 +0000
> > Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > If you can give me some time to eat my lunch

> Sure, not much point in programming on empty stomach ;-)

> Thanks.

Like Stefan said, aliasing make-local-hook to `ignore' will be fine.
All the relevant calls to add-hook set the LOCAL argument to t anyway.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sat, 28 Mar 2020 13:14:42 +0000
> Cc: rms@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Like Stefan said, aliasing make-local-hook to `ignore' will be fine.

OK, using Emacs 23 CC Mode with Emacs 26.3 makes the scroll benchmark
run only 10% slower than in Emacs 23.  So it seems like most of the
slow-down in the scroll benchmark is due to font-lock of CC Mode.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> Like Stefan said, aliasing make-local-hook to `ignore' will be fine.
> All the relevant calls to add-hook set the LOCAL argument to t anyway.

... and even if they don't set the LOCAL argument, `make-local-hook`
still doesn't make any difference.

IOW, contrary to `setq` which sometimes sets the var buffer-locally and
sometimes does it globally depending on whether the var has been made
buffer-local, `add-hook` only uses the LOCAL argument to decide whether
the hook should be affected locally or globally (with the long-standing
ugly exception of those hooks that have been `make-local-variable`,
which I'd love to get rid of).  I changed this back in Emacs-21 ;-)


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Clément Pit-Claudel]

On 28/03/2020 05.58, Eli Zaretskii wrote:
> I think the way to increase performance of C/C++ fontification,
> indentation, etc. is to use syntactic analysis in C (or another
> similarly fast compiled language).  Several programs/libraries are
> available out there for this purpose, and we should encourage
> interested individuals to work on making these used by our major modes
> that support programming languages.  The time we could do that with
> ad-hoc regexps and the likes of syntax-ppss has gone, IMO.

I've looked, but I don't really know of any such libraries.  The problem that cc-mode has to solve most of the time is fontifying an invalid C program (since, unless the programmer is using a structural editing mode), code that is being typed is invalid most of the time.  Most language parsing facilities out there do not have great error recovery, so they are not very good at giving semantic annotations useful for syntax highlighting broken code.

The only great example I know is the OCaml parser written in Menhir, which is incredibly robust.  The most extreme one I know in the other direction is agda-mode, which only displays syntax highlighting when the buffer contains syntactically valid, type-correct code.

I wonder if Stefan's smie could be used to do syntax highlighting.

Clément.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sat, 28 Mar 2020 10:43:00 -0400
> 
> On 28/03/2020 05.58, Eli Zaretskii wrote:
> > I think the way to increase performance of C/C++ fontification,
> > indentation, etc. is to use syntactic analysis in C (or another
> > similarly fast compiled language).  Several programs/libraries are
> > available out there for this purpose, and we should encourage
> > interested individuals to work on making these used by our major modes
> > that support programming languages.  The time we could do that with
> > ad-hoc regexps and the likes of syntax-ppss has gone, IMO.
> 
> I've looked, but I don't really know of any such libraries.

What about tree-sitter?

> The problem that cc-mode has to solve most of the time is fontifying an invalid C program (since, unless the programmer is using a structural editing mode), code that is being typed is invalid most of the time.  Most language parsing facilities out there do not have great error recovery, so they are not very good at giving semantic annotations useful for syntax highlighting broken code.

AFAIK, tree-sitter's use cases include the above.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Clément Pit-Claudel]

On 28/03/2020 11.03, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Sat, 28 Mar 2020 10:43:00 -0400
>>
>> On 28/03/2020 05.58, Eli Zaretskii wrote:
>>> I think the way to increase performance of C/C++ fontification,
>>> indentation, etc. is to use syntactic analysis in C (or another
>>> similarly fast compiled language).  Several programs/libraries are
>>> available out there for this purpose, and we should encourage
>>> interested individuals to work on making these used by our major modes
>>> that support programming languages.  The time we could do that with
>>> ad-hoc regexps and the likes of syntax-ppss has gone, IMO.
>>
>> I've looked, but I don't really know of any such libraries.
> 
> What about tree-sitter?

Neat! It looks promising.  I forgot about this one.
It seems that Atom uses it, so that's a pretty strong endorsement too!
Could it be used with Emacs 25's modules? Otherwise, how can users define new languages without recompiling Emacs?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Sat, 28 Mar 2020 11:17:34 -0400
> 
> > What about tree-sitter?
> 
> Neat! It looks promising.  I forgot about this one.
> It seems that Atom uses it, so that's a pretty strong endorsement too!
> Could it be used with Emacs 25's modules? Otherwise, how can users define new languages without recompiling Emacs?

I'd start building Emacs with that library, and figuring out how to
interface it from Lisp for our purposes.  Only after that I'd worry
about making that a module and/or the ability to add languages.  "The
perfect is the enemy of the good", and many languages already have
support files ready to be used.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 17:17, Clément Pit-Claudel wrote:
> Could it be used with Emacs 25's modules?

Apparently so: https://github.com/karlotness/tree-sitter.el

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 18:10, Dmitry Gutov wrote:
> On 28.03.2020 17:17, Clément Pit-Claudel wrote:
>> Could it be used with Emacs 25's modules?
> 
> Apparently so: https://github.com/karlotness/tree-sitter.el

Sorry, this one is more active: 
https://github.com/ubolonton/emacs-tree-sitter

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 28 Mar 2020 18:11:05 +0200
> Cc: emacs-devel@gnu.org
> 
> On 28.03.2020 18:10, Dmitry Gutov wrote:
> > On 28.03.2020 17:17, Clément Pit-Claudel wrote:
> >> Could it be used with Emacs 25's modules?
> > 
> > Apparently so: https://github.com/karlotness/tree-sitter.el
> 
> Sorry, this one is more active: 
> https://github.com/ubolonton/emacs-tree-sitter

Do any of those support font-lock?  Because I think that's the context
of Clément's question about modules.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 18:25, Eli Zaretskii wrote:
>> From: Dmitry Gutov<dgutov@yandex.ru>
>> Date: Sat, 28 Mar 2020 18:11:05 +0200
>> Cc:emacs-devel@gnu.org
>>
>> On 28.03.2020 18:10, Dmitry Gutov wrote:
>>> On 28.03.2020 17:17, Clément Pit-Claudel wrote:
>>>> Could it be used with Emacs 25's modules?
>>> Apparently so:https://github.com/karlotness/tree-sitter.el
>> Sorry, this one is more active:
>> https://github.com/ubolonton/emacs-tree-sitter
> Do any of those support font-lock?  Because I think that's the context
> of Clément's question about modules.

Apparently they aim to replace it with a new specialized jit-highlighter:

https://github.com/ubolonton/emacs-tree-sitter/issues/4
https://github.com/ubolonton/emacs-tree-sitter/pull/16

But that's not due to modules' limitations, but because of font-lock's 
limitations (allegedly).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi Richard.

Richard> To tell the people with the strongest commitment to freedom
Richard> to use an old version of Emacs is to choose failure.  We need
Richard> to re-optimize the Emacs master so it gives good performance
Richard> on those machines.

We Agree on that.

>> BTW. On my comparissons that variable was set no nil as Doc says:
>> --8<---------------cut here---------------start------------->8---
>> If nil, use the default decoration (typically the minimum
>> available).  --8<---------------cut
>> here---------------end--------------->8--- Also xdisp.c. Just load
>> c-mode not c++-mode

Richard> What were the results of the comparisons you did with this
Richard> setting?  (I wasn't following this topic until yesterday.)

It was on the first email of this thread:
--8<---------------cut here---------------start------------->8---
https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00511.html
--8<---------------cut here---------------end--------------->8---

Ulrich Mueller has a new comparisson with an up to date machine. On his
comparisson he has it with font-lock enable and disable. I am going to
do the same an update the table accordingly.
--8<---------------cut here---------------start------------->8---
https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00724.html
--8<---------------cut here---------------end--------------->8---

>> With the help of Michael Albinus I have update the emacs23 builtin
>> tramp version to the latest one that supports GNU/Emacs23

Richard> Has Tramp become a lot slower?  Or is it only that the
Richard> current Tramp doesn't work on Emacs 23?

Michael Albinus has answered on this email:
--8<---------------cut here---------------start------------->8---
https://lists.gnu.org/archive/html/emacs-devel/2020-03/msg00721.html
--8<---------------cut here---------------end--------------->8---

Richard> Has Org mode become a lot slower?  Or is it only that the
Richard> current Org mode doesn't work on Emacs 23?

Current org-mode does not work on Emacs23. But I remember You were part
of that discussion in the past. And the conclusion of that discussion
was to support just three versions. But the org-mode guys replied a
shortage of man power for it.

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[rrandresf]

Hi Guys.

Dmitry> Apparently they aim to replace it with a new specialized
Dmitry> jit-highlighter:

I was just remembering the AA-tree on the noverlays branch(from Andreas Politz). I do not
remember the sender but a guy was trying to save and merge code from
that branch to master (before closing the Emacs27 merge window). I think
the idea with this AA-tree was improving actual font-lock with the
AA-tree. I think a jit-highlighter would render with more
performance. Am I right?.

Best Regards
ps: the sender was not successfull on using an AA-tree. But Andreas was.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 23:08, rrandresf@gmail.com wrote:
> I think
> the idea with this AA-tree was improving actual font-lock with the
> AA-tree.

I don't think so. IIUC, the branch's goal was to improve the performance 
of overlays. Not text properties (which font-lock uses).

> I think a jit-highlighter would render with more
> performance.

The current font-lock engine is jit-lock, and it's usually not the 
bottleneck of our syntax functionality. The font-lock rules themselves 
are generally more CPU-intensive.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi Dmitry.

Dmitry> I don't think so. IIUC, the branch's goal was to improve the
Dmitry> performance of overlays. Not text properties (which font-lock
Dmitry> uses).

What about font-lock changing its use of text properties to use in place
overlays?

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 28.03.2020 23:38, andrés ramírez wrote:
> Dmitry> I don't think so. IIUC, the branch's goal was to improve the
> Dmitry> performance of overlays. Not text properties (which font-lock
> Dmitry> uses).
> 
> What about font-lock changing its use of text properties to use in place
> overlays?

Not sure I understand the question. Could you rephrase that?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andres.ramirez]

Hi Dmitry.

Dmitry> don't think
Dmitry> so. IIUC, the branch's goal was to improve the performance of
Dmitry> overlays. Not text properties (which font-lock uses).
>> 
>> What about font-lock changing its use of text properties to use in
>> place overlays?

Dmitry> Not sure I understand the question. Could you rephrase that?

Now: font-lock uses text properties
Posibility: font-lock uses overlays (dismiss text properties)

the AA-tree could be usable on that Posibility?

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 29.03.2020 00:15, andres.ramirez wrote:
> Now: font-lock uses text properties
> Posibility: font-lock uses overlays (dismiss text properties)
> 
> the AA-tree could be usable on that Posibility?

Yes. But then, I'm not sure why you'd want that change.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

    > 23.4       5.436    1.019
    > 24.5      25.482    1.636
    > 25.3      22.370    1.610
    > 26.3      16.533    1.911
    > 27.0.90   17.602    2.339

  > First column is the version, second and third column are the times with
  > font-lock-mode enabled and disabled, respectively.

What this says is that even with font lock disabled, there has been a
lot of slowdown.  About half of the slowdown was in version 24, and
most of the rest was after version 26.

If someone can figure out why, we may be able to speed it up again.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi Guys.

Eli> OK, using Emacs 23 CC Mode with Emacs 26.3 makes the scroll
Eli> benchmark run only 10% slower than in Emacs 23.  So it seems like
Eli> most of the slow-down in the scroll benchmark is due to font-lock
Eli> of CC Mode.

This my updated benchmark:

Year 2016 (allwinner H3 Quad Core CPU @ 1GHz). In all cases, I
used xdisp.c from version 26.3 as the test file.

| EMACS      |       no font-lock | cc-mode from 23.4 (without cc-guess.el) |
|------------+--------------------+-----------------------------------------|
| emacs-23.4 | 22.041109085083008 |                       131.7112021446228 |
| emacs-26-3 |  34.09645438194275 |                       111.6537675857544 |

| EMACS 26.3 |      no font-lock |  built-in cc-mode |
|------------+-------------------+-------------------|
|  | 35.47433280944824 | 261.2694363594055 |

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 28 Mar 2020 18:43:13 +0200
> 
> >>> Apparently so:https://github.com/karlotness/tree-sitter.el
> >> Sorry, this one is more active:
> >> https://github.com/ubolonton/emacs-tree-sitter
> > Do any of those support font-lock?  Because I think that's the context
> > of Clément's question about modules.
> 
> Apparently they aim to replace it with a new specialized jit-highlighter:
> 
> https://github.com/ubolonton/emacs-tree-sitter/issues/4
> https://github.com/ubolonton/emacs-tree-sitter/pull/16
> 
> But that's not due to modules' limitations, but because of font-lock's 
> limitations (allegedly).

I'm not sure what limitations they have in mind: font-lock supports
functions, not just regular expressions.  Why does it have to be
replaced?

In any case, tossing JIT font-lock and going back to using all kinds
of hooks is a non-starter.  Emacs 19 and 20 did that, and we found
this method to have fundamental problems which couldn't be resolved in
satisfactory ways.  That is why Emacs 21 introduced jit-lock.el and
the associated support in the display engine for fontifying what is
about to be displayed.  Going back to hooks (if that is what they
intend to do; I couldn't be sure from reading the discussion) is a
step in the wrong direction.

Likewise using overlays instead of text properties (again, not sure
they actually intend doing that, maybe they just talked about it) is
wrong even if we assume the tree-based implementation that sits on a
branch.

More generally, I'm surprised, to say the least, that such a
significant job is being done without saying even a single word here,
let alone making sure the basic design is deemed reasonable by the
core developers.  Do they really think this will make the acceptance
of the code easier?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> In any case, tossing JIT font-lock and going back to using all kinds
> of hooks is a non-starter.

I think it's just a reflection of the fact that author doesn't know
enough about jit-lock and font-lock.

I sent him some patches that should help.

> More generally, I'm surprised, to say the least, that such
> a significant job is being done without saying even a single word
> here, let alone making sure the basic design is deemed reasonable by
> the core developers.  Do they really think this will make the
> acceptance of the code easier?

I think this risks being read as unwelcoming, which is unlikely to
encourage them to get in touch with emacs-devel.

tree-sitter, like LSP, is something Emacs should embrace.  I think in
the case of tree-sitter we would benefit from participating directly in
its development to make sure it's easy enough to integrate into Emacs
(e.g. doesn't require a Javascript or C compiler on the end-user side).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 29 Mar 2020 12:39:55 -0400
> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org, Dmitry Gutov <dgutov@yandex.ru>
> tree-sitter, like LSP, is something Emacs should embrace.

  https://lists.gnu.org/archive/html/emacs-devel/2020-01/msg00059.html

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 29.03.2020 19:39, Stefan Monnier wrote:
> I think it's just a reflection of the fact that author doesn't know
> enough about jit-lock and font-lock.

Indeed, that font-lock rule can be a function is not very well-known.

> I sent him some patches that should help.

That's mighty good of you.

I was curious and wanted to take a look, though, but couldn't find them 
neither among pull requests, nor the issue comments. :-(

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 29.03.2020 17:30, Eli Zaretskii wrote:

>> Apparently they aim to replace it with a new specialized jit-highlighter:
>>
>> https://github.com/ubolonton/emacs-tree-sitter/issues/4
>> https://github.com/ubolonton/emacs-tree-sitter/pull/16
>>
>> But that's not due to modules' limitations, but because of font-lock's
>> limitations (allegedly).
> 
> I'm not sure what limitations they have in mind: font-lock supports
> functions, not just regular expressions.  Why does it have to be
> replaced?

Hence "allegedly". I haven't looked into this issue, or this project, 
deep enough to say more.

> In any case, tossing JIT font-lock and going back to using all kinds
> of hooks is a non-starter.  Emacs 19 and 20 did that, and we found
> this method to have fundamental problems which couldn't be resolved in
> satisfactory ways.  That is why Emacs 21 introduced jit-lock.el and
> the associated support in the display engine for fontifying what is
> about to be displayed.  Going back to hooks (if that is what they
> intend to do; I couldn't be sure from reading the discussion)

I think they're still experimenting. But whatever they come up with in 
that alternative jit highlighter, could hopefully be ported to jit-lock.

> Likewise using overlays instead of text properties (again, not sure
> they actually intend doing that, maybe they just talked about it) is
> wrong even if we assume the tree-based implementation that sits on a
> branch.

Yup.

> More generally, I'm surprised, to say the least, that such a
> significant job is being done without saying even a single word here,
> let alone making sure the basic design is deemed reasonable by the
> core developers.

There is no point in taking offense at that. I think this is just 
another case of the extended developer community, and especially 
newcomers, being uncomfortable with our old-timey tools. I'm pretty sure 
none of the participants in that project even read your earlier posting 
to emacs-devel about tree-sitter.

And if you ask some of them why they didn't think to ask on the mailing 
list first, the answers will likely be very benign like the story here: 
https://www.reddit.com/r/emacs/comments/fqji2t/any_elisp_projects_looking_for_contributors/flqs6v9/

> Do they really think this will make the acceptance
> of the code easier?

They really hope they won't have to ask.

It's a blessing and a curse of Emacs's extensibility. If this project is 
successful, it is likely to reside entirely outside of Emacs core. Just 
like the lsp-mode project lives outside, and only filed a couple of bug 
reports when the built-in JSON support (which they really have to use) 
showed insufficient performance characteristics.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 29 Mar 2020 21:27:50 +0300
> 
> If this project is successful, it is likely to reside entirely
> outside of Emacs core. Just like the lsp-mode project lives outside,
> and only filed a couple of bug reports when the built-in JSON
> support (which they really have to use) showed insufficient
> performance characteristics.

FWIW, I don't think it's reasonable to have such core functionality
outside of core.  Such external projects can never last as long as
Emacs did and will, because there's not enough resources to sustain
them.  Bringing them into core is the only way to both make them clean
enough and tested enough, and make sure they stay alive for many
years, let alone make sure applications are built based on them.

emacs-tree-sitter and font-lock (was: emacs rendering comparisson between emacs23 and emacs26.3)

[Stefan Monnier]

>> I think it's just a reflection of the fact that author doesn't know
>> enough about jit-lock and font-lock.
> Indeed, that font-lock rule can be a function is not very well-known.

Although it seems fairly clearly documented.
Not sure how we could solve this.

>> I sent him some patches that should help.
> That's mighty good of you.
> I was curious and wanted to take a look, though, but couldn't find them
> neither among pull requests, nor the issue comments. :-(

I don't how to participate in a Github pull request with
Free Software.  IIUC there are some Elisp packages for that, but
I haven't investigated how that would work.
So I sent the patches via email.  I put the main one after my sig.


        Stefan


diff --git a/lisp/tree-sitter-highlight.el b/lisp/tree-sitter-highlight.el
index 53ecb32..3246bd6 100644
--- a/lisp/tree-sitter-highlight.el
+++ b/lisp/tree-sitter-highlight.el
@@ -167,7 +167,6 @@ to faces.  Each function takes no arguments."
 (defvar-local tree-sitter-highlight--injections nil)
 (defvar-local tree-sitter-highlight--injections-query nil)
 (defvar-local tree-sitter-highlight--jit-function nil)
-(defvar-local tree-sitter-highlight--orig-scroll-functions nil)
 (defvar-local tree-sitter-highlight--query nil)
 (defvar-local tree-sitter-highlight--query-cursor nil)
 
@@ -240,19 +239,11 @@ to faces.  Each function takes no arguments."
 
 (defun tree-sitter-highlight--jit (old-tree)
   (when old-tree
-    (let ((changes (ts-changed-ranges old-tree tree-sitter-tree))
-           (wstart (window-start))
-           (wend (window-end)))
-      ;; TODO: Remember what we've highlighted, similar to how font-lock does it.
-      ;;       Already highlighted regions shouldn't be re-highlighted.
-
-      ;; Find changes that are within the current window
-      (mapc #'(lambda (range)
-                (let ((start (aref range 0))
-                       (end (aref range 1)))
-                  ;; TODO: Improve this
-                  (tree-sitter-highlight--highlight (max wstart start) (min wend end))))
-        changes))))
+    (mapc #'(lambda (range)
+              (let ((start (aref range 0))
+                    (end (aref range 1)))
+                (font-lock-flush start end)))
+          (ts-changed-ranges old-tree tree-sitter-tree))))
 
 (defun tree-sitter-highlight--highlight (start end)
   (ts--save-context
@@ -282,9 +273,6 @@ to faces.  Each function takes no arguments."
 ;;               (ts-byte-from-position start)
 ;;               (ts-byte-from-position end)))))
 
-(defun tree-sitter-highlight--highlight-window (_window start)
-  (tree-sitter-highlight--highlight start (window-end nil t)))
-
 (defun tree-sitter-highlight--enable ()
   "Enable `tree-sitter-highlight' in this buffer."
   (run-hooks  'tree-sitter-highlight-setup-functions)
@@ -301,10 +289,8 @@ to faces.  Each function takes no arguments."
       tree-sitter-highlight--injections-query (cadr x)))
   (setq tree-sitter-highlight--capture-names (ts-query-capture-names tree-sitter-highlight--query))
   (setq tree-sitter-highlight--query-cursor (ts-make-query-cursor))
-  (make-variable-buffer-local 'window-scroll-functions)
-  (setq tree-sitter-highlight--orig-scroll-functions window-scroll-functions)
-  (setq window-scroll-functions (cons #'tree-sitter-highlight--highlight-window window-scroll-functions))
-  (tree-sitter-highlight--highlight-window nil (window-start))
+  (add-function :override (local 'font-lock-fontify-region-function)
+                #'tree-sitter-highlight--highlight)
   (add-hook 'tree-sitter-after-change-functions #'tree-sitter-highlight--jit nil t)
   )
 
@@ -314,7 +300,8 @@ to faces.  Each function takes no arguments."
     (remove-text-properties (point-min)
       (point-max)
       '(face nil)))
-  (setq window-scroll-functions tree-sitter-highlight--orig-scroll-functions)
+  (remove-function (local 'font-lock-fontify-region-function)
+                   #'tree-sitter-highlight--highlight)
   (remove-hook 'tree-sitter-after-change-functions #'tree-sitter-highlight--jit t))
 
 (define-minor-mode tree-sitter-highlight-mode

Re: emacs rendering comparisson between emacs23 and emacs26.3

[조성빈]

> 2020. 3. 29. 오후 11:32, Eli Zaretskii <eliz@gnu.org> 작성:
> 
> 
>> 
>> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sat, 28 Mar 2020 18:43:13 +0200
>> 
>>>>> Apparently so:https://github.com/karlotness/tree-sitter.el
>>>> Sorry, this one is more active:
>>>> https://github.com/ubolonton/emacs-tree-sitter
>>> Do any of those support font-lock?  Because I think that's the context
>>> of Clément's question about modules.
>> 
>> Apparently they aim to replace it with a new specialized jit-highlighter:
>> 
>> https://github.com/ubolonton/emacs-tree-sitter/issues/4
>> https://github.com/ubolonton/emacs-tree-sitter/pull/16
>> 
>> But that's not due to modules' limitations, but because of font-lock's 
>> limitations (allegedly).
> 
> I'm not sure what limitations they have in mind: font-lock supports
> functions, not just regular expressions.  Why does it have to be
> replaced?

According to them, font-lock doesn’t provide enough control. This is the exact text quoted from the linked issue:

>> so, we don't need to reinvent the wheel but neither rely on font-lock in a way that limit us, as I see it, we can implement our own syntax highlight functions for each grammar available and let font-lock use them instead of the defaults. Unless, of course, I'm missing something.
> 
> That only covers how to highlight a region. I think the main complexity is in the change-handling mechanisms, i.e. which regions should be re-highlighted, when.

I don’t know about font-lock enough to say whether this is true or not. 

> In any case, tossing JIT font-lock and going back to using all kinds
> of hooks is a non-starter.  Emacs 19 and 20 did that, and we found
> this method to have fundamental problems which couldn't be resolved in
> satisfactory ways.  That is why Emacs 21 introduced jit-lock.el and
> the associated support in the display engine for fontifying what is
> about to be displayed.  Going back to hooks (if that is what they
> intend to do; I couldn't be sure from reading the discussion) is a
> step in the wrong direction.
> 
> Likewise using overlays instead of text properties (again, not sure
> they actually intend doing that, maybe they just talked about it) is
> wrong even if we assume the tree-based implementation that sits on a
> branch.
> 
> More generally, I'm surprised, to say the least, that such a
> significant job is being done without saying even a single word here,
> let alone making sure the basic design is deemed reasonable by the
> core developers.  Do they really think this will make the acceptance
> of the code easier?

Actually I’m surprised that people here on the mailing list expects people to ask here — I think most people aren’t even aware of this list, and even if they do, they prefer working on Github + MELPA, with the community. 
I’m pretty sure they aren’t aiming to merge the code into Emacs — they’re in to write another external package, like lsp-mode or magit.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 29.03.2020 21:39, Eli Zaretskii wrote:
>> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 29 Mar 2020 21:27:50 +0300
>>
>> If this project is successful, it is likely to reside entirely
>> outside of Emacs core. Just like the lsp-mode project lives outside,
>> and only filed a couple of bug reports when the built-in JSON
>> support (which they really have to use) showed insufficient
>> performance characteristics.
> 
> FWIW, I don't think it's reasonable to have such core functionality
> outside of core.  Such external projects can never last as long as
> Emacs did and will, because there's not enough resources to sustain
> them.  Bringing them into core is the only way to both make them clean
> enough and tested enough, and make sure they stay alive for many
> years, let alone make sure applications are built based on them.

I'd also rather it benefits all our users, and the development can take 
advantage of our experience.

But this is the scenario I've seen many times. And yes, with several 
innovative projects abandoned after a few years.

Collaborating with the core developers seems objectively more difficult 
for many such projects, however.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: 조성빈 <pcr910303@icloud.com>
> Date: Mon, 30 Mar 2020 04:04:08 +0900
> Cc: Dmitry Gutov <dgutov@yandex.ru>, cpitclaudel@gmail.com,
>  Emacs-devel@gnu.org
> 
> > I'm not sure what limitations they have in mind: font-lock supports
> > functions, not just regular expressions.  Why does it have to be
> > replaced?
> 
> According to them, font-lock doesn’t provide enough control. This is the exact text quoted from the linked issue:
> 
> >> so, we don't need to reinvent the wheel but neither rely on font-lock in a way that limit us, as I see it, we can implement our own syntax highlight functions for each grammar available and let font-lock use them instead of the defaults. Unless, of course, I'm missing something.
> > 
> > That only covers how to highlight a region. I think the main complexity is in the change-handling mechanisms, i.e. which regions should be re-highlighted, when.

But that's exactly the problem that jit-lock solves, and solves well.
Because only the display engine knows exactly what parts need to be
re-highlighted and when.  IOW, this problem is already solved, all
that's needed is to play by jit-lock's rules.

> > More generally, I'm surprised, to say the least, that such a
> > significant job is being done without saying even a single word here,
> > let alone making sure the basic design is deemed reasonable by the
> > core developers.  Do they really think this will make the acceptance
> > of the code easier?
> 
> Actually I’m surprised that people here on the mailing list expects people to ask here — I think most people aren’t even aware of this list, and even if they do, they prefer working on Github + MELPA, with the community. 

I'm quite sure that everyone knows where the Emacs maintainers can be
contacted.

> I’m pretty sure they aren’t aiming to merge the code into Emacs — they’re in to write another external package, like lsp-mode or magit.

As I wrote in another message, I don't believe doing so is viable for
a core feature such as font-lock or major-mode support in general.
Magit is different, since it's an application, not a core feature
designed to be basis for applications.  And if LSP mode wants a long
life and many applications built upon it, it should come to Emacs core
as well, at least IMNSHO.

Re: emacs-tree-sitter and font-lock

[Dmitry Gutov]

On 29.03.2020 21:54, Stefan Monnier wrote:
>>> I think it's just a reflection of the fact that author doesn't know
>>> enough about jit-lock and font-lock.
>> Indeed, that font-lock rule can be a function is not very well-known.
> 
> Although it seems fairly clearly documented.
> Not sure how we could solve this.

Not sure either.

I once wrote a SO answer describing this, it was well-accepted.

>>> I sent him some patches that should help.
>> That's mighty good of you.
>> I was curious and wanted to take a look, though, but couldn't find them
>> neither among pull requests, nor the issue comments. :-(
> 
> I don't how to participate in a Github pull request with
> Free Software. 

Have you tried just using it through Firefox with JavaScript disabled 
(which is probably the crux of your "only Free Software" complaint)?

I can both log into GitHub this way, and create/comment on issues. And a 
pull request is also "an issue", in that it also has a comments section.

Less essential features don't work, but that's beside the point.

> IIUC there are some Elisp packages for that, but
> I haven't investigated how that would work.
> So I sent the patches via email.  I put the main one after my sig.

Thanks, it looks quite reasonable.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > To tell the people with the strongest commitment to freedom to use an
  > > old version of Emacs is to choose failure.  We need to re-optimize the
  > > Emacs master so it gives good performance on those machines.

  > That is of course the ideal we should always strive to, but it's
  > unreasonable to expect newer versions of Emacs to stay as fast as the
  > oder ones: we do add functionality, and some of that eats up CPU
  > cycles.

Would you please strive to keep Emacs from slowing down
for users who don't use, or who disable, the new functionality?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Sun, 29 Mar 2020 23:42:05 -0400
> 
>   > That is of course the ideal we should always strive to, but it's
>   > unreasonable to expect newer versions of Emacs to stay as fast as the
>   > oder ones: we do add functionality, and some of that eats up CPU
>   > cycles.
> 
> Would you please strive to keep Emacs from slowing down
> for users who don't use, or who disable, the new functionality?

For new features that can be disabled, that is mostly the case
anyway.  But most new features we add don't have such knobs, usually
because we don't envision anyone to want them.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Would you please strive to keep Emacs from slowing down
  > > for users who don't use, or who disable, the new functionality?

  > For new features that can be disabled, that is mostly the case
  > anyway.  But most new features we add don't have such knobs, usually
  > because we don't envision anyone to want them.

My point is, if a feature causes significant slowdown, in general usage,
that in itself is a reason to give it a knob.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

font lock with functions (was: emacs rendering comparisson between emacs23 and emacs26.3)

[David Engster]

> On 29.03.2020 19:39, Stefan Monnier wrote:
>> I think it's just a reflection of the fact that author doesn't know
>> enough about jit-lock and font-lock.
>
> Indeed, that font-lock rule can be a function is not very well-known.

Indeed. Since I'm currently writing a major-mode which makes heavy use
of this feature, let me add my experience.

I found the font-lock documentation in the Emacs Lisp reference manual
to be pretty opaque, but I also wouldn't really know how to improve
it. What is mostly missing are practical examples, but I understand that
a reference manual is probably not the right place for this. I found
some helpful posts online, like this SO answer from you

https://stackoverflow.com/a/14675550

and this blog post

http://www.modernemacs.com/post/advanced-syntax/

Other smaller problems I'm having:

- It is very easy during testing to completely hang your Emacs session
  if one of the font-lock functions hangs. One usually needs to kill
  Emacs in this case.

- Debugging is difficult because you cannot instrument the functions for
  some reason. Luckily, I found 'font-lock-studio', which is incredibly
  useful. We should try to get this into ELPA and advertise it in the
  manual:

   https://github.com/Lindydancer/font-lock-studio

On a more general note: When you use functions for font-locking, I often
find myself in the position that I'm more or less writing a parser, and
I could readily tell font-lock how to fontify several parts of the
buffer with different faces. However, AFAICS, this is not possible with
current font-lock? It seems each rule can always only return one type of
face, or am I missing something? Of course, you can still make this
work, but it makes font-lock pretty slow, because you effectively need
to re-parse the changed portions of the buffer several times. I would
imagine that fontifying based on tree-sitter would run into the same
issue?

-David

Re: font lock with functions

[Adam Porter]

David Engster <deng@randomsample.de> writes:

> It is very easy during testing to completely hang your Emacs session
> if one of the font-lock functions hangs. One usually needs to kill
> Emacs in this case.

Often using "pkill -SIGUSR2 emacs" can help.  If that works to interrupt
Emacs and display a backtrace, you can then switch to the buffer being
font-locked and disable the mode doing the font-locking (i.e. removing
the font-lock keywords).  Then when you close the backtrace, Emacs
should not go back into the font-locking loop in your code.  At least,
this usually worked for me when I was working on prism.el.  Sometimes
some random mashing of C-g helps as well.  And once in a while, running
"pkill -SIGUSR2 emacs" in a shell loop a few hundred times, while
randomly mashing C-g in Emacs, works as a last resort.  :)

If even that doesn't help, then you may want to code in one Emacs
instance and test the code in another.

And, yes, Anders's font-lock packages are very helpful.  I've catalogued
some of those and other tools here:

https://github.com/alphapapa/emacs-package-dev-handbook#highlighting--font-locking

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Mon, 30 Mar 2020 22:28:10 -0400
> 
>   > For new features that can be disabled, that is mostly the case
>   > anyway.  But most new features we add don't have such knobs, usually
>   > because we don't envision anyone to want them.
> 
> My point is, if a feature causes significant slowdown, in general usage,
> that in itself is a reason to give it a knob.

When we know a new feature causes significant slowdown, we either fix
it, or provide a way to disable or work around it.  The problem is, we
don't always know there's slowdown, as it frequently happens only in
specific rare use cases.  (John said some time ago we should have a
benchmarking test suite, but I don't think anyone's working on it.)
Also, several slowdowns each of which is minor can together cause a
significant slowdown, and in that case it is even harder to detect and
fix that.

Re: font lock with functions

[Stefan Monnier]

> - It is very easy during testing to completely hang your Emacs session
>   if one of the font-lock functions hangs. One usually needs to kill
>   Emacs in this case.

`jit-lock-debug-mode` might be helpful there.
If it's not good enough for your situation, please `M-x report-emacs-bug`.

>   We should try to get this into ELPA and advertise it in the manual:
>
>    https://github.com/Lindydancer/font-lock-studio

Agreed.

> On a more general note: When you use functions for font-locking, I often
> find myself in the position that I'm more or less writing a parser, and
> I could readily tell font-lock how to fontify several parts of the
> buffer with different faces. However, AFAICS, this is not possible with
> current font-lock?

You can't use the `font-lock-keywords` special syntax for that, but you
can still do it (by hand).  E.g.

    (defvar foo-font-lock-keywords
        `(("someregexp" (1 'foo-face) (2 'bar-face))
          (my-parsing-function)))

    (defun my-parsing-function (limit)
      ...
      (while ...
        (put-text-property x y 'face 'foobar-face)
        ...)
      ;; Return nil if you don't want font-lock to call you again right away
      ;; to "look for the next match".
      nil)


-- Stefan

Re: font lock with functions

[Dmitry Gutov]

On 31.03.2020 16:20, Stefan Monnier wrote:
>        (while ...
>          (put-text-property x y 'face 'foobar-face)
>          ...)
>        ;; Return nil if you don't want font-lock to call you again right away
>        ;; to "look for the next match".

I was going to suggesting this myself, but isn't this approach even more 
of a hack (i.e. undocumented feature)?

Re: font lock with functions

[Stefan Monnier]

>>        (while ...
>>          (put-text-property x y 'face 'foobar-face)
>>          ...)
>>        ;; Return nil if you don't want font-lock to call you again right away
>>        ;; to "look for the next match".
>
> I was going to suggesting this myself, but isn't this approach even more of
> a hack (i.e. undocumented feature)?

I don't see it as a hack, no.  It would be nice for font-lock to provide
a more direct way to get the same result, but it's what we have.


        Stefan

Re: font lock with functions

[Dmitry Gutov]

On 31.03.2020 17:56, Stefan Monnier wrote:
>>>         (while ...
>>>           (put-text-property x y 'face 'foobar-face)
>>>           ...)
>>>         ;; Return nil if you don't want font-lock to call you again right away
>>>         ;; to "look for the next match".
>> I was going to suggesting this myself, but isn't this approach even more of
>> a hack (i.e. undocumented feature)?
> I don't see it as a hack, no.  It would be nice for font-lock to provide
> a more direct way to get the same result, but it's what we have.

OK, but we should probably document this possibility better, then.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> When we know a new feature causes significant slowdown, we either fix
> it, or provide a way to disable or work around it.  The problem is, we
> don't always know there's slowdown, as it frequently happens only in
> specific rare use cases.  (John said some time ago we should have a
> benchmarking test suite, but I don't think anyone's working on it.)

https://elpa.gnu.org/packages/elisp-benchmarks.html ?

  Andrea

-- 
akrl@sdf.org

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: rms@gnu.org, acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Tue, 31 Mar 2020 16:05:08 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > When we know a new feature causes significant slowdown, we either fix
> > it, or provide a way to disable or work around it.  The problem is, we
> > don't always know there's slowdown, as it frequently happens only in
> > specific rare use cases.  (John said some time ago we should have a
> > benchmarking test suite, but I don't think anyone's working on it.)
> 
> https://elpa.gnu.org/packages/elisp-benchmarks.html ?

(Any idea why this isn't in the ELPA repository?)

This might be a beginning, but the issue at hand is not to benchmark
Emacs Lisp programs, it is to benchmark Emacs commands, so we could
make sure we don't have speed regressions.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: rms@gnu.org, acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 31 Mar 2020 16:05:08 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > When we know a new feature causes significant slowdown, we either fix
>> > it, or provide a way to disable or work around it.  The problem is, we
>> > don't always know there's slowdown, as it frequently happens only in
>> > specific rare use cases.  (John said some time ago we should have a
>> > benchmarking test suite, but I don't think anyone's working on it.)
>> 
>> https://elpa.gnu.org/packages/elisp-benchmarks.html ?
>
> (Any idea why this isn't in the ELPA repository?)

What do you mean?  I think it is (as external package tho).

> This might be a beginning, but the issue at hand is not to benchmark
> Emacs Lisp programs, it is to benchmark Emacs commands, so we could
> make sure we don't have speed regressions.

Yes is just a start.  I think everything is in Emacs core can be
classified as elisp implementation, I would not know where to draw a
line otherwise.

We could add benchmark categories if we start having a number of and we
want to better separate them.

  Andrea

-- 
akrl@sdf.org

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: rms@gnu.org, acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Tue, 31 Mar 2020 18:00:25 +0000
> 
> >> https://elpa.gnu.org/packages/elisp-benchmarks.html ?
> >
> > (Any idea why this isn't in the ELPA repository?)
> 
> What do you mean?  I think it is (as external package tho).

When I checkout the ELPA repository, I don't seem to have it.  Or did
I miss something?

> > This might be a beginning, but the issue at hand is not to benchmark
> > Emacs Lisp programs, it is to benchmark Emacs commands, so we could
> > make sure we don't have speed regressions.
> 
> Yes is just a start.  I think everything is in Emacs core can be
> classified as elisp implementation, I would not know where to draw a
> line otherwise.

I said "commands", and it wasn't an accident.  Those are user-visible
units, so IMO we should start by having benchmarks for frequent
commands.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: rms@gnu.org, acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
>> Date: Tue, 31 Mar 2020 18:00:25 +0000
>> 
>> >> https://elpa.gnu.org/packages/elisp-benchmarks.html ?
>> >
>> > (Any idea why this isn't in the ELPA repository?)
>> 
>> What do you mean?  I think it is (as external package tho).
>
> When I checkout the ELPA repository, I don't seem to have it.  Or did
> I miss something?

Is in the 'externals/elisp-benchmarks' branch.

The ELPA readme explain this (line 231).

>> > This might be a beginning, but the issue at hand is not to benchmark
>> > Emacs Lisp programs, it is to benchmark Emacs commands, so we could
>> > make sure we don't have speed regressions.
>> 
>> Yes is just a start.  I think everything is in Emacs core can be
>> classified as elisp implementation, I would not know where to draw a
>> line otherwise.
>
> I said "commands", and it wasn't an accident.  Those are user-visible
> units, so IMO we should start by having benchmarks for frequent
> commands.

What I'm saying is that I agree with you and I think these new
benchmarks should go in the 'elisp-benchmarks' package too.  Perhaps we
can classify these but I think is good to have one suite collecting all
benchmarks.

Andrea

-- 
akrl@sdf.org

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> What do you mean?  I think it is (as external package tho).
> When I checkout the ELPA repository, I don't seem to have it.  Or did
> I miss something?

You probably need to do

    make externals

> I said "commands", and it wasn't an accident.  Those are user-visible
> units, so IMO we should start by having benchmarks for frequent
> commands.

Indeed, none of the current benchmarks exercise the en/decoding code, or
the rendering code, or the process I/O code, ...



        Stefan

Re: font lock with functions

[Stephen Leake]

David Engster <deng@randomsample.de> writes:

> On a more general note: When you use functions for font-locking, I often
> find myself in the position that I'm more or less writing a parser, 

So use an actual parser; use wisi.

-- 
-- Stephe

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > When we know a new feature causes significant slowdown, we either fix
  > it, or provide a way to disable or work around it.  The problem is, we
  > don't always know there's slowdown, as it frequently happens only in
  > specific rare use cases.

1. If the slowdown cases are truly rare, I don't think they will bother
users very often.

2. I think we should be able to anticipate which changes _might_ cause
noticeable slowdowns.  They will be changes that will come into play
frequently in basic activities such as cursor motion, basic editing
operations, or redisplay.  We may not anticipate 100%, but let's start
trying.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: font lock with functions

[David Engster]

>> - It is very easy during testing to completely hang your Emacs session
>>   if one of the font-lock functions hangs. One usually needs to kill
>>   Emacs in this case.
>
> `jit-lock-debug-mode` might be helpful there.

Thanks. Did not know this one.

> You can't use the `font-lock-keywords` special syntax for that, but you
> can still do it (by hand).  E.g.

>       (while ...
>         (put-text-property x y 'face 'foobar-face)
>         ...)
>       ;; Return nil if you don't want font-lock to call you again right away
>       ;; to "look for the next match".
>       nil)

Oh, I see. This is very helpful.

-David

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Tue, 31 Mar 2020 22:07:38 -0400
> 
>   > When we know a new feature causes significant slowdown, we either fix
>   > it, or provide a way to disable or work around it.  The problem is, we
>   > don't always know there's slowdown, as it frequently happens only in
>   > specific rare use cases.
> 
> 1. If the slowdown cases are truly rare, I don't think they will bother
> users very often.

A frequent case is that the use case is rare on the average, but for
some user(s) it is not so rare.

> 2. I think we should be able to anticipate which changes _might_ cause
> noticeable slowdowns.  They will be changes that will come into play
> frequently in basic activities such as cursor motion, basic editing
> operations, or redisplay.  We may not anticipate 100%, but let's start
> trying.

I think we do try: patch review frequently has these aspects
discussed.  But it is a well known fact that humans are lousy
predictors of code speed, which is why the advice is always to profile
before deciding where to optimize.  I think we miss the slowdowns for
the same reason.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 1. If the slowdown cases are truly rare, I don't think they will bother
  > > users very often.

  > A frequent case is that the use case is rare on the average, but for
  > some user(s) it is not so rare.

Yes.  I don't think we can attain perfection, so I'm suggesting a guideline
so we can do better overall without too much extra work.

  > I think we do try: patch review frequently has these aspects
  > discussed.  But it is a well known fact that humans are lousy
  > predictors of code speed.

I'm not suggesting we try to _predict_ the effect on code speed,
rather than we have a guideline to identify the cases where _if_
the code gets slower it would be hard for a user to escape that.

What we would do, in those cases, is add a switch to turn off
the changes.



-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > What we would do, in those cases, is add a switch to turn off
 > the changes.

What we did, in those cases, was to remove the one major switch to turn
off those changes - 'open-paren-in-column-0-is-defun-start'.  With the
argument that people want to comment out larger regions of code by
simply putting comment delimiters at the beginning and end of that code.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Wed, 01 Apr 2020 22:32:12 -0400
> 
>   > I think we do try: patch review frequently has these aspects
>   > discussed.  But it is a well known fact that humans are lousy
>   > predictors of code speed.
> 
> I'm not suggesting we try to _predict_ the effect on code speed,
> rather than we have a guideline to identify the cases where _if_
> the code gets slower it would be hard for a user to escape that.
> 
> What we would do, in those cases, is add a switch to turn off
> the changes.

Without some efficient method of identifying the cases where this is
probable, I fear we will have to add such switches for almost every
non-trivial change (and for some trivial ones as well).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What we did, in those cases, was to remove the one major switch to turn
  > off those changes - 'open-paren-in-column-0-is-defun-start'.

Can you pleese explain what you mean?

What exactly was "removed"?  The variable
'open-paren-in-column-0-is-defun-start' seems to still exist, and its
default value seems to be t (my .emacs does not set it and it is t).


								  With the
  > argument that people want to comment out larger regions of code by
  > simply putting comment delimiters at the beginning and end of that code.

You have not said concretely which practice this refers to, but the
feature of special meaning for open-paren in column zero affects Emacs
editing commands, not the meaning of the program when executed.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I'm not suggesting we try to _predict_ the effect on code speed,
  > > rather than we have a guideline to identify the cases where _if_
  > > the code gets slower it would be hard for a user to escape that.
  > > 
  > > What we would do, in those cases, is add a switch to turn off
  > > the changes.

  > Without some efficient method of identifying the cases where this is
  > probable, I fear we will have to add such switches for almost every
  > non-trivial change (and for some trivial ones as well).

I am not suggesting that we band over backwards.  But it would be useful
to err on the side of letting people turn these features off.

We don't have to preserve those switches forever.  After the feature
is included in a release for several months, we can ask the users,
"Who finds it useful to disable the XYZ feature?"  If nobody reports a
real need for that switch, we can delete it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > Can you pleese explain what you mean?
 >
 > What exactly was "removed"?  The variable
 > 'open-paren-in-column-0-is-defun-start' seems to still exist, and its
 > default value seems to be t (my .emacs does not set it and it is t).
 >
 >
 > 								  With the
 >    > argument that people want to comment out larger regions of code by
 >    > simply putting comment delimiters at the beginning and end of that code.
 >
 > You have not said concretely which practice this refers to, but the
 > feature of special meaning for open-paren in column zero affects Emacs
 > editing commands, not the meaning of the program when executed.

'open-paren-in-column-0-is-defun-start', if non-nil, conceptually allows
progmodes to avoid scanning an entire buffer in order to get things like
syntax highlighting and code indenting right.  Rather, progmodes are
allowed to find the first or next paren in column zero wrt a given
position and base further decisions on the assumption that such a paren
is on the "top level" of its buffer.

Recent Emacsen either ignore that variable or silently reset it to nil
internally so it doesn't get into their way.  Their progmodes either
always scan an entire buffer from its beginning or use some elaborate,
fragile techniques to find such a top level position.  Moreover, our
underlying mechanism for syntax highlighting always marks the entire
rest of a buffer as dirty after every single editing change.  This has
the consequence that that entire part has to be continuously rescanned
when some of it is shown in another window.

The basic slowness of Emacs over the past years is a direct consequence
of that policy.  Most people don't care because they are using fast
processors that easily compensate the resulting overhead.  People with
old and slow hardware suffer from it.

So since you earlier asked for "a switch to turn off the changes" then
my answer is that such a switch already exists but has been deactivated.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> The basic slowness of Emacs over the past years is a direct consequence
> of that policy.

Do you have some kind of (concrete, statistical, circumstantial, ...)
evidence for that?


        Stefan "whose fastest machine has an Intel i3-4170"

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> The basic slowness of Emacs over the past years is a direct consequence
 >> of that policy.
 >
 > Do you have some kind of (concrete, statistical, circumstantial, ...)
 > evidence for that?

I'd gladly provide such evidence to prove or disprove my claim.  But
customizing 'open-paren-in-column-0-is-defun-start' has no effect here
any more and reconstructing its earlier semantics would consume more
resources than I can afford.

 >          Stefan "whose fastest machine has an Intel i3-4170"

martin "on an AMD Athlon 64 X2 Dual Core Processor 5000+"

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>> The basic slowness of Emacs over the past years is a direct consequence
>>> of that policy.
>> Do you have some kind of (concrete, statistical, circumstantial, ...)
>> evidence for that?
> I'd gladly provide such evidence to prove or disprove my claim.

I understand proving/disproving it is hard.

I'm really asking what are the signs that make you think it's
due to the lack of use of `open-paren-in-column-0-is-defun-start`.

> But customizing 'open-paren-in-column-0-is-defun-start' has no effect
> here any more and reconstructing its earlier semantics would consume
> more resources than I can afford.

Yes, I don't think there's much interested in going back to that.
OTOH if we could find specific cases where the old code was faster
(supposedly due to its use of `open-paren-in-column-0-is-defun-start`),
we could think about how to recover that performance.

E.g. `open-paren-in-column-0-is-defun-start` was used in
`forward-comment` was scanning comments backward to avoid having to use
`parse-partial-sexp` all the way from `point-min` in the worst case.
We replaced that with a parse cache (`syntax-ppss`) which seems to be
just as good in my experience (and significantly better in those
languages where open parens in column 0 are virtually non-existent).
If we can show that this replacement is not "just as good", there are
various options to improve it or replace it, but we'd need to know which
case(s) matter in order to design it.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

On Fri, Apr 03, 2020 at 08:59:54 +0200, martin rudalics wrote:
>  > Can you pleese explain what you mean?

>  > What exactly was "removed"?  The variable
>  > 'open-paren-in-column-0-is-defun-start' seems to still exist, and
>  > its default value seems to be t (my .emacs does not set it and it
>  > is t).


>  >	> With the argument that people want to comment out larger
>  >	> regions of code by simply putting comment delimiters at the
>  >	> beginning and end of that code.

>  > You have not said concretely which practice this refers to, but the
>  > feature of special meaning for open-paren in column zero affects Emacs
>  > editing commands, not the meaning of the program when executed.

> 'open-paren-in-column-0-is-defun-start', if non-nil, conceptually allows
> progmodes to avoid scanning an entire buffer in order to get things like
> syntax highlighting and code indenting right.  Rather, progmodes are
> allowed to find the first or next paren in column zero wrt a given
> position and base further decisions on the assumption that such a paren
> is on the "top level" of its buffer.

This assumption proved to be very problematic.  The fact is, people put
parentheses in column zero inside comments, and nothing we can say or do
will stop them.  Why should it?  these parentheses are perfectly valid
in so many languages.  Most notoriously was bug #22884, where there were
such parentheses in Emacs's own C sources.  And there were quite a lot
of ostensible bugs in CC Mode caused by these parentheses.

> Recent Emacsen either ignore that variable or silently reset it to nil
> internally so it doesn't get into their way.

The one place which matters where o-p-i-c-0-i-d-s used to be used was in
back_comment() in syntax.c.  This was removed for the above reasons.

> Their progmodes either always scan an entire buffer from its beginning
> or use some elaborate, fragile techniques to find such a top level
> position.

Why do you use the word "fragile" here?  Do you have examples of this
code failing?  It seems to me the current way is somewhat less fragile
that the o-p-i-c-0-i-d-s way.

> Moreover, our underlying mechanism for syntax highlighting always
> marks the entire rest of a buffer as dirty after every single editing
> change.

I have always been sceptical of the wisdom of this.  Why invalidate the
entire rest of the buffer, when a typical buffer change will cause at
most local changes to the fontification?  I think this can only be for
ease of coding in the font-lock functions.

Another approach would be somehow to divide a buffer into cells,
something like one cell per function.  A buffer change would then
invalidate the fontification only to the end of the current cell, not
EOB.  This would be faster, but somewhat complicated to implement.

> This has the consequence that that entire part has to be continuously
> rescanned when some of it is shown in another window.

> The basic slowness of Emacs over the past years is a direct consequence
> of that policy.  Most people don't care because they are using fast
> processors that easily compensate the resulting overhead.  People with
> old and slow hardware suffer from it.

How bad is this on your machine, really?

> So since you earlier asked for "a switch to turn off the changes" then
> my answer is that such a switch already exists but has been deactivated.

As already stated, it was deactivated for good reasons.  If we were to
bring it back, I think we would need to add heuristics around
paren-in-column-0 detection to the former rather crass 100% quitting of
back_comment when one is encountered.  I've considered this quite a bit
in the past, and can't conceive of anything both helpful and fast.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Paul Eggert]

On 4/2/20 11:59 PM, martin rudalics wrote:
 > The basic slowness of Emacs over the past years is a direct consequence
 > of that policy.

How many users is that true for? When Emacs is slow for me, it's usually because 
of very long lines. Is that the same issue, or a different one?

On 4/3/20 9:23 AM, martin rudalics wrote:
>  >          Stefan "whose fastest machine has an Intel i3-4170"
> 
> martin "on an AMD Athlon 64 X2 Dual Core Processor 5000+"

Is that the Black Edition 5000+ or the regular one? The Black Edition was quite 
the thing in 2007. :-)

-- Paul "on an AMD Phenom II X4 910e" circa 2010

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I understand proving/disproving it is hard.

We've been through this before.  What made me stumble over it this time
is Bug#40278 where I was not able to reproduce any flickering with the
OP's scenario because here scrolling buffers like dispextern.h and
xdisp.c takes some 10 seconds to complete.  I did not investigate what
precisely makes scrolling so slow.  The shear fact that this happens
exclusively with C buffers is enough evidence for me to point at CC mode
as the major culprit.

Now if we had left in 'open-paren-in-column-0-is-defun-start' and would
still be able to handle its non-nil value in some rudimentary way, with
all the bugs it may cause, we would have at least some evidence at our
hands to say, for example, that "going beyond some open paren is
dangerous in the case at hand" and might allow us to spot some
unrestricted movement in that direction.  But as it is, we have no clue,
no smoking gun, and every evidence about where cc-mode moves to when
analyzing the construct at point must be (re-)constructed in a rather
cumbersome way.

 > I'm really asking what are the signs that make you think it's
 > due to the lack of use of `open-paren-in-column-0-is-defun-start`.

We know that parsing a C buffer from the beginning, using something like
our 'parse-partial-sexp', is straightforward and cannot be responsible
for the scrolling problem I mention above.  So the cause must be in a
backtracking step performed by CC mode, either because something in the
state established by parsing does not allow to fully identify the
construct at where parsing stopped (unlikely IMO) or because scrolling
the buffer did overwrite some important part of the state established by
an earlier parsing step.  Note that I'm only talking about scrolling
such buffers, no buffer modifications that would invalidate an earlier
established state are involved here.

 >> But customizing 'open-paren-in-column-0-is-defun-start' has no effect
 >> here any more and reconstructing its earlier semantics would consume
 >> more resources than I can afford.
 >
 > Yes, I don't think there's much interested in going back to that.
 > OTOH if we could find specific cases where the old code was faster
 > (supposedly due to its use of `open-paren-in-column-0-is-defun-start`),
 > we could think about how to recover that performance.

'open-paren-in-column-0-is-defun-start' was dismounted in many steps
spread over many years.  We know that we can't go back there.

 > E.g. `open-paren-in-column-0-is-defun-start` was used in
 > `forward-comment` was scanning comments backward to avoid having to use
 > `parse-partial-sexp` all the way from `point-min` in the worst case.
 > We replaced that with a parse cache (`syntax-ppss`) which seems to be
 > just as good in my experience (and significantly better in those
 > languages where open parens in column 0 are virtually non-existent).

This goes without saying and is also a good argument in the case of C
header files like dispextern.h where the missing
'open-paren-in-column-0-is-defun-start' can be hardly the cause of the
poor performance.

 > If we can show that this replacement is not "just as good", there are
 > various options to improve it or replace it, but we'd need to know which
 > case(s) matter in order to design it.

'open-paren-in-column-0-is-defun-start' would be one knob to press when
things go wrong in some untraceable way.  It would provide a way to
narrow the current buffer down to the three nearest functions around
point and forget about the rest.  As it is, it has no effect at all.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >  > The basic slowness of Emacs over the past years is a direct consequence
 >  > of that policy.
 >
 > How many users is that true for? When Emacs is slow for me, it's usually because of very long lines. Is that the same issue, or a different one?

As mentioned in my answer to Stefan the most recent issue I noticed is
that mouse-wheel scrolling buffers for xdisp.c and dispextern.h consumes
100% of my CPU and still takes some ten seconds to complete.

 > Is that the Black Edition 5000+ or the regular one? The Black Edition was quite the thing in 2007. :-)

How would I find out?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > This assumption proved to be very problematic.  The fact is, people put
 > parentheses in column zero inside comments, and nothing we can say or do
 > will stop them.  Why should it?  these parentheses are perfectly valid
 > in so many languages.  Most notoriously was bug #22884, where there were
 > such parentheses in Emacs's own C sources.  And there were quite a lot
 > of ostensible bugs in CC Mode caused by these parentheses.

All these problems could have been cured easily: People who want such
parentheses would have set 'open-paren-in-column-0-is-defun-start' to
nil (or used a default value of nil) and everybody would have been
happy.  But throwing out the child with the bathwater by eliminating the
effect of that variable completely has left us no clue as to what makes
scrolling so slow in the present situation.

 >> Recent Emacsen either ignore that variable or silently reset it to nil
 >> internally so it doesn't get into their way.
 >
 > The one place which matters where o-p-i-c-0-i-d-s used to be used was in
 > back_comment() in syntax.c.  This was removed for the above reasons.

IIRC that was just the last nail in the variable's coffin.

 >> Their progmodes either always scan an entire buffer from its beginning
 >> or use some elaborate, fragile techniques to find such a top level
 >> position.
 >
 > Why do you use the word "fragile" here?  Do you have examples of this
 > code failing?  It seems to me the current way is somewhat less fragile
 > that the o-p-i-c-0-i-d-s way.

For me it's fragile because it does not allow me to easily hack it.  I
once gave up after trying in vain to implement a very elementary thing:
Make adjusting the first paragraph of the doc-string of an Elisp routine
not add anything to the first line of that doc-string and thus
invalidate our rules for writing doc-strings.  As things stand, I still
manually insert an empty line after the first one, adjust my paragraph,
and remove that empty line afterwards.

 >> Moreover, our underlying mechanism for syntax highlighting always
 >> marks the entire rest of a buffer as dirty after every single editing
 >> change.
 >
 > I have always been sceptical of the wisdom of this.  Why invalidate the
 > entire rest of the buffer, when a typical buffer change will cause at
 > most local changes to the fontification?  I think this can only be for
 > ease of coding in the font-lock functions.

I recall that more than a decade ago I tried to convince Stefan that a
buffer change that leaves the ppss unchanged should not invalidate the
rest of the buffer.  He initially liked the idea but didn't want to
implement it.

 > Another approach would be somehow to divide a buffer into cells,
 > something like one cell per function.  A buffer change would then
 > invalidate the fontification only to the end of the current cell, not
 > EOB.  This would be faster, but somewhat complicated to implement.

Why?

 > How bad is this on your machine, really?

I already cited the scrolling example in my other posts.  Scrolling
dispextern.h and xdisp.c each take 10 seconds for one turn of the wheel
and consume 100% of the allotted CPU here.

 >> So since you earlier asked for "a switch to turn off the changes" then
 >> my answer is that such a switch already exists but has been deactivated.
 >
 > As already stated, it was deactivated for good reasons.  If we were to
 > bring it back, I think we would need to add heuristics around
 > paren-in-column-0 detection to the former rather crass 100% quitting of
 > back_comment when one is encountered.  I've considered this quite a bit
 > in the past, and can't conceive of anything both helpful and fast.

I still don't understand why it had to be eliminated.  Defaulting it to
nil but respecting a non-nil value would have been completely sufficient
IMHO.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Martin.

On Sat, Apr 04, 2020 at 10:56:27 +0200, martin rudalics wrote:
>  > This assumption proved to be very problematic.  The fact is, people put
>  > parentheses in column zero inside comments, and nothing we can say or do
>  > will stop them.  Why should it?  these parentheses are perfectly valid
>  > in so many languages.  Most notoriously was bug #22884, where there were
>  > such parentheses in Emacs's own C sources.  And there were quite a lot
>  > of ostensible bugs in CC Mode caused by these parentheses.

> All these problems could have been cured easily: People who want such
> parentheses would have set 'open-paren-in-column-0-is-defun-start' to
> nil (or used a default value of nil) and everybody would have been
> happy.  But throwing out the child with the bathwater by eliminating the
> effect of that variable completely has left us no clue as to what makes
> scrolling so slow in the present situation.

Well, there aren't "people who want such parentheses".  They just
happen, without any conscious decision on the user's part.  Maybe, as
you say, changing o-p-i-c-0-i-d-s's default to nil would have been better.

But I'm a bit puzzled by the "scrolling so slow" bit.

[ .... ]

(I hope to get around to commenting on the rest of your post, sometime.)

> I already cited the scrolling example in my other posts.  Scrolling
> dispextern.h and xdisp.c each take 10 seconds for one turn of the wheel
> and consume 100% of the allotted CPU here.

There's something very strange going on, here.  If I do

    (time-it (scroll-up) (sit-for 0))

in a region of xdisp.c where the next screenfull hasn't yet been
fontified, it takes 0.029s on my machine.  For you it's taking 10s.
That's a factor of ~350.  Even taking into account me running on an
optimised build, and you (presumably) on a debug build, that gives a
factor of ~100 difference.

I appreciate you needing to run in a debug build, but what sort of
timing do you get on an optimised build?

I think you said you have a state of the art 2007 processor.  Mine is
from 2017.  Processing power did not speed up by a factor of 100 in that
period.

There is something very strange happening here.

[ .... ]

> I still don't understand why it
> [open-paren-in-column-0-is-defun-start] had to be eliminated.
> Defaulting it to nil but respecting a non-nil value would have been
> completely sufficient IMHO.

Maybe we didn't have your benchmark at the time the change was made,
which I think was around 2015.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > (time-it (scroll-up) (sit-for 0))

To avoid confusion - what is 'time-it'?  And please note that I did not
say that I ran 'scroll-up'.  I did a mouse wheel scroll as the OP of
that bug requested and unfortunately I don't know what mouse wheel
scrolling does here because C-h k on it just gives me

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
   help--read-key-sequence()
   byte-code("\300 C\207" [help--read-key-sequence] 1)
   call-interactively(describe-key nil nil)
   command-execute(describe-key)

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: eliz@gnu.org, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sat, 4 Apr 2020 13:10:43 +0200
> 
>  > (time-it (scroll-up) (sit-for 0))
> 
> To avoid confusion - what is 'time-it'?  And please note that I did not
> say that I ran 'scroll-up'.  I did a mouse wheel scroll as the OP of
> that bug requested and unfortunately I don't know what mouse wheel
> scrolling does here because C-h k on it just gives me
> 
> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
>    help--read-key-sequence()
>    byte-code("\300 C\207" [help--read-key-sequence] 1)
>    call-interactively(describe-key nil nil)
>    command-execute(describe-key)

It runs mwheel-scroll.  Not sure why it signals an error on your
system.

And if I turn the mouse wheel just one click, I don't see Emacs
lagging behind, either.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Martin.

On Sat, Apr 04, 2020 at 13:10:43 +0200, martin rudalics wrote:
>  > (time-it (scroll-up) (sit-for 0))

> To avoid confusion - what is 'time-it'?

    (defmacro time-it (&rest forms)
      "Time the running of a sequence of forms using `float-time'.
    Call like this: \"M-: (time-it (foo ...) (bar ...) ...)\"."
      `(let ((start (float-time)))
        ,@forms
        (- (float-time) start)))

> And please note that I did not say that I ran 'scroll-up'.  I did a
> mouse wheel scroll as the OP of that bug requested ....

Sorry.  I thought you were using poetic language when you wrote "one turn
of the wheel".  I didn't have the context of the bug in my head.

Just for comparison, would you please time (scroll-up) (sit-for 0) in
xdisp.c on your machine, to give us a better sense of the relative
speeds.

For what it's worth, I've just tried a fast mouse-wheel scroll in xdisp.c
in X Windows on my machine.  There was no noticeable lag.

> .... and unfortunately I don't know what mouse wheel scrolling does
> here because C-h k on it just gives me

> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
>    help--read-key-sequence()
>    byte-code("\300 C\207" [help--read-key-sequence] 1)
>    call-interactively(describe-key nil nil)
>    command-execute(describe-key)

This doesn't happen to me.  I get the information about mwheel-scroll.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello again, Martin.

On Sat, Apr 04, 2020 at 10:56:27 +0200, martin rudalics wrote:
>  > This assumption proved to be very problematic.  The fact is, people put
>  > parentheses in column zero inside comments, and nothing we can say or do
>  > will stop them.  Why should it?  these parentheses are perfectly valid
>  > in so many languages.  Most notoriously was bug #22884, where there were
>  > such parentheses in Emacs's own C sources.  And there were quite a lot
>  > of ostensible bugs in CC Mode caused by these parentheses.

> All these problems could have been cured easily: People who want such
> parentheses would have set 'open-paren-in-column-0-is-defun-start' to
> nil (or used a default value of nil) and everybody would have been
> happy.  But throwing out the child with the bathwater by eliminating the
> effect of that variable completely has left us no clue as to what makes
> scrolling so slow in the present situation.

>  >> Recent Emacsen either ignore that variable or silently reset it to nil
>  >> internally so it doesn't get into their way.

>  > The one place which matters where o-p-i-c-0-i-d-s used to be used was in
>  > back_comment() in syntax.c.  This was removed for the above reasons.

> IIRC that was just the last nail in the variable's coffin.

My impression was that this was the single place in the syntax routines
it was ever used, though back_comment is called from several places in
syntax.c.

[ .... ]

>  >> Moreover, our underlying mechanism for syntax highlighting always
>  >> marks the entire rest of a buffer as dirty after every single
>  >> editing change.

>  > I have always been sceptical of the wisdom of this.  Why invalidate
>  > the entire rest of the buffer, when a typical buffer change will
>  > cause at most local changes to the fontification?  I think this can
>  > only be for ease of coding in the font-lock functions.

> I recall that more than a decade ago I tried to convince Stefan that a
> buffer change that leaves the ppss unchanged should not invalidate the
> rest of the buffer.  He initially liked the idea but didn't want to
> implement it.

That sounds like a shame.

>  > Another approach would be somehow to divide a buffer into cells,
>  > something like one cell per function.  A buffer change would then
>  > invalidate the fontification only to the end of the current cell,
>  > not EOB.  This would be faster, but somewhat complicated to
>  > implement.

> Why?

Because we'd need somehow to detect when the cell boundaries change,
possibly coalescing two cells, possibly creating a new one.  This would
involve delicate coding in an after-change-function, possibly in a
before-c-f too.

We'd need to avoid prematurely coalescing two functions (for example,
when typing in an open paren in emacs-lisp-mode).

Also, new data structures would be needed in font-lock.

I'm not saying it couldn't be done, but it would take some effort, and
might be politically difficult, too.

[ .... ]

>  >> So since you earlier asked for "a switch to turn off the changes"
>  >> then my answer is that such a switch already exists but has been
>  >> deactivated.

>  > As already stated, it was deactivated for good reasons.  If we were
>  > to bring it back, I think we would need to add heuristics around
>  > paren-in-column-0 detection to the former rather crass 100% quitting
>  > of back_comment when one is encountered.  I've considered this quite
>  > a bit in the past, and can't conceive of anything both helpful and
>  > fast.

Maybe the following would do it: on every open paren at column 0 within a
string or comment, apply a text property called `syntax-not-defun-start'.
back_comment would skip past any such paren with that property.  It would
be the responsibility of the major mode code to add and remove the
property.  Again, this would mean a moderately intricate
after-change-function, possibly paired with a before-change-function.

This would perhaps remove the trouble from o-p-i-c-0-i-d-s and allow us
to put it back, but I'm not sure Stefan M. would like it.

> I still don't understand why it had to be eliminated.  Defaulting it to
> nil but respecting a non-nil value would have been completely sufficient
> IMHO.

I think you might well be right, there.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> I still don't understand why it had to be eliminated.  Defaulting it to
> nil but respecting a non-nil value would have been completely sufficient
> IMHO.

I can answer why it wasn't changed to nil:

1- o-p-i-c-0-i-d-s had an "invisible" effect on `forward-comment`, where
   the only visible effect was that sometimes it caused
   `forward-comment` to misbehave.  This use was just an optimization.
   Setting the var to nil avoided the misbehavior but could result in
   *much* worse performance in some cases (e.g. calling
   `parse-partial-sexp` from `point-min` once per line skipped when
   scanning text backward, so O(N*M) where N is the size of the buffer
   and M is the distance we're moving).  So just changing the var to nil
   was really not an option, it needed to be replaced by *another*
   optimization.  BTW, I still see not reason to link this other
   optimization, which relies on `syntax-ppss` with the scrolling
   performance problems you are seeing, especially if it's only visible
   in CC-mode buffers and not in things like Javascript (which have an
   almost identical syntax-table).

2- o-p-i-c-0-i-d-s also had (and still has) an effect on
   `beginning-of-defun` and this effect is supposed to be visible and
   predictable.  It's not just an optimization.  So changing the var to
   nil would have changed the behavior of `beginning-of-defun` and there
   was a very clear resistance to that.  Another reason not to just
   change its value to nil.

I also can't imagine how defaulting to nil "would have been completely
sufficient IMHO": setting the var to nil had been an option for decades
but it never enjoyed much popularity.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]

On 4/4/20 1:55 AM, martin rudalics wrote:
>  >  > The basic slowness of Emacs over the past years is a direct consequence
>  >  > of that policy.
>  >
>  > How many users is that true for? When Emacs is slow for me, it's usually 
> because of very long lines. Is that the same issue, or a different one?
> 
> As mentioned in my answer to Stefan the most recent issue I noticed is
> that mouse-wheel scrolling buffers for xdisp.c and dispextern.h consumes
> 100% of my CPU and still takes some ten seconds to complete.

I looked into this, and although it's no doubt fundamentally due to a slow 
algorithm, the slowness is exacerbated if you use -Og (which you appear to be 
using). Stefan's recent message hinted at this. I installed the attached into 
master to try to fix the -Og issue; please give it a try.

The slow algorithm should be fixed too, but I'm no expert there.

>  > Is that the Black Edition 5000+ or the regular one? The Black Edition was 
> quite the thing in 2007. :-)
> 
> How would I find out?

Your BIOS right after cycling power, I expect. It's not high priority to find 
out. As I vaguely recall the main advantage of the Black Edition is that you 
could overclock, and if you had a Black Edition my next suggestion was going to 
be a joke that you should overclock your ancient and slow CPU to make your Emacs 
faster....

[-- Attachment #2: gcc-Og-patch.diff --]
[-- Type: text/x-patch, Size: 2331 bytes --]

diff --git a/src/Makefile.in b/src/Makefile.in
index 552dd2e50a..dfd322553b 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -377,11 +377,14 @@ pdmp :=
 # Flags that might be in WARN_CFLAGS but are not valid for Objective C.
 NON_OBJC_CFLAGS = -Wignored-attributes -Wignored-qualifiers -Wopenmp-simd

+# Cajole GCC into inlining key ops even if it wouldn't normally.
+KEY_OPS_CFLAGS = $(if $(filter -Og,$(CFLAGS)),-DDEFINE_KEY_OPS_AS_MACROS)
+
 # -Demacs makes some files produce the correct version for use in Emacs.
 # MYCPPFLAGS is for by-hand Emacs-specific overrides, e.g.,
 # "make MYCPPFLAGS='-DDBUS_DEBUG'".
-EMACS_CFLAGS=-Demacs $(MYCPPFLAGS) -I. -I$(srcdir) \
-  -I$(lib) -I$(top_srcdir)/lib \
+EMACS_CFLAGS = -Demacs $(KEY_OPS_CFLAGS) $(MYCPPFLAGS) \
+  -I. -I$(srcdir) -I$(lib) -I$(top_srcdir)/lib \
   $(C_SWITCH_MACHINE) $(C_SWITCH_SYSTEM) $(C_SWITCH_X_SITE) \
   $(GNUSTEP_CFLAGS) $(CFLAGS_SOUND) $(RSVG_CFLAGS) $(IMAGEMAGICK_CFLAGS) \
   $(PNG_CFLAGS) $(LIBXML2_CFLAGS) $(DBUS_CFLAGS) \
diff --git a/src/lisp.h b/src/lisp.h
index f223814d8f..7fc3af992e 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -411,15 +411,19 @@ #define lisp_h_XHASH(a) XUFIXNUM_RAW (a)
 # define lisp_h_XTYPE(a) ((enum Lisp_Type) (XLI (a) & ~VALMASK))
 #endif

-/* When compiling via gcc -O0, define the key operations as macros, as
-   Emacs is too slow otherwise.  To disable this optimization, compile
-   with -DINLINING=false.  */
-#if (defined __NO_INLINE__ \
-     && ! defined __OPTIMIZE__ && ! defined __OPTIMIZE_SIZE__ \
-     && ! (defined INLINING && ! INLINING))
-# define DEFINE_KEY_OPS_AS_MACROS true
-#else
-# define DEFINE_KEY_OPS_AS_MACROS false
+/* When DEFINE_KEY_OPS_AS_MACROS, define key operations as macros to
+   cajole the compiler into inlining them; otherwise define them as
+   inline functions as this is cleaner and can be more efficient.
+   The default is true if the compiler is GCC-like and if function
+   inlining is disabled because the compiler is not optimizing or is
+   optimizing for size.  Otherwise the default is false.  */
+#ifndef DEFINE_KEY_OPS_AS_MACROS
+# if (defined __NO_INLINE__ \
+      && ! defined __OPTIMIZE__ && ! defined __OPTIMIZE_SIZE__)
+#  define DEFINE_KEY_OPS_AS_MACROS true
+# else
+#  define DEFINE_KEY_OPS_AS_MACROS false
+# endif
 #endif

 #if DEFINE_KEY_OPS_AS_MACROS

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Paul,

On 05.04.2020 03:07, Paul Eggert wrote:
> I looked into this, and although it's no doubt fundamentally due to a 
> slow algorithm, the slowness is exacerbated if you use -Og (which you 
> appear to be using). Stefan's recent message hinted at this. I installed 
> the attached into master to try to fix the -Og issue; please give it a try.

I seem to be getting a 2x perf improvement from this. Which is not bad.

Speaking of the patch, you might want to update the "Maybe in the year 
2020" sentence as well (again). :-)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 'open-paren-in-column-0-is-defun-start', if non-nil, conceptually allows
  > > progmodes to avoid scanning an entire buffer in order to get things like
  > > syntax highlighting and code indenting right.  Rather, progmodes are
  > > allowed to find the first or next paren in column zero wrt a given
  > > position and base further decisions on the assumption that such a paren
  > > is on the "top level" of its buffer.

  > This assumption proved to be very problematic.

Problematic for whom?  I don't see why.

						    The fact is, people put
  > parentheses in column zero inside comments, and nothing we can say or do
  > will stop them.  Why should it?

Why would we want to "stop them"?  That feature is/was a _convenience_
for users, a speedup.  If you don't appreciate the speedup, so you decide
not to arrange to get it, why should I argue with you?  I can still benefit
from that speedup in my projects. so I am happy and so are you.

				     these parentheses are perfectly valid
  > in so many languages.

Nobody said they were "invalid" in any language.  I think we are
miscommunicating.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> ...  I don't know what mouse wheel
 >> scrolling does here because C-h k on it just gives me
 >>
 >> Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
 >>     help--read-key-sequence()
 >>     byte-code("\300 C\207" [help--read-key-sequence] 1)
 >>     call-interactively(describe-key nil nil)
 >>     command-execute(describe-key)
 >
 > It runs mwheel-scroll.  Not sure why it signals an error on your
 > system.

I see.  C-h k apparently doesn't work with my separate minibuffer frame
setup.  Maybe some snafu with 'mouse-wheel--get-scroll-window'.

 > And if I turn the mouse wheel just one click, I don't see Emacs
 > lagging behind, either.

Right.  The effect appears only after a number of turns.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> To avoid confusion - what is 'time-it'?
 >
 >      (defmacro time-it (&rest forms)
 >        "Time the running of a sequence of forms using `float-time'.
 >      Call like this: \"M-: (time-it (foo ...) (bar ...) ...)\"."
 >        `(let ((start (float-time)))
 >          ,@forms
 >          (- (float-time) start)))

Thanks.  But for 'mwheel-scroll' I would have to construct an EVENT
first.  I'm not yet sure how to that.

 >> And please note that I did not say that I ran 'scroll-up'.  I did a
 >> mouse wheel scroll as the OP of that bug requested ....
 >
 > Sorry.  I thought you were using poetic language when you wrote "one turn
 > of the wheel".  I didn't have the context of the bug in my head.

Please have a look at Bug#40278 which is about mouse wheel scrolling, a
child frame, Windows 10 and dispextern.h.  People in that thread mainly
concentrated on the child frame but here on Debian all I needed to clog
my machine was to display dispextern.h and spin the mouse wheel.

 > Just for comparison, would you please time (scroll-up) (sit-for 0) in
 > xdisp.c on your machine, to give us a better sense of the relative
 > speeds.

With your 'time-it' it gives me 0.32201647758483887 here (seconds?).  I
have not managed yet to put scroll-up's into a loop and redisplay after
each call so I could simulate/time the effect of mouse wheel rolling.

 > For what it's worth, I've just tried a fast mouse-wheel scroll in xdisp.c
 > in X Windows on my machine.  There was no noticeable lag.

"fast" is probably not sufficient.  Here a few rolls suffice to freeze
the Emacs frame for about 10 secs.  Maybe on your machine you need more
rolls.  Try to roll fast until you reach the end of the buffer.

I meanwhile suspect that one culprit might be the progressive squaring
in 'mwheel-scroll'.  But I don't understand that yet and do not see it
documented well.  Also, setting 'mouse-wheel-scroll-amount' to '(1) does
not seem to make any difference either.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I looked into this, and although it's no doubt fundamentally due to a
 > slow algorithm, the slowness is exacerbated if you use -Og (which you
 > appear to be using).

I'm using '-O0 -g3 -no-pie' for builds that I use to debug issues and
'-O3 -no-pie' for builds that I use for daily work.  If I were to change
this (involving some 50 builds on various machines and partitions) I
would need to be convinced that the new options would not change
anything when running the debug builds and would not slow down anything
when running the optimized builds.

 > Stefan's recent message hinted at this. I
 > installed the attached into master to try to fix the -Og issue; please
 > give it a try.

 > The slow algorithm should be fixed too, but I'm no expert there.
 >
 >>  > Is that the Black Edition 5000+ or the regular one? The Black Edition was quite the thing in 2007. :-)
 >>
 >> How would I find out?

 > Your BIOS right after cycling power, I expect. It's not high priority
 > to find out. As I vaguely recall the main advantage of the Black
 > Edition is that you could overclock, and if you had a Black Edition my
 > next suggestion was going to be a joke that you should overclock your
 > ancient and slow CPU to make your Emacs faster....

It offers me an overclock mode with a CPU frequency of 200 MHz and a
PCIE Frequency of 100.  I also have M2 Boost disabled whatever that
means.  Still no idea whether that means that I have a Black Edition.
Maybe I'll install a program called CPU-Z (or CPUID ?) to find out.

martin

Inlining policy (was: emacs rendering comparisson between emacs23 and emacs26.3)

[Andrea Corallo]

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

Paul Eggert <eggert@cs.ucla.edu> writes:

> I looked into this, and although it's no doubt fundamentally due to a
> slow algorithm, the slowness is exacerbated if you use -Og (which you
> appear to be using). Stefan's recent message hinted at this. I
> installed the attached into master to try to fix the -Og issue; please
> give it a try.

Hi Paul,

what about having a new macro to control the key operations we want to
always have inlined at -O0 and -Og instead of falling back into defining
these as macros?

Something like the attached patch.

Going further I think we could also use this to take control of the
subset of functions we really want to have always inlined also when
optimizing.

Don't know maybe this solution of the always_inline attribute has been
already attempted, in case apologies for the noise.

  Andrea

--
akrl@sdf.org

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Introduce-KEY_OP_INLINE-for-key-operations-inlining.patch --]
[-- Type: text/x-diff, Size: 10707 bytes --]

From 902a46835d8c37089de407d847ce1259c477dc00 Mon Sep 17 00:00:00 2001
From: Andrea Corallo <akrl@sdf.org>
Date: Sun, 5 Apr 2020 10:09:43 +0100
Subject: [PATCH] Introduce KEY_OP_INLINE for key operations inlining.

2020-04-05  Andrea Corallo  <akrl@sdf.org>

	* src/lisp.h:
	(XLI, XIL, XLP, XPL, CHECK_FIXNUM, CHECK_SYMBOL, CHECK_TYPE)
	(CONSP, EQ, FLOATP, FIXNUMP, NILP, SET_SYMBOL_VAL)
	(SYMBOL_CONSTANT_P, SYMBOL_TRAPPED_WRITE_P, SYMBOL_VAL, SYMBOLP)
	(TAGGEDP, VECTORLIKEP, XCAR, XCDR, XCONS, XHASH, make_fixnum)
	(XFIXNUM_RAW, XTYPE): Do not define these as macro plus use
	KEY_OP_INLINE in the function definition.
	* src/conf_post.h (ATTRIBUTE_ALWAYS_INLINE): New macro.
	(KEY_OP_INLINE): New macro.
	* src/Makefile.in (DEFINE_KEY_OPS_AS_MACROS): Rename into
	FORCE_INLINE_KEY_OPS.
---
 src/Makefile.in |   2 +-
 src/conf_post.h |  14 +++++++
 src/lisp.h      | 107 ++++++++++++++----------------------------------
 3 files changed, 45 insertions(+), 78 deletions(-)

diff --git a/src/Makefile.in b/src/Makefile.in
index dfd322553b..51c200dc9e 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -378,7 +378,7 @@ pdmp :=
 NON_OBJC_CFLAGS = -Wignored-attributes -Wignored-qualifiers -Wopenmp-simd
 
 # Cajole GCC into inlining key ops even if it wouldn't normally.
-KEY_OPS_CFLAGS = $(if $(filter -Og,$(CFLAGS)),-DDEFINE_KEY_OPS_AS_MACROS)
+KEY_OPS_CFLAGS = $(if $(filter -Og,$(CFLAGS)),-DFORCE_INLINE_KEY_OPS)
 
 # -Demacs makes some files produce the correct version for use in Emacs.
 # MYCPPFLAGS is for by-hand Emacs-specific overrides, e.g.,
diff --git a/src/conf_post.h b/src/conf_post.h
index eb8fb18c00..441e19d829 100644
--- a/src/conf_post.h
+++ b/src/conf_post.h
@@ -240,6 +240,8 @@ #define NO_INLINE __attribute__((noinline))
 #define NO_INLINE
 #endif
 
+#define ATTRIBUTE_ALWAYS_INLINE __attribute__((always_inline))
+
 #if __has_attribute (externally_visible)
 #define EXTERNALLY_VISIBLE __attribute__((externally_visible))
 #else
@@ -446,6 +448,18 @@ #define ATTRIBUTE_MALLOC_SIZE(args) ATTRIBUTE_MALLOC ATTRIBUTE_ALLOC_SIZE (args)
 
 #endif
 
+/* Have key operations inlined at -O0 and -Og not to excessively kill
+   performance.  */
+#ifndef KEY_OP_INLINE
+# if (! defined __OPTIMIZE__ \
+      || (defined FORCE_INLINE_KEY_OPS && ! defined __NO_INLINE__ \
+	  && ! defined __OPTIMIZE_SIZE__))
+#  define KEY_OP_INLINE static inline ATTRIBUTE_UNUSED ATTRIBUTE_ALWAYS_INLINE
+# else
+#  define KEY_OP_INLINE INLINE
+# endif
+#endif
+
 /* 'int x UNINIT;' is equivalent to 'int x;', except it cajoles GCC
    into not warning incorrectly about use of an uninitialized variable.  */
 #if defined GCC_LINT || defined lint
diff --git a/src/lisp.h b/src/lisp.h
index 23ff89a977..168a496d58 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -411,53 +411,6 @@ #define lisp_h_XHASH(a) XUFIXNUM_RAW (a)
 # define lisp_h_XTYPE(a) ((enum Lisp_Type) (XLI (a) & ~VALMASK))
 #endif
 
-/* When DEFINE_KEY_OPS_AS_MACROS, define key operations as macros to
-   cajole the compiler into inlining them; otherwise define them as
-   inline functions as this is cleaner and can be more efficient.
-   The default is true if the compiler is GCC-like and if function
-   inlining is disabled because the compiler is not optimizing or is
-   optimizing for size.  Otherwise the default is false.  */
-#ifndef DEFINE_KEY_OPS_AS_MACROS
-# if (defined __NO_INLINE__ \
-      && ! defined __OPTIMIZE__ && ! defined __OPTIMIZE_SIZE__)
-#  define DEFINE_KEY_OPS_AS_MACROS true
-# else
-#  define DEFINE_KEY_OPS_AS_MACROS false
-# endif
-#endif
-
-#if DEFINE_KEY_OPS_AS_MACROS
-# define XLI(o) lisp_h_XLI (o)
-# define XIL(i) lisp_h_XIL (i)
-# define XLP(o) lisp_h_XLP (o)
-# define XPL(p) lisp_h_XPL (p)
-# define CHECK_FIXNUM(x) lisp_h_CHECK_FIXNUM (x)
-# define CHECK_SYMBOL(x) lisp_h_CHECK_SYMBOL (x)
-# define CHECK_TYPE(ok, predicate, x) lisp_h_CHECK_TYPE (ok, predicate, x)
-# define CONSP(x) lisp_h_CONSP (x)
-# define EQ(x, y) lisp_h_EQ (x, y)
-# define FLOATP(x) lisp_h_FLOATP (x)
-# define FIXNUMP(x) lisp_h_FIXNUMP (x)
-# define NILP(x) lisp_h_NILP (x)
-# define SET_SYMBOL_VAL(sym, v) lisp_h_SET_SYMBOL_VAL (sym, v)
-# define SYMBOL_CONSTANT_P(sym) lisp_h_SYMBOL_CONSTANT_P (sym)
-# define SYMBOL_TRAPPED_WRITE_P(sym) lisp_h_SYMBOL_TRAPPED_WRITE_P (sym)
-# define SYMBOL_VAL(sym) lisp_h_SYMBOL_VAL (sym)
-# define SYMBOLP(x) lisp_h_SYMBOLP (x)
-# define TAGGEDP(a, tag) lisp_h_TAGGEDP (a, tag)
-# define VECTORLIKEP(x) lisp_h_VECTORLIKEP (x)
-# define XCAR(c) lisp_h_XCAR (c)
-# define XCDR(c) lisp_h_XCDR (c)
-# define XCONS(a) lisp_h_XCONS (a)
-# define XHASH(a) lisp_h_XHASH (a)
-# if USE_LSB_TAG
-#  define make_fixnum(n) lisp_h_make_fixnum (n)
-#  define XFIXNUM_RAW(a) lisp_h_XFIXNUM_RAW (a)
-#  define XTYPE(a) lisp_h_XTYPE (a)
-# endif
-#endif
-
-
 /* Define the fundamental Lisp data structures.  */
 
 /* This is the set of Lisp data types.  If you want to define a new
@@ -718,25 +671,25 @@ definitely_will_not_unexec_p (void)
    if pointers differ in width from EMACS_INT; otherwise they are
    no-ops.  */
 
-INLINE EMACS_INT
+KEY_OP_INLINE EMACS_INT
 (XLI) (Lisp_Object o)
 {
   return lisp_h_XLI (o);
 }
 
-INLINE Lisp_Object
+KEY_OP_INLINE Lisp_Object
 (XIL) (EMACS_INT i)
 {
   return lisp_h_XIL (i);
 }
 
-INLINE void *
+KEY_OP_INLINE void *
 (XLP) (Lisp_Object o)
 {
   return lisp_h_XLP (o);
 }
 
-INLINE Lisp_Object
+KEY_OP_INLINE Lisp_Object
 (XPL) (void *p)
 {
   return lisp_h_XPL (p);
@@ -744,7 +697,7 @@ definitely_will_not_unexec_p (void)
 
 /* Extract A's type.  */
 
-INLINE enum Lisp_Type
+KEY_OP_INLINE enum Lisp_Type
 (XTYPE) (Lisp_Object a)
 {
 #if USE_LSB_TAG
@@ -758,13 +711,13 @@ definitely_will_not_unexec_p (void)
 /* True if A has type tag TAG.
    Equivalent to XTYPE (a) == TAG, but often faster.  */
 
-INLINE bool
+KEY_OP_INLINE bool
 (TAGGEDP) (Lisp_Object a, enum Lisp_Type tag)
 {
   return lisp_h_TAGGEDP (a, tag);
 }
 
-INLINE void
+KEY_OP_INLINE void
 (CHECK_TYPE) (int ok, Lisp_Object predicate, Lisp_Object x)
 {
   lisp_h_CHECK_TYPE (ok, predicate, x);
@@ -991,13 +944,13 @@ #define ROUNDUP(x, y) (POWER_OF_2 (y)					\
     ptrdiff_t size;
   };
 
-INLINE bool
+KEY_OP_INLINE bool
 (SYMBOLP) (Lisp_Object x)
 {
   return lisp_h_SYMBOLP (x);
 }
 
-INLINE struct Lisp_Symbol * ATTRIBUTE_NO_SANITIZE_UNDEFINED
+KEY_OP_INLINE struct Lisp_Symbol * ATTRIBUTE_NO_SANITIZE_UNDEFINED
 XSYMBOL (Lisp_Object a)
 {
   eassert (SYMBOLP (a));
@@ -1052,7 +1005,7 @@ c_symbol_p (struct Lisp_Symbol *sym)
     }
 }
 
-INLINE void
+KEY_OP_INLINE void
 (CHECK_SYMBOL) (Lisp_Object x)
 {
   lisp_h_CHECK_SYMBOL (x);
@@ -1146,14 +1099,14 @@ #define FIXNUM_OVERFLOW_P(i) \
 
 #if USE_LSB_TAG
 
-INLINE Lisp_Object
+KEY_OP_INLINE Lisp_Object
 (make_fixnum) (EMACS_INT n)
 {
   eassert (!FIXNUM_OVERFLOW_P (n));
   return lisp_h_make_fixnum_wrap (n);
 }
 
-INLINE EMACS_INT
+KEY_OP_INLINE EMACS_INT
 (XFIXNUM_RAW) (Lisp_Object a)
 {
   return lisp_h_XFIXNUM_RAW (a);
@@ -1173,7 +1126,7 @@ make_ufixnum (EMACS_INT n)
    the lisp_h_* macros are eventually removed.  */
 
 /* Make a fixnum representing the value of the low order bits of N.  */
-INLINE Lisp_Object
+KEY_OP_INLINE Lisp_Object
 make_fixnum (EMACS_INT n)
 {
   eassert (! FIXNUM_OVERFLOW_P (n));
@@ -1195,7 +1148,7 @@ make_fixnum (EMACS_INT n)
 /* Extract A's value as a signed integer.  Unlike XFIXNUM, this works
    on any Lisp object, although the resulting integer is useful only
    for things like hashing when A is not a fixnum.  */
-INLINE EMACS_INT
+KEY_OP_INLINE EMACS_INT
 XFIXNUM_RAW (Lisp_Object a)
 {
   EMACS_INT i = XLI (a);
@@ -1225,13 +1178,13 @@ make_ufixnum (EMACS_INT n)
 
 #endif /* ! USE_LSB_TAG */
 
-INLINE bool
+KEY_OP_INLINE bool
 (FIXNUMP) (Lisp_Object x)
 {
   return lisp_h_FIXNUMP (x);
 }
 
-INLINE EMACS_INT
+KEY_OP_INLINE EMACS_INT
 XFIXNUM (Lisp_Object a)
 {
   eassert (FIXNUMP (a));
@@ -1253,7 +1206,7 @@ XUFIXNUM (Lisp_Object a)
 }
 
 /* Return A's hash, which is in the range 0..INTMASK.  */
-INLINE EMACS_INT
+KEY_OP_INLINE EMACS_INT
 (XHASH) (Lisp_Object a)
 {
   return lisp_h_XHASH (a);
@@ -1270,7 +1223,7 @@ make_fixed_natnum (EMACS_INT n)
 
 /* Return true if X and Y are the same object.  */
 
-INLINE bool
+KEY_OP_INLINE bool
 (EQ) (Lisp_Object x, Lisp_Object y)
 {
   return lisp_h_EQ (x, y);
@@ -1399,13 +1352,13 @@ make_pointer_integer (void *p)
 };
 verify (GCALIGNED (struct Lisp_Cons));
 
-INLINE bool
+KEY_OP_INLINE bool
 (NILP) (Lisp_Object x)
 {
   return lisp_h_NILP (x);
 }
 
-INLINE bool
+KEY_OP_INLINE bool
 (CONSP) (Lisp_Object x)
 {
   return lisp_h_CONSP (x);
@@ -1417,7 +1370,7 @@ CHECK_CONS (Lisp_Object x)
   CHECK_TYPE (CONSP (x), Qconsp, x);
 }
 
-INLINE struct Lisp_Cons *
+KEY_OP_INLINE struct Lisp_Cons *
 (XCONS) (Lisp_Object a)
 {
   return lisp_h_XCONS (a);
@@ -1443,13 +1396,13 @@ xcdr_addr (Lisp_Object c)
 
 /* Use these from normal code.  */
 
-INLINE Lisp_Object
+KEY_OP_INLINE Lisp_Object
 (XCAR) (Lisp_Object c)
 {
   return lisp_h_XCAR (c);
 }
 
-INLINE Lisp_Object
+KEY_OP_INLINE Lisp_Object
 (XCDR) (Lisp_Object c)
 {
   return lisp_h_XCDR (c);
@@ -1652,7 +1605,7 @@ STRING_SET_CHARS (Lisp_Object string, ptrdiff_t newsize)
     Lisp_Object contents[FLEXIBLE_ARRAY_MEMBER];
   } GCALIGNED_STRUCT;
 
-INLINE bool
+KEY_OP_INLINE bool
 (VECTORLIKEP) (Lisp_Object x)
 {
   return lisp_h_VECTORLIKEP (x);
@@ -2174,7 +2127,7 @@ CHAR_TABLE_EXTRA_SLOTS (struct Lisp_Char_Table *ct)
 
 /* Value is name of symbol.  */
 
-INLINE Lisp_Object
+KEY_OP_INLINE Lisp_Object
 (SYMBOL_VAL) (struct Lisp_Symbol *sym)
 {
   return lisp_h_SYMBOL_VAL (sym);
@@ -2199,7 +2152,7 @@ SYMBOL_FWD (struct Lisp_Symbol *sym)
   return sym->u.s.val.fwd;
 }
 
-INLINE void
+KEY_OP_INLINE void
 (SET_SYMBOL_VAL) (struct Lisp_Symbol *sym, Lisp_Object v)
 {
   lisp_h_SET_SYMBOL_VAL (sym, v);
@@ -2250,7 +2203,7 @@ SYMBOL_INTERNED_IN_INITIAL_OBARRAY_P (Lisp_Object sym)
    i.e. it's a constant (e.g. nil, t, :keywords), or it has some
    watching functions.  */
 
-INLINE int
+KEY_OP_INLINE int
 (SYMBOL_TRAPPED_WRITE_P) (Lisp_Object sym)
 {
   return lisp_h_SYMBOL_TRAPPED_WRITE_P (sym);
@@ -2261,7 +2214,7 @@ SYMBOL_INTERNED_IN_INITIAL_OBARRAY_P (Lisp_Object sym)
    write to SYM, should also check whether there are any watching
    functions.  */
 
-INLINE int
+KEY_OP_INLINE int
 (SYMBOL_CONSTANT_P) (Lisp_Object sym)
 {
   return lisp_h_SYMBOL_CONSTANT_P (sym);
@@ -2806,7 +2759,7 @@ XBUFFER_OBJFWD (lispfwd a)
     } u;
   } GCALIGNED_STRUCT;
 
-INLINE bool
+KEY_OP_INLINE bool
 (FLOATP) (Lisp_Object x)
 {
   return lisp_h_FLOATP (x);
@@ -2970,7 +2923,7 @@ CHECK_LIST_END (Lisp_Object x, Lisp_Object y)
   CHECK_TYPE (NILP (x), Qlistp, y);
 }
 
-INLINE void
+KEY_OP_INLINE void
 (CHECK_FIXNUM) (Lisp_Object x)
 {
   lisp_h_CHECK_FIXNUM (x);
-- 
2.17.1

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Richard.

On Sat, Apr 04, 2020 at 23:12:49 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > > 'open-paren-in-column-0-is-defun-start', if non-nil,
>   > > conceptually allows progmodes to avoid scanning an entire buffer
>   > > in order to get things like syntax highlighting and code
>   > > indenting right.  Rather, progmodes are allowed to find the
>   > > first or next paren in column zero wrt a given position and base
>   > > further decisions on the assumption that such a paren is on the
>   > > "top level" of its buffer.

>   > This assumption proved to be very problematic.

> Problematic for whom?

Well, for me for a start.  ;-)  I had to deal with lots of bug reports,
the solution to which was "please put a \ before the ( in your comment".
One of these was in the copyright statement in the Emacs C sources (see
bug #22884).  This bug caused the typing of a "//" to take over 10
seconds to redisplay.

> I don't see why.

Suppose open-paren-in-column-0-is-defun-start is non-nil, and we have
something like:

    {
    /* foo
    (but bar
    */
     }
    ^
  point

.  From the indicated point, do (scan-lists (point) -1 1).

(i) scan-lists moves back and finds the comment end....
(ii) ... and thus calls back_comment.
(iii) back_comment finds the ( in column zero, and wrongly assumes this
  is the beginning of a function.
(iv) back_comment breaks off its scanning, returning "comment start not
  found".
(v) scan_lists carries on the scanning from just before the "*/", now
  blissfully unaware it's actually inside a comment.
(vi) scan_lists finds the (, and returns its position.

The indentation and/or fontification functions in CC Mode are now
confused.

>   > The fact is, people put parentheses in column zero inside
>   > comments, and nothing we can say or do will stop them.  Why should
>   > it?

> Why would we want to "stop them"?  That feature is/was a _convenience_
> for users, a speedup.  If you don't appreciate the speedup, so you decide
> not to arrange to get it, why should I argue with you?  I can still benefit
> from that speedup in my projects. so I am happy and so are you.

o-p-i-c-0-i-d-s is an arcane variable which very few users are aware of.
Their first knowledge of it typically came in my response to their bug
report saying put the backslash before the paren.  There are likely many
more users who have encountered wrong indentation/fontification who have
just not bothered reporting it.

>   > these parentheses are perfectly valid in so many languages.

> Nobody said they were "invalid" in any language.  I think we are
> miscommunicating.

I think I meant that, given their validity, it is up to us as Emacs
developers to arrange that they don't cause trouble, rather than
expecting our users to insert these obtrusive backslashes.

> -- 
> Dr Richard Stallman
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sun, 5 Apr 2020 10:44:24 +0200
> 
>  > And if I turn the mouse wheel just one click, I don't see Emacs
>  > lagging behind, either.
> 
> Right.  The effect appears only after a number of turns.

Can you tell how many times you turn the wheel until Emacs gets stuck?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Martin.

On Fri, Apr 03, 2020 at 08:59:54 +0200, martin rudalics wrote:

[ .... ]

> 'open-paren-in-column-0-is-defun-start', if non-nil, conceptually allows
> progmodes to avoid scanning an entire buffer in order to get things like
> syntax highlighting and code indenting right.  Rather, progmodes are
> allowed to find the first or next paren in column zero wrt a given
> position and base further decisions on the assumption that such a paren
> is on the "top level" of its buffer.

> Recent Emacsen either ignore that variable or silently reset it to nil
> internally so it doesn't get into their way.

Correction to my recent posts here.  o-p-i-c-0-i-d-s still exists and
can still work.

Currently, its operation is blocked by the variable
comment-use-syntax-ppss being non-nil.  I'm not sure what this variable
is for (it was introduced in:

    commit 14b95587520959c5b54356547a0a69932a9bb480
    Author: Stefan Monnier <monnier@iro.umontreal.ca>
    Date:   Tue Dec 12 23:03:00 2017 -0500

        * src/syntax.c (find_defun_start): Use syntax-ppss

        (syms_of_syntax): New variable comment-use-syntax-ppss.

), but setting it to nil does indeed reenable the o-p-i-c-0-i-d-s
behaviour.  :-)

[ .... ]

> So since you earlier asked for "a switch to turn off the changes" then
> my answer is that such a switch already exists but has been deactivated.

Please try setting comment-use-syntax-ppss to nil.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> I think I meant that, given their validity, it is up to us as Emacs
> developers to arrange that they don't cause trouble, rather than
> expecting our users to insert these obtrusive backslashes.

Indeed.  And the reverse is true as well: while it might be OK for
a user to put his open-parens in column 0 to help Emacs be more
responsive, it's not OK for Emacs to be too slow on files where there
are no such open-parens in column 0.  So Emacs has to use some other
mechanism anyway to avoid the slowdown in such files.  Once you have
this mechanism in place, the "open-parens in column 0" heuristic is just
not worth the trouble any more.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > Can you tell how many times you turn the wheel until Emacs gets stuck?

Five or six times where a "turn of the wheel" means to put the finger at
one end of the opening of the wheel and turn the wheel until the finger
is at the opposite end, approximately.  Each such turn seems to provoke
between 5 and 7 subsequent calls of 'mwheel-scroll' here so it
apparently takes 30 to 35 calls to get into those ~10 second freezes.
33 is a typical number I've seen, using a simple counter within
'mwheel-scroll'.  A typical value of the cumulative number of lines to
scroll (AMT in 'mwheel-scroll') then seems to be about 3000.  I would
have to run this more often to get better values.

The first five or so of such turns are seemingly processed normally.
After that, input apparently arrives faster than Emacs is able to
process it and no redisplay takes place until all input has been
consumed.

I cannot nearly simulate the problem using a simple loop that steadily
invokes say (forward-line 50) followed by a (sit-for 0).  50 such
invocations take 16.7 seconds here, 22.7 with (forward-line -50).  And
redisplay takes place as expected here.  So it's likely the mouse wheel
input mounting up that inhibits redisplay.

Do you at least see CPU activity significantly go up when you do such
mouse wheel scrolling?  I rarely use mouse wheel scrolling in Emacs so I
cannot say what is typical and what isn't.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > Please try setting comment-use-syntax-ppss to nil.

This has zero impact here.  Even when going to the end of the buffer
first and scrolling backward afterwards.  I have no idea yet what's
going on here and unless someone else can confirm my observations (Noam
was the only person so far who at least noticed that

   I can see it here, especially on scroll towards the beginning of the
   file (Emacs is maxing out the CPU while doing this).

but apparently nobody else has seen it) I have to dig into this myself.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Sun, 5 Apr 2020 17:50:33 +0200
> 
> I cannot nearly simulate the problem using a simple loop that steadily
> invokes say (forward-line 50) followed by a (sit-for 0).  50 such
> invocations take 16.7 seconds here, 22.7 with (forward-line -50).  And
> redisplay takes place as expected here.  So it's likely the mouse wheel
> input mounting up that inhibits redisplay.

Not forward-line, scroll-up.  That's what wheel.el calls.

> Do you at least see CPU activity significantly go up when you do such
> mouse wheel scrolling?

With what build?  With the -O0 build with --enable-checking, I don't
need the mouse: it's enough to lean on C-v and let the keyboard
auto-repeat do its job -- one execution unit of the CPU maxes out
after 5 to 10 C-v's.

But with a -O2 optimized production build, Emacs keeps up both when I
lean on C-v and when I turn the mouse wheel constantly.

Anyway, you don't need to work too hard to get me started on how slow
CC Mode is, especially when re-indentation is involved in addition to
font-lock.  Which is why I think we should move away of regexp-based
and SMIE-based engines towards fast parsers written in C.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 19:15, Eli Zaretskii wrote:
>> Do you at least see CPU activity significantly go up when you do such
>> mouse wheel scrolling?
> With what build?  With the -O0 build with --enable-checking, I don't
> need the mouse: it's enough to lean on C-v and let the keyboard
> auto-repeat do its job -- one execution unit of the CPU maxes out
> after 5 to 10 C-v's.
> 
> But with a -O2 optimized production build, Emacs keeps up both when I
> lean on C-v and when I turn the mouse wheel constantly.

Same here. And much as I like to see certain modes improved, I'm not 
sure this is only a performance problem. Because no matter how fast 
syntax highlighting is, you can lean on C-v faster.

IIRC, redisplay-dont-pause was supposed to help with things like this. 
But it doesn't (and it's obsolete), and apparently jit-lock does its 
thing anyway. This looks like the interesting part of the profile:

     - scroll-up                                              5657  73%
      - jit-lock-function                                     5444  70%
       - jit-lock-fontify-now                                 5439  70%
        + jit-lock--run-functions                             5431  70%
        + run-with-timer                                         5   0%

Do you guys want to try (setq jit-lock-defer-time 0)? It comes with a 
certain visual downside, though.

Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems 
like a good idea in general, so we might consider going further with it.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:05:00 +0300
> 
> Do you guys want to try (setq jit-lock-defer-time 0)? It comes with a 
> certain visual downside, though.
> 
> Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems 
> like a good idea in general, so we might consider going further with it.

No, I want fontification to work fast enough to not require these
measures.  They are retreat of a kind: we are giving up correctness
because we are annoyed by slowness.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 21:16, Eli Zaretskii wrote:
> No, I want fontification to work fast enough to not require these
> measures.  They are retreat of a kind: we are giving up correctness
> because we are annoyed by slowness.

But what's "fast enough"? Fontification that is faster than the signal 
rate of the keyboard?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:05:00 +0300
> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> IIRC, redisplay-dont-pause was supposed to help with things like this. 

Sorry, missed that part.

No, redisplay-dont-pause wasn't supposed to help when redisplay cannot
keep up.  It is for the opposite use case: when redisplay is so slow
it slows down processing of your commands, like typing, for example.
You'd then set it to nil to speed up command processing, while giving
up correct display at all times.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:20:36 +0300
> 
> On 05.04.2020 21:16, Eli Zaretskii wrote:
> > No, I want fontification to work fast enough to not require these
> > measures.  They are retreat of a kind: we are giving up correctness
> > because we are annoyed by slowness.
> 
> But what's "fast enough"?

To me, "fast enough" means I don't need to stop and wait to see the
result of my typing.  Which happens now when something I type requires
reindentation and refontification, at least in some cases.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 21:27, Eli Zaretskii wrote:
>>> No, I want fontification to work fast enough to not require these
>>> measures.  They are retreat of a kind: we are giving up correctness
>>> because we are annoyed by slowness.
>> But what's "fast enough"?
> To me, "fast enough" means I don't need to stop and wait to see the
> result of my typing.  Which happens now when something I type requires
> reindentation and refontification, at least in some cases.

OK, sure. But that's a different kind of scenario and benchmark.

Certainly a lower bar than having mouse-scrolling never lock up.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> apparently takes 30 to 35 calls to get into those ~10 second freezes.
[...]
> I cannot nearly simulate the problem using a simple loop that steadily
> invokes say (forward-line 50) followed by a (sit-for 0).  50 such
> invocations take 16.7 seconds here, 22.7 with (forward-line -50).

16s in one case a "10 seconds freeze": it sounds to me like you *do*
reproduce the problem.
The only difference is that with actual keyboard input pending the
redisplay is partly prevented which is what you perceive as "freeze",
but the real problem is the time it takes.

> But what's "fast enough"? Fontification that is faster than the signal rate
> of the keyboard?

Yes.  Of course, that depends not only on Emacs's code and the
optimization level, but also on your CPU and your keyboard
repetition setting.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 21:23, Eli Zaretskii wrote:
> No, redisplay-dont-pause wasn't supposed to help when redisplay cannot
> keep up.  It is for the opposite use case: when redisplay is so slow
> it slows down processing of your commands, like typing, for example.
> You'd then set it to nil to speed up command processing, while giving
> up correct display at all times.

What's the difference between redisplay being slow and it being unable 
to "keep up"? I don't understand, sorry.

And in this case, IIUC, jit-lock is part of redisplay. And it's slow 
enough for this to be noticeable.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:42:48 +0300
> 
> > To me, "fast enough" means I don't need to stop and wait to see the
> > result of my typing.  Which happens now when something I type requires
> > reindentation and refontification, at least in some cases.
> 
> OK, sure. But that's a different kind of scenario and benchmark.
> 
> Certainly a lower bar than having mouse-scrolling never lock up.

I'm not sure it is lower, because as I said an optimized Emacs doesn't
lock up when I scroll, either with C-v or the mouse.  But the problems
I described do happen in such builds.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> What's the difference between redisplay being slow and it being unable to
> "keep up"? I don't understand, sorry.

If there's already input waiting (because the next event already
arrived) at the start of redisplay, then redisplay is short-circuited.

Typically, in an auto-repeat circumstance, this will either not happen
or happen on a whole batch of repetition.  So we say "can't keep up"
because processing of one command (including redisplay) takes more time
than the interval until the next command.

> And in this case, IIUC, jit-lock is part of redisplay.

Yes, it often is when scrolling.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 21:55:39 +0300
> 
> On 05.04.2020 21:23, Eli Zaretskii wrote:
> > No, redisplay-dont-pause wasn't supposed to help when redisplay cannot
> > keep up.  It is for the opposite use case: when redisplay is so slow
> > it slows down processing of your commands, like typing, for example.
> > You'd then set it to nil to speed up command processing, while giving
> > up correct display at all times.
> 
> What's the difference between redisplay being slow and it being unable 
> to "keep up"? I don't understand, sorry.

That's not the important difference.  The important difference is that
you want the input processed faster, without waiting for slow
redisplay.

> And in this case, IIUC, jit-lock is part of redisplay. And it's slow 
> enough for this to be noticeable.

It isn't jit-lock that's slow, it's the fontification functions it
calls.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  rudalics@gmx.at,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  acm@muc.de
> Date: Sun, 05 Apr 2020 15:03:14 -0400
> 
> > What's the difference between redisplay being slow and it being unable to
> > "keep up"? I don't understand, sorry.
> 
> If there's already input waiting (because the next event already
> arrived) at the start of redisplay, then redisplay is short-circuited.

Only if you call 'redisplay' from Lisp explicitly.

"Normal" redisplay will probe pending input when it is going to send
the output to the glass, and will not do so if redisplay-don't-pause
is set to nil and there's input available.  IOW, whether input
available is checked not at the start of redisplay, but somewhere in
its middle.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[andrés ramírez]

Hi. 
>> What's the difference between redisplay being slow and it being
>> unable to "keep up"? I don't understand, sorry.

Eli> That's not the important difference.  The important difference is
Eli> that you want the input processed faster, without waiting for
Eli> slow redisplay.

I remember the other day someone on this list. Having a very similar
issue as described above. Eli mentioned something about his graphic
card. But. It happens to me, when I have mesa-git with glamor
enabled. Whet I create an
emacs lucid frame and move to the scratch buffer and I write the
character 'w' (and also other identified characters but 'w' is the worst
case) that character is duplicated and also emacs lags for a few
seconds. It happens also with emacs23. On those cases I fire up qemacs or
xemacs or sxemacs those do not have the same issue.

Best Regards

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: andrés ramírez <rrandresf@gmail.com>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,
> 	rudalics@gmx.at,
> 	acm@muc.de,
> 	rms@gnu.org,
> 	emacs-devel@gnu.org
> Date: Sun, 05 Apr 2020 19:16:05 +0000
> 
> Whet I create an
> emacs lucid frame and move to the scratch buffer and I write the
> character 'w' (and also other identified characters but 'w' is the worst
> case) that character is duplicated and also emacs lags for a few
> seconds.

I suggest to run Emacs under perf and see what processing takes most
of those few seconds.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 22:03, Stefan Monnier wrote:
>> What's the difference between redisplay being slow and it being unable to
>> "keep up"? I don't understand, sorry.
> If there's already input waiting (because the next event already
> arrived) at the start of redisplay, then redisplay is short-circuited.
> 
> Typically, in an auto-repeat circumstance, this will either not happen
> or happen on a whole batch of repetition.  So we say "can't keep up"
> because processing of one command (including redisplay) takes more time
> than the interval until the next command.

This agrees with my previous understanding.

>> And in this case, IIUC, jit-lock is part of redisplay.
> Yes, it often is when scrolling.

So would you agree that short-circuiting and skipping jit-lock while 
somebody leans on C-v is a good idea?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 22:03, Eli Zaretskii wrote:
>>> To me, "fast enough" means I don't need to stop and wait to see the
>>> result of my typing.  Which happens now when something I type requires
>>> reindentation and refontification, at least in some cases.
>> OK, sure. But that's a different kind of scenario and benchmark.
>>
>> Certainly a lower bar than having mouse-scrolling never lock up.
> I'm not sure it is lower, because as I said an optimized Emacs doesn't
> lock up when I scroll, either with C-v or the mouse.  But the problems
> I described do happen in such builds.

It's lower because when you're typing you're usually typing something 
meaningful (meaning limited input rate), and the beginning of the window 
usually stays in place (which speeds up re-fontification).

Whereas when you're scrolling, you can scroll as fast as your keyboard 
allows you, and you're triggering fontification of a whole new 
screen-ful every keystroke. And this is closer to Martin's situation, 
AFAICT.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 22:06, Eli Zaretskii wrote:

>> What's the difference between redisplay being slow and it being unable
>> to "keep up"? I don't understand, sorry.
> 
> That's not the important difference.  The important difference is that
> you want the input processed faster, without waiting for slow
> redisplay.

Two chief ways to do that are:

- To work faster
- To do less work

Ideally, we should take advantage of both. Skipping parts of redisplay 
could help with #2.

>> And in this case, IIUC, jit-lock is part of redisplay. And it's slow
>> enough for this to be noticeable.
> 
> It isn't jit-lock that's slow, it's the fontification functions it
> calls.

Yes, obviously.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Apr 05, 2020 at 21:05:00 +0300, Dmitry Gutov wrote:
> On 05.04.2020 19:15, Eli Zaretskii wrote:
> >> Do you at least see CPU activity significantly go up when you do such
> >> mouse wheel scrolling?
> > With what build?  With the -O0 build with --enable-checking, I don't
> > need the mouse: it's enough to lean on C-v and let the keyboard
> > auto-repeat do its job -- one execution unit of the CPU maxes out
> > after 5 to 10 C-v's.

> > But with a -O2 optimized production build, Emacs keeps up both when I
> > lean on C-v and when I turn the mouse wheel constantly.

> Same here. And much as I like to see certain modes improved, I'm not 
> sure this is only a performance problem. Because no matter how fast 
> syntax highlighting is, you can lean on C-v faster.

> IIRC, redisplay-dont-pause was supposed to help with things like this. 
> But it doesn't (and it's obsolete), and apparently jit-lock does its 
> thing anyway. This looks like the interesting part of the profile:

>      - scroll-up                                              5657  73%
>       - jit-lock-function                                     5444  70%
>        - jit-lock-fontify-now                                 5439  70%
>         + jit-lock--run-functions                             5431  70%
>         + run-with-timer                                         5   0%

> Do you guys want to try (setq jit-lock-defer-time 0)? It comes with a 
> certain visual downside, though.

> Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems 
> like a good idea in general, so we might consider going further with it.

Try this:

diff --git a/src/window.c b/src/window.c
index 8cdad27b66..b23ae41aba 100644
--- a/src/window.c
+++ b/src/window.c
@@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
 
   wset_redisplay (XWINDOW (window));
 
-  if (whole && Vfast_but_imprecise_scrolling)
+  if (Vfast_but_imprecise_scrolling
+      && (whole
+          || EQ (Vthis_command, Qmwheel_scroll)))
     specbind (Qfontification_functions, Qnil);
 
   /* On GUI frames, use the pixel-based version which is much slower
@@ -8173,6 +8175,7 @@ syms_of_window (void)
   DEFSYM (Qmode_line_format, "mode-line-format");
   DEFSYM (Qheader_line_format, "header-line-format");
   DEFSYM (Qtab_line_format, "tab-line-format");
+  DEFSYM (Qmwheel_scroll, "mwheel-scroll");
 
   DEFVAR_LISP ("temp-buffer-show-function", Vtemp_buffer_show_function,
 	       doc: /* Non-nil means call as function to display a help buffer.


This eliminates all the delay I formerly saw while mouse scrolling as
fast as possibly.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Paul Eggert]

On 4/5/20 1:46 AM, martin rudalics wrote:
>  > I looked into this, and although it's no doubt fundamentally due to a
>  > slow algorithm, the slowness is exacerbated if you use -Og (which you
>  > appear to be using).
> 
> I'm using '-O0 -g3 -no-pie' for builds that I use to debug issues and
> '-O3 -no-pie' for builds that I use for daily work.

Ah, my mistake. I thought you were using -Og (some other people were). Oh well, 
my patch does help those other people a bit.

> It offers me an overclock mode with a CPU frequency of 200 MHz and a
> PCIE Frequency of 100.  I also have M2 Boost disabled whatever that
> means.  Still no idea whether that means that I have a Black Edition.

I wouldn't overclock as it decreases reliability enough that it's not worth the 
hassle; my suggestion to overclock was meant more as a joke. Getting a newer 
computer should be a much better investment of your time/cash. If you can't 
afford one perhaps we could talk offline about the possibilities.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 05.04.2020 22:57, Alan Mackenzie wrote:
> Try this:
> 
> diff --git a/src/window.c b/src/window.c
> index 8cdad27b66..b23ae41aba 100644
> --- a/src/window.c
> +++ b/src/window.c
> @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
>   
>     wset_redisplay (XWINDOW (window));
>   
> -  if (whole && Vfast_but_imprecise_scrolling)
> +  if (Vfast_but_imprecise_scrolling
> +      && (whole
> +          || EQ (Vthis_command, Qmwheel_scroll)))
>       specbind (Qfontification_functions, Qnil);
>   
>     /* On GUI frames, use the pixel-based version which is much slower
> @@ -8173,6 +8175,7 @@ syms_of_window (void)
>     DEFSYM (Qmode_line_format, "mode-line-format");
>     DEFSYM (Qheader_line_format, "header-line-format");
>     DEFSYM (Qtab_line_format, "tab-line-format");
> +  DEFSYM (Qmwheel_scroll, "mwheel-scroll");

You probably meant to address Martin, but I can just say that while this 
will almost certainly work, it doesn't address the scenario of flooring 
C-v. Or M-v. Etc. Which seem equivalent.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>> And in this case, IIUC, jit-lock is part of redisplay.
>> Yes, it often is when scrolling.
> So would you agree that short-circuiting and skipping jit-lock while
> somebody leans on C-v is a good idea?

It sounds like a trick question, because what I just described says that
it is what already happens, more or less, and I'm not sure which part of
the difference you're thinking of.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 5 Apr 2020 22:25:16 +0300
> 
> >> Certainly a lower bar than having mouse-scrolling never lock up.
> > I'm not sure it is lower, because as I said an optimized Emacs doesn't
> > lock up when I scroll, either with C-v or the mouse.  But the problems
> > I described do happen in such builds.
> 
> It's lower because when you're typing you're usually typing something 
> meaningful (meaning limited input rate), and the beginning of the window 
> usually stays in place (which speeds up re-fontification).
> 
> Whereas when you're scrolling, you can scroll as fast as your keyboard 
> allows you, and you're triggering fontification of a whole new 
> screen-ful every keystroke.

And yet Emacs does cope with that "higher bar", whereas while I type I
sometimes see unbearably slow responses.  Which in my book doesn't fit
its being "lower bar".

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Sun, 5 Apr 2020 19:57:53 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
>  rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> Try this:
> 
> diff --git a/src/window.c b/src/window.c
> index 8cdad27b66..b23ae41aba 100644
> --- a/src/window.c
> +++ b/src/window.c
> @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
>  
>    wset_redisplay (XWINDOW (window));
>  
> -  if (whole && Vfast_but_imprecise_scrolling)
> +  if (Vfast_but_imprecise_scrolling
> +      && (whole
> +          || EQ (Vthis_command, Qmwheel_scroll)))
>      specbind (Qfontification_functions, Qnil);

That's just to make a point, no?  You don't seriously propose such a
change, right?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Recent Emacsen either ignore that variable or silently reset it to nil
  > internally so it doesn't get into their way.  Their progmodes either
  > always scan an entire buffer from its beginning or use some elaborate,
  > fragile techniques to find such a top level position.  Moreover, our
  > underlying mechanism for syntax highlighting always marks the entire
  > rest of a buffer as dirty after every single editing change.  This has
  > the consequence that that entire part has to be continuously rescanned
  > when some of it is shown in another window.

Does anyone disagree with this specific factual claim?

  > The basic slowness of Emacs over the past years is a direct consequence
  > of that policy.

Does anyone disagree with this general claim?

  > So since you earlier asked for "a switch to turn off the changes" then
  > my answer is that such a switch already exists but has been deactivated.

If those claims are true, I suggest reactivating it.
Why impose the fast=machine mode on everyone?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > This assumption proved to be very problematic.

  > > Problematic for whom?

  > Well, for me for a start.

If it causes a problem only for you personally as a maintainer, I am
sure we could find a way to spare you from spending time on this.  But
we are miscommunicating completely.

I assumed you meant "problematically for some people who are doing
some kinds of things with Emacs", and I would like to know what kinds
of things they are.  The answer that is crucial for deciding whether
this problem ought to have been "fixed" or not.

  > Suppose open-paren-in-column-0-is-defun-start is non-nil, and we have
  > something like:

  >     {
  >     /* foo
  >     (but bar
  >     */
  >      }
  >     ^
  >   point

Now I see the scenario.  Indeed, with
open-paren-in-column-0-is-defun-start non-nil, some users would regard
the resulting behavior as a problem.

I do not see how this constitutes a reason to deliminate that option.
Why is that better than the obvious response: to tell the user,

   Either delate that open-paren, or indent it, or set
   open-paren-in-column-0-is-defun-start to nil.

Does someone want to present an argument leading to the conclusion
that it was better to deactivate that variable?

  > I think I meant that, given their validity, it is up to us as Emacs
  > developers to arrange that they don't cause trouble, rather than
  > expecting our users to insert these obtrusive backslashes.

Why assume that the one recommended way to edit the buffer to avoid
the confusion is inserting a backslash?


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > This assumption proved to be very problematic.

  > > Problematic for whom?

  > Well, for me for a start.

If it causes a problem only for you personally as a maintainer, I am
sure we could find a way to spare you from spending time on this.  But
I think we are miscommunicating over this.

I understood you to mean "problematic for some people who are doing
some kinds of things with Emacs", so I would like to know what kinds
of things they are.  The answer that is crucial for deciding whether
this problem ought to have been "fixed" or not.

But is that what you meant?  Or did you mean something else?

I get the impression that your analysis of the issue is based on
assuming that open-paren-in-column-0-is-defun-start defaults to t.
What if it defaulted to nil?  Would that eliminate the problem
you're concerned about?  I think so.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Indeed.  And the reverse is true as well: while it might be OK for
  > a user to put his open-parens in column 0 to help Emacs be more
  > responsive, it's not OK for Emacs to be too slow on files where there
  > are no such open-parens in column 0.  So Emacs has to use some other
  > mechanism anyway to avoid the slowdown in such files.

The mechanism it has now is effective only on fast machines.

							   Once you have
  > this mechanism in place, the "open-parens in column 0" heuristic is just
  > not worth the trouble any more.

If and when such a mechanism is in place for us free software
activists, we might agree that open-paren-in-column-0-is-defun-start
is not worth while any more.  But it appears that has not happened
yet, so I think that the open-paren-in-column-0-is-defun-start guided
heuristic IS still worth while for us.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Ravine Var]

martin rudalics <rudalics@gmx.at> writes:
> I meanwhile suspect that one culprit might be the progressive squaring
> in 'mwheel-scroll'.  But I don't understand that yet and do not see it
> documented well.  Also, setting 'mouse-wheel-scroll-amount' to '(1) does
> not seem to make any difference either.

I have set mouse-wheel-progressive-speed to nil to get somewhat
reasonable scrolling behavior. Perhaps this tweak can make a difference
in your machine...

-R

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I appreciate you needing to run in a debug build,

I run a debug build only only when I try to investigate an error
reported on bug-gnu-emacs or when I have to run Emacs under GDB.  Maybe
optimized builds can deliver acceptable behavior now when run under GDB.
But the last time I tried, behavior was not acceptable.  And what about
our advice that users should run Emacs under GDB all the time to detect
the cause of a crash?

 > but what sort of
 > timing do you get on an optimised build?

An optimized build runs reasonably fast here so I didn't bother so far.

 > I think you said you have a state of the art 2007 processor.  Mine is
 > from 2017.  Processing power did not speed up by a factor of 100 in that
 > period.

That's why I wondered that nobody else sees this problem.  Does really
nobody run a debug build nowadays?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I also can't imagine how defaulting to nil "would have been completely
 > sufficient IMHO": setting the var to nil had been an option for decades
 > but it never enjoyed much popularity.

Defaulting to nil would have meant that people who want "correct" syntax
under all conditions would never have to care about this.  OTOH, people
who did care about performance more would have had the possibility to
set this option to non-nil and get the necessary feedback via
font-locking (highlighting open parens in column zero red when they are
in a string or a comment) when they accidentally inserted such a paren
or scrolled into it.  Why was that highlighting functionality removed?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> I cannot nearly simulate the problem using a simple loop that steadily
 >> invokes say (forward-line 50) followed by a (sit-for 0).  50 such
 >> invocations take 16.7 seconds here, 22.7 with (forward-line -50).  And
 >> redisplay takes place as expected here.  So it's likely the mouse wheel
 >> input mounting up that inhibits redisplay.
 >
 > Not forward-line, scroll-up.  That's what wheel.el calls.

I know.  But I wanted to not relate the behavior to things like the
window configuration.

 >> Do you at least see CPU activity significantly go up when you do such
 >> mouse wheel scrolling?
 >
 > With what build?  With the -O0 build with --enable-checking, I don't
 > need the mouse: it's enough to lean on C-v and let the keyboard
 > auto-repeat do its job -- one execution unit of the CPU maxes out
 > after 5 to 10 C-v's.

OK.  Then people see the problem and it's just that they do not build
with -O0.

 > But with a -O2 optimized production build, Emacs keeps up both when I
 > lean on C-v and when I turn the mouse wheel constantly.

Same here.

 > Anyway, you don't need to work too hard to get me started on how slow
 > CC Mode is, especially when re-indentation is involved in addition to
 > font-lock.  Which is why I think we should move away of regexp-based
 > and SMIE-based engines towards fast parsers written in C.

As long as we are not there (and IMO even after that) Emacs should be
able to do its SMIE parsing in a practical way: Restrict backward and
forward parsing to the smallest reasonable code fragment around point.
And reasonable would mean the smallest enclosing fragment delimited by
two parens in column zero it can find in either direction (which can be
still quite large when viewing functions like redisplay_internal).

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> I cannot nearly simulate the problem using a simple loop that steadily
 >> invokes say (forward-line 50) followed by a (sit-for 0).  50 such
 >> invocations take 16.7 seconds here, 22.7 with (forward-line -50).
 >
 > 16s in one case a "10 seconds freeze": it sounds to me like you *do*
 > reproduce the problem.

Not really.  In the 16s case redisplay kicks in and displays the buffer
250 times.  In the 10s case redisplay stops after five times.

 > The only difference is that with actual keyboard input pending the
 > redisplay is partly prevented which is what you perceive as "freeze",
 > but the real problem is the time it takes.

What does get fontified when redisplay freezes?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > I have set mouse-wheel-progressive-speed to nil to get somewhat
 > reasonable scrolling behavior. Perhaps this tweak can make a difference
 > in your machine...

It doesn't help.  I already tried setting ‘mouse-wheel-scroll-amount’ to
'(1) which should do about the same.  The interference of these two
options is not yet clear to me.

Thanks, martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Richard.

On Sun, Apr 05, 2020 at 22:36:22 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]

>   > >   > This assumption proved to be very problematic.

>   > > Problematic for whom?

>   > Well, for me for a start.

> If it causes a problem only for you personally as a maintainer, I am
> sure we could find a way to spare you from spending time on this.  But
> we are miscommunicating completely.

> I assumed you meant "problematically for some people who are doing
> some kinds of things with Emacs", and I would like to know what kinds
> of things they are.  The answer that is crucial for deciding whether
> this problem ought to have been "fixed" or not.

Yes, that is what I meant.  The "kind of things" is nothing more than
writing comments in C (etc.) source code.  Sooner or later, such a
comment has a parenthesis in it, and sooner or later (e.g. by M-q), this
will end up in column 0, when (as is often the case) the user has no
" *" comment prefix.

>   > Suppose open-paren-in-column-0-is-defun-start is non-nil, and we have
>   > something like:

>   >     {
>   >     /* foo
>   >     (but bar
>   >     */
>   >      }
>   >     ^
>   >   point

> Now I see the scenario.  Indeed, with
> open-paren-in-column-0-is-defun-start non-nil, some users would regard
> the resulting behavior as a problem.

[ Please do M-: (setq comment-use-syntax-ppss nil) to reenable
o-p-i-c-0-i-d-s's action. ]

I think "some" is an understatement.  With show-paren-mode enabled, and
with point after the }, it visibly mismatches the ( rather than matching
the {.

If this was all that happened, it might not be too bad.  But
(forward-comment -1) fails on such a beast.  scan-lists (backwards)
fails, too.  We're talking about low level Emacs primitives being buggy.
This is surely worse than higher level functions being buggy.

> I do not see how this constitutes a reason to deliminate that option.
> Why is that better than the obvious response: to tell the user,

>    Either delate that open-paren, or indent it, or set
>    open-paren-in-column-0-is-defun-start to nil.

Because it is highly non-trivial to detect this happening in the code.
To insert code to detect this would cause a slow down, likely a big slow
down.

> Does someone want to present an argument leading to the conclusion
> that it was better to deactivate that variable?

This has, I think, been much discussed on this list in years past.  I
actually implemented a solution whereby comments and strings would have
been marked by a text property in forward scanning, and back_comment
would merely examine that property.  This scheme wasn't accepted for
Emacs, and it would have suffered the same from needing large scale
scanning.

My position at the moment is that the o-p-i-c-0-i-d-s mechanism should
remain active, but that we should fix the bugs in it such that the
backward scanning would only stop at a column-0 paren not in a comment
or string.  I outlined one way of doing this in the thread called "A
proposal for the future of open-paren-in-column-0-is-defun-start", but
nobody has responded to the thrust of this post at all, as yet.

>   > I think I meant that, given their validity, it is up to us as Emacs
>   > developers to arrange that they don't cause trouble, rather than
>   > expecting our users to insert these obtrusive backslashes.

> Why assume that the one recommended way to edit the buffer to avoid
> the confusion is inserting a backslash?

This was the recommendation which used to be in the Emacs manual.  (I
think I took it out.)  When bug #22884 hit, Paul Eggert "solved" it by
refilling the copyright text such that the paren wasn't in column 0
anymore.

> -- 
> Dr Richard Stallman
> Chief GNUisance of the GNU Project (https://gnu.org)
> Founder, Free Software Foundation (https://fsf.org)
> Internet Hall-of-Famer (https://internethalloffame.org)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Mon, Apr 06, 2020 at 05:27:22 +0300, Eli Zaretskii wrote:
> > Date: Sun, 5 Apr 2020 19:57:53 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
> >  rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org

> > Try this:

> > diff --git a/src/window.c b/src/window.c
> > index 8cdad27b66..b23ae41aba 100644
> > --- a/src/window.c
> > +++ b/src/window.c
> > @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)

> >    wset_redisplay (XWINDOW (window));

> > -  if (whole && Vfast_but_imprecise_scrolling)
> > +  if (Vfast_but_imprecise_scrolling
> > +      && (whole
> > +          || EQ (Vthis_command, Qmwheel_scroll)))
> >      specbind (Qfontification_functions, Qnil);

> That's just to make a point, no?  You don't seriously propose such a
> change, right?

Well, the point is that extending the domain of
fast-but-imprecise-scrolling isn't difficult.

As to whether or not it's a good idea, probably not in such a crude
manner.  I think I would favour such an extension, yes, but I don't feel
strongly about it.  It would solve the problem of the 10 second wait at
the end of crazed mouse wheel scrolling, though.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 05.04.2020 23:48, Dmitry Gutov wrote:
> 
> You probably meant to address Martin, but I can just say that while this 
> will almost certainly work, it doesn't address the scenario of flooring 
> C-v. Or M-v. Etc. Which seem equivalent.

Apologies. I read the patch wrong. Apparently before this patch the 
optimization didn't [always] apply to mouse scrolling?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, eliz@gnu.org, rrandresf@gmail.com, emacs-devel@gnu.org
> Date: Sun, 05 Apr 2020 22:36:20 -0400
> 
>   > Recent Emacsen either ignore that variable or silently reset it to nil
>   > internally so it doesn't get into their way.  Their progmodes either
>   > always scan an entire buffer from its beginning or use some elaborate,
>   > fragile techniques to find such a top level position.  Moreover, our
>   > underlying mechanism for syntax highlighting always marks the entire
>   > rest of a buffer as dirty after every single editing change.  This has
>   > the consequence that that entire part has to be continuously rescanned
>   > when some of it is shown in another window.
> 
> Does anyone disagree with this specific factual claim?

I'm not sure what is "the claim" here, but I want to point out a small
inaccuracy: redisplay doesn't "continuously rescan the entire rest of
the buffer", it only rescans the part(s) shown in windows.  (It might
happen that some major mode's font-lock definitions end up rescanning
much more, but that's a separate issue.)  And frankly, what would we
like Emacs to do instead?  A change in a buffer can potentially affect
the fontification of the rest of the buffer, and I don't think we
would like Emacs to fail to update other windows showing the same
buffer -- that would be against Emacs's description as "real-time"
editor.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Mon, 6 Apr 2020 11:05:11 +0200
> 
>  >> Do you at least see CPU activity significantly go up when you do such
>  >> mouse wheel scrolling?
>  >
>  > With what build?  With the -O0 build with --enable-checking, I don't
>  > need the mouse: it's enough to lean on C-v and let the keyboard
>  > auto-repeat do its job -- one execution unit of the CPU maxes out
>  > after 5 to 10 C-v's.
> 
> OK.  Then people see the problem and it's just that they do not build
> with -O0.

I build with -O0 all the time, it's just that, like you, I never use
such a build for any serious work, only for debugging and exploring of
new features.

> As long as we are not there (and IMO even after that) Emacs should be
> able to do its SMIE parsing in a practical way: Restrict backward and
> forward parsing to the smallest reasonable code fragment around point.
> And reasonable would mean the smallest enclosing fragment delimited by
> two parens in column zero it can find in either direction (which can be
> still quite large when viewing functions like redisplay_internal).

That's for the major modes to implement, the jit-lock infrastructure
calls the fontification functions on a small region around the area of
interest of the display engine.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>,
>  rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> Date: Mon, 6 Apr 2020 15:43:17 +0300
> 
> On 05.04.2020 23:48, Dmitry Gutov wrote:
> > 
> > You probably meant to address Martin, but I can just say that while this 
> > will almost certainly work, it doesn't address the scenario of flooring 
> > C-v. Or M-v. Etc. Which seem equivalent.
> 
> Apologies. I read the patch wrong. Apparently before this patch the 
> optimization didn't [always] apply to mouse scrolling?

It affects any scroll command, but only partially.

People often forget that scrolling is implemented partially in the
command that is invoked by the likes of C-v, and partially in the
display engine.  Binding a variable in the command only affects the
former part, but not the latter.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 16:39, Eli Zaretskii wrote:
> People often forget that scrolling is implemented partially in the
> command that is invoked by the likes of C-v, and partially in the
> display engine.  Binding a variable in the command only affects the
> former part, but not the latter.

I thought binding fontification-functions to nil was done there exactly 
with that intent: to affect what the display engine subsequently does. 
Isn't that right?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   > Indeed.  And the reverse is true as well: while it might be OK for
>   > a user to put his open-parens in column 0 to help Emacs be more
>   > responsive, it's not OK for Emacs to be too slow on files where there
>   > are no such open-parens in column 0.  So Emacs has to use some other
>   > mechanism anyway to avoid the slowdown in such files.
>
> The mechanism it has now is effective only on fast machines.

Do you have any evidence to back that claim?


        Stefan "whose main work machines are Core 2 Duo"

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 6 Apr 2020 16:55:11 +0300
> 
> On 06.04.2020 16:39, Eli Zaretskii wrote:
> > People often forget that scrolling is implemented partially in the
> > command that is invoked by the likes of C-v, and partially in the
> > display engine.  Binding a variable in the command only affects the
> > former part, but not the latter.
> 
> I thought binding fontification-functions to nil was done there exactly 
> with that intent: to affect what the display engine subsequently does. 

Can't be done, because redisplay runs only _after_ the scroll command
returns, at which point this binding is undone.  You can easily see
this in a debugger: put a breakpoint inside redisplay_internal right
after the specbind line is executed, then type

  (gdb) p Fsymbol_value (Qfontification_functions)

You will see the value is a cons cell, not nil.

What that binding does is affect the portions of display code that are
run as part of window_scroll_pixel_based, which the scroll command
calls to calculate a suitable position of window-start by simulating
display.  That display code runs with fontifications disabled, which
makes it faster.

All the scroll commands in Emacs do is more or less calculate the
window-start point; the rest is done when redisplay kicks in.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   > Recent Emacsen either ignore that variable or silently reset it to nil
>   > internally so it doesn't get into their way.  Their progmodes either
>   > always scan an entire buffer from its beginning or use some elaborate,
>   > fragile techniques to find such a top level position.  Moreover, our
>   > underlying mechanism for syntax highlighting always marks the entire
>   > rest of a buffer as dirty after every single editing change.  This has
>   > the consequence that that entire part has to be continuously rescanned
>   > when some of it is shown in another window.
>
> Does anyone disagree with this specific factual claim?

There are several claims above which can be interpreted in many
different ways, some of them correspond to reality but many others
don't.  So if you don't know the actual facts behind the above claims
you're likely to infer incorrect conclusions.

>   > The basic slowness of Emacs over the past years is a direct consequence
>   > of that policy.
> Does anyone disagree with this general claim?

I think I made it clear that I do.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 05:36, Richard Stallman wrote:
> If and when such a mechanism is in place for us free software
> activists, we might agree that open-paren-in-column-0-is-defun-start
> is not worth while any more.  But it appears that has not happened
> yet, so I think that the open-paren-in-column-0-is-defun-start guided
> heuristic IS still worth while for us.

This question (whether syntax-ppss is "too slow") was raised a couple of 
times already, but in all previous discussions, as well as in this one, 
nobody has done the bare minimum research (e.g. profiling some 
real-world scenario) to prove this claim. Nor, if it indeed showed up in 
profiler reports, examined which part of the current mechanism is the 
bottleneck. Nor tried to fix that part somehow.

This is not how we should approach performance problems.

The only arguments I see are nostalgic (e.g., somehow, Emacs was faster 
years ago, maybe? the files were smaller, and the C standard was simpler?)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> 16s in one case a "10 seconds freeze": it sounds to me like you *do*
>> reproduce the problem.
> Not really.  In the 16s case redisplay kicks in and displays the buffer
> 250 times.  In the 10s case redisplay stops after five times.

Yes, but the underlying problem remains: it takes 10-16s to do the work,
so if the corresponding input events come in over a period shorter than
10-16s Emacs won't be able to keep up.

>> The only difference is that with actual keyboard input pending the
>> redisplay is partly prevented which is what you perceive as "freeze",
>> but the real problem is the time it takes.
> What does get fontified when redisplay freezes?

The text gets fontified by the scrolling routine because the size of the
text depends on the fontification, so correct scrolling needs to fontify
the text before it can compute the destination of the scroll.

That's where `fast-but-imprecise-scrolling` comes into play.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> As long as we are not there (and IMO even after that) Emacs should be
> able to do its SMIE parsing in a practical way: Restrict backward and
> forward parsing to the smallest reasonable code fragment around point.
> And reasonable would mean the smallest enclosing fragment delimited by
> two parens in column zero it can find in either direction (which can be
> still quite large when viewing functions like redisplay_internal).

Could people refrain from "X should do Y" when they have no idea how
X currently works?

SMIE's indentation works by parsing backward, so that it stops parsing
as soon as it found the info needed to determine the indentation to use.

It almost always stops long before reaching the nearest "open paren in
column 0".


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: acm@muc.de,  Eli Zaretskii <eliz@gnu.org>,  rrandresf@gmail.com,
>   rms@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 06 Apr 2020 11:25:43 -0400
> 
> >> 16s in one case a "10 seconds freeze": it sounds to me like you *do*
> >> reproduce the problem.
> > Not really.  In the 16s case redisplay kicks in and displays the buffer
> > 250 times.  In the 10s case redisplay stops after five times.
> 
> Yes, but the underlying problem remains: it takes 10-16s to do the work,
> so if the corresponding input events come in over a period shorter than
> 10-16s Emacs won't be able to keep up.

I think "the work" of a single scroll-up takes less (much less) than
10-16s.  But when several C-v keypresses accumulate (because the user
types them faster than Emacs can process them), their collective work
takes a long time to process.

> The text gets fontified by the scrolling routine because the size of the
> text depends on the fontification, so correct scrolling needs to fontify
> the text before it can compute the destination of the scroll.
> 
> That's where `fast-but-imprecise-scrolling` comes into play.

Right.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> >> 16s in one case a "10 seconds freeze": it sounds to me like you *do*
>> >> reproduce the problem.
>> > Not really.  In the 16s case redisplay kicks in and displays the buffer
>> > 250 times.  In the 10s case redisplay stops after five times.
>> 
>> Yes, but the underlying problem remains: it takes 10-16s to do the work,
>> so if the corresponding input events come in over a period shorter than
>> 10-16s Emacs won't be able to keep up.
>
> I think "the work" of a single scroll-up takes less (much less) than
> 10-16s.

Indeed, IIUC his measurement of 10-16s is for the equivalent of about
50 scrolls.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 17:44, Eli Zaretskii wrote:

>> I thought binding fontification-functions to nil was done there exactly
>> with that intent: to affect what the display engine subsequently does.
> 
> Can't be done, because redisplay runs only _after_ the scroll command
> returns, at which point this binding is undone.  You can easily see
> this in a debugger: put a breakpoint inside redisplay_internal right
> after the specbind line is executed, then type
> 
>    (gdb) p Fsymbol_value (Qfontification_functions)
> 
> You will see the value is a cons cell, not nil.
> 
> What that binding does is affect the portions of display code that are
> run as part of window_scroll_pixel_based, which the scroll command
> calls to calculate a suitable position of window-start by simulating
> display.  That display code runs with fontifications disabled, which
> makes it faster.
> 
> All the scroll commands in Emacs do is more or less calculate the
> window-start point; the rest is done when redisplay kicks in.

So what happens then? Does it trigger jit-lock anyway?

If so, I don't understand how we get the apparent performance 
improvement from using fast-but-imprecise-scrolling.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> As long as we are not there (and IMO even after that) Emacs should be
 >> able to do its SMIE parsing in a practical way: Restrict backward and
 >> forward parsing to the smallest reasonable code fragment around point.
 >> And reasonable would mean the smallest enclosing fragment delimited by
 >> two parens in column zero it can find in either direction (which can be
 >> still quite large when viewing functions like redisplay_internal).
 >
 > Could people refrain from "X should do Y" when they have no idea how
 > X currently works?
 >
 > SMIE's indentation works by parsing backward, so that it stops parsing
 > as soon as it found the info needed to determine the indentation to use.
 >
 > It almost always stops long before reaching the nearest "open paren in
 > column 0".

I did interpret Eli's remark about "SMIE-based engines" as the simple
minded approach used by CC-mode which to my knowledge often does not
stop long before reaching the nearest "open paren in column 0".  I did
not intend to relate this to smie.el but it's obvious that I created
that impression.  Sorry for that.

martin

Re: Inlining policy

[Stefan Monnier]

> what about having a new macro to control the key operations we want to
> always have inlined at -O0 and -Og instead of falling back into defining
> these as macros?
>
> Something like the attached patch.

That sounds even better than what we have now since it removed the need
for the duplicate definition (once as macro and once as function).

I haven't checked whether it has the same performance impact, and
neither do I know how it compares w.r.t. "portability/reliability", but
judging just from the patch, I really like it.

Paul?


        Stefan

Re: Inlining policy

[Paul Eggert]

On 4/6/20 11:15 AM, Stefan Monnier wrote:

> I haven't checked whether it has the same performance impact, and
> neither do I know how it compares w.r.t. "portability/reliability", but
> judging just from the patch, I really like it.
> 
> Paul?

Although I like the idea, it significantly hurts performance with -O0 (GCC 
9.3.1, x86-64) because it causes GCC to emit excess instructions to save and 
restore inline function arguments to the stack.

I'm testing a patch to use the idea for -Og but not -O0. Here it seems to help 
performance. Unfortunately this means we still need to keep those macros, for -O0.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 05:16, Stefan Monnier wrote:
>>>> And in this case, IIUC, jit-lock is part of redisplay.
>>> Yes, it often is when scrolling.
>> So would you agree that short-circuiting and skipping jit-lock while
>> somebody leans on C-v is a good idea?
> It sounds like a trick question, because what I just described says that
> it is what already happens, more or less, and I'm not sure which part of
> the difference you're thinking of.

We do that when fast-but-imprecise-scrolling is on, right?

But with it off (which is the default), fontification functions are 
still forced to run during every scroll-up command, don't they?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 06.04.2020 05:25, Eli Zaretskii wrote:
>>>> Certainly a lower bar than having mouse-scrolling never lock up.
>>> I'm not sure it is lower, because as I said an optimized Emacs doesn't
>>> lock up when I scroll, either with C-v or the mouse.  But the problems
>>> I described do happen in such builds.
>> It's lower because when you're typing you're usually typing something
>> meaningful (meaning limited input rate), and the beginning of the window
>> usually stays in place (which speeds up re-fontification).
>>
>> Whereas when you're scrolling, you can scroll as fast as your keyboard
>> allows you, and you're triggering fontification of a whole new
>> screen-ful every keystroke.
> And yet Emacs does cope with that "higher bar", whereas while I type I
> sometimes see unbearably slow responses.  Which in my book doesn't fit
> its being "lower bar".

At least it's lower in theory. But certain implementation approaches can 
make it more difficult.

Would I be correct to assume that you see only see this in CC Mode based 
major mode, and only when editing big enough files? Such as xdisp.c 
mentioned many times recently.

Re: Inlining policy

[Andrea Corallo]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 4/6/20 11:15 AM, Stefan Monnier wrote:
>
>> I haven't checked whether it has the same performance impact, and
>> neither do I know how it compares w.r.t. "portability/reliability", but
>> judging just from the patch, I really like it.
>>
>> Paul?
>
> Although I like the idea, it significantly hurts performance with -O0
> (GCC 9.3.1, x86-64) because it causes GCC to emit excess instructions
> to save and restore inline function arguments to the stack.

True

https://godbolt.org/z/rfKxo7

Actually I think moving the lisp_h_* into the functions could also make
these easier to debug in case.  I find a little odd to optimize for
speed -O0, in the end is the exact reason why -Og exists.  But I
understand there's a trade-off here.

Thanks for reviewing

  Andrea

-- 
akrl@sdf.org

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Apr 06, 2020 at 17:55:50 +0300, Dmitry Gutov wrote:
> On 06.04.2020 05:36, Richard Stallman wrote:
> > If and when such a mechanism is in place for us free software
> > activists, we might agree that open-paren-in-column-0-is-defun-start
> > is not worth while any more.  But it appears that has not happened
> > yet, so I think that the open-paren-in-column-0-is-defun-start guided
> > heuristic IS still worth while for us.

> This question (whether syntax-ppss is "too slow") was raised a couple of 
> times already, but in all previous discussions, as well as in this one, 
> nobody has done the bare minimum research (e.g. profiling some 
> real-world scenario) to prove this claim. Nor, if it indeed showed up in 
> profiler reports, examined which part of the current mechanism is the 
> bottleneck. Nor tried to fix that part somehow.

> This is not how we should approach performance problems.

> The only arguments I see are nostalgic (e.g., somehow, Emacs was faster 
> years ago, maybe? the files were smaller, and the C standard was simpler?)

I've attempted to simulate one of Martin's scenarios, and set up my tty
frame with two side by side windows, each 65 lines deep and wide enough
for xdisp.c not to need continuation lines.  xdisp.c is displayed in both
windows, from Line 401 at the LHS, and from Line 35162 on the RHS.  In
the LH window point is at BOL 435 (the opening brace for function
fill_column_indicator_column) and in the RH window point is at BOL 35159
(a random line in start_hourglass).

With various combinations of the pertinent variables, I run

    (time-it (insert "{") (other-window 1) (insert " ") (other-window 1)
    (sit-for 0))

.  After each such timing I undo the buffer changes before the next
timing.

Here are the results:
                         open-paren-in-column-zero-is-defun-start
			      nil                    t
comment-use-syntax-ppss
        nil                  0.319s                0.264s
	 t                   0.319s                0.227s

.  Bearing in mind that c-u-s-p being t suppresses the action of
o-p-i-c-0-i-d-s in back_comment, but not in beginning-of-defun, it seems
like the o-p-i-c-0-i-d-s mechanism does indeed speed up some scenarios
in C Mode, significantly but not massively.

IMAO, It would be nice to have the code testing o-p-i-c-0-i-d-s (both
places) able to ignore spurious cases of parens in literals.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>>>> And in this case, IIUC, jit-lock is part of redisplay.
>>>> Yes, it often is when scrolling.
>>> So would you agree that short-circuiting and skipping jit-lock while
>>> somebody leans on C-v is a good idea?
>> It sounds like a trick question, because what I just described says that
                                                 ^
                                                you (or "is")

>> it is what already happens, more or less, and I'm not sure which part of
>> the difference you're thinking of.
> We do that when fast-but-imprecise-scrolling is on, right?

At this point I don't really know "that" refers to.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 00:27, Stefan Monnier wrote:
>>>>>> And in this case, IIUC, jit-lock is part of redisplay.
>>>>> Yes, it often is when scrolling.
>>>> So would you agree that short-circuiting and skipping jit-lock while
>>>> somebody leans on C-v is a good idea?
>>> It sounds like a trick question, because what I just described says that
>                                                   ^
>                                                  you (or "is")
> 
>>> it is what already happens, more or less, and I'm not sure which part of
>>> the difference you're thinking of.
>> We do that when fast-but-imprecise-scrolling is on, right?
> At this point I don't really know "that" refers to.

that = skip jit-lock when input is pending.

If not, where else its performance improvement comes from?

Re: Inlining policy

[Stefan Monnier]

> Actually I think moving the lisp_h_* into the functions could also make
> these easier to debug in case.  I find a little odd to optimize for
> speed -O0, in the end is the exact reason why -Og exists.  But I
> understand there's a trade-off here.

FWIW, I use `-Og` nowadays and can't remember the last time I used
`-O0`, so I'd be happy with a solution that doesn't work so well for
`-O0` as long as it works well for `-Og`.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 06.04.2020 22:36, Alan Mackenzie wrote:

> I've attempted to simulate one of Martin's scenarios, and set up my tty
> frame with two side by side windows, each 65 lines deep and wide enough
> for xdisp.c not to need continuation lines.  xdisp.c is displayed in both
> windows, from Line 401 at the LHS, and from Line 35162 on the RHS.  In
> the LH window point is at BOL 435 (the opening brace for function
> fill_column_indicator_column) and in the RH window point is at BOL 35159
> (a random line in start_hourglass).
> 
> With various combinations of the pertinent variables, I run
> 
>      (time-it (insert "{") (other-window 1) (insert " ") (other-window 1)
>      (sit-for 0))
> 
> .  After each such timing I undo the buffer changes before the next
> timing.

So you chose the absolute worst scenario for syntax-ppss, right? Where 
its cache would be unavoidably clobbered at least once. I'm guessing the 
difference between measurements is ~ how long (parse-partial-sexp 
(point-min) (point-max)) takes on your machine in that file.

How about some other, more common scenarios?

Like Martin just scrolling the buffer (we can simulate that with several 
calls to scroll-up).

Or Eli just typing in some large file.

I think these two cases featured more prominently in the recent 
discussions, with associated complaints about performance.

> Here are the results:
>                           open-paren-in-column-zero-is-defun-start
> 			      nil                    t
> comment-use-syntax-ppss
>          nil                  0.319s                0.264s
> 	 t                   0.319s                0.227s
> 
> .  Bearing in mind that c-u-s-p being t suppresses the action of
> o-p-i-c-0-i-d-s in back_comment, but not in beginning-of-defun, it seems
> like the o-p-i-c-0-i-d-s mechanism does indeed speed up some scenarios
> in C Mode, significantly but not massively.
> 
> IMAO, It would be nice to have the code testing o-p-i-c-0-i-d-s (both
> places) able to ignore spurious cases of parens in literals.

I can't imagine anybody doing the 
type-switch-window-to-the-same-buffer-type routine frequently, but if 
you really want to improve that use case, I think the first takeaway 
that the necessary knobs are still present. A text property check 
wouldn't slow the routines significantly (I think), but you'd have to 
benchmark that as well, and also the extra code in CC Mode which would 
scan the whole buffer and apply those properties. In the first 
implementation, at least, you could do that using the normal 
before/after-change-functions, not "super" ones.

Here's a better idea to try, though: keep comment-use-syntax-ppss set to 
t. And also use syntax-ppss to find the beginning of defun in 
c-beginning-of-defun. But to make it honor the o-p-i-c-0-i-d-s, set 
syntax-begin-function to a new function that contains whatever "fast" 
logic you prefer. E.g. scanning the buffer back for open parens in 
column 1 without certain text property applied. I'm not 100% sure why 
this var is marked obsolete now (probably because it creates murky 
semantics), but this is the simplest way to add this behavior, I think.

In any case, I'm afraid all this won't bring any dramatic improvement in 
the more usual scenarios which I mentioned above. And we'll be back to 
looking for other ways to improve performance.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> So you chose the absolute worst scenario for syntax-ppss, right? Where its
> cache would be unavoidably clobbered at least once. I'm guessing the
> difference between measurements is ~ how long (parse-partial-sexp 
> (point-min) (point-max)) takes on your machine in that file.

FWIW, `syntax-ppss` has not been optimized for this case at all, so
I wouldn't be surprised if it takes noticeably more than

    (parse-partial-sexp (point-min) (point-max))

because it performs that computation in chunks, and the overhead for
each chunk is likely non-negligible.
[ Note: this is just a wild guess on my part.  I never bothered to
  measure it: `syntax-ppss` is a fairly naive implementation with a lot
  of room for improvement.  ]

IOW if it takes the same time as (parse-partial-sexp (point-min) (point-max))
then I'm quite happy ;-)

> How about some other, more common scenarios?

I agree that it's probably not very representative.

>> Here are the results:
>>                           open-paren-in-column-zero-is-defun-start
>> 			      nil                    t
>> comment-use-syntax-ppss
>>          nil                  0.319s                0.264s
>> 	 t                   0.319s                0.227s
>> .  Bearing in mind that c-u-s-p being t suppresses the action of
>> o-p-i-c-0-i-d-s in back_comment, but not in beginning-of-defun, it seems
>> like the o-p-i-c-0-i-d-s mechanism does indeed speed up some scenarios
>> in C Mode, significantly but not massively.
>> IMAO, It would be nice to have the code testing o-p-i-c-0-i-d-s (both
>> places) able to ignore spurious cases of parens in literals.

The problem here is that in order to decide whether or not it's *still*
spurious after the change near BOB, you basically have to compute the
equivalent of `syntax-ppss`.  So we're back to square one.

What could be done is to change `syntax-ppss` so it optimizes for the
case where the buffer changes do not impact the way parsing is done
"further down" (i.e. not for the case where the change just opened (or
closed) a string/comment).  Not sure it's worth the trouble, OTOH.

> 1 without certain text property applied. I'm not 100% sure why this var is
> marked obsolete now (probably because it creates murky semantics), but this
> is the simplest way to add this behavior, I think.

Yes, the problem is the murky semantics (including the fact that you
can't rely on it being used, even though some code does want to rely on
it, typically in multi-mode setups where it's abused to impose
a boundary).

> In any case, I'm afraid all this won't bring any dramatic improvement in the
> more usual scenarios which I mentioned above. And we'll be back to looking
> for other ways to improve performance.

That's also my impression.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> that = skip jit-lock when input is pending.

AFAIK jit-lock is never skipped: it's (part of) the redisplay which is
skipped (and that in turn may include calls to jit-lock).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 02:43, Stefan Monnier wrote:
>> that = skip jit-lock when input is pending.
> AFAIK jit-lock is never skipped: it's (part of) the redisplay which is
> skipped (and that in turn may include calls to jit-lock).

This sounds like splitting hairs; if the part containing it is skipped, 
then jit-lock is skipped as well, obviously.

Does that happen when fast-but-imprecise-scrolling is nil as well?

Or is the situation that redisplay skips jit-lock when there's pending 
input (a good thing), but scrolling commands go back and force it 
anyway, to compute the new positions to scroll to?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > The mechanism it has now is effective only on fast machines.

  > Do you have any evidence to back that claim?

Others have posted the data here.  It is not crucial that I find it
myself.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, that is what I meant.  The "kind of things" is nothing more than
  > writing comments in C (etc.) source code.  Sooner or later, such a
  > comment has a parenthesis in it, and sooner or later (e.g. by M-q), this
  > will end up in column 0,

I am surprised, because that never happens to me.  I don't put a * at
the start of each comment line, but I do indent each comment line
and that should be enough to make sure no openparen fills into column 0.

What do you think of such indentation as a solution for this problem.

  > My position at the moment is that the o-p-i-c-0-i-d-s mechanism should
  > remain active, but that we should fix the bugs in it such that the
  > backward scanning would only stop at a column-0 paren not in a comment
  > or string.  I outlined one way of doing this in the thread called "A
  > proposal for the future of open-paren-in-column-0-is-defun-start", but
  > nobody has responded to the thrust of this post at all, as yet.

If this can be done, I don't think I would argue against it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>> that = skip jit-lock when input is pending.
>> AFAIK jit-lock is never skipped: it's (part of) the redisplay which is
>> skipped (and that in turn may include calls to jit-lock).
> This sounds like splitting hairs; if the part containing it is skipped, then
> jit-lock is skipped as well, obviously.
>
> Does that happen when fast-but-imprecise-scrolling is nil as well?

Skipping redisplay when there's pending input happens regardless of
`fast-but-imprecise-scrolling`.

Now here's where "splitting hairs" becomes important: while the choice
to skip redisplay is not affected by `fast-but-imprecise-scrolling`, the
choice to skip jit-lock *is*.

> Or is the situation that redisplay skips jit-lock when there's pending input
> (a good thing), but scrolling commands go back and force it anyway, to
> compute the new positions to scroll to?

That's right (tho, AFAIK the scrolling commands cause jit-lock *before*
redisplay happens, so "go back" is not quite accurate).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   > > The mechanism it has now is effective only on fast machines.
>   > Do you have any evidence to back that claim?
> Others have posted the data here.

Which data are you referring to?  Maybe I missed it as well?

The only data I've seen so far that shows any effect of
open-paren-in-column-0 is the one recently (as in "after your above
messages") posted by Alan, and this data says nothing about slow or fast
machines, shows a relatively small performance difference (not
negligible, but within a factor 2), and considers a situation expected
to be rather rare.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   > Yes, that is what I meant.  The "kind of things" is nothing more than
>   > writing comments in C (etc.) source code.  Sooner or later, such a
>   > comment has a parenthesis in it, and sooner or later (e.g. by M-q), this
>   > will end up in column 0,
>
> I am surprised, because that never happens to me.

Of course that never happens to you: you're the one who introduced the
open-paren-in-column-0 heuristic, so you're clearly biased.

> What do you think of such indentation as a solution for this problem.

That presumes you have control over the code (and coding conventions).


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems
 >> like a good idea in general, so we might consider going further with it.
 >
 > Try this:
 >
 > diff --git a/src/window.c b/src/window.c
 > index 8cdad27b66..b23ae41aba 100644
 > --- a/src/window.c
 > +++ b/src/window.c
 > @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
 >
 >     wset_redisplay (XWINDOW (window));
 >
 > -  if (whole && Vfast_but_imprecise_scrolling)
 > +  if (Vfast_but_imprecise_scrolling
 > +      && (whole
 > +          || EQ (Vthis_command, Qmwheel_scroll)))
 >       specbind (Qfontification_functions, Qnil);
 >
 >     /* On GUI frames, use the pixel-based version which is much slower
 > @@ -8173,6 +8175,7 @@ syms_of_window (void)
 >     DEFSYM (Qmode_line_format, "mode-line-format");
 >     DEFSYM (Qheader_line_format, "header-line-format");
 >     DEFSYM (Qtab_line_format, "tab-line-format");
 > +  DEFSYM (Qmwheel_scroll, "mwheel-scroll");
 >
 >     DEFVAR_LISP ("temp-buffer-show-function", Vtemp_buffer_show_function,
 >   	       doc: /* Non-nil means call as function to display a help buffer.
 >
 >
 > This eliminates all the delay I formerly saw while mouse scrolling as
 > fast as possibly.

I tried that now and it makes no noticeable difference here.  Maybe I
did something wrong - all I checked via GDB was that it does perform the
specbinding.  Maybe that's not sufficient.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >>    > Recent Emacsen either ignore that variable or silently reset it to nil
 >>    > internally so it doesn't get into their way.  Their progmodes either
 >>    > always scan an entire buffer from its beginning or use some elaborate,
 >>    > fragile techniques to find such a top level position.  Moreover, our
 >>    > underlying mechanism for syntax highlighting always marks the entire
 >>    > rest of a buffer as dirty after every single editing change.  This has
 >>    > the consequence that that entire part has to be continuously rescanned
 >>    > when some of it is shown in another window.
 >>
 >> Does anyone disagree with this specific factual claim?
 >
 > I'm not sure what is "the claim" here, but I want to point out a small
 > inaccuracy: redisplay doesn't "continuously rescan the entire rest of
 > the buffer", it only rescans the part(s) shown in windows.

My text didn't say that _redisplay_ has to do that.  It talks about a
"consequence" and here the consequence is that jit-lock triggers an
eventual rescan of the rest of the buffer down to the part that is
displayed in another window.

 > (It might
 > happen that some major mode's font-lock definitions end up rescanning
 > much more, but that's a separate issue.)  And frankly, what would we
 > like Emacs to do instead?  A change in a buffer can potentially affect
 > the fontification of the rest of the buffer, and I don't think we
 > would like Emacs to fail to update other windows showing the same
 > buffer -- that would be against Emacs's description as "real-time"
 > editor.

It depends on what we consider a failure: I do not think that inserting
a "/*" in my window showing the top of my C buffer should cause the
window showing the bottom of my buffer consider everything part of a
comment starting at that "/*".  I think that the comment should stop at
the first paren in column zero after that "/*" even if that paren is
shown in the window showing the top of the buffer and even if that's
syntactically wrong (and hence the compiler would see it differently).

So what I would vote for in this case is that font-lock would specially
highlight the open paren that "stops seeing the comment as comment" so
the user is aware of the fact.  Would that be so hard to implement?
Even with a slight change in 'parse-partial-sexp' and 'syntax-ppss' that
would accompany it.  Optionally, by default turned off, so nobody who
wants the correct syntactic interpretation would be bothered.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Martin.

On Tue, Apr 07, 2020 at 10:31:08 +0200, martin rudalics wrote:
>  >> Or, alternatively, (setq fast-but-imprecise-scrolling t). This var seems
>  >> like a good idea in general, so we might consider going further with it.

>  > Try this:

>  > diff --git a/src/window.c b/src/window.c
>  > index 8cdad27b66..b23ae41aba 100644
>  > --- a/src/window.c
>  > +++ b/src/window.c
>  > @@ -5475,7 +5475,9 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)

>  >     wset_redisplay (XWINDOW (window));

>  > -  if (whole && Vfast_but_imprecise_scrolling)
>  > +  if (Vfast_but_imprecise_scrolling
>  > +      && (whole
>  > +          || EQ (Vthis_command, Qmwheel_scroll)))
>  >       specbind (Qfontification_functions, Qnil);

>  >     /* On GUI frames, use the pixel-based version which is much slower
>  > @@ -8173,6 +8175,7 @@ syms_of_window (void)
>  >     DEFSYM (Qmode_line_format, "mode-line-format");
>  >     DEFSYM (Qheader_line_format, "header-line-format");
>  >     DEFSYM (Qtab_line_format, "tab-line-format");
>  > +  DEFSYM (Qmwheel_scroll, "mwheel-scroll");

>  >     DEFVAR_LISP ("temp-buffer-show-function", Vtemp_buffer_show_function,
>  >   	       doc: /* Non-nil means call as function to display a help buffer.


>  > This eliminates all the delay I formerly saw while mouse scrolling as
>  > fast as possibly.

> I tried that now and it makes no noticeable difference here.  Maybe I
> did something wrong - all I checked via GDB was that it does perform
> the specbinding.  Maybe that's not sufficient.

That's not so good.  The way it works is by bypassing the fontification
in the scrolling code (which is only there to calculate how many lines to
scroll, particularly for variable height fonts).  The only fontification
should happen for redisplay, and that only for when there's no input
event waiting.

So if all works well, only isolated small segments of the buffer
(screenfuls) get fontified and displayed and no long queue of buffer
portions for fontification should build up.

I don't know why it's not working for you.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Martin,

On 07.04.2020 11:31, martin rudalics wrote:
> I tried that now and it makes no noticeable difference here.  Maybe I
> did something wrong - all I checked via GDB was that it does perform the
> specbinding.  Maybe that's not sufficient.

Did you both apply the patch and set fast-but-imprecise-scrolling to t?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 15:07, Dmitry Gutov wrote:
> Hi Martin,
> 
> On 07.04.2020 11:31, martin rudalics wrote:
>> I tried that now and it makes no noticeable difference here.  Maybe I
>> did something wrong - all I checked via GDB was that it does perform the
>> specbinding.  Maybe that's not sufficient.
> 
> Did you both apply the patch and set fast-but-imprecise-scrolling to t?

Sorry, nevermind. If the specbinding was applied, then you did.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> I tried that now and it makes no noticeable difference here.  Maybe I
 >> did something wrong - all I checked via GDB was that it does perform
 >> the specbinding.  Maybe that's not sufficient.
 >
 > That's not so good.  The way it works is by bypassing the fontification
 > in the scrolling code (which is only there to calculate how many lines to
 > scroll, particularly for variable height fonts).  The only fontification
 > should happen for redisplay, and that only for when there's no input
 > event waiting.
 >
 > So if all works well, only isolated small segments of the buffer
 > (screenfuls) get fontified and displayed and no long queue of buffer
 > portions for fontification should build up.
 >
 > I don't know why it's not working for you.

As a matter of fact, it is slightly better.  Still, Emacs locks down
after a few scrolls in a similar way as without it.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > Did you both apply the patch and set fast-but-imprecise-scrolling to t?

Sure.  I also verified that

      specbind (Qfontification_functions, Qnil);

gets executed.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 11:32, martin rudalics wrote:
> It depends on what we consider a failure: I do not think that inserting
> a "/*" in my window showing the top of my C buffer should cause the
> window showing the bottom of my buffer consider everything part of a
> comment starting at that "/*".  I think that the comment should stop at
> the first paren in column zero after that "/*" even if that paren is
> shown in the window showing the top of the buffer and even if that's
> syntactically wrong (and hence the compiler would see it differently).

To be accurate, the comment started by '/*' would end at the first 
occurrence of '*/' somewhere down the buffer.

If you insert or close a paren, however, that would definitely change 
syntax state at EOB.

The former case is possible to support better in syntax-ppss, FWIW. The 
latter we probably never will (in syntax-ppss).

> So what I would vote for in this case is that font-lock would specially
> highlight the open paren that "stops seeing the comment as comment" so
> the user is aware of the fact.  Would that be so hard to implement?
> Even with a slight change in 'parse-partial-sexp' and 'syntax-ppss' that
> would accompany it.  Optionally, by default turned off, so nobody who
> wants the correct syntactic interpretation would be bothered.

According to Alan's experiment, you can get this behavior now already 
simply by setting comment-use-syntax-ppss to nil and 
open-paren-in-column-0-is-defun-start to t.

Does it improve your experience significantly?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 15:45, martin rudalics wrote:
>  > So if all works well, only isolated small segments of the buffer
>  > (screenfuls) get fontified and displayed and no long queue of buffer
>  > portions for fontification should build up.
>  >
>  > I don't know why it's not working for you.
> 
> As a matter of fact, it is slightly better.  Still, Emacs locks down
> after a few scrolls in a similar way as without it.

And can it similarly lock down for a long time? Or does it unfreeze quickly?

What about other major modes? If you create a big JS file and try the 
same experiment in js-mode? Or create a long Elisp file and do the same 
in emacs-lisp-mode (you can copy-paste a lot of time the contents of 
some existing file)? Or take any of those files, switch to 
fundamental-mode and do the same.

Does scrolling also lock up in a similar manner?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rrandresf@gmail.com, emacs-devel@gnu.org, acm@muc.de
> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 7 Apr 2020 10:32:07 +0200
> 
>  > (It might
>  > happen that some major mode's font-lock definitions end up rescanning
>  > much more, but that's a separate issue.)  And frankly, what would we
>  > like Emacs to do instead?  A change in a buffer can potentially affect
>  > the fontification of the rest of the buffer, and I don't think we
>  > would like Emacs to fail to update other windows showing the same
>  > buffer -- that would be against Emacs's description as "real-time"
>  > editor.
> 
> It depends on what we consider a failure: I do not think that inserting
> a "/*" in my window showing the top of my C buffer should cause the
> window showing the bottom of my buffer consider everything part of a
> comment starting at that "/*".

If that is what the language syntax rules say, then yes, we should
paint everything in comment face.  If you disagree, then let's agree
to disagree: I consider "inaccurate" display one of the worst things
an editor can do to its users.

> So what I would vote for in this case is that font-lock would specially
> highlight the open paren that "stops seeing the comment as comment" so
> the user is aware of the fact.

How would that help if that parentheses is not shown in any window?
And even if it is shown, do we really expect users to pay attention to
that, and immediately understand that this parentheses with an unusual
face explains the wrong display?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: martin rudalics <rudalics@gmx.at>
> Date: Tue, 7 Apr 2020 14:45:42 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, rrandresf@gmail.com,
>  Dmitry Gutov <dgutov@yandex.ru>, rms@gnu.org, emacs-devel@gnu.org
> 
> As a matter of fact, it is slightly better.  Still, Emacs locks down
> after a few scrolls in a similar way as without it.

Maybe showing profiles for when Emacs locks up with and without that
setting could give some hints about what's going on?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 02:41, Stefan Monnier wrote:
>> So you chose the absolute worst scenario for syntax-ppss, right? Where its
>> cache would be unavoidably clobbered at least once. I'm guessing the
>> difference between measurements is ~ how long (parse-partial-sexp
>> (point-min) (point-max)) takes on your machine in that file.
> 
> FWIW, `syntax-ppss` has not been optimized for this case at all, so
> I wouldn't be surprised if it takes noticeably more than
> 
>      (parse-partial-sexp (point-min) (point-max))
> 
> because it performs that computation in chunks, and the overhead for
> each chunk is likely non-negligible.

The times it takes here, in an optimized build, fluctuates from 0.02 to 
0.05. So indeed, the difference is larger. But the same order of 
magnitude, which is still pretty good, IMO.

>>> Here are the results:
>>>                            open-paren-in-column-zero-is-defun-start
>>> 			      nil                    t
>>> comment-use-syntax-ppss
>>>           nil                  0.319s                0.264s
>>> 	 t                   0.319s                0.227s
>>> .  Bearing in mind that c-u-s-p being t suppresses the action of
>>> o-p-i-c-0-i-d-s in back_comment, but not in beginning-of-defun, it seems
>>> like the o-p-i-c-0-i-d-s mechanism does indeed speed up some scenarios
>>> in C Mode, significantly but not massively.
>>> IMAO, It would be nice to have the code testing o-p-i-c-0-i-d-s (both
>>> places) able to ignore spurious cases of parens in literals.
> 
> The problem here is that in order to decide whether or not it's *still*
> spurious after the change near BOB, you basically have to compute the
> equivalent of `syntax-ppss`.  So we're back to square one.

That's a good point. In this particular scenario all column-1 parens 
down in the buffer should be invalidated.

> What could be done is to change `syntax-ppss` so it optimizes for the
> case where the buffer changes do not impact the way parsing is done
> "further down" (i.e. not for the case where the change just opened (or
> closed) a string/comment).  Not sure it's worth the trouble, OTOH.

Yup. Not sure either, but let's see what feedback others give.

Or we could try indefinitely supporting the "worse-is-better" semantics 
offered by o-p-i-c-0-i-d-s if, again, it's really in demand.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 6 Apr 2020 20:16:43 +0300
> 
> > What that binding does is affect the portions of display code that are
> > run as part of window_scroll_pixel_based, which the scroll command
> > calls to calculate a suitable position of window-start by simulating
> > display.  That display code runs with fontifications disabled, which
> > makes it faster.
> > 
> > All the scroll commands in Emacs do is more or less calculate the
> > window-start point; the rest is done when redisplay kicks in.
> 
> So what happens then? Does it trigger jit-lock anyway?

Basically, yes.  But with a twist, see below.

> If so, I don't understand how we get the apparent performance improvement from using fast-but-imprecise-scrolling.

To explain that, I need to step back and describe how jit-lock
mechanism works, from the display engine POV, and then how the scroll
commands work.  (See also a large commentary in keyboard.c around line
250.)

For simplicity, let's assume we have a buffer with no overlays, no
display strings, and no other complications, only faces.  Let's also
assume the text is entirely L2R, so we can ignore what bidi.c does.

When the display engine starts working on displaying a window, it
begins from the window-start position, loads all the faces and other
display-related stuff, then starts layout by moving one character at a
time in the direction of increasing buffer positions, and computing
the metrics of each character as determined by the face(s) at each
position.  It stops when it gets to a buffer position where the
'fontified' text property changes its value.  If the new value of
'fontified' is nil, the display engine calls fontification-functions.
jit-lock adds its function to that hook, so jit-lock is called.  What
jit-lock does, in a nutshell, is call the current buffer's
fontification function(s) on a chunk of text starting at the position
for which it was called, and it puts on that chunk a value of t for
the 'fontified' text property.  (Where that chunk ended is the
position where the 'fontified' property's value will again change to
nil, and if that position is inside the window, jit-lock will be
called again when the display engine gets to that position.)  After
fontification-functions return, the display engine processes the faces
in the chunk that has been fontified; then it continues marching
through the buffer.

Next, I need to describe how scroll commands work.  Assuming you typed
C-v, the scroll command computes the new window-start point.  It does
so by beginning at the current window-start, then moving through the
buffer text until it reaches the point that is N pixels down, where N
is the height of the window.  To move through the text, the scroll
command uses functions from the display engine that "simulate"
display, i.e. they perform the same layout calculations, but don't
prepare anything to be sent to the glass.  However -- and this is the
important part -- this "display simulation" uses the same code
described above that looks for positions where the 'fontified'
property is nil and calls fontification-functions to fontify such
text.  This is necessary, because fontification can change the metrics
of the text on display, and so any pixel-wise calculations and
decisions whether given text is or isn't inside a window must be sure
the text is fontified and has the right faces in order to produce
accurate results.

When the scroll command is done computing the new window-start point,
it sets a special flag telling redisplay to obey that window-start
value (otherwise redisplay might decide to compute window-start on its
own), and returns.

Then redisplay kicks in, and does its thing described above.  It will
find the entire text until the new window-start already fontified, and
also some text after the new window-start (because jit-lock works in
chunks, and usually fontifies slightly more than strictly needed).  So
displaying the new contents of the window will need to fontify only
those parts visible in the window that were not already fontified when
the scroll command looked for the new window-start.

If you lean on C-v, then the scroll command is repeatedly called to
find the next window-start, and repeatedly fontifies the text it goes
through.  If you scroll N screen-fulls, the fontification of the first
N-1 screen-fulls (if they were never displayed before, which happens
when you visit a file and immediately start scrolling) is "wasted
effort" for the purposes of the final window redisplay, because those
N-1 screen-fulls will not be displayed.  Only the fontification of the
last screen-full is "useful", because it saves some, sometimes all, of
the fontification job of the stuff to be eventually displayed when you
stop scrolling.

Enter fast-but-imprecise-scrolling.  What it does is disable jit-lock
during the part where the scroll command walked the buffer to find the
next window-start position.  Crucially, it also disables putting the
non-nil 'fontified' property on the text traversed by the scroll
command (because that is done by jit-lock).  So when you stop
scrolling, redisplay of the stuff that ends up in the window will have
to fontify all of what is visible in the window.

Bottom line: fast-but-imprecise-scrolling saves the work of jit-lock
when scrolling more than one screen-full, and is more-or-less useless
when scrolling one screen-full.  IOW, it is a "solution" for the use
case of leaning on C-v or M-v to scroll far into the buffer that was
never before displayed.

Of course, if fast-but-imprecise-scrolling makes the job of scroll
commands so much easier that Emacs is capable of keeping up with the
keyboard auto-repeat rate, then every screen-full you scroll through
_will_ be displayed, and therefore will be fontified by the calls to
jit-lock from redisplay itself -- and that will again make scrolling
slower.  So fast-but-imprecise-scrolling is most efficient when it
makes scroll commands faster, but not too fast...

Sorry for the long email, I hope this tricky issue is now at least a
tad more clear.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 6 Apr 2020 21:59:29 +0300
> 
> Would I be correct to assume that you see only see this in CC Mode based 
> major mode, and only when editing big enough files? Such as xdisp.c 
> mentioned many times recently.

The former is probably correct.  But not the latter: there are some
constructs in C that make CC Mode notoriously slow, even in much
smaller buffers.  And just to make "notoriously slow" quantifiable: I
mean cases where typing RET takes half a second or a second.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 17:30, Eli Zaretskii wrote:
>> Cc: rudalics@gmx.at, acm@muc.de, rrandresf@gmail.com, rms@gnu.org,
>>   emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Mon, 6 Apr 2020 21:59:29 +0300
>>
>> Would I be correct to assume that you see only see this in CC Mode based
>> major mode, and only when editing big enough files? Such as xdisp.c
>> mentioned many times recently.
> 
> The former is probably correct.  But not the latter: there are some
> constructs in C that make CC Mode notoriously slow, even in much
> smaller buffers.  And just to make "notoriously slow" quantifiable: I
> mean cases where typing RET takes half a second or a second.

In that case, there are several issues at play.

Scenarios that are made slower if the file is large are probably caused 
by the same architectural decisions that make (find-file "src/xdisp.c") 
slow-ish. All those could likely be improved if CC Mode migrated to 
using syntax-propertize-function, with its on-demand approach to parsing.

Scenarios which are slow no matter what probably deserve a separate bug 
report each, and are likely fixable without significant redesign.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Tue, Apr 07, 2020 at 18:04:13 +0300, Dmitry Gutov wrote:

> In that case, there are several issues at play.

> Scenarios that are made slower if the file is large are probably caused 
> by the same architectural decisions that make (find-file "src/xdisp.c") 
> slow-ish. All those could likely be improved if CC Mode migrated to 
> using syntax-propertize-function, with its on-demand approach to parsing.

Please would you stop saying this.  It's not true, and it gets very
wearing.  The crude syntax-propertize-function mechanism would have to
be considerably enhanced before it could support CC Mode's features.
And even then, it would be unlikely to be faster than the current CC
Mode.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 7 Apr 2020 18:04:13 +0300
> Cc: rudalics@gmx.at, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
>  acm@muc.de
> 
> Scenarios which are slow no matter what probably deserve a separate bug 
> report each

Agreed, let's get that ball rolling.

> and are likely fixable without significant redesign.

Unfortunately, I'm not as optimistic as you on that one.

So here's the first bug report about a random slowness in CC Mode:

  . visit src/w32.c on the emacs-27 branch and go to line 8747
  . C-e DEL -- this deletes the last 'e' of "else"
  . the following line gets immediately fontified as a function name
  . type 'e' -- then watch how the fontification of the next line to
    its original face takes almost 1 sec -- this is the problem

FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
Emacs 26.3 takes only slightly less time in this case.

(I will file a formal bug report about this if Alan wants me to.)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 07.04.2020 20:42, Alan Mackenzie wrote:

>> Scenarios that are made slower if the file is large are probably caused
>> by the same architectural decisions that make (find-file "src/xdisp.c")
>> slow-ish. All those could likely be improved if CC Mode migrated to
>> using syntax-propertize-function, with its on-demand approach to parsing.
> 
> Please would you stop saying this.  It's not true, and it gets very
> wearing.  The crude syntax-propertize-function mechanism would have to
> be considerably enhanced before it could support CC Mode's features.

I will shut up about it now (saying it twice it plenty), but I am pretty 
confident saying that if you manage to migrate to s-p-f, file opening 
time will go down. Performance while typing is likely to improve too, at 
least when the same buffer is not shown in another window, many thousand 
lines later. It's not a given, but I'd really like to see you try 
collaborating with Stefan. At least for a prototype, to have a better 
idea of the associated performance tradeoffs. He posted a preliminary 
patch in the past, FWIW.

"Considerable enhancement" can also be a part of that discussion.

> And even then, it would be unlikely to be faster than the current CC
> Mode.

Some scenarios can become slower, that's for sure. But the more common 
ones can get faster. We won't know until we try.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 07.04.2020 22:22, Eli Zaretskii wrote:

>> and are likely fixable without significant redesign.
> 
> Unfortunately, I'm not as optimistic as you on that one.

In any case, s-p-f is unlikely to help there. ¯\_(ツ)_/¯

> So here's the first bug report about a random slowness in CC Mode:

Thank you.

>    . visit src/w32.c on the emacs-27 branch and go to line 8747
>    . C-e DEL -- this deletes the last 'e' of "else"
>    . the following line gets immediately fontified as a function name
>    . type 'e' -- then watch how the fontification of the next line to
>      its original face takes almost 1 sec -- this is the problem

I can and can't reproduce this at the same time.

Meaning, I do see "fontification of the next line to its original face" 
take some time. But if I don't wait for it to turn black again and 
continue typing, nothing bad happens: Emacs continues reacting to input 
just as promptly, and the next line turns black soon enough.

And if I type e.g. 'e RET', or 'e SPC', the next line turns black 
seemingly instantly.

> FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
> Emacs 26.3 takes only slightly less time in this case.

'emacs -Q' and a '-Og -g3' build of 27.0.90 here.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Tue, Apr 07, 2020 at 22:22:43 +0300, Eli Zaretskii wrote:
> > From: Dmitry Gutov <dgutov@yandex.ru>
> > Date: Tue, 7 Apr 2020 18:04:13 +0300
> > Cc: rudalics@gmx.at, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
> >  acm@muc.de

> > Scenarios which are slow no matter what probably deserve a separate bug 
> > report each

> Agreed, let's get that ball rolling.

> > and are likely fixable without significant redesign.

> Unfortunately, I'm not as optimistic as you on that one.

> So here's the first bug report about a random slowness in CC Mode:

>   . visit src/w32.c on the emacs-27 branch and go to line 8747
>   . C-e DEL -- this deletes the last 'e' of "else"
>   . the following line gets immediately fontified as a function name
>   . type 'e' -- then watch how the fontification of the next line to
>     its original face takes almost 1 sec -- this is the problem

OK, I'm on Line 8759, where I've got:

  else
    cp->status = STATUS_READ_FAILED;

(I haven't updated for a few days.)  Is this the same place?

> FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
> Emacs 26.3 takes only slightly less time in this case.

Is this not jit-lock-context-time?

It takes 0.5s on my optimised build.  j-l-context-time is 0.5 here.
I'll admit I can't change the delay by changing j-l-c-t, but that might
be due to not reinitialising jit-lock properly.

However, first setting font-lock-support-mode to nil, then
reinitialising font-lock, deleting then reinserting that "e", the line
after the else _never_ gets its (lack of) face back.

> (I will file a formal bug report about this if Alan wants me to.)

No, it's fine like this!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Tue, Apr 07, 2020 at 22:45:25 +0300, Dmitry Gutov wrote:
> Hi Alan,

> On 07.04.2020 20:42, Alan Mackenzie wrote:

> >> Scenarios that are made slower if the file is large are probably caused
> >> by the same architectural decisions that make (find-file "src/xdisp.c")
> >> slow-ish. All those could likely be improved if CC Mode migrated to
> >> using syntax-propertize-function, with its on-demand approach to parsing.

> > Please would you stop saying this.  It's not true, and it gets very
> > wearing.  The crude syntax-propertize-function mechanism would have to
> > be considerably enhanced before it could support CC Mode's features.

> I will shut up about it now (saying it twice it plenty), but I am pretty 
> confident saying that if you manage to migrate to s-p-f, file opening 
> time will go down.

I'm sure it would.  If file opening time were really a concern, a hybrid
algorithm would perhaps be the best way: apply the text properties first
in a lazy fashion, and thereafter treat them with care, as CC Mode
currently does.  But this would merely transfer the start up time to the
time taken in early scrolls forward.

> Performance while typing is likely to improve too, at least when the
> same buffer is not shown in another window, many thousand lines later.

What makes you think this?  Anyway, the case you cite has to be handled
well, too.

> It's not a given, but I'd really like to see you try collaborating
> with Stefan. At least for a prototype, to have a better idea of the
> associated performance tradeoffs. He posted a preliminary patch in the
> past, FWIW.

> "Considerable enhancement" can also be a part of that discussion.

The syntax-propertize-function mechanism works by erasing ALL
syntax-table properties after a change point, then reapplying them
lazily.  Considering that s-t properties have an overwhelmingly local
effect, this is very wasteful of processor time.

Consider, for example, editing within a large C++ raw string, a common
occurrence.  You yourself reported as a bug sluggish performance here in
mid 2016.  The cause was erasing too many s-t text properties at a
buffer change.  I think we were talking about 1 second per typed
character in the scenario you gave.  There are typically lots of these
properties in a raw string, in particular on " characters.

Consider(2) a C++ template: excusing my C++ syntax knowledge, type in

   template class foo < bar, baz >= bar>

, perhaps typing in the odd newline inside the template (a common
occurrence), or nesting further templates inside it (also a common
occurrence).  Note how the parenthesis text properties are added and
removed as you type.  All these modification are necessary, and they are
largely _before_ the point of insertion, not after it.

Splatting all the properties after the point of change would be
unnecessary and wasteful.

> > And even then, it would be unlikely to be faster than the current CC
> > Mode.

> Some scenarios can become slower, that's for sure. But the more common 
> ones can get faster. We won't know until we try.

Trying would be a _lot_ of work.  How is one to handle the common
example scenarios above?  Well, you'd have to enhance the
syntax-propertize-function with a means of determining a start position
for erasing s-t props, and also a stop position.  Once you do that,
you're effectively doing what CC Mode currently does, so where's the
speed advantage coming from?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > 
  > >   > Recent Emacsen either ignore that variable or silently reset it to nil
  > >   > internally so it doesn't get into their way.  Their progmodes either
  > >   > always scan an entire buffer from its beginning or use some elaborate,
  > >   > fragile techniques to find such a top level position.  Moreover, our
  > >   > underlying mechanism for syntax highlighting always marks the entire
  > >   > rest of a buffer as dirty after every single editing change.  This has
  > >   > the consequence that that entire part has to be continuously rescanned
  > >   > when some of it is shown in another window.
  > > 
  > > Does anyone disagree with this specific factual claim?

  > I'm not sure what is "the claim" here,

The factual statements in the paragraph I quoted, above.

					   but I want to point out a small
  > inaccuracy: redisplay doesn't "continuously rescan the entire rest of
  > the buffer", it only rescans the part(s) shown in windows.	(It might
  > happen that some major mode's font-lock definitions end up rescanning
  > much more, but that's a separate issue.)

Isn't that separate issue the issue we are talking about.

					      And frankly, what would we
  > like Emacs to do instead?

It would scan only from the last open-paren-in-column-zero, as it did
in the past.

			       A change in a buffer can potentially affect
  > the fontification of the rest of the buffer, and I don't think we
  > would like Emacs to fail to update other windows showing the same
  > buffer

I have a feeling there has been a change of topic here, but I can't be
sure.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > Yes, that is what I meant.  The "kind of things" is nothing more than
  > >   > writing comments in C (etc.) source code.  Sooner or later, such a
  > >   > comment has a parenthesis in it, and sooner or later (e.g. by M-q), this
  > >   > will end up in column 0,
  > >
  > > I am surprised, because that never happens to me.

  > Of course that never happens to you: you're the one who introduced the
  > open-paren-in-column-0 heuristic, so you're clearly biased.

Please focus on the issue not on personalities.  Anyone can be mistaken,
but don't argue that an objective claim is wrong based on who made it.

What matters is whether one can get good results from
open-paren-in-column-0, good enough to make it usable and useful.

I found the open-paren-in-column-0 heuristic satisfactory because in
practice I saw that open-parens did not drift into column zero.  Other
people have said that this does happen to them, and they can't avoid
it.

They, and I, were using the same editor, with the same heuristic, and
getting totally different results.  Why is that?  Surely that is pertinent.
I conjecture it is because of indentation conventions, but if that is not
the reason, what is?

  > > What do you think of such indentation as a solution for this problem.

  > That presumes you have control over the code (and coding conventions).

It presumes that the developers adopt the practice of indenting the
text of block comments, or can be persuaded to do so.

IF my conjecture is correct, open-paren-in-column-0 is useful for
programs where block comments are indented.  Is that conjecture
correct?  How about if we finish comparing notes, and see.

If that is how things are, it should be useful to support
open-paren-in-column-0-is-defun-start = t rather than make it a no-op.
If the code you edit does not indent block comments, you would
be happier with open-paren-in-column-0-is-defun-start = nil,
but you don't need that feature to be no-op'ed.

(I did not start indenting the text of block comments because of
the open-paren-in-column-0 heuristic.  I did it because that looks nice.)

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

You've written an extremely clear explanation of how fontification and
scrolling interact.  I'm impressed.

  > Of course, if fast-but-imprecise-scrolling makes the job of scroll
  > commands so much easier that Emacs is capable of keeping up with the
  > keyboard auto-repeat rate, then every screen-full you scroll through
  > _will_ be displayed, and therefore will be fontified by the calls to
  > jit-lock from redisplay itself -- and that will again make scrolling
  > slower.  So fast-but-imprecise-scrolling is most efficient when it
  > makes scroll commands faster, but not too fast...

I think that is true, if "faster" and "slower" are interpreted in
terms of CPU time used.  But if all of that work, including the
fontification of each screenful, is so quick that Emacs never falls
behind, I think it won't seem like "slowness" to the user -- who never
has to _wait_ for it.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> I found the open-paren-in-column-0 heuristic satisfactory because in
> practice I saw that open-parens did not drift into column zero.

Yes, it was a good heuristic when we needed one and when a heuristic was
good enough.  Nowadays it's not good enough any more.  Users who
experience a misbehavior because some "stray" open-paren in column-0
don't think "oh silly me, I shouldn't do that".  Instead they think
"Duh!  Emacs is stupid!".

> It presumes that the developers adopt the practice of indenting the
> text of block comments, or can be persuaded to do so.

What about those who can't or don't want to?

Should Emacs provide poor behavior when open parens in column 0 occur
for non-defuns?  Should Emacs provide poor performance when the buffer
doesn't have any open-parens in column 0 (as is the common case in many
languages)?

For these reasons, we came up with a different approach, not based on
a heuristic.  To make it work sufficiently efficiently it relies
on caching.  It's not always as efficient as the old heuristic, but on
the average it is more efficient, in the worst case it's a bit less
efficient, and in the best case it's a lot more efficient.

OK, I've wasted enough time in this thread, so I'll stop here.

I just recommend anyone who wants to hack on this problem to first do
some careful profiling (both at the Elisp level and the C level).
Then see if maybe the performance improvement that they're aiming for
can't be obtained simply by optimizing syntax-ppss (e.g. it's a pretty
simple code so it would not be hard to move it to C, if needed).

Finally, I'd urge anyone who's not willing to spend this effort to stop
arguing here about what they think should be done, because the likelihood
that the profiler would prove them wrong is just much too high.


        Stefan

Re: Inlining policy

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

On 4/5/20 3:23 AM, Andrea Corallo wrote:
> what about having a new macro to control the key operations we want to
> always have inlined at -O0 and -Og instead of falling back into defining
> these as macros?

I looked into this some more, and found some problems with using 
__attribute__((always_inline)), such as GCC complaining that it can't inline 
some functions in some cases on some platforms. It was a pain to suss out what 
would improve performance while avoiding the warnings, and while fiddling with 
this a bit I found a simpler change that had even better performance than 
__attribute__((always_inline)) did, at least on one platform for my standard 
benchmark of 'make compile-always'. So I installed the two attached patches 
instead. Together they improve Emacs CPU performance for 'gcc -Og' by about 60% 
on my benchmark.

I kept the DEFINE_KEY_OPS_AS_MACROS stuff because of the -O0 performance issues 
that we discussed earlier. Once we no longer need to worry about -O0 performance 
when debugging, we can get rid of that stuff.

Thanks for suggesting inlining as a way to improve Emacs performance when 
compiled with -Og.

[-- Attachment #2: 0001-Improve-inlining-when-compiling-with-Og.patch --]
[-- Type: text/x-patch, Size: 2538 bytes --]

From b9b32074cecb0e0d20cf527eadacbb0b7f5cced9 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 7 Apr 2020 16:57:04 -0700
Subject: [PATCH 1/2] Improve inlining when compiling with -Og
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* src/conf_post.h (EXTERN_INLINE) [!EMACS_EXTERN_INLINE]:
Make it static inline, not merely static.  This is a worthwhile
performance improvement on my two platforms A and B (see below).
On my platform A this change improves user+system CPU performance of
‘make compile-always’ by 52% on an -Og build, and by 1.4% on the
default -O2 build.  On my platform B this improves the same benchmark
by 41% on an -Og build, and by -0.8% on the default -O2 build.
That "-0.8%" is a small negative for this change, and I recall that
it is why I didn't make this change earlier.  However, Platform B uses
an older GCC so we needn't worry overmuch about this small negative.

With this change the performance advantage of -O2 over -Og has dropped
on platform A; formerly -O2 was 87% faster than -Og, and now it is
only 25% faster.  On platform B the performance advantage of -O2 over
-Og has dropped from being 62% faster to being 14% faster.

Platform A is GCC 9.3.1 20200317 (Red Hat 9.3.1-1) on Fedora 31
x86-64 (AMD Phenom II X4 910e, circa 2010).  Platform B is GCC (Ubuntu
7.5.0-3ubuntu1~18.04) 7.5.0 on Ubuntu 18.04.4 (Intel Xeon E3-1225 V2,
circa 2012).

This patch was inspired by a suggestion by Andrea Corallo in:
https://lists.gnu.org/r/emacs-devel/2020-04/msg00263.html
---
 src/conf_post.h | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/src/conf_post.h b/src/conf_post.h
index eb8fb18c00..79fb4cfe8a 100644
--- a/src/conf_post.h
+++ b/src/conf_post.h
@@ -432,15 +432,13 @@ extern int emacs_setenv_TZ (char const *);
 
 #else
 
-/* Use 'static' instead of 'extern inline' because 'static' typically
-   has better performance for Emacs.  Do not use the 'inline' keyword,
-   as modern compilers inline automatically.  ATTRIBUTE_UNUSED
-   pacifies gcc -Wunused-function.  */
+/* Use 'static inline' instead of 'extern inline' because 'static inline'
+   has much better performance for Emacs when compiled with 'gcc -Og'.  */
 
 # ifndef INLINE
 #  define INLINE EXTERN_INLINE
 # endif
-# define EXTERN_INLINE static ATTRIBUTE_UNUSED
+# define EXTERN_INLINE static inline
 # define INLINE_HEADER_BEGIN
 # define INLINE_HEADER_END
 
-- 
2.17.1


[-- Attachment #3: 0002-Revert-my-KEY_OPS_CFLAGS-change-to-src-Makefile.in.patch --]
[-- Type: text/x-patch, Size: 1777 bytes --]

From c36792bd79f3a97f292ab0612aa5c3ab1b4d9e70 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 7 Apr 2020 20:52:28 -0700
Subject: [PATCH 2/2] Revert my KEY_OPS_CFLAGS change to src/Makefile.in
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Now that -Og inlining has been improved this is no longer helpful.
* src/Makefile.in (KEY_OPS_CFLAGS): Remove.  All uses removed.
This improved CPU performance of ‘make compile-always’ by 5% on my
platform, which was gcc -Og, GCC 9.3.1 20200317 (Red Hat 9.3.1-1),
Fedora 31 x86-64 (AMD Phenom II X4 910e, circa 2010).
---
 src/Makefile.in | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/src/Makefile.in b/src/Makefile.in
index dfd322553b..552dd2e50a 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -377,14 +377,11 @@ pdmp :=
 # Flags that might be in WARN_CFLAGS but are not valid for Objective C.
 NON_OBJC_CFLAGS = -Wignored-attributes -Wignored-qualifiers -Wopenmp-simd
 
-# Cajole GCC into inlining key ops even if it wouldn't normally.
-KEY_OPS_CFLAGS = $(if $(filter -Og,$(CFLAGS)),-DDEFINE_KEY_OPS_AS_MACROS)
-
 # -Demacs makes some files produce the correct version for use in Emacs.
 # MYCPPFLAGS is for by-hand Emacs-specific overrides, e.g.,
 # "make MYCPPFLAGS='-DDBUS_DEBUG'".
-EMACS_CFLAGS = -Demacs $(KEY_OPS_CFLAGS) $(MYCPPFLAGS) \
-  -I. -I$(srcdir) -I$(lib) -I$(top_srcdir)/lib \
+EMACS_CFLAGS=-Demacs $(MYCPPFLAGS) -I. -I$(srcdir) \
+  -I$(lib) -I$(top_srcdir)/lib \
   $(C_SWITCH_MACHINE) $(C_SWITCH_SYSTEM) $(C_SWITCH_X_SITE) \
   $(GNUSTEP_CFLAGS) $(CFLAGS_SOUND) $(RSVG_CFLAGS) $(IMAGEMAGICK_CFLAGS) \
   $(PNG_CFLAGS) $(LIBXML2_CFLAGS) $(DBUS_CFLAGS) \
-- 
2.17.1

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Tue, 7 Apr 2020 20:10:18 +0000
> Cc: Dmitry Gutov <dgutov@yandex.ru>, rudalics@gmx.at, rrandresf@gmail.com,
>   emacs-devel@gnu.org, rms@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> OK, I'm on Line 8759, where I've got:
> 
>   else
>     cp->status = STATUS_READ_FAILED;
> 
> (I haven't updated for a few days.)  Is this the same place?

Yup.

> > FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
> > Emacs 26.3 takes only slightly less time in this case.
> 
> Is this not jit-lock-context-time?

I have no idea.  But if you are saying that this is "normal", then I
disagree: I don't see why the change in fontification after DEL is
instantaneous, while the change after typing 'e' takes such annoyingly
long time.  Is jit-lock-context-time not involved in the former? if
so, why?

> However, first setting font-lock-support-mode to nil, then
> reinitialising font-lock, deleting then reinserting that "e", the line
> after the else _never_ gets its (lack of) face back.

So this is still rather a mystery, isn't it?

Thanks.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 7 Apr 2020 23:05:04 +0300
> Cc: rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org, acm@muc.de,
>  emacs-devel@gnu.org
> 
> I can and can't reproduce this at the same time.
> 
> Meaning, I do see "fontification of the next line to its original face" 
> take some time. But if I don't wait for it to turn black again and 
> continue typing, nothing bad happens: Emacs continues reacting to input 
> just as promptly, and the next line turns black soon enough.

Whether this is "bad" depends on what you use the fontification for.
For me, it's a sign of syntax correctness (among other things), so
waiting for the confirmation of the syntax for such a relatively long
time is quite "bad".

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: rudalics@gmx.at, rrandresf@gmail.com, emacs-devel@gnu.org,
> 	acm@muc.de
> Date: Tue, 07 Apr 2020 22:29:21 -0400
> 
>   > >   > Recent Emacsen either ignore that variable or silently reset it to nil
>   > >   > internally so it doesn't get into their way.  Their progmodes either
>   > >   > always scan an entire buffer from its beginning or use some elaborate,
>   > >   > fragile techniques to find such a top level position.  Moreover, our
>   > >   > underlying mechanism for syntax highlighting always marks the entire
>   > >   > rest of a buffer as dirty after every single editing change.  This has
>   > >   > the consequence that that entire part has to be continuously rescanned
>   > >   > when some of it is shown in another window.
> 
> 					   but I want to point out a small
>   > inaccuracy: redisplay doesn't "continuously rescan the entire rest of
>   > the buffer", it only rescans the part(s) shown in windows.	(It might
>   > happen that some major mode's font-lock definitions end up rescanning
>   > much more, but that's a separate issue.)
> 
> Isn't that separate issue the issue we are talking about.

I'm not sure.  The text says "scan an entire buffer" and "the entire
part has to be continuously rescanned", which could be interpreted as
though this rescanning is the main culprit for the slowness.  I wanted
to point out that these aspects are not forced by the infrastructure
that causes the displayed text to be fontified just in time, but by
some of the major mode's fontification functions called by the
infrastructure.  It is IMO important to understand where to look for
code that needs to be sped up, to avoid wasting effort on optimizing
places that don't contribute significantly to the slowdown.

> 					      And frankly, what would we
>   > like Emacs to do instead?
> 
> It would scan only from the last open-paren-in-column-zero, as it did
> in the past.

That would produce incorrect display, as I explained later:

> 			       A change in a buffer can potentially affect
>   > the fontification of the rest of the buffer, and I don't think we
>   > would like Emacs to fail to update other windows showing the same
>   > buffer
> 
> I have a feeling there has been a change of topic here, but I can't be
> sure.

I surely didn't mean to change the topic.  If the argument was that
stopping considering the fontification at some arbitrary point in the
buffer will improve performance, my point was that it OTOH might
produce inaccurate display, which I consider not a good idea.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: dgutov@yandex.ru, acm@muc.de, rrandresf@gmail.com,
> 	emacs-devel@gnu.org, rudalics@gmx.at
> Date: Tue, 07 Apr 2020 22:29:30 -0400
> 
> You've written an extremely clear explanation of how fontification and
> scrolling interact.  I'm impressed.

Thanks.  I thought it was important to explain this in so much detail
so that everyone interested in this issue will be on the same page.
What happens in this use case is quite complex, and many people don't
know about these details and therefore are very surprised by some
effects of some changes.

>   > Of course, if fast-but-imprecise-scrolling makes the job of scroll
>   > commands so much easier that Emacs is capable of keeping up with the
>   > keyboard auto-repeat rate, then every screen-full you scroll through
>   > _will_ be displayed, and therefore will be fontified by the calls to
>   > jit-lock from redisplay itself -- and that will again make scrolling
>   > slower.  So fast-but-imprecise-scrolling is most efficient when it
>   > makes scroll commands faster, but not too fast...
> 
> I think that is true, if "faster" and "slower" are interpreted in
> terms of CPU time used.

The CPU time is one major factor, of course.  But not the only one,
because the keyboard auto-repeat rate is a separate factor which can
be controlled and changed independently of the CPU speed.

> But if all of that work, including the fontification of each
> screenful, is so quick that Emacs never falls behind, I think it
> won't seem like "slowness" to the user -- who never has to _wait_
> for it.

That's true, to some extent, but my point was that the elapsed time
will become slower when each screenful is actually displayed while
scrolling.  It isn't an important point, though, just a side comment.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Eli.

On Wed, Apr 08, 2020 at 09:15:49 +0300, Eli Zaretskii wrote:
> > Date: Tue, 7 Apr 2020 20:10:18 +0000
> > Cc: Dmitry Gutov <dgutov@yandex.ru>, rudalics@gmx.at, rrandresf@gmail.com,
> >   emacs-devel@gnu.org, rms@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > OK, I'm on Line 8759, where I've got:

> >   else
> >     cp->status = STATUS_READ_FAILED;

> > (I haven't updated for a few days.)  Is this the same place?

> Yup.

> > > FTR, this is in Emacs 27.0.90 built with -Og, but a -O2 compilation of
> > > Emacs 26.3 takes only slightly less time in this case.

> > Is this not jit-lock-context-time?

> I have no idea.  But if you are saying that this is "normal", then I
> disagree: I don't see why the change in fontification after DEL is
> instantaneous, while the change after typing 'e' takes such annoyingly
> long time.  Is jit-lock-context-time not involved in the former? if
> so, why?

Is the problem more the asymmetry between the deletion and the
insertion than the actual time taken after the reinsertion?  We've had
context fontification since Emacs 21.

I admit I don't remember in detail why after the deletion the font
locking region is extended, but not after the reinsertion.  Possibly in
the former case, context fontification failed to do the job right.

> > However, first setting font-lock-support-mode to nil, then
> > reinitialising font-lock, deleting then reinserting that "e", the line
> > after the else _never_ gets its (lack of) face back.

> So this is still rather a mystery, isn't it?

It's context fontification.  I'm quite sure of that, now.  To restart
jit-lock with a new j-l-context-time involves explicitly getting rid of
a timer and allowing it to restart.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Wed, 8 Apr 2020 07:02:15 +0000
> Cc: dgutov@yandex.ru, rudalics@gmx.at, rrandresf@gmail.com,
>   emacs-devel@gnu.org, rms@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > Is this not jit-lock-context-time?
> 
> > I have no idea.  But if you are saying that this is "normal", then I
> > disagree: I don't see why the change in fontification after DEL is
> > instantaneous, while the change after typing 'e' takes such annoyingly
> > long time.  Is jit-lock-context-time not involved in the former? if
> > so, why?
> 
> Is the problem more the asymmetry between the deletion and the
> insertion than the actual time taken after the reinsertion?

Not the asymmetry, the slow refontifciation in the latter case.  It
won't help me if both refontifciations were "symmetrically" slow.

> > > However, first setting font-lock-support-mode to nil, then
> > > reinitialising font-lock, deleting then reinserting that "e", the line
> > > after the else _never_ gets its (lack of) face back.
> 
> > So this is still rather a mystery, isn't it?
> 
> It's context fontification.  I'm quite sure of that, now.  To restart
> jit-lock with a new j-l-context-time involves explicitly getting rid of
> a timer and allowing it to restart.

Then perhaps the solution is to reduce the default value of
jit-lock-context-time?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Date: Wed, 08 Apr 2020 10:10:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rudalics@gmx.at, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
>  dgutov@yandex.ru
> 
> > It's context fontification.  I'm quite sure of that, now.  To restart
> > jit-lock with a new j-l-context-time involves explicitly getting rid of
> > a timer and allowing it to restart.
> 
> Then perhaps the solution is to reduce the default value of
> jit-lock-context-time?

Btw, I cannot wrap my head around this "deferred context
fontification" stuff; never could.

The doc string of jit-lock-contextually says:

  If t, means fontification occurs on those lines modified and all
  subsequent lines.  This means those subsequent lines are refontified to reflect
  their new syntactic context, after `jit-lock-context-time' seconds.
  If any other value, e.g., `syntax-driven', means syntactically true
  fontification occurs only if syntactic fontification is performed using the
  buffer mode's syntax table, i.e., only if `font-lock-keywords-only' is nil.

I understand what t means and does, but what does non-nil, non-t value
do, and how is it different from t? in particular, does it also wait for
jit-lock-context-time of idleness?

The doc string of jit-lock-mode also seems to document only the t
value:

  - Deferred context fontification if `jit-lock-contextually' is
    non-nil.  This means fontification updates the buffer corresponding to
    true syntactic context, after `jit-lock-context-time' seconds of Emacs
    idle time, while Emacs remains idle.  Otherwise, fontification occurs
    on modified lines only, and subsequent lines can remain fontified
    corresponding to previous syntactic contexts.  This is useful where
    strings or comments span lines.

The code doesn't seem to start the jit-lock-context-timer if the value
is non-t, or am I missing something?  Since the default value is
'syntax-driven', where does that 0.5-second delay we all see come
from?  I guess you (Alan) have this figured out, so please share what
you've learned, and let's update the documentation to benefit from
that.

Thanks.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > To be accurate, the comment started by '/*' would end at the first occurrence of '*/' somewhere down the buffer.

If it appears before the open brace in column zero, yes.

 > If you insert or close a paren, however, that would definitely change syntax state at EOB.

It would still only get as far as to the next brace in column zero.

 > According to Alan's experiment, you can get this behavior now already simply by setting comment-use-syntax-ppss to nil and open-paren-in-column-0-is-defun-start to t.
 >
 > Does it improve your experience significantly?

No.  Fast and imprecise scrolling doesn't improve anything either here.
I'm surprised how people get improved behavior from these.  But I agree
with Stefan to shut up here from now on.  I don't have the time to
conduct an in-depth analysis of this problem and spotting a possible
culprit without such an analysis is, in fact, futile.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> As a matter of fact, it is slightly better.  Still, Emacs locks down
 >> after a few scrolls in a similar way as without it.
 >
 > And can it similarly lock down for a long time? Or does it unfreeze quickly?

It doesn't lock down always.  But when it locks down it takes about the
same time to recover.

 > What about other major modes? If you create a big JS file and try the
 > same experiment in js-mode? Or create a long Elisp file and do the
 > same in emacs-lisp-mode (you can copy-paste a lot of time the contents
 > of some existing file)? Or take any of those files, switch to
 > fundamental-mode and do the same.

I don't use JS so I have no idea how to experiment with it.  Scrolling a
large Elisp file never locks down Emacs.  But note that the problem with
scrolling happens already with dispextern.h whose size is about a tenth
of that of xdisp.c.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> It depends on what we consider a failure: I do not think that inserting
 >> a "/*" in my window showing the top of my C buffer should cause the
 >> window showing the bottom of my buffer consider everything part of a
 >> comment starting at that "/*".
 >
 > If that is what the language syntax rules say, then yes, we should
 > paint everything in comment face.  If you disagree, then let's agree
 > to disagree: I consider "inaccurate" display one of the worst things
 > an editor can do to its users.

Turning the tables, I don't consider inaccurate editing (like inserting
an unbalanced paren or delimiter) one of the worst things a user can do
to its editor.  For me, an open paren in column zero is some sort of an
agreement between me and the editor we both would heed as accurately as
we can.  People like you who don't care would not even pay attention to
it.  People like me who do care might profit from it.  It certainly
isn't a cure of all problems and might not even help with scrolling
buffers.  But it would provide one knob a user could press to find out.

 >> So what I would vote for in this case is that font-lock would specially
 >> highlight the open paren that "stops seeing the comment as comment" so
 >> the user is aware of the fact.
 >
 > How would that help if that parentheses is not shown in any window?

Why bother?  The comment would appear highlighted as comment down to
that parenthesis or to the end of the window, whatever comes first.

 > And even if it is shown, do we really expect users to pay attention to
 > that, and immediately understand that this parentheses with an unusual
 > face explains the wrong display?

I certainly would.  In my last commit I did insert such a paren on line
8022 of xfns.c and didn't notice it.  OTOH, the average user who never
customized 'open-paren-in-column-0-is-defun-start' would not be aware of
it.

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 > It stops when it gets to a buffer position where the
 > 'fontified' text property changes its value.  If the new value of
 > 'fontified' is nil, the display engine calls fontification-functions.
 > jit-lock adds its function to that hook, so jit-lock is called.  What
 > jit-lock does, in a nutshell, is call the current buffer's
 > fontification function(s) on a chunk of text starting at the position
 > for which it was called, and it puts on that chunk a value of t for
 > the 'fontified' text property.  (Where that chunk ended is the
 > position where the 'fontified' property's value will again change to
 > nil, and if that position is inside the window, jit-lock will be
 > called again when the display engine gets to that position.)  After
 > fontification-functions return, the display engine processes the faces
 > in the chunk that has been fontified; then it continues marching
 > through the buffer.

IIRC fontification can also modify text before the position where the
'fontified' property changed to nil.  Or is that wrong?

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[martin rudalics]

 >> Is this not jit-lock-context-time?
 >
 > I have no idea.  But if you are saying that this is "normal", then I
 > disagree: I don't see why the change in fontification after DEL is
 > instantaneous, while the change after typing 'e' takes such annoyingly
 > long time.  Is jit-lock-context-time not involved in the former? if
 > so, why?

For me that behavior is "normal" in the sense that I see it happen here
all the time, when editing C files, on the average once in a minute.
But since it never disrupts my typing I simply got used to it.

What's bothering me more is that when I put an overlay somewhere in a
buffer and eventually remove it, underlying text properties provided by
the font locking mechanism are never restored unless I explicitly edit
the line where such properties should be seen.  Apparently so, because
font locking does not trigger when overlays get removed because that
does not count as a buffer change.  Another thing I got used to ...

martin

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: dgutov@yandex.ru, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Wed, 8 Apr 2020 10:38:30 +0200
> 
> What's bothering me more is that when I put an overlay somewhere in a
> buffer and eventually remove it, underlying text properties provided by
> the font locking mechanism are never restored unless I explicitly edit
> the line where such properties should be seen.  Apparently so, because
> font locking does not trigger when overlays get removed because that
> does not count as a buffer change.

Overlay changes do count as buffer changes, for the purposes of
redisplay, so a bug report with a recipe will be appreciated.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 08.04.2020 11:36, martin rudalics wrote:
> I don't use JS so I have no idea how to experiment with it.  Scrolling a
> large Elisp file never locks down Emacs.

OK, thanks. That does point to CC Mode, rather than some general display 
engine problems.

> But note that the problem with
> scrolling happens already with dispextern.h whose size is about a tenth
> of that of xdisp.c.

Yeah, scrolling performance shouldn't be affected by the size of the 
file anyway. It's only editing performance that could.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 8 Apr 2020 13:37:08 +0300
> 
> > But note that the problem with
> > scrolling happens already with dispextern.h whose size is about a tenth
> > of that of xdisp.c.
> 
> Yeah, scrolling performance shouldn't be affected by the size of the 
> file anyway.

Scrolling performance _can_ be affected by the size of the buffer
(especially if the major mode's fontifications look far back), but
only when you scroll close to its end.  When you start scrolling from
the beginning, all buffers will behave similarly.  I think what martin
says is that the problem starts happening almost immediately, so being
some relatively short distance from the file's beginning already
produces the slowdown.  Which then will affect smaller files as well,
of course.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: martin rudalics <rudalics@gmx.at>
> Date: Wed, 8 Apr 2020 10:37:46 +0200
> 
>  > It stops when it gets to a buffer position where the
>  > 'fontified' text property changes its value.  If the new value of
>  > 'fontified' is nil, the display engine calls fontification-functions.
>  > jit-lock adds its function to that hook, so jit-lock is called.  What
>  > jit-lock does, in a nutshell, is call the current buffer's
>  > fontification function(s) on a chunk of text starting at the position
>  > for which it was called, and it puts on that chunk a value of t for
>  > the 'fontified' text property.  (Where that chunk ended is the
>  > position where the 'fontified' property's value will again change to
>  > nil, and if that position is inside the window, jit-lock will be
>  > called again when the display engine gets to that position.)  After
>  > fontification-functions return, the display engine processes the faces
>  > in the chunk that has been fontified; then it continues marching
>  > through the buffer.
> 
> IIRC fontification can also modify text before the position where the
> 'fontified' property changed to nil.  Or is that wrong?

What is actually fontified depends on the buffer's
fontification-functions; jit-lock just calls those functions with a
region that starts at the first position in the window that has its
'fontified' property nil, and ends jit-lock-chunk-size (500 bytes by
default) after that.  However, the 'fontified' property will be set by
jit-lock to t only for text in that chunk, as jit-lock has no way of
knowing what was actually fontified.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 08.04.2020 14:22, Eli Zaretskii wrote:
> Scrolling performance_can_  be affected by the size of the buffer
> (especially if the major mode's fontifications look far back)

That's true, of course, but we have various options to avoid that, so 
wouldn't expect this particular failing from a mature major mode package.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> The doc string of jit-lock-contextually says:
>
>   If t, means fontification occurs on those lines modified and all
>   subsequent lines.  This means those subsequent lines are refontified to reflect
>   their new syntactic context, after `jit-lock-context-time' seconds.
>   If any other value, e.g., `syntax-driven', means syntactically true
>   fontification occurs only if syntactic fontification is performed using the
>   buffer mode's syntax table, i.e., only if `font-lock-keywords-only' is nil.
>
> I understand what t means and does, but what does non-nil, non-t value
> do, and how is it different from t?

This is a global variable.  The non-nil non-t value means "treat it as
nil in some buffers and as t in others, depending on
`font-lock-keywords-only`" (well, more or less: other users of jit-lock
than font-lock can also affect it, but as a first approximation that's
about it).

> in particular, does it also wait for jit-lock-context-time
> of idleness?

Yes.

> The code doesn't seem to start the jit-lock-context-timer if the value
> is non-t, or am I missing something?

Yes, you're missing:

    (when (and contextual jit-lock-contextually)
      (setq-local jit-lock-contextually t))

which will turn the non-nil non-t value into t when applicable.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> What's bothering me more is that when I put an overlay somewhere in a
> buffer and eventually remove it, underlying text properties provided by
> the font locking mechanism are never restored

Overlays should usually not affect the way font-lock *puts* properties.
So when you remove the overlay, there should be no need to re-run
font-lock, and the mere redisplay should make the underlying text
properties (previously installed by font-lock) visible.

But all this depends on specifics, so without knowing what kind of
overlay you're putting/removing when/where/how it's hard to know where's
the problem.

IOW: make it a bug report.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> IIRC fontification can also modify text before the position where the
> 'fontified' property changed to nil.

It can do basically anything it likes, so yes, that can happen.
That can also cause the display to be incorrect because the redisplay
has already rendered that text.

> Or is that wrong?

It's not wrong, but if you *need* it, then you may have to do extra
work to make sure it's handled correctly.

E.g. jit-lock clients can tell jit-lock which part of the buffer they
actually fontified by returning a value of the form
`(jit-lock-bounds START . END)`.  START can then find its way to the
following code (under the name `loose-beg`):

               ;; The redisplay engine has already rendered the buffer up-to
               ;; `orig-start' and won't notice if the above jit-lock-functions
               ;; changed the appearance of any part of the buffer prior
               ;; to that.  So if `loose-beg' is before `orig-start', we need to
               ;; cause a new redisplay cycle after this one so that the changes
               ;; are properly reflected on screen.
               ;; To make such repeated redisplay happen less often, we can
               ;; eagerly extend the refontified region with
               ;; jit-lock-after-change-extend-region-functions.
               (when (< loose-beg orig-start)
                 (run-with-timer 0 nil #'jit-lock-force-redisplay
                                 (copy-marker loose-beg)
                                 (copy-marker orig-start)))


-- Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: acm@muc.de,  rudalics@gmx.at,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  dgutov@yandex.ru
> Date: Wed, 08 Apr 2020 10:22:29 -0400
> 
> > I understand what t means and does, but what does non-nil, non-t value
> > do, and how is it different from t?
> 
> This is a global variable.  The non-nil non-t value means "treat it as
> nil in some buffers and as t in others, depending on
> `font-lock-keywords-only`" (well, more or less: other users of jit-lock
> than font-lock can also affect it, but as a first approximation that's
> about it).
> 
> > in particular, does it also wait for jit-lock-context-time
> > of idleness?
> 
> Yes.

Thanks.

> > The code doesn't seem to start the jit-lock-context-timer if the value
> > is non-t, or am I missing something?
> 
> Yes, you're missing:
> 
>     (when (and contextual jit-lock-contextually)
>       (setq-local jit-lock-contextually t))
> 
> which will turn the non-nil non-t value into t when applicable.

This is in jit-lock-register, which is not called from jit-lock.el.
Does this mean that non-nil, non-t value of jit-lock-contextually is
like nil, unless the major mode calls jit-lock-register with its 2nd
argument non-nil?  And if so, why does the doc string mention
font-lock-keywords-only and doesn't mention jit-lock-register?  They
seem to be at least theoretically independent?  If not, where's their
connection?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> This is in jit-lock-register, which is not called from jit-lock.el.
> Does this mean that non-nil, non-t value of jit-lock-contextually is
> like nil, unless the major mode calls jit-lock-register with its 2nd
> argument non-nil?

Yes.

> And if so, why does the doc string mention font-lock-keywords-only and
> doesn't mention jit-lock-register?

I think that's because that's written for the end-user rather than for
the programmer: the end user doesn't know `jit-lock-register`, but it
may know `font-lock-keywords-only`.

But maybe it's really just an artifact of history (originally, jit-lock
and font-lock were more tightly linked).

> They seem to be at least theoretically independent?  If not, where's
> their connection?

`font-lock-keywords-only` ends up passed by font-lock as
2nd arg to `jit-lock-register`.


        Stefan


PS: BTW, maybe `jit-lock-contextually` should really be part of
font-lock rather than jit-lock.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: acm@muc.de,  rudalics@gmx.at,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  dgutov@yandex.ru
> Date: Wed, 08 Apr 2020 11:16:29 -0400
> 
> > This is in jit-lock-register, which is not called from jit-lock.el.
> > Does this mean that non-nil, non-t value of jit-lock-contextually is
> > like nil, unless the major mode calls jit-lock-register with its 2nd
> > argument non-nil?
> 
> Yes.
> 
> > And if so, why does the doc string mention font-lock-keywords-only and
> > doesn't mention jit-lock-register?
> 
> I think that's because that's written for the end-user rather than for
> the programmer: the end user doesn't know `jit-lock-register`, but it
> may know `font-lock-keywords-only`.
> 
> But maybe it's really just an artifact of history (originally, jit-lock
> and font-lock were more tightly linked).
> 
> > They seem to be at least theoretically independent?  If not, where's
> > their connection?
> 
> `font-lock-keywords-only` ends up passed by font-lock as
> 2nd arg to `jit-lock-register`.

Thanks, I tried to clarify the docs using this information.

RE: emacs rendering comparisson between emacs23 and emacs26.3

[Drew Adams]

> Whether this is "bad" depends on what you use the fontification for.
> For me, it's a sign of syntax correctness (among other things), so
> waiting for the confirmation of the syntax for such a relatively long
> time is quite "bad".

I don't mean to distract from the discussion.

But what you say there seems useful to me: syntax
highlighting can serve more than one purpose.

Given that, could one solution/approach perhaps be
to let users choose (e.g. by variable, possibly
buffer-local) the behavior they want in this regard?

For example, some users, or users in some contexts
(e.g. some modes), might prefer to sacrifice some
degree of fidelity (correctness) between syntax and
fontification, if that provides better performance
in those contexts.

Again, if this question/comment is a distraction,
feel free to ignore.

Re: Inlining policy

[Andrea Corallo]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 4/5/20 3:23 AM, Andrea Corallo wrote:
>> what about having a new macro to control the key operations we want to
>> always have inlined at -O0 and -Og instead of falling back into defining
>> these as macros?
>
> I looked into this some more, and found some problems with using
> __attribute__((always_inline)), such as GCC complaining that it can't
> inline some functions in some cases on some platforms. It was a pain
> to suss out what would improve performance while avoiding the
> warnings, and while fiddling with this a bit I found a simpler change
> that had even better performance than __attribute__((always_inline))
> did, at least on one platform for my standard benchmark of 'make
> compile-always'. So I installed the two attached patches
> instead. Together they improve Emacs CPU performance for 'gcc -Og' by
> about 60% on my benchmark.

Is this a problem you have experienced compiling one of our builds or
something you have found can potentially happen in general?  Anyway
should be easy to ask GCC to silence the specific warning in case.

> Thanks for suggesting inlining as a way to improve Emacs performance
> when compiled with -Og.

Welcome

Andrea

--
akrl@sdf.org

Re: Inlining policy

[Paul Eggert]

On 4/8/20 11:28 AM, Andrea Corallo wrote:
> Is this a problem you have experienced compiling one of our builds or
> something you have found can potentially happen in general?  Anyway
> should be easy to ask GCC to silence the specific warning in case.

I found it when compiling GCC.

Although I initially went down the route of silencing the warning, I stopped 
doing that once I found the simpler and faster-code solution that didn't involve 
using __attribute__ ((always_inline)).

Re: Inlining policy

[Andrea Corallo]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 4/8/20 11:28 AM, Andrea Corallo wrote:
>> Is this a problem you have experienced compiling one of our builds or
>> something you have found can potentially happen in general?  Anyway
>> should be easy to ask GCC to silence the specific warning in case.
>
> I found it when compiling GCC.
>
> Although I initially went down the route of silencing the warning, I
> stopped doing that once I found the simpler and faster-code solution
> that didn't involve using __attribute__ ((always_inline)).

Understand.  A last word on this, always_inline can come back into play
when optimizing for perf also -O2.

Thanks

  Andrea

-- 
akrl@sdf.org

Re: Inlining policy

[Paul Eggert]

On 4/8/20 11:44 AM, Paul Eggert wrote:
> I found it when compiling GCC.

Ooops, I meant to write "when compiling Emacs with GCC".

Re: Inlining policy

[Andrea Corallo]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 4/8/20 11:44 AM, Paul Eggert wrote:
>> I found it when compiling GCC.
>
> Ooops, I meant to write "when compiling Emacs with GCC".

At this point the ultimate goal is to compile GCC with Emacs :)

-- 
akrl@sdf.org

Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 4857 bytes --]

Hi Eli,

On 07.04.2020 17:26, Eli Zaretskii wrote:

>> If so, I don't understand how we get the apparent performance improvement from using fast-but-imprecise-scrolling.
> 
> To explain that, I need to step back and describe how jit-lock
> mechanism works, from the display engine POV, and then how the scroll
> commands work.  (See also a large commentary in keyboard.c around line
> 250.) <...>

Thank you, what was a lot of additional detail.

The overall approach seems well-considered, but maybe there are ways to 
make it more... predictable, so to speak. Or more helpful.

> If you lean on C-v, then the scroll command is repeatedly called to
> find the next window-start, and repeatedly fontifies the text it goes
> through.  If you scroll N screen-fulls, the fontification of the first
> N-1 screen-fulls (if they were never displayed before, which happens
> when you visit a file and immediately start scrolling) is "wasted
> effort" for the purposes of the final window redisplay, because those
> N-1 screen-fulls will not be displayed.  Only the fontification of the
> last screen-full is "useful", because it saves some, sometimes all, of
> the fontification job of the stuff to be eventually displayed when you
> stop scrolling.

As you noted, when fontification takes a considerable amount of time, 
redisplay should be "free", relatively speaking. And if we didn't skip 
it, it would provide the user with the feedback that something is indeed 
happening.

> Enter fast-but-imprecise-scrolling.  What it does is disable jit-lock
> during the part where the scroll command walked the buffer to find the
> next window-start position.  Crucially, it also disables putting the
> non-nil 'fontified' property on the text traversed by the scroll
> command (because that is done by jit-lock).  So when you stop
> scrolling, redisplay of the stuff that ends up in the window will have
> to fontify all of what is visible in the window.

The obvious downside here is that it's "imprecise". And that it doesn't 
really know whether redisplay is going to be skipped, even though it 
could take a pretty good guess (using the input_was_pending variable, 
for example). If the user has leaned on C-v, they are probably not 
looking for much precision. But they might want it if they just pressed 
C-v once.

> Of course, if fast-but-imprecise-scrolling makes the job of scroll
> commands so much easier that Emacs is capable of keeping up with the
> keyboard auto-repeat rate, then every screen-full you scroll through
> _will_ be displayed, and therefore will be fontified by the calls to
> jit-lock from redisplay itself -- and that will again make scrolling
> slower.  So fast-but-imprecise-scrolling is most efficient when it
> makes scroll commands faster, but not too fast...

In my testing with an '-Og -g3' build (from emacs-27; master is much 
faster with the recent tweaks) it allows for redisplay once every 
several screenfuls or so. So it seems like a good tradeoff, in those 
particular conditions.

I wonder if we could make it more coordinated. The two problems I see are:

1. Scrolling commands can skip jit-lock when redisplay is going to apply 
it (loss of precision for no actual perf improvement), or they don't 
skip it (when fast-but-... is nil) even though redisplay will (precision 
seems unimportant in most pending-input scenarios).
2. If we accumulate a large enough input queue of C-v keypresses, Emacs 
won't react to anything (even to C-g, even hit three times) until it 
processes these events. Even though the user most probably wanted it to 
stop scrolling just as soon as they stopped leaning on C-v.

I think #2 can have good solutions, but I only have rough suggestions 
about that.

But let's look at #1. We could fix the discrepancy in two ways: either 
skip jit-lock in window_scroll if and only if redisplay will likely be 
skipped (as shown in the attached patch) (a). Or never skip it there, as 
well as indicate to redisplay that it shouldn't be skipped either (b).

(a) will increase reactivity, as well as make it harder to overwhelm the 
input queue, which will be perceived by a number of users (who manage to 
hit these conditions and have slow enough machines) as better 
responsiveness.

(b) will let the user see what's happening instead, even though the 
scrolling will become slower. I think this is a plus in its own right, 
because scrolling is a visual activity; there's little point in doing it 
if you can't see where you scrolled to. We can probably implement this 
by setting input_was_pending unconditionally to false inside window_scroll.

Logically, I think I'd prefer (b), especially if a solution for #2 was 
added as well. Because as soon as #2 is solved, (a)'s benefit isn't so 
clear anymore.

But either (a) or (b), both look more responsive to me than the current 
state of affairs.

[-- Attachment #2: fast_scroll_on_pending_input.diff --]
[-- Type: text/x-patch, Size: 1228 bytes --]

diff --git a/src/keyboard.c b/src/keyboard.c
index bf1f5da22d..3ea8ba7619 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -281,7 +281,7 @@ #define GROW_RAW_KEYBUF							\
    with the input rate, but if it can keep up just enough that there's no
    input_pending when we begin the command, then redisplay is not skipped
    which results in better feedback to the user.  */
-static bool input_was_pending;
+bool input_was_pending;
 
 /* Circular buffer for pre-read keyboard input.  */
 
diff --git a/src/window.c b/src/window.c
index ff17cd88f3..5f7a08ec8c 100644
--- a/src/window.c
+++ b/src/window.c
@@ -5466,6 +5466,8 @@ window_internal_height (struct window *w)
    means don't signal an error if we try to move over BEGV or ZV,
    respectively.  */
 
+extern bool input_was_pending;
+
 static void
 window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
 {
@@ -5475,7 +5477,7 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
 
   wset_redisplay (XWINDOW (window));
 
-  if (whole && Vfast_but_imprecise_scrolling)
+  if (whole && input_was_pending)
     specbind (Qfontification_functions, Qnil);
 
   /* On GUI frames, use the pixel-based version which is much slower

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It presumes that the developers adopt the practice of indenting the
  > > text of block comments, or can be persuaded to do so.

  > What about those who can't or don't want to?

Provided they do not set open-paren-in-column-0-is-defun-start to t,
nothing will change for them.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It would scan only from the last open-paren-in-column-zero, as it did
  > > in the past.

  > That would produce incorrect display, as I explained later:

  > > 			       A change in a buffer can potentially affect
  > >   > the fontification of the rest of the buffer, and I don't think we
  > >   > would like Emacs to fail to update other windows showing the same
  > >   > buffer

I could not see how that relates to the issue.  That point was not
explicitly stated.  That's why I said it seemed like a change in
topic.

Now you've explained how they relate, so now I understand your point.

I am suggesting that we once again support a mode in which users
signal the start of a defun with an open-delimiter in column zero.
For CC modes, this would have to be explicitly enabled.

If you are editing code written in the usual GNU style 
then you can enable that mode and get good results.

If you are editing code NOT written in that style
then don't enable the mode.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: acm@muc.de, rudalics@gmx.at, eliz@gnu.org, rrandresf@gmail.com,
> 	emacs-devel@gnu.org
> Date: Wed, 08 Apr 2020 23:10:06 -0400
> 
>   > > It presumes that the developers adopt the practice of indenting the
>   > > text of block comments, or can be persuaded to do so.
> 
>   > What about those who can't or don't want to?
> 
> Provided they do not set open-paren-in-column-0-is-defun-start to t,
> nothing will change for them.

I think there might be a miscommunication here.  I think Stefan
alluded to editing source files outside of Emacs (more generally,
outside the GNU Project), where the style is different and not under
any control by us.  Those source files could violate this convention,
and we will not be able to do anything about that, except adapt Emacs
to handle them reasonably well.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 9 Apr 2020 03:32:13 +0300
> 
> > Of course, if fast-but-imprecise-scrolling makes the job of scroll
> > commands so much easier that Emacs is capable of keeping up with the
> > keyboard auto-repeat rate, then every screen-full you scroll through
> > _will_ be displayed, and therefore will be fontified by the calls to
> > jit-lock from redisplay itself -- and that will again make scrolling
> > slower.  So fast-but-imprecise-scrolling is most efficient when it
> > makes scroll commands faster, but not too fast...
> 
> In my testing with an '-Og -g3' build (from emacs-27; master is much 
> faster with the recent tweaks) it allows for redisplay once every 
> several screenfuls or so. So it seems like a good tradeoff, in those 
> particular conditions.

Tradeoffs of what?  You are probably scrolling in a buffer that uses
a single font all over, with fontifications changing only the colors.
While this is a very frequent scenario, it is by no means the only one
Emacs needs to support.  Imagine a buffer where many lines have
characters that become much taller than the default after
fontifications -- in such a buffer fast-but-imprecise-scrolling will
cause you get to the wrong screenful, because it will consider each
character to be displayed with the default face.

> -  if (whole && Vfast_but_imprecise_scrolling)
> +  if (whole && input_was_pending)
>      specbind (Qfontification_functions, Qnil);

I don't like this, because it's unconditional.  In my view, the
tradeoff proposed by fast-but-imprecise-scrolling is a bad tradeoff,
something I personally will never take, even if Emacs locks up for
many seconds in some situation.  If you suggest to _add_ the
pending-input condition to fast-but-imprecise-scrolling being set,
then I'm okay with that optional behavior being smarter (assuming this
doesn't have downsides we aren't aware of yet), if users of
fast-but-imprecise-scrolling agree with it.

More generally, I don't think I see how even this proposal makes
something clearly better.  We already have jit-lock-defer-time; people
who have slow machines are advised to set that to something like 0.1
or 0.25, and they can have scrolling that is way faster than with
fast-but-imprecise-scrolling (and with the same tradeoff of making
scrolling "imprecise").  Why invent kludges when we already have a
better solution that was there since Emacs 21?

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> More generally, I don't think I see how even this proposal makes
> something clearly better.  We already have jit-lock-defer-time; people
> who have slow machines are advised to set that to something like 0.1
> or 0.25, and they can have scrolling that is way faster than with
> fast-but-imprecise-scrolling (and with the same tradeoff of making
> scrolling "imprecise").  Why invent kludges when we already have a
> better solution that was there since Emacs 21?

BTW, you can also set `jit-lock-defer-time` to 0 in which case jit-lock
is deferred iff there's input pending.
[ This refinement of `jit-lock-defer-time` was introduced last time
  this kind of discussion took place, which is also the time
  `fast-but-imprecise-scrolling` was introduced.  ]


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 09.04.2020 16:19, Stefan Monnier wrote:
> BTW, you can also set `jit-lock-defer-time` to 0 in which case jit-lock
> is deferred iff there's input pending.
> [ This refinement of `jit-lock-defer-time` was introduced last time
>    this kind of discussion took place, which is also the time
>    `fast-but-imprecise-scrolling` was introduced.  ]

Thanks. I tried it (and mentioned in the parent thread), but it seems 
like in addition to the downsides of f-b-i-s (imprecise scrolling, which 
apparently makes it a bad candidate for default behavior) it adds some 
extra flickering from time to time (I figured because some redisplays 
occurred before the idle timer had the chance to run). It's not as 
obvious in an 'emacs -Q' session, but with my custom config it's more 
apparent. And that flickering occurs the most while I'm flooring C-v.

Is it possible that the conditions for skipping redisplay might return 
false while input-pending-p returns t? Then redisplay will occur, but if 
that coincides with an unfontified screenful, the screen will be 
rendered unfontified? That doesn't explain the additional frames where 
that highlighting is added, however.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> Thanks. I tried it (and mentioned in the parent thread), but it seems like
> in addition to the downsides of f-b-i-s (imprecise scrolling, which
> apparently makes it a bad candidate for default behavior) it adds some 
> extra flickering from time to time (I figured because some redisplays
> occurred before the idle timer had the chance to run). It's not as 
> obvious in an 'emacs -Q' session, but with my custom config it's more
> apparent. And that flickering occurs the most while I'm flooring C-v.

I think the flickering is the direct effect of the defer'd jit-lock: you
first see the non-jit-lock'd text and only after that the idle timer
runs jit-lock and updates the display accordingly.

> Is it possible that the conditions for skipping redisplay might return
> false while input-pending-p returns t?

Very much so, yes: the tests are sufficiently different and the possible
circumstances sufficiently varied that it's almost unavoidable.

E.g. usually the redisplay is skipped based on the presence of input at
the *beginning* of the command, whereas the `jit-lock-defer-time` thingy
samples the input later, when jit-lock is called.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 9 Apr 2020 21:37:08 +0300
> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> 
> On 09.04.2020 16:19, Stefan Monnier wrote:
> > BTW, you can also set `jit-lock-defer-time` to 0 in which case jit-lock
> > is deferred iff there's input pending.
> > [ This refinement of `jit-lock-defer-time` was introduced last time
> >    this kind of discussion took place, which is also the time
> >    `fast-but-imprecise-scrolling` was introduced.  ]
> 
> Thanks. I tried it (and mentioned in the parent thread), but it seems 
> like in addition to the downsides of f-b-i-s (imprecise scrolling, which 
> apparently makes it a bad candidate for default behavior) it adds some 
> extra flickering from time to time (I figured because some redisplays 
> occurred before the idle timer had the chance to run).

Which is why I don't think using 0 is such a good idea.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 09.04.2020 22:20, Eli Zaretskii wrote:
> Which is why I don't think using 0 is such a good idea.

None of the non-nil values look sufficiently usable to me.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 09.04.2020 22:16, Stefan Monnier wrote:
>> Thanks. I tried it (and mentioned in the parent thread), but it seems like
>> in addition to the downsides of f-b-i-s (imprecise scrolling, which
>> apparently makes it a bad candidate for default behavior) it adds some
>> extra flickering from time to time (I figured because some redisplays
>> occurred before the idle timer had the chance to run). It's not as
>> obvious in an 'emacs -Q' session, but with my custom config it's more
>> apparent. And that flickering occurs the most while I'm flooring C-v.
> 
> I think the flickering is the direct effect of the defer'd jit-lock:

It's weird, though, that it's not so obvious with the 'emacs -Q' build.

> you
> first see the non-jit-lock'd text and only after that the idle timer
> runs jit-lock and updates the display accordingly.

And that it happens in the middle of scrolling. All the while C-v is 
still pressed.

And also: jit-lock-function supposedly has a provision for that sort of 
circumstances: jit-lock-defer-time=0 specially, doing fontification 
immediately, as long as input-pending-p=>nil.

>> Is it possible that the conditions for skipping redisplay might return
>> false while input-pending-p returns t?
> 
> Very much so, yes: the tests are sufficiently different and the possible
> circumstances sufficiently varied that it's almost unavoidable.
> 
> E.g. usually the redisplay is skipped based on the presence of input at
> the *beginning* of the command, whereas the `jit-lock-defer-time` thingy
> samples the input later, when jit-lock is called.

The condition for skipping redisplay based in the presence of inputs 
before and after, to be a tad more accurate. But the difference is 
there, to be sure.

So thus far fast-but-imprecise-scrolling seems to provide better 
behavior. I would love to remove that variable, though, and just tell 
people to use jit-lock-defer-time *if* the jit-lock behavior can be 
improved.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> > BTW, you can also set `jit-lock-defer-time` to 0 in which case jit-lock
>> > is deferred iff there's input pending.
>> > [ This refinement of `jit-lock-defer-time` was introduced last time
>> >    this kind of discussion took place, which is also the time
>> >    `fast-but-imprecise-scrolling` was introduced.  ]
>> 
>> Thanks. I tried it (and mentioned in the parent thread), but it seems
>> like in addition to the downsides of f-b-i-s (imprecise scrolling, which
>> apparently makes it a bad candidate for default behavior) it adds some
>> extra flickering from time to time (I figured because some redisplays
>> occurred before the idle timer had the chance to run).
>
> Which is why I don't think using 0 is such a good idea.

It's definitely not good enough to be used as a default value, agreed.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> So thus far fast-but-imprecise-scrolling seems to provide better
> behavior. I would love to remove that variable, though, and just tell 
> people to use jit-lock-defer-time *if* the jit-lock behavior can
> be improved.

Making (setq jit-lock-defer-time 0) check input-pending-p was
a quick hack.  It worked well enough to be useful.

I agree that it would be good to provide something better, which works
harder at trying to make sure that jit-lock is skipped iff redisplay is
skipped.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 09.04.2020 22:46, Stefan Monnier wrote:
> Making (setq jit-lock-defer-time 0) check input-pending-p was
> a quick hack.  It worked well enough to be useful.
> 
> I agree that it would be good to provide something better, which works
> harder at trying to make sure that jit-lock is skipped iff redisplay is
> skipped.

Considering the predicate is composite, its result changes over time, 
and jit-lock and redisplay happen not exactly at the same time, it seems 
difficult.

The approach taken by f-b-i-s originally seems easier to "make right". 
Not necessarily exactly like in my last patch, but maybe in that exact 
place in the code.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 3575 bytes --]

Hi Eli,

You have addressed only about 30% of my previous email.

I'll have refer to the remaining 70% anyway.

On 09.04.2020 10:46, Eli Zaretskii wrote:
>> In my testing with an '-Og -g3' build (from emacs-27; master is much
>> faster with the recent tweaks) it allows for redisplay once every
>> several screenfuls or so. So it seems like a good tradeoff, in those
>> particular conditions.
> 
> Tradeoffs of what?  You are probably scrolling in a buffer that uses
> a single font all over, with fontifications changing only the colors.
> While this is a very frequent scenario, it is by no means the only one
> Emacs needs to support.  Imagine a buffer where many lines have
> characters that become much taller than the default after
> fontifications -- in such a buffer fast-but-imprecise-scrolling will
> cause you get to the wrong screenful, because it will consider each
> character to be displayed with the default face.

You will have the wrong screenful, yes. But how wrong exactly? If the 
user floored C-v, they are not looking for precision. That's my main 
justification for the approach (a) presented in that last patch, and 
it's the aspect that the said patch improves: instead of having 
imprecise scrolling always, they only have that when input arrives too fast.

>> -  if (whole && Vfast_but_imprecise_scrolling)
>> +  if (whole && input_was_pending)
>>       specbind (Qfontification_functions, Qnil);
> 
> I don't like this, because it's unconditional.  In my view, the
> tradeoff proposed by fast-but-imprecise-scrolling is a bad tradeoff,
> something I personally will never take, even if Emacs locks up for
> many seconds in some situation.  If you suggest to _add_ the
> pending-input condition to fast-but-imprecise-scrolling being set,
> then I'm okay with that optional behavior being smarter (assuming this
> doesn't have downsides we aren't aware of yet), if users of
> fast-but-imprecise-scrolling agree with it.

These are all rough patches. I'm more interested in a discussion than in 
a rubber-stamp. And I'd be surprised if fast-but-imprecise-scrolling has 
a significant number of users yet.

> More generally, I don't think I see how even this proposal makes
> something clearly better.

There were two different proposals in that email. And one general 
feature request.

That patch was for (a). I'm inclined to prefer (b), even though the 
implementation is dirtier. See attached patch that switches between (a) 
and (b) based on the value of fast-but-imprecise-scrolling. The (b) 
behavior becomes the default.

Curiously, I think that this patch demonstrates that while 
fast-but-imprecise-scrolling makes scrolling faster (a), the result 
looks less responsive than the (new) default behavior (b). And that one 
looks more responsive than the current one.

Neither of them deals with the backed-up input queue, alas (#2).

> We already have jit-lock-defer-time; people
> who have slow machines are advised to set that to something like 0.1
> or 0.25, and they can have scrolling that is way faster than with
> fast-but-imprecise-scrolling (and with the same tradeoff of making
> scrolling "imprecise").  Why invent kludges when we already have a
> better solution that was there since Emacs 21?

As explained in a separate subthread, jit-lock-defer-time=0 has worse 
side-effects than fast-but-imprecise-scrolling (while including its 
downside). And I won't even mention values higher than 0.

But, of course, if we can make jit-lock-defer-time=0 non-flickery, we 
could recommend that instead of fast-but-imprecise-scrolling=t.

[-- Attachment #2: more_responsive_scrolling.diff --]
[-- Type: text/x-patch, Size: 1455 bytes --]

diff --git a/src/keyboard.c b/src/keyboard.c
index bf1f5da22d..3ea8ba7619 100644
--- a/src/keyboard.c
+++ b/src/keyboard.c
@@ -281,7 +281,7 @@ #define GROW_RAW_KEYBUF							\
    with the input rate, but if it can keep up just enough that there's no
    input_pending when we begin the command, then redisplay is not skipped
    which results in better feedback to the user.  */
-static bool input_was_pending;
+bool input_was_pending;
 
 /* Circular buffer for pre-read keyboard input.  */
 
diff --git a/src/window.c b/src/window.c
index ff17cd88f3..5e61d9f528 100644
--- a/src/window.c
+++ b/src/window.c
@@ -5466,6 +5466,8 @@ window_internal_height (struct window *w)
    means don't signal an error if we try to move over BEGV or ZV,
    respectively.  */
 
+extern bool input_was_pending;
+
 static void
 window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
 {
@@ -5475,8 +5477,15 @@ window_scroll (Lisp_Object window, EMACS_INT n, bool whole, bool noerror)
 
   wset_redisplay (XWINDOW (window));
 
-  if (whole && Vfast_but_imprecise_scrolling)
-    specbind (Qfontification_functions, Qnil);
+  if (Vfast_but_imprecise_scrolling)
+    {
+      if (input_was_pending)
+        specbind (Qfontification_functions, Qnil);
+    }
+  else
+    {
+      input_was_pending = false;
+    }
 
   /* On GUI frames, use the pixel-based version which is much slower
      than the line-based one but can handle varying line heights.  */

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >   > > It presumes that the developers adopt the practice of indenting the
  > >   > > text of block comments, or can be persuaded to do so.
  > > 
  > >   > What about those who can't or don't want to?
  > > 
  > > Provided they do not set open-paren-in-column-0-is-defun-start to t,
  > > nothing will change for them.

  > I think there might be a miscommunication here.  I think Stefan
  > alluded to editing source files outside of Emacs (more generally,
  > outside the GNU Project), where the style is different and not under
  > any control by us.

I think so too.  I responded to that by saying people should not turn on
this mode for those files:

  > > Provided they do not set open-paren-in-column-0-is-defun-start to t,
  > > nothing will change for them.


-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> I think so too.  I responded to that by saying people should not turn on
> this mode for those files:

Performance has to be be good in files without open-parens in column-0.
So setting this var *should* provide no significant performance benefit
(otherwise it means performance is not good enough in files without
open-parens in column-0 and we should fix that).

A setting that is not just a performance heuristic would be fine,
e.g. something that causes the syntax-ppss state to be forcibly reset to
"toplevel" whenever we bump into an open-paren in column 0 (which seems
to be something Martin wants, so that a non-closed comment doesn't leak
past an open-paren in column-0).  But that's a completely separate issue
(beside the superficial connection that they both have to do with parens
in column-0).

But this too long thread is about performance, and when it comes to
performance any heuristic based on open-parens in-column-0 is
a non-starter AFAIC because they only cover a subset of cases and
we need something that covers the other cases as well.
And AFAIK we already have such a "something", so there's no need to
settle for this half-solution, no matter how cheap it may be nor how
attached to it some people might be.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 08.04.2020 05:09, Alan Mackenzie wrote:

>> I will shut up about it now (saying it twice it plenty), but I am pretty
>> confident saying that if you manage to migrate to s-p-f, file opening
>> time will go down.
> 
> I'm sure it would.  If file opening time were really a concern, a hybrid
> algorithm would perhaps be the best way: apply the text properties first
> in a lazy fashion, and thereafter treat them with care, as CC Mode
> currently does.

s-p-f would help with the first step, and as for "treating them with 
care", it would be good for us all to see how much of that is really 
needed. Improving syntax-propertize is not out of the question, and it 
might benefit several major modes, not just the CC collection.

> But this would merely transfer the start up time to the
> time taken in early scrolls forward.

Not really. The start up scans the whole buffer, doesn't it? The early 
scrolls forward would still scan only a fraction of it.

>> Performance while typing is likely to improve too, at least when the
>> same buffer is not shown in another window, many thousand lines later.
> 
> What makes you think this?

Inserting characters can alter the syntax state of the whole buffer. At 
least that's true for some of them. Full buffer scan sounds inevitable 
in those cases.

>> "Considerable enhancement" can also be a part of that discussion.
> 
> The syntax-propertize-function mechanism works by erasing ALL
> syntax-table properties after a change point, then reapplying them
> lazily.

That's not right. It only erases syntax-table properties in a chunk 
before calling syntax-propertize-function on the same range of 
positions. IOW, is overwrites them lazily as well.

 > Considering that s-t properties have an overwhelmingly local
 > effect, this is very wasteful of processor time.

It would have been. As you can see, it's not a difficult problem to fix, 
even if it were still present.

> Consider, for example, editing within a large C++ raw string, a common
> occurrence.  You yourself reported as a bug sluggish performance here in
> mid 2016.  The cause was erasing too many s-t text properties at a
> buffer change.  I think we were talking about 1 second per typed
> character in the scenario you gave.  There are typically lots of these
> properties in a raw string, in particular on " characters.

I'm pretty sure I have thought of that example because it's an instance 
of a syntax problem that's easy enough to solve within 
syntax-propertize-function framework.

> Consider(2) a C++ template: excusing my C++ syntax knowledge, type in
> 
>     template class foo < bar, baz >= bar>
> 
> , perhaps typing in the odd newline inside the template (a common
> occurrence), or nesting further templates inside it (also a common
> occurrence).  Note how the parenthesis text properties are added and
> removed as you type.  All these modification are necessary, and they are
> largely _before_ the point of insertion, not after it.

The current implementation of applying these properties can probably be 
transferred into a syntax-propertize-function with only modest changes.

>> Some scenarios can become slower, that's for sure. But the more common
>> ones can get faster. We won't know until we try.
> 
> Trying would be a _lot_ of work.  How is one to handle the common
> example scenarios above?

Stefan has offered to help. And I'm sure he could answer the follow-up 
questions much better than I.

> Well, you'd have to enhance the
> syntax-propertize-function with a means of determining a start position
> for erasing s-t props, and also a stop position.

The real-world uses of s-p-f out there already solve syntax problems of 
comparable complexity. And move the start position, among other things.

> Once you do that,
> you're effectively doing what CC Mode currently does, so where's the
> speed advantage coming from?

 From doing things more lazily, is how I see it. But I'm not an expert 
on CC Mode architecture.

Among other benefits, moving it to a standard-ish framework like s-p-f 
could (possibly) simplify its code, as well as make it more approachable 
for other developers already familiar with how most other major modes 
are written. So far I wouldn't even know where to start fixing bugs in 
it, and IMHO CC Mode currently has bus factor = 1. It's not great for 
its future. I suspect it's not ideal for you either.

Simply collaborating with one other developer on an overhaul project 
(whether it succeeds or not; perhaps partially) can improve on that.

   Cheers.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, rudalics@gmx.at,
>  rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 9 Apr 2020 22:28:12 +0300
> 
> On 09.04.2020 22:20, Eli Zaretskii wrote:
> > Which is why I don't think using 0 is such a good idea.
> 
> None of the non-nil values look sufficiently usable to me.

What values did you try?

Both 0.1 and 0.25 work for me: scrolling is continuous, without any
flickering and without any fontifications of the screenfuls that I
scroll across, and the only times scrolling pauses for a brief moment
is when Emacs GCs.

If this isn't what happens for you, I think it would be good to find
the customizations which affect this behavior, we might be missing
something in this picture.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, acm@muc.de, rrandresf@gmail.com,
>  emacs-devel@gnu.org, rms@gnu.org, rudalics@gmx.at
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 9 Apr 2020 22:30:05 +0300
> 
> So thus far fast-but-imprecise-scrolling seems to provide better 
> behavior.

Not here, it isn't.  Scrolling gets stuck very often and for prolonged
times in an unoptimized build.  In an optimized build, it gets stuck
less often, but the scrolling is significantly slower than with
jit-lock-defer-time set to 0.1.

Did you try fast-but-imprecise-scrolling in an optimized build or in
an unoptimized one?

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 10 Apr 2020 01:17:51 +0300
> 
> You have addressed only about 30% of my previous email.

I don't have anything to say about the rest.

> > Tradeoffs of what?  You are probably scrolling in a buffer that uses
> > a single font all over, with fontifications changing only the colors.
> > While this is a very frequent scenario, it is by no means the only one
> > Emacs needs to support.  Imagine a buffer where many lines have
> > characters that become much taller than the default after
> > fontifications -- in such a buffer fast-but-imprecise-scrolling will
> > cause you get to the wrong screenful, because it will consider each
> > character to be displayed with the default face.
> 
> You will have the wrong screenful, yes. But how wrong exactly? If the 
> user floored C-v, they are not looking for precision.

Depends on what you mean by "precision".  They might very well miss
some parts of the buffer entirely, i.e. never see them on display.
This may or may not be important, depending on the use case.  An
editor is not supposed to skip portions of the buffer when scrolling
continuously.

> > We already have jit-lock-defer-time; people
> > who have slow machines are advised to set that to something like 0.1
> > or 0.25, and they can have scrolling that is way faster than with
> > fast-but-imprecise-scrolling (and with the same tradeoff of making
> > scrolling "imprecise").  Why invent kludges when we already have a
> > better solution that was there since Emacs 21?
> 
> As explained in a separate subthread, jit-lock-defer-time=0 has worse 
> side-effects than fast-but-imprecise-scrolling (while including its 
> downside). And I won't even mention values higher than 0.

Here, it behaves much better.  So maybe some other factors are at work
in your configuration.  I think we should try making the solid
solutions work as intended before we look for kludges.  The "normal"
way of doing something when Emacs is idle is by using an idle timer,
and that's what jit-lock-defer-time does.  So from my POV it is the
solution for such problems.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 09:13, Eli Zaretskii wrote:
> Not here, it isn't.  Scrolling gets stuck very often and for prolonged
> times in an unoptimized build.  In an optimized build, it gets stuck
> less often, but the scrolling is significantly slower than with
> jit-lock-defer-time set to 0.1.
> 
> Did you try fast-but-imprecise-scrolling in an optimized build or in
> an unoptimized one?

As usual, '-Og -g3'.

But I'm comparing fast-but-imprecise-scrolling with jit-lock-defer-time 
set to 0, which is supposed to be its direct counterpart. And the latter 
adds flickering.

Higher values add flickering too, even more noticeably. But that seems 
unavoidable.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> Depends on what you mean by "precision".  They might very well miss
> some parts of the buffer entirely, i.e. never see them on display.

But that happens regardless of whether the scroll is computed correctly,
because redisplay itself is skipped.

> This may or may not be important, depending on the use case.  An
> editor is not supposed to skip portions of the buffer when scrolling
> continuously.

If we can arrange for the "skip jit-lock during scroll" to ensure that
the subsequent redisplay also skips jit-lock (or is itself skipped),
then this should not happen, right?
E.g. in the case where jit-lock-defer-time > 0.

> Here, it behaves much better.  So maybe some other factors are at work
> in your configuration.

I think different definitions of "better" are at play here as well.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, rrandresf@gmail.com,
>  emacs-devel@gnu.org, rms@gnu.org, rudalics@gmx.at
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 10 Apr 2020 15:38:05 +0300
> 
> But I'm comparing fast-but-imprecise-scrolling with jit-lock-defer-time 
> set to 0, which is supposed to be its direct counterpart. And the latter 
> adds flickering.
> 
> Higher values add flickering too, even more noticeably. But that seems 
> unavoidable.

It doesn't flicker here.  Maybe the double-buffering is the reason?

With -Og, if you set jit-lock-defer-time to 0.5, doesn't it scroll
much faster? does it flicker then?

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Fri, 10 Apr 2020 09:00:04 -0400
> 
> > Depends on what you mean by "precision".  They might very well miss
> > some parts of the buffer entirely, i.e. never see them on display.
> 
> But that happens regardless of whether the scroll is computed correctly,
> because redisplay itself is skipped.

It isn't skipped here if I use jit-lock-defer-time.  Maybe your
keyboard auto-repeat rate is too high?  Mine is set to about 50/sec.

> > This may or may not be important, depending on the use case.  An
> > editor is not supposed to skip portions of the buffer when scrolling
> > continuously.
> 
> If we can arrange for the "skip jit-lock during scroll" to ensure that
> the subsequent redisplay also skips jit-lock (or is itself skipped),
> then this should not happen, right?
> E.g. in the case where jit-lock-defer-time > 0.

Indeed, it doesn't here.

> > Here, it behaves much better.  So maybe some other factors are at work
> > in your configuration.
> 
> I think different definitions of "better" are at play here as well.

In the context of the above, "better" means "without flickering".

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> > Depends on what you mean by "precision".  They might very well miss
>> > some parts of the buffer entirely, i.e. never see them on display.
>> But that happens regardless of whether the scroll is computed correctly,
>> because redisplay itself is skipped.
> It isn't skipped here if I use jit-lock-defer-time.

I was talking about `fast-but-imprecise-scrolling`, not about
`jit-lock-defer-time` (I thought that's what your use of "precision"
implied).


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 16:34, Eli Zaretskii wrote:
> It doesn't flicker here.  Maybe the double-buffering is the reason?

Double-buffering decreases the number of display updates, so it hardly 
can be the cause of this.

As we've discussed with Stefan, the implementation is likely the reason.

> With -Og, if you set jit-lock-defer-time to 0.5, doesn't it scroll
> much faster? does it flicker then?

It doesn't scroll much faster than with the value of 0 (that already 
scrolls fast enough).

It doesn't flicker while scrolling because it simply shows unfontified 
screenfuls of code. Which is unacceptable for default behavior. And I'd 
hesitate to recommend it to anyone even to deal with performance problems.

It *does* flicker afterwards (e.g. 0.5 sec after I depress C-v), because 
that's what deferred jit-lock does: it applies syntax highlighting with 
a delay. So I type, wait 0.5sec, and syntax highlighting arrives. It's a 
less-aggravating kind of flicker because of its laid-back pace, but I 
still wouldn't call it acceptable user experience.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 09:08, Eli Zaretskii wrote:
>> Cc:monnier@iro.umontreal.ca,acm@muc.de,rudalics@gmx.at,
>>   rrandresf@gmail.com,rms@gnu.org,emacs-devel@gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>> Date: Thu, 9 Apr 2020 22:28:12 +0300
>>
>> On 09.04.2020 22:20, Eli Zaretskii wrote:
>>> Which is why I don't think using 0 is such a good idea.
>> None of the non-nil values look sufficiently usable to me.
> What values did you try?
> 
> Both 0.1 and 0.25 work for me: scrolling is continuous, without any
> flickering and without any fontifications of the screenfuls that I
> scroll across, and the only times scrolling pauses for a brief moment
> is when Emacs GCs.

Same. We just have different definitions of the word "usable". I think 
I've explained mine.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>>> > Depends on what you mean by "precision".  They might very well miss
>>> > some parts of the buffer entirely, i.e. never see them on display.
>>> But that happens regardless of whether the scroll is computed correctly,
>>> because redisplay itself is skipped.
>> It isn't skipped here if I use jit-lock-defer-time.
> I was talking about `fast-but-imprecise-scrolling`, not about
> `jit-lock-defer-time` (I thought that's what your use of "precision"
> implied).

Please disregard this answer, I was confused about what I had written.
I think my earlier answer was not about `fast-but-imprecise-scrolling`
specifically but about the case where you lean on `C-v`.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> It *does* flicker afterwards (e.g. 0.5 sec after I depress C-v), because
> that's what deferred jit-lock does: it applies syntax highlighting with 
> a delay. So I type, wait 0.5sec, and syntax highlighting arrives.  It's
> a less-aggravating kind of flicker because of its laid-back pace, but I 
> still wouldn't call it acceptable user experience.

And in theory it should be possible to make something like
(setq jit-lock-defer-mode 0) avoid this flicker because the last
repetition of `C-v` is executed with `input-pending-p` returning nil, so
we should be able to fully jit-lock right away with no deferral at
that point.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 09:47, Eli Zaretskii wrote:
>> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>>   emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 10 Apr 2020 01:17:51 +0300
>>
>> You have addressed only about 30% of my previous email.
> 
> I don't have anything to say about the rest.

Not even about the last patch I sent?

>>> Tradeoffs of what?  You are probably scrolling in a buffer that uses
>>> a single font all over, with fontifications changing only the colors.
>>> While this is a very frequent scenario, it is by no means the only one
>>> Emacs needs to support.  Imagine a buffer where many lines have
>>> characters that become much taller than the default after
>>> fontifications -- in such a buffer fast-but-imprecise-scrolling will
>>> cause you get to the wrong screenful, because it will consider each
>>> character to be displayed with the default face.
>>
>> You will have the wrong screenful, yes. But how wrong exactly? If the
>> user floored C-v, they are not looking for precision.
> 
> Depends on what you mean by "precision".  They might very well miss
> some parts of the buffer entirely, i.e. never see them on display.
> This may or may not be important, depending on the use case.  An
> editor is not supposed to skip portions of the buffer when scrolling
> continuously.

And that's what my last patch helps it avoid doing, by default.

Speaking of the loss of precision, though. You said you'd never make 
that choice, and yet you recommend the use of jit-lock-defer-time. It 
has the very same problem, doesn't it?

>> As explained in a separate subthread, jit-lock-defer-time=0 has worse
>> side-effects than fast-but-imprecise-scrolling (while including its
>> downside). And I won't even mention values higher than 0.
> 
> Here, it behaves much better.  So maybe some other factors are at work
> in your configuration.  I think we should try making the solid
> solutions work as intended before we look for kludges.  The "normal"
> way of doing something when Emacs is idle is by using an idle timer,
> and that's what jit-lock-defer-time does.  So from my POV it is the
> solution for such problems.

When I said "worse", I didn't mean to say "slower", I meant worse in how 
it looks. Performance-wise, jit-lock-defer-time=0 and 
fast-but-imprecise-scrolling=t work about the same here.

And I'd like for jit-lock-defer-time=0 to work better, so we can remove 
the extra customization option, but it's just not there yet.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Fri, 10 Apr 2020 10:29:07 -0400
> 
> >> > Depends on what you mean by "precision".  They might very well miss
> >> > some parts of the buffer entirely, i.e. never see them on display.
> >> But that happens regardless of whether the scroll is computed correctly,
> >> because redisplay itself is skipped.
> > It isn't skipped here if I use jit-lock-defer-time.
> 
> I was talking about `fast-but-imprecise-scrolling`

Now I'm confused: fast-but-imprecise-scrolling doesn't skip redisplay.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, rrandresf@gmail.com,
>  emacs-devel@gnu.org, rms@gnu.org, rudalics@gmx.at
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 10 Apr 2020 17:34:57 +0300
> 
> On 10.04.2020 16:34, Eli Zaretskii wrote:
> > It doesn't flicker here.  Maybe the double-buffering is the reason?
> 
> Double-buffering decreases the number of display updates, so it hardly 
> can be the cause of this.

What else could be the reason of differences between what we see on
our respective systems?

> As we've discussed with Stefan, the implementation is likely the reason.

That sounds like a request to stop interfering with the discussion
between you and Stefan.

> It doesn't flicker while scrolling because it simply shows unfontified 
> screenfuls of code. Which is unacceptable for default behavior.

Showing unfontified text _is_ the intended effect of
jit-lock-defer-time.  On slow machines, I fail to see how this could
be worse than having Emacs hang for many seconds.

> And I'd hesitate to recommend it to anyone even to deal with
> performance problems.

Not sure why would you hesitate.

> It *does* flicker afterwards (e.g. 0.5 sec after I depress C-v), because 
> that's what deferred jit-lock does: it applies syntax highlighting with 
> a delay.

Highlighting shouldn't flicker; it doesn't here.  It just paints the
text with colors.  Redisplay is smart enough to redraw on the glass
only those parts whose colors have changed, and that shouldn't cause
any flicker.

> So I type, wait 0.5sec, and syntax highlighting arrives. It's a 
> less-aggravating kind of flicker because of its laid-back pace, but I 
> still wouldn't call it acceptable user experience.

I guess we disagree here.  And I don't really see what better solution
could be provided, when Emacs is unable to keep up with keyboard
auto-repeat.  The only solution is to do less, and jit-lock-defer-time
does just that.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 18:36, Eli Zaretskii wrote:
>>>>> Depends on what you mean by "precision".  They might very well miss
>>>>> some parts of the buffer entirely, i.e. never see them on display.
>>>> But that happens regardless of whether the scroll is computed correctly,
>>>> because redisplay itself is skipped.
>>> It isn't skipped here if I use jit-lock-defer-time.
>> I was talking about `fast-but-imprecise-scrolling`
> Now I'm confused: fast-but-imprecise-scrolling doesn't skip redisplay.

Allow me to annotate one of the previous replies:

   regardless of whether the scroll is computed correctly

     (i.e. whether fast-but-imprecise-scrolling is nil or t)

   redisplay itself is skipped.

And it can also be skipped on slow enough machines, even with font-lock 
disabled altogether.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 10 Apr 2020 17:45:43 +0300
> 
> On 10.04.2020 09:47, Eli Zaretskii wrote:
> >> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
> >>   emacs-devel@gnu.org
> >> From: Dmitry Gutov <dgutov@yandex.ru>
> >> Date: Fri, 10 Apr 2020 01:17:51 +0300
> >>
> >> You have addressed only about 30% of my previous email.
> > 
> > I don't have anything to say about the rest.
> 
> Not even about the last patch I sent?

What would you like me to say about it?

> > Depends on what you mean by "precision".  They might very well miss
> > some parts of the buffer entirely, i.e. never see them on display.
> > This may or may not be important, depending on the use case.  An
> > editor is not supposed to skip portions of the buffer when scrolling
> > continuously.
> 
> And that's what my last patch helps it avoid doing, by default.

It does?  Then maybe I didn't understand what it does.

> Speaking of the loss of precision, though. You said you'd never make 
> that choice, and yet you recommend the use of jit-lock-defer-time. It 
> has the very same problem, doesn't it?

Yes.  And that's why I personally don't use jit-lock-defer-time even
on slow machines.  But if someone is prepared to pay that price,
jit-lock-defer-time is IMO a better solution because it is much faster
(and also much cleaner, if you ask me).

> When I said "worse", I didn't mean to say "slower", I meant worse in how 
> it looks. Performance-wise, jit-lock-defer-time=0 and 
> fast-but-imprecise-scrolling=t work about the same here.

On my system, jit-lock-defer-time=0.1 is much faster that
fast-but-imprecise-scrolling=t.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 17:45, Stefan Monnier wrote:
>> It*does*  flicker afterwards (e.g. 0.5 sec after I depress C-v), because
>> that's what deferred jit-lock does: it applies syntax highlighting with
>> a delay. So I type, wait 0.5sec, and syntax highlighting arrives.  It's
>> a less-aggravating kind of flicker because of its laid-back pace, but I
>> still wouldn't call it acceptable user experience.
> And in theory it should be possible to make something like
> (setq jit-lock-defer-mode 0) avoid this flicker because the last
> repetition of `C-v` is executed with `input-pending-p` returning nil, so
> we should be able to fully jit-lock right away with no deferral at
> that point.

When the last repetition is executed, the behavior seems fine already if 
the value is 0. What I see is "flickering" during scrolling, while C-v 
is still pressed. Maybe it's the effect of fontified screenfuls being 
interleaved with unfontified ones.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> When the last repetition is executed, the behavior seems fine already if the
> value is 0. What I see is "flickering" during scrolling, while C-v is still
> pressed.

Hmmm... I don't know why you see "flickering" during the scrolling, indeed.

And I'm surprised you don't see "flickering" on the last repetition: the
way jit-lock-defer-time works, when input-pending-p is non-nil, jit-lock
marks the chunk as fontified (which is a lie), and this chunk will
always go past `window-end`, so on the last repetition the beginning of
the window is not passed to jit-lock at all (because it was part of the
last chunk of the previous call to jit-lock, with input-pending = true),
and it only gets fontified a bit later via the timer.

That's one of the main advantages of the way
`fast-but-imprecise-scrolling` works: it doesn't lie about having
fontified that chunk.

Along the lines of what you proposed, I just tried the patch below,
which is a bit like (setq jit-lock-defer-mode 0) but without the idle timer.
It's not quite right either, tho (e.g. `emacs -Q src/xdisp.c` fails to font-lock
the initial window, tho it does get font-locked as soon as I move the
cursor).


        Stefan


diff --git a/src/xdisp.c b/src/xdisp.c
index 193cc372b0..230e8e2182 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -4097,6 +4097,7 @@ handle_fontified_prop (struct it *it)
 {
   Lisp_Object prop, pos;
   enum prop_handled handled = HANDLED_NORMALLY;
+  extern bool input_was_pending;
 
   if (!NILP (Vmemory_full))
     return handled;
@@ -4108,6 +4109,7 @@ handle_fontified_prop (struct it *it)
   if (!STRINGP (it->string)
       && it->s == NULL
       && !NILP (Vfontification_functions)
+      && !input_was_pending
       && !NILP (Vrun_hooks)
       && (pos = make_fixnum (IT_CHARPOS (*it)),
 	  prop = Fget_char_property (pos, Qfontified, Qnil),

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 10 Apr 2020 10:45:08 -0400
> Cc: rms@gnu.org, emacs-devel@gnu.org, rudalics@gmx.at, rrandresf@gmail.com,
>  acm@muc.de, Eli Zaretskii <eliz@gnu.org>
> 
> And in theory it should be possible to make something like
> (setq jit-lock-defer-mode 0) avoid this flicker because the last
> repetition of `C-v` is executed with `input-pending-p` returning nil, so
> we should be able to fully jit-lock right away with no deferral at
> that point.

Redisplay is not instantaneous, so even if you check at some point
through redisplay and find that no input is pending, this can change
while redisplay is in progress.

In any case, this talk about checking whether input is available seems
to mean we want to resurrect the nil value of redisplay-dont-pause,
doesn't it?

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 18:47, Eli Zaretskii wrote:

>> On 10.04.2020 16:34, Eli Zaretskii wrote:
>>> It doesn't flicker here.  Maybe the double-buffering is the reason?
>>
>> Double-buffering decreases the number of display updates, so it hardly
>> can be the cause of this.
> 
> What else could be the reason of differences between what we see on
> our respective systems?

Maybe the fact that you see updating the highlighting after the text has 
been displayed as okay, and I don't. It feels like flickering to me, a 
visual aggravation. Thus, if I see that during scrolling, it's also 
"flickering".

>> It doesn't flicker while scrolling because it simply shows unfontified
>> screenfuls of code. Which is unacceptable for default behavior.
> 
> Showing unfontified text _is_ the intended effect of
> jit-lock-defer-time.  On slow machines, I fail to see how this could
> be worse than having Emacs hang for many seconds.

There can be other options.

And on *very slow* machines, okay, that can be a solution. Even so, a 
low enough machine might not keep up with scrolling even if font-lock is 
skipped, so it's not a panacea.

>> And I'd hesitate to recommend it to anyone even to deal with
>> performance problems.
> 
> Not sure why would you hesitate.

Because it feels like giving up. "Here's a solution, please disregard 
that it comes with obvious downsides".

>> It *does* flicker afterwards (e.g. 0.5 sec after I depress C-v), because
>> that's what deferred jit-lock does: it applies syntax highlighting with
>> a delay.
> 
> Highlighting shouldn't flicker; it doesn't here.  It just paints the
> text with colors.  Redisplay is smart enough to redraw on the glass
> only those parts whose colors have changed, and that shouldn't cause
> any flicker.

That's what I call flickering in this particular discussion.

When delay is small (e.g. <= 0.1), the visual effect is closer to what 
we would call flickering traditionally (fast visual changes in display 
where there shouldn't be any). When the delay is larger, the effect is 
somewhat less aggravating, but at the same time the lack of syntax 
highlighting for prolonged times is more obvious. That's like a feature 
breakage.

>> So I type, wait 0.5sec, and syntax highlighting arrives. It's a
>> less-aggravating kind of flicker because of its laid-back pace, but I
>> still wouldn't call it acceptable user experience.
> 
> I guess we disagree here.  And I don't really see what better solution
> could be provided, when Emacs is unable to keep up with keyboard
> auto-repeat.  The only solution is to do less, and jit-lock-defer-time
> does just that.

If you go back to my last patch and try it, you might see that both the 
default behavior, as well as the new behavior with f-b-i-s on, *look* 
more responsive. And if you go to the first email in this subthread, 
there are justifications for each change there.

None of it deals with keyboard auto-repeat. On the other hand, having 
input keys fire as fast as they can is not a hard requirement. Ideally, 
Emacs should stop scrolling as soon as I release the 'v' key. No matter 
how many screenfuls it had managed to scroll through in the meantime.

(And there are other commands where such behavior would also be preferable).

I think that can be implemented, in the GUI Emacs, at least.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Performance has to be be good in files without open-parens in column-0.

We would certainly like that to be the case.  Is that the case now,
even on the machines we can use in the Free World?

If so, we don't have a problem.

If not, an improvement for GNU-formatted code is better than nothing.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> We would certainly like that to be the case.  Is that the case now,
> even on the machines we can use in the Free World?

In my experience on my good old T60 and T61, yes.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 20:46, Stefan Monnier wrote:
>> When the last repetition is executed, the behavior seems fine already if the
>> value is 0. What I see is "flickering" during scrolling, while C-v is still
>> pressed.
> 
> Hmmm... I don't know why you see "flickering" during the scrolling, indeed.
> 
> And I'm surprised you don't see "flickering" on the last repetition: the
> way jit-lock-defer-time works, when input-pending-p is non-nil, jit-lock
> marks the chunk as fontified (which is a lie), and this chunk will
> always go past `window-end`, so on the last repetition the beginning of
> the window is not passed to jit-lock at all (because it was part of the
> last chunk of the previous call to jit-lock, with input-pending = true),
> and it only gets fontified a bit later via the timer.

Perhaps the answer is the same as why I do see mostly fontified 
screenfuls while scrolling, even with jit-lock-defer-time=0. Like... 
idle timers manage to run from time to time?

Similarly, I usually don't see a flicker right after visiting a file. 
Only sometimes. Do you?

> That's one of the main advantages of the way
> `fast-but-imprecise-scrolling` works: it doesn't lie about having
> fontified that chunk.

Is there a downside to that strategy? What would happen if jit-lock 
didn't apply the 'defer' value in this case?

> Along the lines of what you proposed, I just tried the patch below,
> which is a bit like (setq jit-lock-defer-mode 0) but without the idle timer.
> It's not quite right either, tho (e.g. `emacs -Q src/xdisp.c` fails to font-lock
> the initial window, tho it does get font-locked as soon as I move the
> cursor).

I also had to make input_was_pending non-static to make it work.

The result is I see a lot more unfontified screenfuls when scrolling 
(which is a minus), but scrolling is even faster now (but it was fast 
with previous patches or f-b-i-s already). Further, it almost certainly 
suffers from the same same problem that makes 
fast-but-imprecise-scrolling apparently unsuitable for being a default 
(loss of precision). The main plus there being that it *can* keep up 
with the keyboard's repeat rate.

But I wonder if anyone would want to scroll this quickly when they can't 
read this fast anyway (or even briefly scan through the contents of the 
windows that fly by).

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, rrandresf@gmail.com,
>  emacs-devel@gnu.org, rms@gnu.org, rudalics@gmx.at
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 11 Apr 2020 03:21:52 +0300
> 
> > What else could be the reason of differences between what we see on
> > our respective systems?
> 
> Maybe the fact that you see updating the highlighting after the text has 
> been displayed as okay, and I don't. It feels like flickering to me, a 
> visual aggravation. Thus, if I see that during scrolling, it's also 
> "flickering".

That's not flickering, at least not in our accepted terminology.
Flickering is redrawing of the entire frame, without any changes to
the contents.  That's not what happens here: Emacs only redraws the
parts that have non-default colors, and it doesn't redraw the tool
bar, the menu bar, and the mode line (only the line number and the
percent are updated in the mode line).

> > Showing unfontified text _is_ the intended effect of
> > jit-lock-defer-time.  On slow machines, I fail to see how this could
> > be worse than having Emacs hang for many seconds.
> 
> There can be other options.
> 
> And on *very slow* machines, okay, that can be a solution. Even so, a 
> low enough machine might not keep up with scrolling even if font-lock is 
> skipped, so it's not a panacea.

On such slow machines, fast-but-imprecise-scrolling is not a solution,
either; IME it actually is worse than jit-lock-defer-time.

> >> And I'd hesitate to recommend it to anyone even to deal with
> >> performance problems.
> > 
> > Not sure why would you hesitate.
> 
> Because it feels like giving up. "Here's a solution, please disregard 
> that it comes with obvious downsides".

What are the alternatives?  If the system is slow, we must cause Emacs
do less so that it doesn't appear as hanging up.

> If you go back to my last patch and try it, you might see that both the 
> default behavior, as well as the new behavior with f-b-i-s on, *look* 
> more responsive.

Perhaps with optimized builds on your fast machine, they do.  But we
are specifically discussing slow systems here.

> None of it deals with keyboard auto-repeat. On the other hand, having 
> input keys fire as fast as they can is not a hard requirement.

Keyboard auto-repeat (or the equivalent repeated turning of the mouse
wheel) was what started this thread.  Let's not change the subject.

> Ideally, 
> Emacs should stop scrolling as soon as I release the 'v' key. No matter 
> how many screenfuls it had managed to scroll through in the meantime.

jit-lock-defer-time satisfies this requirement, at least here, whereas
fast-but-imprecise-scrolling doesn't.

> (And there are other commands where such behavior would also be preferable).

Which commands are those?  Are you sure the current discussion about
jit-lock and its slowdown is relevant for explaining the slowdown of
those commands?

Re: Scrolling commands and skipping redisplay

[Alan Mackenzie]

On Thu, Apr 09, 2020 at 03:32:13 +0300, Dmitry Gutov wrote:

[ .... ]

I think it would be useful to have a quick summary of six possibilities
for fast scrolling of a buffer caused by holding down C-v.

I observed these phenomena on a master branch build with CFLAGS=-Og -g3
before the recent optimisations for this.  I'm running on a Linux
virtual terminal, with 65 lines of text fitting in a window.

In all the tests I held down <page down> for a few seconds, then let it
go.  (<page down> is equivalent to C-v.)

Disclosure: I implemented fast-but-imprecise-scrolling.

(i) (eq jit-lock-defer-timer nil)
  a - f-b-i-s nil:
    One or two screenfuls get displayed; then Emacs appears to hang.
    On releasing <page down> Emacs does nothing and is unresponsive for
    a long time (up to 10 or 20 seconds).  It then displays the position
    it has reached.
  b - f-b-i-s t:
    Emacs continually displays a different screen, always fontified, but
    jerkily.  On releasing <page down>, one or two more screens are
    displayed, and Emacs becomes responsonsive again almost instantly.

(ii) (eq jit-lock-defer-timer 0)
  a - f-b-i-s nil:
    Screenfuls get continually displayed, all fontified, but jerkily.
    On releasing <page down>, the scrolling continues for a long time
    (10 or 20 seconds), and the last screen is displayed, before Emacs
    becomes responsive to user input.
  b - f-b-i-s t:
    Screenfuls get continually displayed, but more slowly that in (ii)a,
    and jerkily.  Screenfuls are partially fontified when they are first
    displayed, and the fontification gets completed an instant later,
    before scrolling further.  On releasing <page down>, Emacs becomes
    responsive again almost instantly.

(iii) (eq jit-lock-defer-time 0.1)
  a - f-b-i-s nil:
    Screenfuls are displayed steadily and rapidly, unfontified.  On
    releasing the key, the scrolling stops instantly, and there is a
    longish (0.1s) delay until the same screen gets redisplayed
    fontified.  Emacs becomes responsive instantly.
  b - f-b-i-s t:
    Indistinguishable from (iii)(a).

My personal view: As defaults, I think (i)a and (ii)a are bad, due
to the long intervals Emacs appears to have hung in.  Of (i)b and (ii)b,
I think (i)b is better, since it avoids the "twinkling" of (ii)b.

I dislike (iii)a,b because of the appearance of non-fontified screens,
but particularly because the last screen gets fontified in situ after
first appearing unfontified.  I think the non-fontification looks poor
to a user, even though the scrolling is smooth.

If it were up to me to chose a default setting, I would prefer (i)b,
closely followed by (ii)b.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Scrolling commands and skipping redisplay

[Eli Zaretskii]

> Date: Sat, 11 Apr 2020 13:21:26 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> 
> My personal view: As defaults, I think (i)a and (ii)a are bad, due
> to the long intervals Emacs appears to have hung in.  Of (i)b and (ii)b,
> I think (i)b is better, since it avoids the "twinkling" of (ii)b.
> 
> I dislike (iii)a,b because of the appearance of non-fontified screens,
> but particularly because the last screen gets fontified in situ after
> first appearing unfontified.  I think the non-fontification looks poor
> to a user, even though the scrolling is smooth.
> 
> If it were up to me to chose a default setting, I would prefer (i)b,
> closely followed by (ii)b.

Disclosure: I didn't implement any of these options.

I dislike (i)b and (ii)b because they are jerky, and because when I
stop leaning on C-v, I have a lot of screen to watch scrolling, until
Emacs starts executing new commands.  I guess my auto-repeat rate is
higher than Alan's. or maybe it's because I'm trying this in a GUI
frame, not a TTY frame.

My favorite (if one needs to alleviate the default, where Emacs hangs
-- I never actually lean on C-v for prolonged times, except for
testing this very issue) is (iii)a, because it gives me an
uninterrupted scrolling and immediate response once I release C-v.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> Perhaps the answer is the same as why I do see mostly fontified screenfuls
> while scrolling, even with jit-lock-defer-time=0.  Like... idle timers manage
> to run from time to time?

That's possible, but if so, I think it might be a bug in itself.

> Similarly, I usually don't see a flicker right after visiting a file.
> Only sometimes.  Do you?

I can't remember seeing a delayed highlight when visiting a file, no.
[ Tho I'm not completely sure which config you're thinking of.
  I assume you're talking about (setq jit-lock-defer-time 0).  ]

>> That's one of the main advantages of the way
>> `fast-but-imprecise-scrolling` works: it doesn't lie about having
>> fontified that chunk.
> Is there a downside to that strategy?
> What would happen if jit-lock didn't apply the 'defer' value in this case?

Not sure what would happen, but it would mean jit-lock's
fontification-function doesn't hold its end of the bargain with the redisplay.
Maybe it would cause jit-lock to be called right back on the next
character?

> The result is I see a lot more unfontified screenfuls when scrolling (which
> is a minus), but scrolling is even faster now (but it was fast with previous
> patches or f-b-i-s already). Further, it almost certainly suffers from the
> same same problem that makes fast-but-imprecise-scrolling apparently
> unsuitable for being a default (loss of precision).

Normally, input_was_pending should be the same during scroll than during
the subsequent redisplay, so there should not be any loss of precision, no.

> The main plus there being that it *can* keep up with the keyboard's
> repeat rate.

Aha!

> But I wonder if anyone would want to scroll this quickly when they can't
> read this fast anyway (or even briefly scan through the contents of the
> windows that fly by).

I think we should aim for "scrolling always keeps up".  When
I lean on C-v, it's often because I'm looking for something which I can
recognize visually, so when display is skipped it defeats the purpose.
Of course, sometimes the "something which I can recognize" needs to be
font-locked to be recognizable in which case skipping jit-lock may also
defeat the purpose (unless it had been font-locked earlier), but still,
something is better than nothing.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Noam Postavsky]

On Tue, 7 Apr 2020 at 04:45, martin rudalics <rudalics@gmx.at> wrote:

> So what I would vote for in this case is that font-lock would specially
> highlight the open paren that "stops seeing the comment as comment" so
> the user is aware of the fact.  Would that be so hard to implement?
> Even with a slight change in 'parse-partial-sexp' and 'syntax-ppss' that
> would accompany it.  Optionally, by default turned off, so nobody who
> wants the correct syntactic interpretation would be bothered.

I think Bug#21871 "Emacs Lisp Mode (at least): spurious parens in
column 0 don't get bold red highlighting." is related. Perhaps the
patch I posted at https://debbugs.gnu.org/21871#57 could help?

Re: Scrolling commands and skipping redisplay

[Alan Mackenzie]

Hello, Eli.

On Sat, Apr 11, 2020 at 16:45:02 +0300, Eli Zaretskii wrote:
> > Date: Sat, 11 Apr 2020 13:21:26 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> > 
> > My personal view: As defaults, I think (i)a and (ii)a are bad, due
> > to the long intervals Emacs appears to have hung in.  Of (i)b and (ii)b,
> > I think (i)b is better, since it avoids the "twinkling" of (ii)b.
> > 
> > I dislike (iii)a,b because of the appearance of non-fontified screens,
> > but particularly because the last screen gets fontified in situ after
> > first appearing unfontified.  I think the non-fontification looks poor
> > to a user, even though the scrolling is smooth.
> > 
> > If it were up to me to chose a default setting, I would prefer (i)b,
> > closely followed by (ii)b.

> Disclosure: I didn't implement any of these options.

:-)

> I dislike (i)b and (ii)b because they are jerky, and because when I
> stop leaning on C-v, I have a lot of screen to watch scrolling, until
> Emacs starts executing new commands.  I guess my auto-repeat rate is
> higher than Alan's. or maybe it's because I'm trying this in a GUI
> frame, not a TTY frame.

It sounds like your time for scrolling a screen, even without
fontification, is close to your auto-repeat time.  My repeat rate is 30
characters/second.

> My favorite (if one needs to alleviate the default, where Emacs hangs
> -- I never actually lean on C-v for prolonged times, except for
> testing this very issue) is (iii)a, because it gives me an
> uninterrupted scrolling and immediate response once I release C-v.

Each one of our sets of (dis)likes is reasonable.  That's a strong
argument for retaining both jit-lock deferred fontification and
fast-but-imprecise-scrolling.

What seems not so good is the difficulty a user, on experiencing the
problem of 15 second hangs, would have in finding both of the above
solutions.  What do you think of adding a paragraph like the following
after the description of f-b-i-scrolling on the page "Scrolling" in the
Emacs manual:

    As an alternative to setting `fast-but-imprecise-scrolling' you
    might prefer to enable jit-lock deferred fontification (See Font
    Lock).  To do this, customize `jit-lock-defer-time' to a small
    positive number such as 0.1.  This gives you less jerky scrolling,
    but the buffer contents on any scroll operation into a fresh portion
    of the buffer will at first be unfontified,

?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Scrolling commands and skipping redisplay

[Eli Zaretskii]

> Date: Sat, 11 Apr 2020 17:04:04 +0000
> Cc: dgutov@yandex.ru, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > I dislike (i)b and (ii)b because they are jerky, and because when I
> > stop leaning on C-v, I have a lot of screen to watch scrolling, until
> > Emacs starts executing new commands.  I guess my auto-repeat rate is
> > higher than Alan's. or maybe it's because I'm trying this in a GUI
> > frame, not a TTY frame.
> 
> It sounds like your time for scrolling a screen, even without
> fontification, is close to your auto-repeat time.

_Only_ without fontification.  With fontifications, at least in CC
Mode, Emacs gets stuck after several screenfuls in -O0 and -Og builds,
which means the time to scroll is longer than the auto-repeat
interval.

> My repeat rate is 30 characters/second.

It's about 50 here.

>     As an alternative to setting `fast-but-imprecise-scrolling' you
>     might prefer to enable jit-lock deferred fontification (See Font
>     Lock).  To do this, customize `jit-lock-defer-time' to a small
>     positive number such as 0.1.  This gives you less jerky scrolling,
>     but the buffer contents on any scroll operation into a fresh portion
>     of the buffer will at first be unfontified,

No objections.

Re: Scrolling commands and skipping redisplay

[Alan Mackenzie]

Hello, Eli.

On Sat, Apr 11, 2020 at 20:15:55 +0300, Eli Zaretskii wrote:
> > Date: Sat, 11 Apr 2020 17:04:04 +0000
> > Cc: dgutov@yandex.ru, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > It sounds like your time for scrolling a screen, even without
> > fontification, is close to your auto-repeat time.

> _Only_ without fontification.  With fontifications, at least in CC
> Mode, Emacs gets stuck after several screenfuls in -O0 and -Og builds,
> which means the time to scroll is longer than the auto-repeat
> interval.

This would presumably happen in emacs-lisp-mode, too, if you ever did
auto-repeated C-v's.

> >     As an alternative to setting `fast-but-imprecise-scrolling' you
> >     might prefer to enable jit-lock deferred fontification (See Font
> >     Lock).  To do this, customize `jit-lock-defer-time' to a small
> >     positive number such as 0.1.  This gives you less jerky scrolling,
> >     but the buffer contents on any scroll operation into a fresh portion
> >     of the buffer will at first be unfontified,

> No objections.

OK, I've tidied it up a bit, and pushed it to the emacs-27 branch.

It's looking like the gnu.org mailing server is down.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 11.04.2020 16:45, Stefan Monnier wrote:
>> Perhaps the answer is the same as why I do see mostly fontified screenfuls
>> while scrolling, even with jit-lock-defer-time=0.  Like... idle timers manage
>> to run from time to time?
> 
> That's possible, but if so, I think it might be a bug in itself.

Any suggestions for debugging it?

FWIW, it happens with 'emacs -Q' here.

>> Similarly, I usually don't see a flicker right after visiting a file.
>> Only sometimes.  Do you?
> 
> I can't remember seeing a delayed highlight when visiting a file, no.
> [ Tho I'm not completely sure which config you're thinking of.
>    I assume you're talking about (setq jit-lock-defer-time 0).  ]

Yes. Shouldn't it happen, if we follow the same logic?

>>> That's one of the main advantages of the way
>>> `fast-but-imprecise-scrolling` works: it doesn't lie about having
>>> fontified that chunk.
>> Is there a downside to that strategy?
>> What would happen if jit-lock didn't apply the 'defer' value in this case?
> 
> Not sure what would happen, but it would mean jit-lock's
> fontification-function doesn't hold its end of the bargain with the redisplay.
> Maybe it would cause jit-lock to be called right back on the next
> character?

But it seems to work all right during "simulated" redisplay in 
scroll_command, doesn't it? When we bind fontifications-functions to nil 
(with f-b-i-s enabled). Or does that work fine only because there are no 
fontification functions to call?

>> The result is I see a lot more unfontified screenfuls when scrolling (which
>> is a minus), but scrolling is even faster now (but it was fast with previous
>> patches or f-b-i-s already). Further, it almost certainly suffers from the
>> same same problem that makes fast-but-imprecise-scrolling apparently
>> unsuitable for being a default (loss of precision).
> 
> Normally, input_was_pending should be the same during scroll than during
> the subsequent redisplay, so there should not be any loss of precision, no.

No, the same loss of precision as associated with f-b-i-s. Since 
font-lock won't occur while scrolling, whatever different-sized fonts, 
etc, that could be applied, won't be. So the scrolling positions will be 
imprecise.

>> But I wonder if anyone would want to scroll this quickly when they can't
>> read this fast anyway (or even briefly scan through the contents of the
>> windows that fly by).
> 
> I think we should aim for "scrolling always keeps up".  When
> I lean on C-v, it's often because I'm looking for something which I can
> recognize visually, so when display is skipped it defeats the purpose.
> Of course, sometimes the "something which I can recognize" needs to be
> font-locked to be recognizable in which case skipping jit-lock may also
> defeat the purpose (unless it had been font-locked earlier), but still,
> something is better than nothing.

The other way to do this can be:

a) make sure redisplay is never skipped while scrolling,
b) somehow throttle input events in this kind of situations.

How to define "this kind of sitiations" is an open question thus far, 
but it should be possible to to differentiate between the user hitting 
the keys and the user holding them, at least in graphical environments.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 11.04.2020 10:55, Eli Zaretskii wrote:

>> Maybe the fact that you see updating the highlighting after the text has
>> been displayed as okay, and I don't. It feels like flickering to me, a
>> visual aggravation. Thus, if I see that during scrolling, it's also
>> "flickering".
> 
> That's not flickering, at least not in our accepted terminology.
> Flickering is redrawing of the entire frame, without any changes to
> the contents.

Allow me to disagree: flickering can be localized. We've have 
discussions in the past about stuff flickering around the cursor, for 
example (due to running timers and the absence of double buffering).

In any case, I called it that because, for lower values of the 
delay-time, is feels about about the same, and is similarly annoying.

Better names welcome.

>> And on *very slow* machines, okay, that can be a solution. Even so, a
>> low enough machine might not keep up with scrolling even if font-lock is
>> skipped, so it's not a panacea.
> 
> On such slow machines, fast-but-imprecise-scrolling is not a solution,
> either; IME it actually is worse than jit-lock-defer-time.

In doesn't mess with fontification. So the users would lose rather 
little (most of the time) if such fast-but-imprecise-scrolling was 
enabled by default, especially with the proposed improvement.

IOW, the scrolling speed-up it offers is limited (but could be enough in 
optimized builds for most of us, and even for -Og builds of current 
master), but the the downside it brings is the smallest among all the 
options we've had discussed here.

My point above there was rather different: removing features to 
compensate for a high keyboard repeat rate just doesn't seem like a good 
solution.

>> Because it feels like giving up. "Here's a solution, please disregard
>> that it comes with obvious downsides".
> 
> What are the alternatives?  If the system is slow, we must cause Emacs
> do less so that it doesn't appear as hanging up.

I agree with "do less", of course.

>> If you go back to my last patch and try it, you might see that both the
>> default behavior, as well as the new behavior with f-b-i-s on, *look*
>> more responsive.
> 
> Perhaps with optimized builds on your fast machine, they do.  But we
> are specifically discussing slow systems here.

Have you tried them? Do they not look more responsive [while scrolling 
continues] than the default behavior?

>> None of it deals with keyboard auto-repeat. On the other hand, having
>> input keys fire as fast as they can is not a hard requirement.
> 
> Keyboard auto-repeat (or the equivalent repeated turning of the mouse
> wheel) was what started this thread.  Let's not change the subject.

I'm not changing it.

There are two main problems, mentioned from the beginning:

- The user doesn't see what's going on (redisplay stops updating).
- Emacs continues to do something long after the key have been released, 
or the mouse wheel stopped rotating.

>> Ideally,
>> Emacs should stop scrolling as soon as I release the 'v' key. No matter
>> how many screenfuls it had managed to scroll through in the meantime.
> 
> jit-lock-defer-time satisfies this requirement, at least here, whereas
> fast-but-imprecise-scrolling doesn't.

(setq jit-lock-defer-time 0) doesn't satisfy that requirement on my 
machine with '-Og -g3'.

>> (And there are other commands where such behavior would also be preferable).
> 
> Which commands are those?

'undo' and 'backward-delete' come to mind, both of which have repeatable 
bindings in my config. The first one especially can take arbitrary 
amounts of time. 'previous-line' and 'next-line' too (their misbehavior 
is usually unnoticed, but just try (setq scroll-conservatively 1)).

> Are you sure the current discussion about
> jit-lock and its slowdown is relevant for explaining the slowdown of
> those commands?

It would be relevant if we look into other solutions rather than trying 
to speed those commands' execution.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 10.04.2020 18:56, Eli Zaretskii wrote:
>>> Depends on what you mean by "precision".  They might very well miss
>>> some parts of the buffer entirely, i.e. never see them on display.
>>> This may or may not be important, depending on the use case.  An
>>> editor is not supposed to skip portions of the buffer when scrolling
>>> continuously.
>> And that's what my last patch helps it avoid doing, by default.
> It does?  Then maybe I didn't understand what it does.

The "else" branch (meaning the new default behavior) makes sure 
redisplay is never skipped during scrolling commands.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> Normally, input_was_pending should be the same during scroll than during
>> the subsequent redisplay, so there should not be any loss of precision, no.
> No, the same loss of precision as associated with f-b-i-s. Since font-lock
> won't occur while scrolling, whatever different-sized fonts, etc, that could
> be applied, won't be. So the scrolling positions will be imprecise.

It is precise in the sense that it will skip exactly the amount
corresponding to what is displayed.  Because jit-lock is skipped, what
is displayed is different, but the scrolling itself is "precise".


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, rrandresf@gmail.com,
>  emacs-devel@gnu.org, rms@gnu.org, rudalics@gmx.at
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 12 Apr 2020 05:51:58 +0300
> 
> On 11.04.2020 10:55, Eli Zaretskii wrote:
> 
> >> Maybe the fact that you see updating the highlighting after the text has
> >> been displayed as okay, and I don't. It feels like flickering to me, a
> >> visual aggravation. Thus, if I see that during scrolling, it's also
> >> "flickering".
> > 
> > That's not flickering, at least not in our accepted terminology.
> > Flickering is redrawing of the entire frame, without any changes to
> > the contents.
> 
> Allow me to disagree

I was talking about our terminology.  Disagreeing with accepted
terminology will just increase confusion, as happened here.  I Only
now understand what you mean by that.

> IOW, the scrolling speed-up it offers is limited (but could be enough in 
> optimized builds for most of us, and even for -Og builds of current 
> master), but the the downside it brings is the smallest among all the 
> options we've had discussed here.

In your opinion; mine differs.

> My point above there was rather different: removing features to 
> compensate for a high keyboard repeat rate just doesn't seem like a good 
> solution.

We do remove features in other similar situations, for example in
so-long-mode.

> >> If you go back to my last patch and try it, you might see that both the
> >> default behavior, as well as the new behavior with f-b-i-s on, *look*
> >> more responsive.
> > 
> > Perhaps with optimized builds on your fast machine, they do.  But we
> > are specifically discussing slow systems here.
> 
> Have you tried them? Do they not look more responsive [while scrolling 
> continues] than the default behavior?

Yes.  No.

> >> Ideally,
> >> Emacs should stop scrolling as soon as I release the 'v' key. No matter
> >> how many screenfuls it had managed to scroll through in the meantime.
> > 
> > jit-lock-defer-time satisfies this requirement, at least here, whereas
> > fast-but-imprecise-scrolling doesn't.
> 
> (setq jit-lock-defer-time 0) doesn't satisfy that requirement on my 
> machine with '-Og -g3'.

Don't use zero, use something like 0.05 or 0.1.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 12 Apr 2020 06:07:17 +0300
> 
> On 10.04.2020 18:56, Eli Zaretskii wrote:
> >>> Depends on what you mean by "precision".  They might very well miss
> >>> some parts of the buffer entirely, i.e. never see them on display.
> >>> This may or may not be important, depending on the use case.  An
> >>> editor is not supposed to skip portions of the buffer when scrolling
> >>> continuously.
> >> And that's what my last patch helps it avoid doing, by default.
> > It does?  Then maybe I didn't understand what it does.
> 
> The "else" branch (meaning the new default behavior) makes sure 
> redisplay is never skipped during scrolling commands.

I cannot imagine why would we want to exempt only the scrolling
commands from the input_was_pending trick.  And if we want to exempt
all of them, then isn't that tantamount to just removing that
variable?  And if we remove it, what to do ab out the scenario
described by Stefan in the large comment to that variable?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Fri, Apr 10, 2020 at 06:33:13 +0300, Dmitry Gutov wrote:
> Hi Alan,

> On 08.04.2020 05:09, Alan Mackenzie wrote:

> >> I will shut up about it now (saying it twice it plenty), but I am pretty
> >> confident saying that if you manage to migrate to s-p-f, file opening
> >> time will go down.

> > I'm sure it would.  If file opening time were really a concern, a hybrid
> > algorithm would perhaps be the best way: apply the text properties first
> > in a lazy fashion, and thereafter treat them with care, as CC Mode
> > currently does.

> s-p-f would help with the first step, and as for "treating them with 
> care", it would be good for us all to see how much of that is really 
> needed. Improving syntax-propertize is not out of the question, and it 
> might benefit several major modes, not just the CC collection.

> > But this would merely transfer the start up time to the
> > time taken in early scrolls forward.

> Not really. The start up scans the whole buffer, doesn't it? The early 
> scrolls forward would still scan only a fraction of it.

I'm thinking more about "scrolling" to the function in the file that one
wants to work on or look at.  On average, this will be a little more
than half way through the file (there is often a large comment block at
BOB).  So you'd only be saving about half of CC Mode's start-up scan.

> >> Performance while typing is likely to improve too, at least when the
> >> same buffer is not shown in another window, many thousand lines later.

> > What makes you think this?

> Inserting characters can alter the syntax state of the whole buffer. At 
> least that's true for some of them. Full buffer scan sounds inevitable 
> in those cases.

Full buffer scans are very unusual.  Inserting a " where every
subsequent line ended with a backslash might do that.  Inserting a C++ <
wouldn't - its effect is limited to up to the next brace or semicolon.

Inserting a C++ raw string opener does typically necessitate a full
scan (a search for a matching closer), but that would also be the case
using syntax-propertize.

> >> "Considerable enhancement" can also be a part of that discussion.

> > The syntax-propertize-function mechanism works by erasing ALL
> > syntax-table properties after a change point, then reapplying them
> > lazily.

> That's not right. It only erases syntax-table properties in a chunk 
> before calling syntax-propertize-function on the same range of 
> positions. IOW, is overwrites them lazily as well.

Sorry, I was mistaken there.  The bounds for erasing and re-applying the
s-t props are determined (except in simple cases) by
syntax-propertize-extend-region-functions.

So, we would merely be moving functions from
c-get-state-before-change-functions and c-before-font-lock-functions
(effectively lists of before-/after-change functions) to
s-p-extend-region-f, together with adaptation.  Would you agree that
such a change to CC Mode would be largely pointless if some of these
functions had to remain on c-get-state-b-c-f and c-before-f-l-f?

But the way s-p-extend-region-f functions are called is to keep calling
them repeatedly until they've all said "no change" together.  This would
dramatically slow down CC Mode, where currently these functions are each
called exactly once.

Also, the syntax-propertize mechanism is weaker than CC Mode's: When it
is run, there is no way of knowing whether it's being called as a change
function, and if it is, OLD-LEN is discarded.  How can it have access to
variables set in before-change-functions?  (An example of such is
c-raw-string-end-delim-disrupted.  In before change, it is set when the
existing raw string end delimiter is about to cease to be.  In after
change, the fact of this flag being nil means we don't need to search
for an alternative closing delimiter, etc.  This change can obviously
not be detected in an after-change function.)

>  > Considering that s-t properties have an overwhelmingly local
>  > effect, this is very wasteful of processor time.

> It would have been. As you can see, it's not a difficult problem to fix, 
> even if it were still present.

The lack of full information (see above) in the syntax-propertize
mechanism is a problem.

> > Consider, for example, editing within a large C++ raw string, a common
> > occurrence.  You yourself reported as a bug sluggish performance here in
> > mid 2016.  The cause was erasing too many s-t text properties at a
> > buffer change.  I think we were talking about 1 second per typed
> > character in the scenario you gave.  There are typically lots of these
> > properties in a raw string, in particular on " characters.

> I'm pretty sure I have thought of that example because it's an instance 
> of a syntax problem that's easy enough to solve within 
> syntax-propertize-function framework.

Having actually gone through all the issues and implemented raw strings,
I can't agree with you there.  There are all sorts of subtleties which
necessitate intimate cooperation between the before-change-functions and
after-change-functions.  Such cooperation seems to be excluded by the
syntax-propertize mechanism.

> > Consider(2) a C++ template: excusing my C++ syntax knowledge, type in

> >     template class foo < bar, baz >= bar>

> > , perhaps typing in the odd newline inside the template (a common
> > occurrence), or nesting further templates inside it (also a common
> > occurrence).  Note how the parenthesis text properties are added and
> > removed as you type.  All these modification are necessary, and they are
> > largely _before_ the point of insertion, not after it.

> The current implementation of applying these properties can probably be 
> transferred into a syntax-propertize-function with only modest changes.

Maybe, but with a slowdown.  More of these properties will get erased
than needed (with nested template forms), and they will all need to get
put back again.

> >> Some scenarios can become slower, that's for sure. But the more common
> >> ones can get faster. We won't know until we try.

Other than starting up a buffer, we still haven't identified any
specific scenarios where speed up might happen.

> > Trying would be a _lot_ of work.  How is one to handle the common
> > example scenarios above?

> Stefan has offered to help. And I'm sure he could answer the follow-up 
> questions much better than I.

I've tried quite a few optimisations over the years.  Some have been
successful, but all too often I've put in a lot of work, then at the end
of it the profiler tells me It's just been a waste of time.  I strongly
suspect that that would be the result here, too.

> > Well, you'd have to enhance the syntax-propertize-function with a
> > means of determining a start position for erasing s-t props, and
> > also a stop position.

> The real-world uses of s-p-f out there already solve syntax problems of 
> comparable complexity. And move the start position, among other things.

OK.  I was mistaken there.

> > Once you do that, you're effectively doing what CC Mode currently
> > does, so where's the speed advantage coming from?

>  From doing things more lazily, is how I see it. But I'm not an expert 
> on CC Mode architecture.

> Among other benefits, moving it to a standard-ish framework like s-p-f 
> could (possibly) simplify its code, as well as make it more approachable 
> for other developers already familiar with how most other major modes 
> are written. So far I wouldn't even know where to start fixing bugs in 
> it, and IMHO CC Mode currently has bus factor = 1. It's not great for 
> its future. I suspect it's not ideal for you either.

I don't think the syntax-propertize mechanism is all that brilliant.
It's too constrained, and places too many restrictions on what can be
done with the syntax-table text property.  For example, (from
syntax.el):

(defvar syntax-propertize-function nil
  ;; Rather than a -functions hook, this is a -function because it's easier
  ;; to do a single scan than several scans: with multiple scans, one cannot
  ;; assume that the text before point has been propertized, so syntax-ppss
  ;; gives unreliable results (and stores them in its cache to boot, so we'd
  ;; have to flush that cache between each function, and we couldn't use
  ;; syntax-ppss-flush-cache since that would not only flush the cache but also
  ;; reset syntax-propertize--done which should not be done in this case).

From my point of view, "multiple scans" are _much_ easier.  They are
prohibited here only because syntax-ppss and syntax-propertize-function
have got themselves tied up in a tight knot.  One answer would be not to
use syntax-ppss inside a s-p-function.  (CC Mode doesn't use syntax-ppss
at all).  Another answer would be to give the responsibility of removing
the s-t text properties to the s-p-function.

> Simply collaborating with one other developer on an overhaul project 
> (whether it succeeds or not; perhaps partially) can improve on that.

But take a massive amount of time.

>    Cheers.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 12.04.2020 18:34, Alan Mackenzie wrote:
>>> But this would merely transfer the start up time to the
>>> time taken in early scrolls forward.
> 
>> Not really. The start up scans the whole buffer, doesn't it? The early
>> scrolls forward would still scan only a fraction of it.
> 
> I'm thinking more about "scrolling" to the function in the file that one
> wants to work on or look at.  On average, this will be a little more
> than half way through the file (there is often a large comment block at
> BOB).  So you'd only be saving about half of CC Mode's start-up scan.

Yes, on average it's only 2x benefit, but then again, if the buffer 
opens at its beginning first, the extra initialization will be spread 
across user commands. Which should have some additional positive effect 
on apparent responsiveness.

>> Inserting characters can alter the syntax state of the whole buffer. At
>> least that's true for some of them. Full buffer scan sounds inevitable
>> in those cases.
> 
> Full buffer scans are very unusual.  Inserting a " where every
> subsequent line ended with a backslash might do that.  Inserting a C++ <
> wouldn't - its effect is limited to up to the next brace or semicolon.
> 
> Inserting a C++ raw string opener does typically necessitate a full
> scan (a search for a matching closer), but that would also be the case
> using syntax-propertize.

Not really. It would just mark the opener as a string opener (maybe with 
some extra text property), and that's that. Then font-lock would fontify 
the following text as string contents (until the end of the window or a 
little bit farther). Then you type the closer, it only has to scan a 
little far back (it'll call syntax-ppss to find the string opener), the 
closer is propertized as appropriate, and that's that. No full buffer 
scans at any step.

I recall that fontifying the rest of the buffer as text after a simple 
string opener could be a sore topic for you, but raw strings should be 
rare enough (aren't they?), or if they are not, fontification logic 
could opt to do something different, while syntax-table properties will 
be applied the "correct" way.

> So, we would merely be moving functions from
> c-get-state-before-change-functions and c-before-font-lock-functions
> (effectively lists of before-/after-change functions) to
> s-p-extend-region-f, together with adaptation. Would you agree that
> such a change to CC Mode would be largely pointless if some of these
> functions had to remain on c-get-state-b-c-f and c-before-f-l-f?

Yes, I think before-change-functions should become empty. Or much emptier.

> But the way s-p-extend-region-f functions are called is to keep calling
> them repeatedly until they've all said "no change" together.  This would
> dramatically slow down CC Mode, where currently these functions are each
> called exactly once.

Here's a simple solution: create one function special for CC Mode that 
would do that, and add it to s-p-extend-region-f.

But there might be even better way to do that. I'm not the best person 
to discuss that with.

> Also, the syntax-propertize mechanism is weaker than CC Mode's: When it
> is run, there is no way of knowing whether it's being called as a change
> function, and if it is, OLD-LEN is discarded.  How can it have access to
> variables set in before-change-functions?  (An example of such is
> c-raw-string-end-delim-disrupted.  In before change, it is set when the
> existing raw string end delimiter is about to cease to be.  In after
> change, the fact of this flag being nil means we don't need to search
> for an alternative closing delimiter, etc.  This change can obviously
> not be detected in an after-change function.)

As we seem to agree, before-change-functions should not be needed. 
Neither should be the code that tracks the details of the edits that the 
user makes. That alone can simplify some things.

>> I'm pretty sure I have thought of that example because it's an instance
>> of a syntax problem that's easy enough to solve within
>> syntax-propertize-function framework.
> 
> Having actually gone through all the issues and implemented raw strings,
> I can't agree with you there.  There are all sorts of subtleties which
> necessitate intimate cooperation between the before-change-functions and
> after-change-functions.  Such cooperation seems to be excluded by the
> syntax-propertize mechanism.

It encourages a different approach. Again: there are examples of raw 
strings support in other major modes.

>>> Consider(2) a C++ template: excusing my C++ syntax knowledge, type in
> 
>>>      template class foo < bar, baz >= bar>
> 
>>> , perhaps typing in the odd newline inside the template (a common
>>> occurrence), or nesting further templates inside it (also a common
>>> occurrence).  Note how the parenthesis text properties are added and
>>> removed as you type.  All these modification are necessary, and they are
>>> largely _before_ the point of insertion, not after it.
> 
>> The current implementation of applying these properties can probably be
>> transferred into a syntax-propertize-function with only modest changes.
> 
> Maybe, but with a slowdown.  More of these properties will get erased
> than needed (with nested template forms), and they will all need to get
> put back again.

We won't really know until we can measure the result.

>>>> Some scenarios can become slower, that's for sure. But the more common
>>>> ones can get faster. We won't know until we try.
> 
> Other than starting up a buffer, we still haven't identified any
> specific scenarios where speed up might happen.

When before-changes-functions only contains (t syntax-ppss-flush-cache), 
that can visibly change the performance tradeoffs.

> I don't think the syntax-propertize mechanism is all that brilliant.
> It's too constrained, and places too many restrictions on what can be
> done with the syntax-table text property.  For example, (from
> syntax.el):

I wouldn't say it's perfect either. But it's proven helpful over the 
years, and provided a base design for a lot of major mode implementations.

> (defvar syntax-propertize-function nil
>    ;; Rather than a -functions hook, this is a -function because it's easier
>    ;; to do a single scan than several scans: with multiple scans, one cannot
>    ;; assume that the text before point has been propertized, so syntax-ppss
>    ;; gives unreliable results (and stores them in its cache to boot, so we'd
>    ;; have to flush that cache between each function, and we couldn't use
>    ;; syntax-ppss-flush-cache since that would not only flush the cache but also
>    ;; reset syntax-propertize--done which should not be done in this case).
> 
>  From my point of view, "multiple scans" are _much_ easier.  They are
> prohibited here only because syntax-ppss and syntax-propertize-function
> have got themselves tied up in a tight knot.  One answer would be not to
> use syntax-ppss inside a s-p-function.  (CC Mode doesn't use syntax-ppss
> at all).  Another answer would be to give the responsibility of removing
> the s-t text properties to the s-p-function.

I think we could extend the customizability in that direction. But first 
we'd have to see clear evidence that the current design is not good 
enough (e.g. CC Mode has been reimplemented on top of it, and the result 
is decidedly not fast enough).

>> Simply collaborating with one other developer on an overhaul project
>> (whether it succeeds or not; perhaps partially) can improve on that.
> 
> But take a massive amount of time.

You could consider it an investment.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 12.04.2020 08:03, Stefan Monnier wrote:
>>> Normally, input_was_pending should be the same during scroll than during
>>> the subsequent redisplay, so there should not be any loss of precision, no.
>> No, the same loss of precision as associated with f-b-i-s. Since font-lock
>> won't occur while scrolling, whatever different-sized fonts, etc, that could
>> be applied, won't be. So the scrolling positions will be imprecise.
> It is precise in the sense that it will skip exactly the amount
> corresponding to what is displayed.  Because jit-lock is skipped, what
> is displayed is different, but the scrolling itself is "precise".

Hmm, good point.

Overall, this approach might be better than nothing (better than 
stalling display, I think), especially if font-lock right after a file 
has been visited can be fixed.

As default behavior, though, users or mid-power systems might prefer if 
we throttle keyboard input in such conditions instead.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 12.04.2020 10:10, Eli Zaretskii wrote:
> I cannot imagine why would we want to exempt only the scrolling
> commands from the input_was_pending trick.

Scrolling is certainly special at least in some way:

- Implementation-wise, not all commands simulate redisplay during their 
operation. Given that the heavy lifting has probably already been done, 
maybe skipping redisplay doesn't help save a lot of work anyway? At 
least without extra tricks like jit-lock-defer-time which speed up both 
simulated and "real" redisplay, making skipping redisplay less essential 
anyway.

- As we talked about "precision", the user needs to see the buffer 
contents to see where they are scrolling to, and where to stop. If the 
number of screens you need to scroll is >1, you can likely never decide 
on the necessary number of scrolls in advance. This is not the case for 
many other Emacs commands. E.g., if I use find-file, I usually don't 
need to see the prompt, and I can start typing the desired file name in 
advance.

> And if we remove it, what to do ab out the scenario
> described by Stefan in the large comment to that variable?

In a way, the comment supports my point:

   ...all the hard work of jit-locking is "wasted"...

The current approach helps avoid this most of the time, but it errs on 
the side of skipping. I suggest it might be better to err on the side of 
not skipping, for a certain set of commands at least.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 16 Apr 2020 05:18:54 +0300
> 
> On 12.04.2020 10:10, Eli Zaretskii wrote:
> > I cannot imagine why would we want to exempt only the scrolling
> > commands from the input_was_pending trick.
> 
> Scrolling is certainly special at least in some way:
> 
> - Implementation-wise, not all commands simulate redisplay during their 
> operation.

I think you will find that many more do than you seem to assume.  Even
just redisplaying a window does this in many cases (to find the proper
place for window-start position).  C-n and C-p do as well.  And when
scroll-conservatively is in effect, almost every command that moves
point does.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>   ;; Rather than a -functions hook, this is a -function because it's easier
>   ;; to do a single scan than several scans: with multiple scans, one cannot
[...]
> From my point of view, "multiple scans" are _much_ easier.

Obviously it's a matter of opinion.

Luckily, you have a Turing complete language at hand, so
`syntax-propertize-function` does not *impose* a single scan, it just
encourages it (especially with `syntax-propertize-rules`).

Your `syntax-propertize-function` can definitely use multiple scans if
it wants (and it's simply up to you to be careful about the fact that
until the last scan, `syntax-ppss` (and `parse-partial-sexp`) will
obviously not give the "final" answer, and if you use `syntax-ppss`
there, it's then your responsability to manually flush its cache with
`syntax-ppss-flush-cache`).

Really, you can use `syntax-propertize-function` with CC-mode without
significant changes to the code (i.e. reusing the code mostly unchanged
for the actual work).  The basic idea is to have CC-mode pretend that
the buffer only extends as far as `syntax-propertize--done`:

- when opening the file, pretend it's empty (i.e. do the initial scan
  as if `point-max` were equal to `point-min`).
- when `syntax-propertize-function` is called with arguments BEG and END,
  treat it as if the old code was faced with the following sequence of
  operations:
  1- delete all the text between BEG and `point-max`.
  2- insert at the end of file the new text found between BEG and END.

Clearly, the old code is supposed to handle correctly all three elements
(an empty file, deletion of text at EOB, and insertion of text at EOB),
so the old code will be able to handle those requests correctly without
any need for a redesign (you just have to "hide" the rest of the buffer
from the corresponding code, which might be doable with narrowing).

Of course the code of CC-mode is designed to handle many more operations
than deletion and insertion at EOB, so after that first step you could
try and adjust the code accordingly, but that can only make it simpler,
and is not necessary for a first cut.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 16.04.2020 08:22, Eli Zaretskii wrote:
>> Scrolling is certainly special at least in some way:
>>
>> - Implementation-wise, not all commands simulate redisplay during their
>> operation.
> I think you will find that many more do than you seem to assume.  Even
> just redisplaying a window does this in many cases (to find the proper
> place for window-start position).  C-n and C-p do as well.  And when
> scroll-conservatively is in effect, almost every command that moves
> point does.

Would that be... almost all of them?

(Indeed, the fact that (setq scroll-conservatively 1) doesn't work as 
well as one would expect is a real bug IMO).

In any case, if there is a way to dynamically detect these cases and 
disable redisplay skipping for them, I'd like to try that out.

Even more if there was a way to put the results of "simulated redisplay" 
to use in the "real" redisplay later.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> - Implementation-wise, not all commands simulate redisplay during their 
>>   operation.
> I think you will find that many more do than you seem to assume.  Even
> just redisplaying a window does this in many cases (to find the proper
> place for window-start position).  C-n and C-p do as well.

char-based, word-based, and sexp-based motion doesn't.  But line and
motion that depends on "visual size" indeed typically does.

> And when scroll-conservatively is in effect, almost every command that
> moves point does.

Really?  I thought `scroll-conservatively` only affects the redisplay
code and not the execution of "normal" commands (i.e. commands which
don't themselves rely on simulating redisplay).


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 16 Apr 2020 23:13:16 +0300
> 
> > I think you will find that many more do than you seem to assume.  Even
> > just redisplaying a window does this in many cases (to find the proper
> > place for window-start position).  C-n and C-p do as well.  And when
> > scroll-conservatively is in effect, almost every command that moves
> > point does.
> 
> Would that be... almost all of them?

I don't think I understand the question.  Please elaborate.

> (Indeed, the fact that (setq scroll-conservatively 1) doesn't work as 
> well as one would expect is a real bug IMO).

I'm not aware of any bugs in that, so please describe what you think
is a bug in more detail.

> In any case, if there is a way to dynamically detect these cases and 
> disable redisplay skipping for them, I'd like to try that out.

Yes, jit-lock-defer-time is that way.  But you already know that.

If that's not what you meant, then what exactly would you like to be
able to detect?  The move_it_* functions are the entry point to the
display simulation code, and they are used all over the place.

> Even more if there was a way to put the results of "simulated redisplay" 
> to use in the "real" redisplay later.

That happens automatically, since the 'fontified' property they
produce is left on the buffer text.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Thu, 16 Apr 2020 16:20:56 -0400
> 
> >> - Implementation-wise, not all commands simulate redisplay during their 
> >>   operation.
> > I think you will find that many more do than you seem to assume.  Even
> > just redisplaying a window does this in many cases (to find the proper
> > place for window-start position).  C-n and C-p do as well.
> 
> char-based, word-based, and sexp-based motion doesn't.  But line and
> motion that depends on "visual size" indeed typically does.

Also some functions that have nothing to do with movement, like
posn-at-point.  In a nutshell, any function that needs to consider
layout, and any command that uses such functions.

> > And when scroll-conservatively is in effect, almost every command that
> > moves point does.
> 
> Really?  I thought `scroll-conservatively` only affects the redisplay
> code and not the execution of "normal" commands (i.e. commands which
> don't themselves rely on simulating redisplay).

Like I said: redisplay itself calls those functions internally in many
situations, and scroll-conservatively > 100 makes it use that much
more.  When I said above "almost every command", I meant redisplay
triggered after commands that move point.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> Really?  I thought `scroll-conservatively` only affects the redisplay
>> code and not the execution of "normal" commands (i.e. commands which
>> don't themselves rely on simulating redisplay).
>
> Like I said: redisplay itself calls those functions internally in many
> situations, and scroll-conservatively > 100 makes it use that much
> more.  When I said above "almost every command", I meant redisplay
> triggered after commands that move point.

But my understanding was that this part of the discussion was focused on
the distinction between "may run redisplay/jit-lock during the execution
of the command itself" vs "only runs jit-lock/redisplay once the command
itself is over".

AFAIK `scroll-conservatively` only affects the redisplay so it can't
affect the above distinction.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Fri, 17 Apr 2020 11:13:27 -0400
> 
> > Like I said: redisplay itself calls those functions internally in many
> > situations, and scroll-conservatively > 100 makes it use that much
> > more.  When I said above "almost every command", I meant redisplay
> > triggered after commands that move point.
> 
> But my understanding was that this part of the discussion was focused on
> the distinction between "may run redisplay/jit-lock during the execution
> of the command itself" vs "only runs jit-lock/redisplay once the command
> itself is over".

That was not my understanding, but if I misunderstood, apologies.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 17.04.2020 18:25, Eli Zaretskii wrote:
>> From: Stefan Monnier<monnier@iro.umontreal.ca>
>> Cc:dgutov@yandex.ru,acm@muc.de,rrandresf@gmail.com,
>>    emacs-devel@gnu.org,rms@gnu.org,rudalics@gmx.at
>> Date: Fri, 17 Apr 2020 11:13:27 -0400
>>
>>> Like I said: redisplay itself calls those functions internally in many
>>> situations, and scroll-conservatively > 100 makes it use that much
>>> more.  When I said above "almost every command", I meant redisplay
>>> triggered after commands that move point.
>> But my understanding was that this part of the discussion was focused on
>> the distinction between "may run redisplay/jit-lock during the execution
>> of the command itself" vs "only runs jit-lock/redisplay once the command
>> itself is over".
> That was not my understanding, but if I misunderstood, apologies.

If the command does not simulate redisplay, and only redisplay itself 
does something (extra? special? I'm not sure), then I don't think I 
would apply.

Not according to the reasoning I outlined, at least.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 17.04.2020 09:11, Eli Zaretskii wrote:

>>> I think you will find that many more do than you seem to assume.  Even
>>> just redisplaying a window does this in many cases (to find the proper
>>> place for window-start position).  C-n and C-p do as well.  And when
>>> scroll-conservatively is in effect, almost every command that moves
>>> point does.
>>
>> Would that be... almost all of them?
> 
> I don't think I understand the question.  Please elaborate.

I think the majority of commands either move point, or have the 
potential to do so. Even those that do completing-read, for example.

>> (Indeed, the fact that (setq scroll-conservatively 1) doesn't work as
>> well as one would expect is a real bug IMO).
> 
> I'm not aware of any bugs in that, so please describe what you think
> is a bug in more detail.

1. (setq scroll-conservatively 1)

2. Visit xdisp.c.

3. Lean on C-n.

4. From time to time, you will see point jump into the middle of the 
screen, when font-lock takes too long, apparently (or maybe it 
corresponds to GC pauses).

This shouldn't happen because next-line only moves by one line.

>> In any case, if there is a way to dynamically detect these cases and
>> disable redisplay skipping for them, I'd like to try that out.
> 
> Yes, jit-lock-defer-time is that way.  But you already know that.

No, it speeds up redisplay. It doesn't make *sure* that redisplay is not 
skipped.

To do that, need to modify the condition of skipping redisplay.

> If that's not what you meant, then what exactly would you like to be
> able to detect?  The move_it_* functions are the entry point to the
> display simulation code, and they are used all over the place.

Perhaps detect the cases when move_it_* was used inside the command's 
implementation?

>> Even more if there was a way to put the results of "simulated redisplay"
>> to use in the "real" redisplay later.
> 
> That happens automatically, since the 'fontified' property they
> produce is left on the buffer text.

That might be sufficient if we assume that font-lock is always the 
slowest part of redisplay (for some commands it might not always be 
true, but I'm guessing here).

But with that assumption, the extra condition of not skipping redisplay 
could be simplified (?) to only checking whether the current screen has 
been fontified already (i.e. 'fontified' is non-nil). If it's fontified, 
we won't skip redisplay.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 17 Apr 2020 20:04:04 +0300
> 
> 1. (setq scroll-conservatively 1)
> 
> 2. Visit xdisp.c.
> 
> 3. Lean on C-n.
> 
> 4. From time to time, you will see point jump into the middle of the 
> screen, when font-lock takes too long, apparently (or maybe it 
> corresponds to GC pauses).
> 
> This shouldn't happen because next-line only moves by one line.

And it doesn't.  What happens is that Emacs is sometimes unable to
keep up with input, so when it comes to displaying the next screenful,
point is already more than 1 line below the window's end.  So Emacs
recenters.

That's why the special value of 101 was introduced: so that this
recentering never happens.

> >> In any case, if there is a way to dynamically detect these cases and
> >> disable redisplay skipping for them, I'd like to try that out.
> > 
> > Yes, jit-lock-defer-time is that way.  But you already know that.
> 
> No, it speeds up redisplay. It doesn't make *sure* that redisplay is not 
> skipped.
> 
> To do that, need to modify the condition of skipping redisplay.

Adding such a condition would be against Emacs design, which enters
redisplay only when Emacs is idle.  As long as Emacs is not idle,
redisplay will be skipped, that's how Emacs is designed.

> > If that's not what you meant, then what exactly would you like to be
> > able to detect?  The move_it_* functions are the entry point to the
> > display simulation code, and they are used all over the place.
> 
> Perhaps detect the cases when move_it_* was used inside the command's 
> implementation?
> 
> >> Even more if there was a way to put the results of "simulated redisplay"
> >> to use in the "real" redisplay later.
> > 
> > That happens automatically, since the 'fontified' property they
> > produce is left on the buffer text.
> 
> That might be sufficient if we assume that font-lock is always the 
> slowest part of redisplay (for some commands it might not always be 
> true, but I'm guessing here).

Nothing else can be kept from the move_it_* phase, because the command
that is running might not yet have finished, and so not all of the
changes that affect display are known.

> But with that assumption, the extra condition of not skipping redisplay 
> could be simplified (?) to only checking whether the current screen has 
> been fontified already (i.e. 'fontified' is non-nil). If it's fontified, 
> we won't skip redisplay.

I don't think forcing redisplay in the way that you suggest is a good
idea, see above.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 17.04.2020 20:35, Eli Zaretskii wrote:
>> Cc:acm@muc.de,rudalics@gmx.at,rrandresf@gmail.com,rms@gnu.org,
>>   emacs-devel@gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>> Date: Fri, 17 Apr 2020 20:04:04 +0300
>>
>> 1. (setq scroll-conservatively 1)
>>
>> 2. Visit xdisp.c.
>>
>> 3. Lean on C-n.
>>
>> 4. From time to time, you will see point jump into the middle of the
>> screen, when font-lock takes too long, apparently (or maybe it
>> corresponds to GC pauses).
>>
>> This shouldn't happen because next-line only moves by one line.
> And it doesn't.

What doesn't? It clearly does happen. And 'next-line' only moves by one 
line (the docstring says so, at least).

> What happens is that Emacs is sometimes unable to
> keep up with input, so when it comes to displaying the next screenful,
> point is already more than 1 line below the window's end.  So Emacs
> recenters.

I know. But these are internal details, which shouldn't affect the 
observable behavior this much.

And I'm not convinced that, if redisplay didn't happen, Emacs can't 
decide that scrolling happened anyway. After all, if that was the case, 
then in the original problem scenario (floor C-v, see Emacs freeze after 
a while), after Emacs unfreezes, the window start would have been the 
same (or almost the same) as when Emacs stopped updating the screen.

> That's why the special value of 101 was introduced: so that this
> recentering never happens.

This bug basically makes lower values (which I personally would 
otherwise choose to use, probably even the value of 1) frustratingly 
unreliable. That's both bad and non-obvious. Emacs clearly doesn't honor 
scroll-conservatively's description.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 17 Apr 2020 20:52:32 +0300
> 
> >> 4. From time to time, you will see point jump into the middle of the
> >> screen, when font-lock takes too long, apparently (or maybe it
> >> corresponds to GC pauses).
> >>
> >> This shouldn't happen because next-line only moves by one line.
> > And it doesn't.
> 
> What doesn't?

It doesn't happen when next-line only moves by one line, then stops
(as opposed to continue moving by one more line).

> > What happens is that Emacs is sometimes unable to
> > keep up with input, so when it comes to displaying the next screenful,
> > point is already more than 1 line below the window's end.  So Emacs
> > recenters.
> 
> I know. But these are internal details, which shouldn't affect the 
> observable behavior this much.

The display engine has no idea what happened or why.  All it knows is
that point is more than one line below the window-bottom.  That's the
basics of the MVC design of Emacs: redisplay is uncoupled from the
changes made by the commands.

That said, if you can find a way of avoiding that, by all means
propose a patch, I'm sure many users will be happier.

> And I'm not convinced that, if redisplay didn't happen, Emacs can't 
> decide that scrolling happened anyway.

What do you mean by "scrolling happened"?  Scrolling happens because
the display engine decides to move the window-start point.  But you
seem to use this in some other meaning.

> After all, if that was the case, then in the original problem
> scenario (floor C-v, see Emacs freeze after a while), after Emacs
> unfreezes, the window start would have been the same (or almost the
> same) as when Emacs stopped updating the screen.

What happens when Emacs "freezes" is that point moves, but the display
is not updated to reflect that.  When redisplay does have an
opportunity to kick in, it looks at the position of point, and decides
where to put the window-start given that (by default, the window-start
will be half-window above the line containing point).

> > That's why the special value of 101 was introduced: so that this
> > recentering never happens.
> 
> This bug basically makes lower values (which I personally would 
> otherwise choose to use, probably even the value of 1) frustratingly 
> unreliable.

Larger values become more and more reliable.

> Emacs clearly doesn't honor scroll-conservatively's description.

The description says:

  Scroll up to this many lines, to bring point back on screen.
  If point moves off-screen, redisplay will scroll by up to
  ‘scroll-conservatively’ lines in order to bring point just barely
  onto the screen again.  If that cannot be done, then redisplay
  recenters point as usual.^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ^^^^^^^^^^^^^^^^^^^^^^^^

So I think we are okay wrt the description.  But once again, patches
are welcome to improve this behavior.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 17.04.2020 21:48, Eli Zaretskii wrote:

>>>> 4. From time to time, you will see point jump into the middle of the
>>>> screen, when font-lock takes too long, apparently (or maybe it
>>>> corresponds to GC pauses).
>>>>
>>>> This shouldn't happen because next-line only moves by one line.
>>> And it doesn't.
>>
>> What doesn't?
> 
> It doesn't happen when next-line only moves by one line, then stops
> (as opposed to continue moving by one more line).

But next-line only moves by one line. It's the other invocations of it 
that perform subsequent movements. The fact that Emacs sometimes chooses 
to "merge" several invocations into one is a speed optimization. One 
that's not obvious to the casual user. And even many advanced ones, I'm 
fairly sure.

>>> What happens is that Emacs is sometimes unable to
>>> keep up with input, so when it comes to displaying the next screenful,
>>> point is already more than 1 line below the window's end.  So Emacs
>>> recenters.
>>
>> I know. But these are internal details, which shouldn't affect the
>> observable behavior this much.
> 
> The display engine has no idea what happened or why.  All it knows is
> that point is more than one line below the window-bottom.  That's the
> basics of the MVC design of Emacs: redisplay is uncoupled from the
> changes made by the commands.

It seems that scrolling is rather coupled to redisplay, however. Which 
is something I've never been aware of in the many years of using Emacs.

There are different directions in which that could be changed, but maybe 
this one could be improved without changing the architecture.

> That said, if you can find a way of avoiding that, by all means
> propose a patch, I'm sure many users will be happier.

The current implementation is too rich for my blood (starting with no 
documentation for try_scrolling's arguments). But here's the idea:

At the beginning of each command, we can save the value of point. If, at 
the end, we're still in the same buffer, and the new value of point 
fails the scroll-conservatively check, count the lines between the 
previous value of point and the new one. If that number is <= 
scroll-conservatively, then scroll "conservatively" anyway.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Thu, Apr 16, 2020 at 04:43:23 +0300, Dmitry Gutov wrote:
> Hi Alan,

> On 12.04.2020 18:34, Alan Mackenzie wrote:
> >>> But this would merely transfer the start up time to the
> >>> time taken in early scrolls forward.

> >> Not really. The start up scans the whole buffer, doesn't it? The early
> >> scrolls forward would still scan only a fraction of it.

> > I'm thinking more about "scrolling" to the function in the file that one
> > wants to work on or look at.  On average, this will be a little more
> > than half way through the file (there is often a large comment block at
> > BOB).  So you'd only be saving about half of CC Mode's start-up scan.

> Yes, on average it's only 2x benefit, but then again, if the buffer 
> opens at its beginning first, the extra initialization will be spread 
> across user commands. Which should have some additional positive effect 
> on apparent responsiveness.

Will it?  A slightly faster appearance of the buffer at line 1 will be
offset by a slightly more sluggish response when doing M-x imenu foo.  And
"slightly" here means a fraction of a second on a modern machine,
presumably somewhat longer on an older one.

> >> Inserting characters can alter the syntax state of the whole buffer. At
> >> least that's true for some of them. Full buffer scan sounds inevitable
> >> in those cases.

> > Full buffer scans are very unusual.  Inserting a " where every
> > subsequent line ended with a backslash might do that.  Inserting a C++ <
> > wouldn't - its effect is limited to up to the next brace or semicolon.

> > Inserting a C++ raw string opener does typically necessitate a full
> > scan (a search for a matching closer), but that would also be the case
> > using syntax-propertize.

> Not really. It would just mark the opener as a string opener (maybe with 
> some extra text property), and that's that.

You don't know whether it's an unterminated raw string (the usual case)
until you've scanned for a potential closing delimiter.  This affects the
font locking.  (An unterminated opening delimiter gets
font-lock-warning-face, a terminated one doesn't.)

This is the sort of feature which I'm not willing to sacrifice.

> Then font-lock would fontify the following text as string contents
> (until the end of the window or a little bit farther). Then you type
> the closer, it only has to scan a little far back (it'll call
> syntax-ppss to find the string opener), the closer is propertized as
> appropriate, and that's that. No full buffer scans at any step.

> I recall that fontifying the rest of the buffer as text after a simple 
> string opener could be a sore topic for you, but raw strings should be 
> rare enough (aren't they?), or if they are not, fontification logic 
> could opt to do something different, while syntax-table properties will 
> be applied the "correct" way.

I'm not sure what you mean by "as text".  I've no reason to think raw
strings are at all rare.  I've had several bug reports for them.  I'm not
sure what you mean by "fontification logic ... something different" - do
you mean in the raw string case?  Or by "while s-t properties will be
applied in the "correct" way" - these properties are currently correctly
applied (modulo any remaining bugs).

> > So, we would merely be moving functions from
> > c-get-state-before-change-functions and c-before-font-lock-functions
> > (effectively lists of before-/after-change functions) to
> > s-p-extend-region-f, together with adaptation. Would you agree that
> > such a change to CC Mode would be largely pointless if some of these
> > functions had to remain on c-get-state-b-c-f and c-before-f-l-f?

> Yes, I think before-change-functions should become empty. Or much emptier.

It can't become empty.  after-change-functions is fine for dealing with
insertions, but can't do much after a deletion.  Consider the case where
you're in a string and all you know is that 5 characters have been
deleted.  Those characters might have been )foo", so after checking the
beginning of the string starts off with R"bar(, you've then got to scan a
long way forward looking for )bar".  Effectively every deletion within a
string would involve scanning to the end of that string.

By contrast, in a before-change-function, a quick check will show that
)foo", which is about to be deleted is here not a raw string delimiter,
so no special action is needed.

> > But the way s-p-extend-region-f functions are called is to keep calling
> > them repeatedly until they've all said "no change" together.  This would
> > dramatically slow down CC Mode, where currently these functions are each
> > called exactly once.

> Here's a simple solution: create one function special for CC Mode that 
> would do that, and add it to s-p-extend-region-f.

> But there might be even better way to do that. I'm not the best person 
> to discuss that with.

Sorry for not thinking it through, that's not a problem at all, really.

> > Also, the syntax-propertize mechanism is weaker than CC Mode's: When it
> > is run, there is no way of knowing whether it's being called as a change
> > function, and if it is, OLD-LEN is discarded.  How can it have access to
> > variables set in before-change-functions?  (An example of such is
> > c-raw-string-end-delim-disrupted.  In before change, it is set when the
> > existing raw string end delimiter is about to cease to be.  In after
> > change, the fact of this flag being nil means we don't need to search
> > for an alternative closing delimiter, etc.  This change can obviously
> > not be detected in an after-change function.)

> As we seem to agree, before-change-functions should not be needed. 

I don't agree at all.  before-change-functions is needed (see above).

> Neither should be the code that tracks the details of the edits that the 
> user makes. That alone can simplify some things.

At the cost of complicating other things, and slowing things down.  If
you have no b-c-f, you have no information about a deletion beyond its
OLD-LEN.  So you have to do a lot more checking and scanning in an a-c-f
to recover that lost information.

> >> I'm pretty sure I have thought of that example because it's an instance
> >> of a syntax problem that's easy enough to solve within
> >> syntax-propertize-function framework.

> > Having actually gone through all the issues and implemented raw strings,
> > I can't agree with you there.  There are all sorts of subtleties which
> > necessitate intimate cooperation between the before-change-functions and
> > after-change-functions.  Such cooperation seems to be excluded by the
> > syntax-propertize mechanism.

> It encourages a different approach. Again: there are examples of raw 
> strings support in other major modes.

Do any of them have the syntactical complexities of C++'s raw strings?

> >>> Consider(2) a C++ template: excusing my C++ syntax knowledge, type in

> >>>      template class foo < bar, baz >= bar>

> >>> , perhaps typing in the odd newline inside the template (a common
> >>> occurrence), or nesting further templates inside it (also a common
> >>> occurrence).  Note how the parenthesis text properties are added and
> >>> removed as you type.  All these modification are necessary, and they are
> >>> largely _before_ the point of insertion, not after it.

> >> The current implementation of applying these properties can probably be
> >> transferred into a syntax-propertize-function with only modest changes.

> > Maybe, but with a slowdown.  More of these properties will get erased
> > than needed (with nested template forms), and they will all need to get
> > put back again.

> We won't really know until we can measure the result.

What's the point in investing all the effort to make the change, when
there's not even a prediction of a speed up?

> >>>> Some scenarios can become slower, that's for sure. But the more common
> >>>> ones can get faster. We won't know until we try.

> > Other than starting up a buffer, we still haven't identified any
> > specific scenarios where speed up might happen.

> When before-changes-functions only contains (t syntax-ppss-flush-cache), 
> that can visibly change the performance tradeoffs.

It can indeed, for the worse.  See above.

> > I don't think the syntax-propertize mechanism is all that brilliant.
> > It's too constrained, and places too many restrictions on what can be
> > done with the syntax-table text property.  For example, (from
> > syntax.el):

> I wouldn't say it's perfect either. But it's proven helpful over the 
> years, and provided a base design for a lot of major mode implementations.

Your first sentence is the pertinent one, here.

And I'm not sure where the proof of the syntax-propertize mechanism being
helpful is.  Has anybody but its originator positively chosen to use it,
whilst being aware of the alternatives?

And the mechanism has caused difficult to debug bugs, such as bug #40661.

To become usable for CC Mode, it would need to provide something on
before-change-functions to complement what's on a-c-f, and it would need
to provide some control to the major mode over which syntax-table
properties get erased.

> > (defvar syntax-propertize-function nil
> >    ;; Rather than a -functions hook, this is a -function because it's easier
> >    ;; to do a single scan than several scans: with multiple scans, one cannot
> >    ;; assume that the text before point has been propertized, so syntax-ppss
> >    ;; gives unreliable results (and stores them in its cache to boot, so we'd
> >    ;; have to flush that cache between each function, and we couldn't use
> >    ;; syntax-ppss-flush-cache since that would not only flush the cache but also
> >    ;; reset syntax-propertize--done which should not be done in this case).

> >  From my point of view, "multiple scans" are _much_ easier.  They are
> > prohibited here only because syntax-ppss and syntax-propertize-function
> > have got themselves tied up in a tight knot.  One answer would be not to
> > use syntax-ppss inside a s-p-function.  (CC Mode doesn't use syntax-ppss
> > at all).  Another answer would be to give the responsibility of removing
> > the s-t text properties to the s-p-function.

> I think we could extend the customizability in that direction. But first 
> we'd have to see clear evidence that the current design is not good 
> enough (e.g. CC Mode has been reimplemented on top of it, and the result 
> is decidedly not fast enough).

Before we even start reimplementing CC Mode, we'd want some evidence that
the current design of syntax-propertize is adequate.  I see quite clearly
that it is not.

> >> Simply collaborating with one other developer on an overhaul project
> >> (whether it succeeds or not; perhaps partially) can improve on that.

> > But take a massive amount of time.

> You could consider it an investment.

As yet, I don't honestly see how.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> The current implementation is too rich for my blood (starting with no
> documentation for try_scrolling's arguments). But here's the idea:
>
> At the beginning of each command, we can save the value of point. If, at the
> end, we're still in the same buffer, and the new value of point fails the
> scroll-conservatively check, count the lines between the previous value of
> point and the new one. If that number is <= scroll-conservatively, then
> scroll "conservatively" anyway.

Another approach could be:

- distinguish the user's setting of scroll-conservatively from the one
  that applies to the current redisplay.
- increase the redisplay's current value of scroll-conservatively each
  time redisplay is skipped.
- reset the redisplay's current value of scroll-conservatively to the
  variable's value every time redisplay took place.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 19.04.2020 20:12, Alan Mackenzie wrote:

>> Yes, on average it's only 2x benefit, but then again, if the buffer
>> opens at its beginning first, the extra initialization will be spread
>> across user commands. Which should have some additional positive effect
>> on apparent responsiveness.
> 
> Will it?  A slightly faster appearance of the buffer at line 1 will be
> offset by a slightly more sluggish response when doing M-x imenu foo.  And
> "slightly" here means a fraction of a second on a modern machine,
> presumably somewhat longer on an older one.

M-x file-file followed by M-x imenu should take ~as much as before. And 
if the user doesn't go for Imenu or scrolling to EOB right away, they 
get extra performance.

I don't know how slow it's for the average user, but there certainly are 
some with slow machines out there. Or people compiling without 
optimizations.

>>> Inserting a C++ raw string opener does typically necessitate a full
>>> scan (a search for a matching closer), but that would also be the case
>>> using syntax-propertize.
> 
>> Not really. It would just mark the opener as a string opener (maybe with
>> some extra text property), and that's that.
> 
> You don't know whether it's an unterminated raw string (the usual case)
> until you've scanned for a potential closing delimiter.

Is C++ syntax so ambiguous? Can R"( mean something else?

> This affects the
> font locking.  (An unterminated opening delimiter gets
> font-lock-warning-face, a terminated one doesn't.)

If everything after R"( is fontified as a string, it serves as a 
"warning" of sort as well.

> This is the sort of feature which I'm not willing to sacrifice.

Is it worth a full buffer scan every time you write a new raw string 
literal?

>> Then font-lock would fontify the following text as string contents
>> (until the end of the window or a little bit farther). Then you type
>> the closer, it only has to scan a little far back (it'll call
>> syntax-ppss to find the string opener), the closer is propertized as
>> appropriate, and that's that. No full buffer scans at any step.
> 
>> I recall that fontifying the rest of the buffer as text after a simple
>> string opener could be a sore topic for you, but raw strings should be
>> rare enough (aren't they?), or if they are not, fontification logic
>> could opt to do something different, while syntax-table properties will
>> be applied the "correct" way.
> 
> I'm not sure what you mean by "as text".

Sorry, I meant "as string".

> I've no reason to think raw
> strings are at all rare.  I've had several bug reports for them.  I'm not
> sure what you mean by "fontification logic ... something different" - do
> you mean in the raw string case?

I mean that if a raw string is unterminated, the default behavior should 
be to fontify the rest of the buffer as string. But then again, you 
could choose some different highliting in font-lock rules.

> Or by "while s-t properties will be
> applied in the "correct" way" - these properties are currently correctly
> applied (modulo any remaining bugs).

In that case, you should always be able to find the beginning of the 
current string literal (raw or plain) using syntax-ppss.

>> Yes, I think before-change-functions should become empty. Or much emptier.
> 
> It can't become empty.  after-change-functions is fine for dealing with
> insertions, but can't do much after a deletion.  Consider the case where
> you're in a string and all you know is that 5 characters have been
> deleted.  Those characters might have been )foo", so after checking the
> beginning of the string starts off with R"bar(, you've then got to scan a
> long way forward looking for )bar".  Effectively every deletion within a
> string would involve scanning to the end of that string.

This is an example of extra complexity you have to retain to implement 
the above feature the way you do.

It's probably also an example of how before/after-change-functions 
essentially duplicate the knowledge of language syntax. I'm guessing 
here, but to make it work like that, you need to have multiple functions 
"understand" the raw string syntax.

Whereas with syntax-propertize-function, that knowledge is concentrated 
in one place (maybe two, if font-lock rules do something unusual). This 
way, the code is simplified.

>>> Having actually gone through all the issues and implemented raw strings,
>>> I can't agree with you there.  There are all sorts of subtleties which
>>> necessitate intimate cooperation between the before-change-functions and
>>> after-change-functions.  Such cooperation seems to be excluded by the
>>> syntax-propertize mechanism.
> 
>> It encourages a different approach. Again: there are examples of raw
>> strings support in other major modes.
> 
> Do any of them have the syntactical complexities of C++'s raw strings?

You tell me:

https://docs.ruby-lang.org/en/2.4.0/syntax/literals_rdoc.html#label-Percent+Strings
https://rosettacode.org/wiki/Here_document#Ruby
https://github.com/atom/language-ruby/issues/109#issue-98715165

>>> Maybe, but with a slowdown.  More of these properties will get erased
>>> than needed (with nested template forms), and they will all need to get
>>> put back again.
> 
>> We won't really know until we can measure the result.
> 
> What's the point in investing all the effort to make the change, when
> there's not even a prediction of a speed up?

In principle, the speed-up will come from:

- Deferred execution (where several buffer changes can be handled 
together and not right away),
- No parsing the buffer contents much farther than the current window, 
in most cases. Which can speed up the majority of user actions. The 
exceptions will remain slower, but that is often a good tradeoff.

> And I'm not sure where the proof of the syntax-propertize mechanism being
> helpful is.  Has anybody but its originator positively chosen to use it,
> whilst being aware of the alternatives?

The alternatives being reinventing the relevant logic from zero in each 
major mode? And writing syntax caching logic each time?

> To become usable for CC Mode, it would need to provide something on
> before-change-functions to complement what's on a-c-f, and it would need
> to provide some control to the major mode over which syntax-table
> properties get erased.

Not something I can comment on.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Sun, 19 Apr 2020 13:17:11 -0400
> 
> - distinguish the user's setting of scroll-conservatively from the one
>   that applies to the current redisplay.
> - increase the redisplay's current value of scroll-conservatively each
>   time redisplay is skipped.
> - reset the redisplay's current value of scroll-conservatively to the
>   variable's value every time redisplay took place.

How is this different from when scroll-conservatively is 101?

IOW, you need to answer the following question: if moving point into
view means scrolling the display by more than the value of
scroll-conservatively, then what do you want Emacs to do instead of
putting point at the last screen line of the window?  If your answer
is "put it on the last line anyway", then you effectively asked to
treat any value of scroll-conservatively as if it were > 100, and we
already provide that.  You should ask yourself what do users expect to
see when they set scroll-conservatively to a number smaller than 101.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 19 Apr 2020 20:09:04 +0300
> 
> > It doesn't happen when next-line only moves by one line, then stops
> > (as opposed to continue moving by one more line).
> 
> But next-line only moves by one line. It's the other invocations of it 
> that perform subsequent movements. The fact that Emacs sometimes chooses 
> to "merge" several invocations into one is a speed optimization.

It's not a "merge" and not an optimization, it's how Emacs was
designed.  It only enters redisplay when it's idle.  If you invoke
commands one after another quickly enough, more than one command will
be executed before the next redisplay cycle.

> > The display engine has no idea what happened or why.  All it knows is
> > that point is more than one line below the window-bottom.  That's the
> > basics of the MVC design of Emacs: redisplay is uncoupled from the
> > changes made by the commands.
> 
> It seems that scrolling is rather coupled to redisplay, however.

Not sure what you mean by "coupled" here.  As I explained, scrolling
is _implemented_ in redisplay.  The scrolling commands just calculate
where the window-start should be, they don't scroll anything.

> Which is something I've never been aware of in the many years of
> using Emacs.

With Emacs, one learns something new every day.  That includes
myself, of course.

> At the beginning of each command, we can save the value of point. If, at 
> the end, we're still in the same buffer, and the new value of point 
> fails the scroll-conservatively check, count the lines between the 
> previous value of point and the new one. If that number is <= 
> scroll-conservatively, then scroll "conservatively" anyway.

And if not, then what?

Anyway, scrolling commands don't normally move point, they move the
window-start.  Point then moves because redisplay brings it into
view.  So I don't think your algorithm will work, at least not in the
usual case.

What you describe might be relevant to "scrolling" that happens if you
lean on C-n, not on C-v.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> You don't know whether it's an unterminated raw string (the usual case)
>> until you've scanned for a potential closing delimiter.
> Is C++ syntax so ambiguous? Can R"( mean something else?

No it has nothing to do with the language's syntax.

It's just a UI choice: Alan likes to highlight the opening string
delimiter of unterminated strings [ Not sure why CC-mode doesn't do the
same for unterminated comments, tho maybe it's just an oversight.  ]

I think this should not vary between major modes, IOW it should be
a separate feature that works on all major modes.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> - distinguish the user's setting of scroll-conservatively from the one
>>   that applies to the current redisplay.
>> - increase the redisplay's current value of scroll-conservatively each
>>   time redisplay is skipped.
>> - reset the redisplay's current value of scroll-conservatively to the
>>   variable's value every time redisplay took place.
>
> How is this different from when scroll-conservatively is 101?

It *will* recenter when you do C-2 C-n.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 19.04.2020 22:06, Eli Zaretskii wrote:

>>> It doesn't happen when next-line only moves by one line, then stops
>>> (as opposed to continue moving by one more line).
>>
>> But next-line only moves by one line. It's the other invocations of it
>> that perform subsequent movements. The fact that Emacs sometimes chooses
>> to "merge" several invocations into one is a speed optimization.
> 
> It's not a "merge" and not an optimization, it's how Emacs was
> designed.  It only enters redisplay when it's idle.  If you invoke
> commands one after another quickly enough, more than one command will
> be executed before the next redisplay cycle.

The fact that Emacs's behavior can depend on when redisplay happens, and 
the user cannot reliably control it, is problematic IMHO.

>> At the beginning of each command, we can save the value of point. If, at
>> the end, we're still in the same buffer, and the new value of point
>> fails the scroll-conservatively check, count the lines between the
>> previous value of point and the new one. If that number is <=
>> scroll-conservatively, then scroll "conservatively" anyway.
> 
> And if not, then what?

Then give up and move point to the middle of the window.

> Anyway, scrolling commands don't normally move point, they move the
> window-start.  Point then moves because redisplay brings it into
> view.  So I don't think your algorithm will work, at least not in the
> usual case.

I'm a lot less worried about scrolling "conservatively" when the user 
actually invokes the scrolling commands. In those scenarios, it's fairly 
natural to expect that point moves however Emacs prefers (but we also 
have scroll-preserve-screen-position).

Stefan's idea might improve on scrolling commands' behavior as well in 
this regard, but it could be slightly harder to implement.

> What you describe might be relevant to "scrolling" that happens if you
> lean on C-n, not on C-v.

C-n, or C-M-e, or C-s. These are indeed scenarios when the current 
behavior can be most surprising. The new users tend to lean on C-n and 
C-p a lot at first, I think, and that's when it bites the most. But even 
now I lean on C-n from time to time when speed-reading code.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 17.04.2020 20:35, Eli Zaretskii wrote:

>>>> In any case, if there is a way to dynamically detect these cases and
>>>> disable redisplay skipping for them, I'd like to try that out.
>>>
>>> Yes, jit-lock-defer-time is that way.  But you already know that.
>>
>> No, it speeds up redisplay. It doesn't make *sure* that redisplay is not
>> skipped.
>>
>> To do that, need to modify the condition of skipping redisplay.
> 
> Adding such a condition would be against Emacs design, which enters
> redisplay only when Emacs is idle.  As long as Emacs is not idle,
> redisplay will be skipped, that's how Emacs is designed.

As we've determined, certain commands force redisplay to be performed 
anyway (though partially).

If performing it in full in those cases will take just a little more CPU 
time, the result might be Emacs that looks more responsive under load, 
and just as fast.

>>>> Even more if there was a way to put the results of "simulated redisplay"
>>>> to use in the "real" redisplay later.
>>>
>>> That happens automatically, since the 'fontified' property they
>>> produce is left on the buffer text.
>>
>> That might be sufficient if we assume that font-lock is always the
>> slowest part of redisplay (for some commands it might not always be
>> true, but I'm guessing here).
> 
> Nothing else can be kept from the move_it_* phase, because the command
> that is running might not yet have finished, and so not all of the
> changes that affect display are known.

It might be possible to construct some "cache key" that would allow us 
to check whether the remainder of the command changed something that 
would affect display. And if it didn't, use the last saved result of 
move_it_*.

>> But with that assumption, the extra condition of not skipping redisplay
>> could be simplified (?) to only checking whether the current screen has
>> been fontified already (i.e. 'fontified' is non-nil). If it's fontified,
>> we won't skip redisplay.
> 
> I don't think forcing redisplay in the way that you suggest is a good
> idea, see above.

Because... it will result in worse performance?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 02:21, Stefan Monnier wrote:
>>> You don't know whether it's an unterminated raw string (the usual case)
>>> until you've scanned for a potential closing delimiter.
>> Is C++ syntax so ambiguous? Can R"( mean something else?
> No it has nothing to do with the language's syntax.
> 
> It's just a UI choice: Alan likes to highlight the opening string
> delimiter of unterminated strings [ Not sure why CC-mode doesn't do the
> same for unterminated comments, tho maybe it's just an oversight.  ]
> 
> I think this should not vary between major modes, IOW it should be
> a separate feature that works on all major modes.

If it can be done reliably outside of CC Mode (i.e. without relying on 
knowing the particulars of the buffer's programming language), then it's 
certainly another argument for using 
syntax-propertize-function/syntax-ppss framework.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Sun, 19 Apr 2020 19:22:41 -0400
> 
> >> - distinguish the user's setting of scroll-conservatively from the one
> >>   that applies to the current redisplay.
> >> - increase the redisplay's current value of scroll-conservatively each
> >>   time redisplay is skipped.
> >> - reset the redisplay's current value of scroll-conservatively to the
> >>   variable's value every time redisplay took place.
> >
> > How is this different from when scroll-conservatively is 101?
> 
> It *will* recenter when you do C-2 C-n.

So "C-2 C-n" would behave differently from "C-n C-n", or from running
a macro that records 2 C-n in a row?  Is that a good thing?

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 20 Apr 2020 06:17:55 +0300
> 
> > It's not a "merge" and not an optimization, it's how Emacs was
> > designed.  It only enters redisplay when it's idle.  If you invoke
> > commands one after another quickly enough, more than one command will
> > be executed before the next redisplay cycle.
> 
> The fact that Emacs's behavior can depend on when redisplay happens, and 
> the user cannot reliably control it, is problematic IMHO.

Please be specific: what Emacs's behavior can depend on when redisplay
happens, apart from whether something was or wasn't updated on
display?

> >> At the beginning of each command, we can save the value of point. If, at
> >> the end, we're still in the same buffer, and the new value of point
> >> fails the scroll-conservatively check, count the lines between the
> >> previous value of point and the new one. If that number is <=
> >> scroll-conservatively, then scroll "conservatively" anyway.
> > 
> > And if not, then what?
> 
> Then give up and move point to the middle of the window.

But that's again "not good" according to your opinion, isn't it?  Or
maybe you assume this will solve most of the cases when we currently
recenter under scroll-conservatively < 101?

> > Anyway, scrolling commands don't normally move point, they move the
> > window-start.  Point then moves because redisplay brings it into
> > view.  So I don't think your algorithm will work, at least not in the
> > usual case.
> 
> I'm a lot less worried about scrolling "conservatively" when the user 
> actually invokes the scrolling commands.

Ahem... the thread's title begins with "scrolling commands",so I
thought we were still discussing that?

> > What you describe might be relevant to "scrolling" that happens if you
> > lean on C-n, not on C-v.
> 
> C-n, or C-M-e, or C-s. These are indeed scenarios when the current 
> behavior can be most surprising.

Is it?  Just out of habit, I suppose.  What Emacs does make a lot of
sense: it shows you the optimal amount of context.  By contrast,
scroll-conservatively shows you only one side of that context, and
usually the wrong side.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 20 Apr 2020 06:27:17 +0300
> 
> >> To do that, need to modify the condition of skipping redisplay.
> > 
> > Adding such a condition would be against Emacs design, which enters
> > redisplay only when Emacs is idle.  As long as Emacs is not idle,
> > redisplay will be skipped, that's how Emacs is designed.
> 
> As we've determined, certain commands force redisplay to be performed 
> anyway (though partially).

That's not redisplay, that's JIT fontification that happens as side
effect of redisplay.  It can _also_ happen as side effect of other
commands.  But that's not redisplay.

> If performing it in full in those cases will take just a little more CPU 
> time, the result might be Emacs that looks more responsive under load, 
> and just as fast.

If you can find a way of reusing that information afterwards (more
than redisplay itself already does), yes.  But AFAIU you are talking
about a thorough redesign of the Emacs display engine.  It already
attempts to reuse every piece of display that can be kept.  For
example, if you type C-n on the bottom-most screen line, and you have
scroll-conservatively set, the display engine will redraw exactly one
screen line, and all the rest will be scrolled (via a direct Xlib
call) on the glass, without regenerating the glyphs.

Of course, if you invoke a command that moves to a place that has no
overlap with the text of the previous screenful, no such optimizations
are possible, unless the new screen has some parts of it that happen
to be identical to the old one (yes, Emacs does that optimization as
well).

> > Nothing else can be kept from the move_it_* phase, because the command
> > that is running might not yet have finished, and so not all of the
> > changes that affect display are known.
> 
> It might be possible to construct some "cache key" that would allow us 
> to check whether the remainder of the command changed something that 
> would affect display. And if it didn't, use the last saved result of 
> move_it_*.

You will see in redisplay_window and its subroutines a lot of code
that attempts to do exactly that: figure out what has changed and what
hasn't.  You are in effect saying that this must be radically
improved, in which case I'd happily agree, but please believe me that
it is not a trivial job, certainly not if you want to stay within the
current design of the display engine.  You will probably need to
design and implement various new fields of the window and buffer
objects that will cache more data about the previous state of buffer
and display than what they already do, make sure these caches are
reset when no longer valid, then build new redisplay optimizations
based on those, find the conditions where the optimizations can and
cannot be used, etc.

> >> But with that assumption, the extra condition of not skipping redisplay
> >> could be simplified (?) to only checking whether the current screen has
> >> been fontified already (i.e. 'fontified' is non-nil). If it's fontified,
> >> we won't skip redisplay.
> > 
> > I don't think forcing redisplay in the way that you suggest is a good
> > idea, see above.
> 
> Because... it will result in worse performance?

No, because you will fight the design, and the design will fight back.
That's all I can say.  IMO, if we want to make the Emacs display more
efficient, we need to redesign it.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> It *will* recenter when you do C-2 C-n.
> So "C-2 C-n" would behave differently from "C-n C-n",

Yes.

> or from running a macro that records 2 C-n in a row?
> Is that a good thing?

AFAIK that's the reason why scroll-conservatively is a number rather
than a boolean.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 17:23, Eli Zaretskii wrote:
> So "C-2 C-n" would behave differently from "C-n C-n", or from running
> a macro that records 2 C-n in a row?  Is that a good thing?

Is scroll-conservatively is 1, 'C-2 C-n' should behave differently at 
the bottom of the window from 'C-n C-n'.

If a macro recording doesn't behave the same as the original 
invocations, I'd call it a bug as well. Though probably of a lesser 
priority.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
>  rudalics@gmx.at
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 20 Apr 2020 19:42:27 +0300
> 
> On 20.04.2020 17:23, Eli Zaretskii wrote:
> > So "C-2 C-n" would behave differently from "C-n C-n", or from running
> > a macro that records 2 C-n in a row?  Is that a good thing?
> 
> Is scroll-conservatively is 1, 'C-2 C-n' should behave differently at 
> the bottom of the window from 'C-n C-n'.

Why? they both move point by 2 lines.

> If a macro recording doesn't behave the same as the original 
> invocations, I'd call it a bug as well. Though probably of a lesser 
> priority.

We'll have to disagree, I guess.  I personally don't care much about
scroll-conservatively, although I did spend some hours making it
behave like others requested.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 19:51, Eli Zaretskii wrote:
> Why? they both move point by 2 lines.

First of all, it already does. The difference comes up only when Emacs 
is too busy to re-render.

And C-u 1001 C-n followed by C-u 1000 C-p also moves point by 1 line, 
just like C-n does, yet I hope you would not expect it to behave exactly 
the same WRT scroll-conservatively.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Andreas Schwab]

On Apr 20 2020, Dmitry Gutov wrote:

> And C-u 1001 C-n followed by C-u 1000 C-p also moves point by 1 line,

No, it moves by 2001 lines.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 18:01, Eli Zaretskii wrote:

>>> It's not a "merge" and not an optimization, it's how Emacs was
>>> designed.  It only enters redisplay when it's idle.  If you invoke
>>> commands one after another quickly enough, more than one command will
>>> be executed before the next redisplay cycle.
>>
>> The fact that Emacs's behavior can depend on when redisplay happens, and
>> the user cannot reliably control it, is problematic IMHO.
> 
> Please be specific: what Emacs's behavior can depend on when redisplay
> happens, apart from whether something was or wasn't updated on
> display?

The window's scroll position.

Again: you (setq scroll-conservatively 1) and lean on C-n for a little 
while. Depending on whether Emacs manages to redisplay after every 
command invocation, the point will, or will not, scroll to the middle of 
the window. It could do that several times as well.

>>>> At the beginning of each command, we can save the value of point. If, at
>>>> the end, we're still in the same buffer, and the new value of point
>>>> fails the scroll-conservatively check, count the lines between the
>>>> previous value of point and the new one. If that number is <=
>>>> scroll-conservatively, then scroll "conservatively" anyway.
>>>
>>> And if not, then what?
>>
>> Then give up and move point to the middle of the window.
> 
> But that's again "not good" according to your opinion, isn't it?

Why wouldn't my suggestion be good enough?

> Or
> maybe you assume this will solve most of the cases when we currently
> recenter under scroll-conservatively < 101?

All important ones, yes.

>>> Anyway, scrolling commands don't normally move point, they move the
>>> window-start.  Point then moves because redisplay brings it into
>>> view.  So I don't think your algorithm will work, at least not in the
>>> usual case.
>>
>> I'm a lot less worried about scrolling "conservatively" when the user
>> actually invokes the scrolling commands.
> 
> Ahem... the thread's title begins with "scrolling commands",so I
> thought we were still discussing that?

The amount of times I have mentioned 'C-n' might have persuaded otherwise.

>>> What you describe might be relevant to "scrolling" that happens if you
>>> lean on C-n, not on C-v.
>>
>> C-n, or C-M-e, or C-s. These are indeed scenarios when the current
>> behavior can be most surprising.
> 
> Is it?  Just out of habit, I suppose.  What Emacs does make a lot of
> sense: it shows you the optimal amount of context.  By contrast,
> scroll-conservatively shows you only one side of that context, and
> usually the wrong side.

Are we going to argue whether scroll-conservatively is useful? I hope 
not. And if didn't assign value to that "optimal amount of context" 
thing you mention, I'd be arguing in favor of making 
scroll-conservatively a boolean instead.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, acm@muc.de, rrandresf@gmail.com,
>  emacs-devel@gnu.org, rms@gnu.org, rudalics@gmx.at
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 20 Apr 2020 19:56:22 +0300
> 
> On 20.04.2020 19:51, Eli Zaretskii wrote:
> > Why? they both move point by 2 lines.
> 
> First of all, it already does. The difference comes up only when Emacs 
> is too busy to re-render.

No.  If you type C-n twice fast enough, Emacs will behave the same as
if you type C-2 C-n.

> And C-u 1001 C-n followed by C-u 1000 C-p also moves point by 1 line, 
> just like C-n does, yet I hope you would not expect it to behave exactly 
> the same WRT scroll-conservatively.

If you type these two commands quickly so that there's no intervening
redisplay, Emacs _will_ behave the same, and I would expect it to.

And I don't see any point in continue arguing about this, since we
have very different expectations and different perspective.  Let's
agree to disagree.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> > So "C-2 C-n" would behave differently from "C-n C-n", or from running
>> > a macro that records 2 C-n in a row?  Is that a good thing?
>> Is scroll-conservatively is 1, 'C-2 C-n' should behave differently at 
>> the bottom of the window from 'C-n C-n'.
> Why?

Because that's how `scroll-conservatively` was defined.

> they both move point by 2 lines.

Overall, yes.  But one of the commands moves by 2 lines where the other
(identical) two only move by one line.

> We'll have to disagree, I guess.  I personally don't care much about
> scroll-conservatively, although I did spend some hours making it
> behave like others requested.

Indeed, you implemented the "boolean" interpretation of
`scroll-conservatively` which seems to be what most users wanted.
But this discussion is about the case where `scroll-conservatively` is
purposefully set to a small non-0 value meaning that the user expects
conservative scrolling only when it's just a matter of moving by
a few lines (e.g. by 1 line).

I don't care much about `scroll-conservatively` either, FWIW (I'm not
in love with the default "recenter" either by the way, but
`scroll-conservatively` isn't any better.  Maybe we can get closer to my
"ideal", but I think my real ideal requires a lot of intelligence and
predictive power).


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 20 Apr 2020 20:04:06 +0300
> 
> >> The fact that Emacs's behavior can depend on when redisplay happens, and
> >> the user cannot reliably control it, is problematic IMHO.
> > 
> > Please be specific: what Emacs's behavior can depend on when redisplay
> > happens, apart from whether something was or wasn't updated on
> > display?
> 
> The window's scroll position.
> 
> Again: you (setq scroll-conservatively 1) and lean on C-n for a little 
> while. Depending on whether Emacs manages to redisplay after every 
> command invocation, the point will, or will not, scroll to the middle of 
> the window. It could do that several times as well.

Yes.  because that's what scroll-conservatively = 1 means.

> >> Then give up and move point to the middle of the window.
> > 
> > But that's again "not good" according to your opinion, isn't it?
> 
> Why wouldn't my suggestion be good enough?

Because you don't want to see recentering, ever:

> > Or
> > maybe you assume this will solve most of the cases when we currently
> > recenter under scroll-conservatively < 101?
> 
> All important ones, yes.

See? so set scroll-conservatively to 101, and Bob's your uncle.  Yours
truly has worked hard and long to have that never recenter, including
in some exotic use cases.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Mon, 20 Apr 2020 13:39:18 -0400
> 
> >> > So "C-2 C-n" would behave differently from "C-n C-n", or from running
> >> > a macro that records 2 C-n in a row?  Is that a good thing?
> >> Is scroll-conservatively is 1, 'C-2 C-n' should behave differently at 
> >> the bottom of the window from 'C-n C-n'.
> > Why?
> 
> Because that's how `scroll-conservatively` was defined.

No.  Its definition is spelled out in the doc string.

> > they both move point by 2 lines.
> 
> Overall, yes.  But one of the commands moves by 2 lines where the other
> (identical) two only move by one line.

Which brings me back to the point where I explained this, several
messages up-thread.  So I guess I can now say I'm out of this
discussion, since I already said everything I could.  Let's agree to
disagree.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> First of all, it already does. The difference comes up only when Emacs 
>> is too busy to re-render.
> No.  If you type C-n twice fast enough, Emacs will behave the same as
> if you type C-2 C-n.


Actually, I think if Emacs is "idle" before typing those two `C-n`,
Emacs will not behave the same as `C-2 C-n` unless they really reach
Emacs at the same time (which seems almost impossible unless you're
doing it over something like an SSH connection where some delay might
end up bundling them together).  This is because the `input_was_pending`
var is set at the *beginning* of processing the first `C-n` and at that
point, the second should not have arrived yet.  So AFAICT the redisplay
between the two will not be skipped.

If you do `C-n C-n C-n` quickly, the redisplay between the second and
the third might very well be skipped, OTOH (since the second and third
`C-n` might arrive while we're processing the first, in which case
`input_was_pending` will be true when processing the second).

> And I don't see any point in continue arguing about this, since we
> have very different expectations and different perspective.  Let's
> agree to disagree.

Do you think my suggestion to "accumulate" `scroll-conservatively` when
redisplay is skipped might be acceptable (I don't actually know if it
will/can work, OTOH).


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Mon, 20 Apr 2020 13:47:10 -0400
> 
> Do you think my suggestion to "accumulate" `scroll-conservatively` when
> redisplay is skipped might be acceptable (I don't actually know if it
> will/can work, OTOH).

I very much doubt this is worth the complexity.  How many people even
set scroll-conservatively to something smaller than 101?  My guess
would be zero.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Mon, 20 Apr 2020 13:47:10 -0400
> 
> If you do `C-n C-n C-n` quickly, the redisplay between the second and
> the third might very well be skipped, OTOH (since the second and third
> `C-n` might arrive while we're processing the first, in which case
> `input_was_pending` will be true when processing the second).

Btw, "line" in this context is actually the amount of pixels equal to
the canonical line height, not a physical line.  When the display
engine counts "lines" for the purpose of comparison with the value of
scroll-conservatively, it actually counts pixels and then divides that
by the canonical line height, it doesn't count screen lines.  So if
some lines in the buffer are displayed taller than the default face,
you will see a lot of recentering regardless of what you do.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 20:39, Eli Zaretskii wrote:
> Because you don't want to see recentering, ever:

I do. When I jump far enough below the current screen.

IOW, C-u 2 C-n at the bottom of the window should recenter.

C-n, pressed twice, shouldn't.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 20:35, Eli Zaretskii wrote:
> No.  If you type C-n twice fast enough, Emacs will behave the same as
> if you type C-2 C-n.

That's the bug.

> If you type these two commands quickly so that there's no intervening
> redisplay, Emacs_will_  behave the same, and I would expect it to.

It will behave differently from the "normal" case. Which is when there 
is an intervening redisplay.

> And I don't see any point in continue arguing about this, since we
> have very different expectations and different perspective.  Let's
> agree to disagree.

It's your prerogative, but IMHO it's one of the "papercut" bugs that mar 
the initial user experience for people who come from other text editors 
(with existing expectations).

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 21:19, Eli Zaretskii wrote:
> Btw, "line" in this context is actually the amount of pixels equal to
> the canonical line height, not a physical line.  When the display
> engine counts "lines" for the purpose of comparison with the value of
> scroll-conservatively, it actually counts pixels and then divides that
> by the canonical line height, it doesn't count screen lines.  So if
> some lines in the buffer are displayed taller than the default face,
> you will see a lot of recentering regardless of what you do.

I wonder if this was a result of some feature request.

Sounds unnecessarily complex, IMHO.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> by the canonical line height, it doesn't count screen lines.  So if
> some lines in the buffer are displayed taller than the default face,
> you will see a lot of recentering regardless of what you do.

Good point.  So any "unusual" character which ends up selecting
a different font that happens to be slightly taller will disrupt the
"conservative" scroll.

Tho, I guess we should consider "1" as meaning "strictly smaller than
2.0", in which case a line would have to be twice as tall as a normal
line before we cause a recenter.

> I very much doubt this is worth the complexity.

Fair enough.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rrandresf@gmail.com, emacs-devel@gnu.org, rms@gnu.org,
>  rudalics@gmx.at
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 20 Apr 2020 21:29:07 +0300
> 
> On 20.04.2020 21:19, Eli Zaretskii wrote:
> > Btw, "line" in this context is actually the amount of pixels equal to
> > the canonical line height, not a physical line.  When the display
> > engine counts "lines" for the purpose of comparison with the value of
> > scroll-conservatively, it actually counts pixels and then divides that
> > by the canonical line height, it doesn't count screen lines.  So if
> > some lines in the buffer are displayed taller than the default face,
> > you will see a lot of recentering regardless of what you do.
> 
> I wonder if this was a result of some feature request.

No.  Scrolling-related variables always worked this way, since Emacs
21.  You will see the same with scroll-margin, for example.

> Sounds unnecessarily complex, IMHO.

If you think about this for a while, you will realize that it's the
only sane way of doing that, in an editor that is required to handle
lines which can contain anything: small and large characters, images,
you name it.  Any other way, and you will have a myriad of special
conditions and complex decisions.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> If it can be done reliably outside of CC Mode (i.e. without relying on
> knowing the particulars of the buffer's programming language),

It can be done reliably, but it would most likely want help from the
major mode (e.g. the C syntax tables give the syntax of strings but
don't indicate that those strings are not allowed to span multiple
lines).


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 21:52, Eli Zaretskii wrote:
> No.  Scrolling-related variables always worked this way, since Emacs
> 21.  You will see the same with scroll-margin, for example.
> 
>> Sounds unnecessarily complex, IMHO.
> If you think about this for a while, you will realize that it's the
> only sane way of doing that, in an editor that is required to handle
> lines which can contain anything: small and large characters, images,
> you name it.  Any other way, and you will have a myriad of special
> conditions and complex decisions.

Fair enough.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Mon, 20 Apr 2020 14:51:07 -0400
> 
> > by the canonical line height, it doesn't count screen lines.  So if
> > some lines in the buffer are displayed taller than the default face,
> > you will see a lot of recentering regardless of what you do.
> 
> Good point.  So any "unusual" character which ends up selecting
> a different font that happens to be slightly taller will disrupt the
> "conservative" scroll.

A tall character or an image, yes.

> Tho, I guess we should consider "1" as meaning "strictly smaller than
> 2.0", in which case a line would have to be twice as tall as a normal
> line before we cause a recenter.

No, I think the inequality is done in pixels, if I'm not
misremembering.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Stefan.

On Mon, Apr 20, 2020 at 15:01:14 -0400, Stefan Monnier wrote:
> > If it can be done reliably outside of CC Mode (i.e. without relying on
> > knowing the particulars of the buffer's programming language),

> It can be done reliably, but it would most likely want help from the
> major mode (e.g. the C syntax tables give the syntax of strings but
> don't indicate that those strings are not allowed to span multiple
> lines).

How about adding support to the syntax table?

For example, a new syntax class called something like "line string
quote".  One of these would begin a @def{line string}, and another one
or an unescaped NL (whichever came first) would end it.

Easy enough to implement, at least for forward scanning.  back_comment
might cause some fun.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Stefan.

On Sun, Apr 19, 2020 at 19:21:59 -0400, Stefan Monnier wrote:
> >> You don't know whether it's an unterminated raw string (the usual case)
> >> until you've scanned for a potential closing delimiter.
> > Is C++ syntax so ambiguous? Can R"( mean something else?

> No it has nothing to do with the language's syntax.

> It's just a UI choice: Alan likes to highlight the opening string
> delimiter of unterminated strings [ Not sure why CC-mode doesn't do the
> same for unterminated comments, tho maybe it's just an oversight.  ]

There isn't really such a thing as an unterminated comment.  A line
comment extends to the next unescaped EOL, a block comment extends to
the next */ or EOB.  The syntax routines and font locking already handle
it well.

> I think this should not vary between major modes, IOW it should be
> a separate feature that works on all major modes.


>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 23:06, Alan Mackenzie wrote:
> A line
> comment extends to the next unescaped EOL, a block comment extends to
> the next */ or EOB.

And an unterminated raw string doesn't?

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Alan Mackenzie]

Hello, Dmitry.

On Sun, Apr 19, 2020 at 21:12:57 +0300, Dmitry Gutov wrote:
> On 19.04.2020 20:12, Alan Mackenzie wrote:

[ .... ]

> >>> Inserting a C++ raw string opener does typically necessitate a full
> >>> scan (a search for a matching closer), but that would also be the
> >>> case using syntax-propertize.

> >> Not really. It would just mark the opener as a string opener (maybe with
> >> some extra text property), and that's that.

> > You don't know whether it's an unterminated raw string (the usual case)
> > until you've scanned for a potential closing delimiter.

> Is C++ syntax so ambiguous? Can R"( mean something else?

C++ syntax is ambiguous in several places, but not here.  R"foo(,
assuming it's not itself inside a literal, means exactly one thing.  But
you don't know whether it starts an unterminated string unless you seach
for a closing delimiter.

> > This affects the font locking.  (An unterminated opening delimiter
> > gets font-lock-warning-face, a terminated one doesn't.)

> If everything after R"( is fontified as a string, it serves as a 
> "warning" of sort as well.

That means putting syntax-table properties on every " after the R"foo(,
otherwise that string would only extend to the first such ".

But it wouldn't work well, since it wouldn't tell the user whether her
string was already closed.

> > This is the sort of feature which I'm not willing to sacrifice.

> Is it worth a full buffer scan every time you write a new raw string 
> literal?

Yes, definitely.

> >> Then font-lock would fontify the following text as string contents
> >> (until the end of the window or a little bit farther). Then you type
> >> the closer, it only has to scan a little far back (it'll call
> >> syntax-ppss to find the string opener), the closer is propertized as
> >> appropriate, and that's that. No full buffer scans at any step.

> >> I recall that fontifying the rest of the buffer as text after a simple
> >> string opener could be a sore topic for you, but raw strings should be
> >> rare enough (aren't they?), or if they are not, fontification logic
> >> could opt to do something different, while syntax-table properties will
> >> be applied the "correct" way.

> > I'm not sure what you mean by "as text".

> Sorry, I meant "as string".

OK.

Same procedure for a simple string - if it's a terminated string the "
gets font-lock-string-face, if it's not it gets f-l-warning face.

> > I've no reason to think raw strings are at all rare.  I've had
> > several bug reports for them.  I'm not sure what you mean by
> > "fontification logic ... something different" - do you mean in the
> > raw string case?

> I mean that if a raw string is unterminated, the default behavior should 
> be to fontify the rest of the buffer as string. But then again, you 
> could choose some different highliting in font-lock rules.

The current strategy is to fontify the unterminated R"foo( with
warning-face, and let the devil deal with the rest of the string (i.e. no
attempt is made to apply syntax-table properties).  The first portion of
the raw string will indeed get string-face.

As soon as the closing delimiter is typed, the warning-face is removed
from the opener and syntax-table text properties applied throughout the
string.  The entire string then gets string-face.

[ .... ]

> >> Yes, I think before-change-functions should become empty. Or much
> >> emptier.

> > It can't become empty.  after-change-functions is fine for dealing
> > with insertions, but can't do much after a deletion.  Consider the
> > case where you're in a string and all you know is that 5 characters
> > have been deleted.  Those characters might have been )foo", so after
> > checking the beginning of the string starts off with R"bar(, you've
> > then got to scan a long way forward looking for )bar".  Effectively
> > every deletion within a string would involve scanning to the end of
> > that string.

> This is an example of extra complexity you have to retain to implement 
> the above feature the way you do.

It will become more complex and slower, if information from
before-change-functions is ignored, or discarded.  The alternative is,
after each deletion, to scan forward checking that the terminating
delimiter still exists.  This is slower and more complicated than
checking in b-c-f whether it's about to be removed.

> It's probably also an example of how before/after-change-functions 
> essentially duplicate the knowledge of language syntax. I'm guessing 
> here, but to make it work like that, you need to have multiple functions 
> "understand" the raw string syntax.

b/a-c-f implement the language syntax.  It's one of the places the
language is codified.  The mechanism is in several functions, yes.  If
you're interested, go into cc-engine.el and search for "raw string".

> Whereas with syntax-propertize-function, that knowledge is concentrated 
> in one place (maybe two, if font-lock rules do something unusual). This 
> way, the code is simplified.

No, it gets complicated, assuming no loss of functionality.  A given
amount of functionality would get squashed into a smaller place.  The
current implementation (of C++ raw strings) is optimised such that normal
insertion and deletion don't cause the s-t properties on the entire
string to be modified.  That requires details of the buffer before the
insertions and deletions.

[ .... ]

> >>> Maybe, but with a slowdown.  More of these properties will get erased
> >>> than needed (with nested template forms), and they will all need to get
> >>> put back again.

> >> We won't really know until we can measure the result.

> > What's the point in investing all the effort to make the change, when
> > there's not even a prediction of a speed up?

> In principle, the speed-up will come from:

> - Deferred execution (where several buffer changes can be handled 
> together and not right away),

I've never been wholly convinced by laziness.  Sooner or later these
changes need to be handled, and delaying them is not going to accelerate
them.

> - No parsing the buffer contents much farther than the current window, 
> in most cases. Which can speed up the majority of user actions. The 
> exceptions will remain slower, but that is often a good tradeoff.

This will involve loss of functionality, as already noted.  And bugs;
whilst typing in normal text, CC Mode has to search backwards for a safe
place, otherwise context fontification can mess things up.  This is an
area where optimisation would be useful.

> > And I'm not sure where the proof of the syntax-propertize mechanism
> > being helpful is.  Has anybody but its originator positively chosen
> > to use it, whilst being aware of the alternatives?

> The alternatives being reinventing the relevant logic from zero in each 
> major mode? And writing syntax caching logic each time?

Or writing and using a better framework.

The question remains: has anybody other than Stefan M. freely chosen to
use syntax-ppss and syntax-propertize-function, whilst being aware of
their disadvantages and of alternatives?

Remember, that for an extended period of time syntax-ppss didn't work
properly, and even now it doesn't do the right thing in narrowed buffers,
at least for a programming mode such as CC Mode.  The syntax-propertize
mechanism erases s-t p's in a manner not under the control of the major
mode, which means the major mode needs to implement workarounds (which
are liable to be slow).

> > To become usable for CC Mode, it would need to provide something on
> > before-change-functions to complement what's on a-c-f, and it would need
> > to provide some control to the major mode over which syntax-table
> > properties get erased.

> Not something I can comment on.

Hmmm.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> Tho, I guess we should consider "1" as meaning "strictly smaller than
>> 2.0", in which case a line would have to be twice as tall as a normal
>> line before we cause a recenter.
>
> No, I think the inequality is done in pixels, if I'm not
> misremembering.

What I meant is that if the setting is 1 and the line height is, say, 13
pixels, we could consider any displacement of strictly less than 26
pixels to be "within the limits".  Or we could accept a floating point
setting and let the user make that choice.


        Stefan

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

>> It can be done reliably, but it would most likely want help from the
>> major mode (e.g. the C syntax tables give the syntax of strings but
>> don't indicate that those strings are not allowed to span multiple
>> lines).
> How about adding support to the syntax table?

I don't know of any language that defines strings this way, so I'd
rather not.

[ Furthermore, I think there are too many subtly different ways to
  (dis)allow multiline strings in programming languages, so handling it
  in something like `syntax-propertize` is preferable at least until we
  have a firm grasp of what is the design space.  ]

It's OK to try and detect the situation and highlight it as an error,
but that's quite different from defining it as "the official syntax".


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 20.04.2020 18:16, Eli Zaretskii wrote:

>> As we've determined, certain commands force redisplay to be performed
>> anyway (though partially).
> 
> That's not redisplay, that's JIT fontification that happens as side
> effect of redisplay.  It can _also_ happen as side effect of other
> commands.  But that's not redisplay.

A certain amount of work that's also part of redisplay. IIRC 
posn-at-point also "simulates redisplay". Not sure if scrolling commands 
use it, but others do.

I'm saying this because my current impression is still that redisplay is 
skipped sometimes (when Emacs is busy) for performance reasons. And the 
cases above are when they might apply to a lesser extent.

>> If performing it in full in those cases will take just a little more CPU
>> time, the result might be Emacs that looks more responsive under load,
>> and just as fast.
> 
> If you can find a way of reusing that information afterwards (more
> than redisplay itself already does), yes.  But AFAIU you are talking
> about a thorough redesign of the Emacs display engine.

Hopefully not.

> It already
> attempts to reuse every piece of display that can be kept.  For
> example, if you type C-n on the bottom-most screen line, and you have
> scroll-conservatively set, the display engine will redraw exactly one
> screen line, and all the rest will be scrolled (via a direct Xlib
> call) on the glass, without regenerating the glyphs.

That's an interesting tidbit. But it seems to also say that redisplay 
for C-n is usually quite fast. And maybe we don't gain much by skipping 
it in that case.

>> It might be possible to construct some "cache key" that would allow us
>> to check whether the remainder of the command changed something that
>> would affect display. And if it didn't, use the last saved result of
>> move_it_*.
> 
> You will see in redisplay_window and its subroutines a lot of code
> that attempts to do exactly that: figure out what has changed and what
> hasn't.  You are in effect saying that this must be radically
> improved, in which case I'd happily agree, but please believe me that
> it is not a trivial job, certainly not if you want to stay within the
> current design of the display engine.  You will probably need to
> design and implement various new fields of the window and buffer
> objects that will cache more data about the previous state of buffer
> and display than what they already do, make sure these caches are
> reset when no longer valid, then build new redisplay optimizations
> based on those, find the conditions where the optimizations can and
> cannot be used, etc.

So I wouldn't be correct to say that the display is only dependent on 
the current window configuration (shown buffers, window sizes, window 
scroll positions), buffer text which can be checked via 
buffer-modified-tick and the overlays which... might have a similar 
"counter" added?

But mode-lines and header-lines aren't amenable to caching, right? That 
complicates things.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Alan,

On 21.04.2020 00:19, Alan Mackenzie wrote:

>> Is C++ syntax so ambiguous? Can R"( mean something else?
> 
> C++ syntax is ambiguous in several places, but not here.  R"foo(,
> assuming it's not itself inside a literal, means exactly one thing.  But
> you don't know whether it starts an unterminated string unless you seach
> for a closing delimiter.

Then you can put a certain syntax-table value on it either way. Not 
(string-to-syntax "\""), though.

>>> This affects the font locking.  (An unterminated opening delimiter
>>> gets font-lock-warning-face, a terminated one doesn't.)
> 
>> If everything after R"( is fontified as a string, it serves as a
>> "warning" of sort as well.
> 
> That means putting syntax-table properties on every " after the R"foo(,
> otherwise that string would only extend to the first such ".

Nope. You will put (string-to-syntax "|") on the the raw string opener. 
It doesn't match "ordinary" quotes.

> Same procedure for a simple string - if it's a terminated string the "
> gets font-lock-string-face, if it's not it gets f-l-warning face.

As discussed in nearby messages, this should be doable in a generic way 
outside of CC Mode.

>> I mean that if a raw string is unterminated, the default behavior should
>> be to fontify the rest of the buffer as string. But then again, you
>> could choose some different highliting in font-lock rules.
> 
> The current strategy is to fontify the unterminated R"foo( with
> warning-face, and let the devil deal with the rest of the string (i.e. no
> attempt is made to apply syntax-table properties).  The first portion of
> the raw string will indeed get string-face.
> 
> As soon as the closing delimiter is typed, the warning-face is removed
> from the opener and syntax-table text properties applied throughout the
> string.  The entire string then gets string-face.

That would change. But within limits that are acceptable for you, hopefully.

>>> It can't become empty.  after-change-functions is fine for dealing
>>> with insertions, but can't do much after a deletion.  Consider the
>>> case where you're in a string and all you know is that 5 characters
>>> have been deleted.  Those characters might have been )foo", so after
>>> checking the beginning of the string starts off with R"bar(, you've
>>> then got to scan a long way forward looking for )bar".  Effectively
>>> every deletion within a string would involve scanning to the end of
>>> that string.
> 
>> This is an example of extra complexity you have to retain to implement
>> the above feature the way you do.
> 
> It will become more complex and slower, if information from
> before-change-functions is ignored, or discarded.  The alternative is,
> after each deletion, to scan forward checking that the terminating
> delimiter still exists.  This is slower and more complicated than
> checking in b-c-f whether it's about to be removed.

Removal of micro-optimizations will make things slower only when they 
apply to actual bottlenecks. And those can change with the change of the 
approach.

>> It's probably also an example of how before/after-change-functions
>> essentially duplicate the knowledge of language syntax. I'm guessing
>> here, but to make it work like that, you need to have multiple functions
>> "understand" the raw string syntax.
> 
> b/a-c-f implement the language syntax.  It's one of the places the
> language is codified.  The mechanism is in several functions, yes.  If
> you're interested, go into cc-engine.el and search for "raw string".

Your confirmation is good enough for me for now, thank you.

>> Whereas with syntax-propertize-function, that knowledge is concentrated
>> in one place (maybe two, if font-lock rules do something unusual). This
>> way, the code is simplified.
> 
> No, it gets complicated, assuming no loss of functionality.  A given
> amount of functionality would get squashed into a smaller place.  The
> current implementation (of C++ raw strings) is optimised such that normal
> insertion and deletion don't cause the s-t properties on the entire
> string to be modified.  That requires details of the buffer before the
> insertions and deletions.

True. So maybe Emacs will do "extra" work. It still might turn out to be 
faster for most user interactions.

>> In principle, the speed-up will come from:
> 
>> - Deferred execution (where several buffer changes can be handled
>> together and not right away),
> 
> I've never been wholly convinced by laziness.  Sooner or later these
> changes need to be handled, and delaying them is not going to accelerate
> them.

It's the difference between keeping the whole buffer up-to-date and only 
doing that for a limited range of chars.

>> - No parsing the buffer contents much farther than the current window,
>> in most cases. Which can speed up the majority of user actions. The
>> exceptions will remain slower, but that is often a good tradeoff.
> 
> This will involve loss of functionality, as already noted.  And bugs;
> whilst typing in normal text, CC Mode has to search backwards for a safe
> place, otherwise context fontification can mess things up.  This is an
> area where optimisation would be useful.

That's where syntax-ppss comes in.

>>> And I'm not sure where the proof of the syntax-propertize mechanism
>>> being helpful is.  Has anybody but its originator positively chosen
>>> to use it, whilst being aware of the alternatives?
> 
>> The alternatives being reinventing the relevant logic from zero in each
>> major mode? And writing syntax caching logic each time?
> 
> Or writing and using a better framework.

Well, um. You didn't write a generic framework that would help all major 
mode writers yourself. It's a bit too much to expect that from others 
either.

> The question remains: has anybody other than Stefan M. freely chosen to
> use syntax-ppss and syntax-propertize-function, whilst being aware of
> their disadvantages and of alternatives?

I'm not him.

> Remember, that for an extended period of time syntax-ppss didn't work
> properly, and even now it doesn't do the right thing in narrowed buffers,
> at least for a programming mode such as CC Mode.  The syntax-propertize
> mechanism erases s-t p's in a manner not under the control of the major
> mode, which means the major mode needs to implement workarounds (which
> are liable to be slow).

Narrowing is a tricky thing. I'd wager to say that the vast majority of 
our users don't use it, or use it in very specific ways. So any related 
bugs might go unnoticed or take quite some time to be uncovered.

Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> The question remains: has anybody other than Stefan M. freely chosen to
> use syntax-ppss and syntax-propertize-function, whilst being aware of
> their disadvantages and of alternatives?

Not quite sure I can answer that.

I can point to cperl-mode which used the exact same approach as
`syntax-propertize` (without using `syntax-propertize` because it didn't
exist yet) and I can point to many major modes that use
`syntax-propertize` successfully without my involvement.  Perl is one of
the languages that requires the most intense help from `syntax-table
text properties.

AFAIK CC-mode is the only "alternative" and last I checked there's only
one human on this planet who really knows what it does.

> The syntax-propertize mechanism erases s-t p's in a manner not under
> the control of the major mode, which means the major mode needs to
> implement workarounds (which are liable to be slow).

I have no idea what you're referring to.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dgutov@yandex.ru,  acm@muc.de,  rrandresf@gmail.com,
>   emacs-devel@gnu.org,  rms@gnu.org,  rudalics@gmx.at
> Date: Mon, 20 Apr 2020 17:50:20 -0400
> 
> What I meant is that if the setting is 1 and the line height is, say, 13
> pixels, we could consider any displacement of strictly less than 26
> pixels to be "within the limits".

That would mean the last line we scroll into the window will be only
partially visible, and that is a no-no in scrolling.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 21 Apr 2020 04:27:19 +0300
> 
> On 20.04.2020 18:16, Eli Zaretskii wrote:
> 
> >> As we've determined, certain commands force redisplay to be performed
> >> anyway (though partially).
> > 
> > That's not redisplay, that's JIT fontification that happens as side
> > effect of redisplay.  It can _also_ happen as side effect of other
> > commands.  But that's not redisplay.
> 
> A certain amount of work that's also part of redisplay.

I suggest to use a common terminology, otherwise we will just confuse
each other.  Let's agree to call "redisplay" only what happens when
Emacs calls one of the two functions: redisplay_internal and
echo_area_display.  OK?  Redisplay calls many functions, including
fontification-functions, but those functions themselves are not to be
called "redisplay", because they don't display anything.

> IIRC posn-at-point also "simulates redisplay". Not sure if scrolling
> commands use it, but others do.

Many commands do, much more than what you mentioned.  If we call all
of them "redisplay" we will completely muddy the waters.

> I'm saying this because my current impression is still that redisplay is 
> skipped sometimes (when Emacs is busy) for performance reasons.

Only if input is available (or if it arrives during redisplay,
assuming you set redisplay-dont-pause to a nil value).

> > It already
> > attempts to reuse every piece of display that can be kept.  For
> > example, if you type C-n on the bottom-most screen line, and you have
> > scroll-conservatively set, the display engine will redraw exactly one
> > screen line, and all the rest will be scrolled (via a direct Xlib
> > call) on the glass, without regenerating the glyphs.
> 
> That's an interesting tidbit. But it seems to also say that redisplay 
> for C-n is usually quite fast.

It is fast.  By default, it will scroll half the window and redraw the
other half (due to recentering).

> > You will see in redisplay_window and its subroutines a lot of code
> > that attempts to do exactly that: figure out what has changed and what
> > hasn't.  You are in effect saying that this must be radically
> > improved, in which case I'd happily agree, but please believe me that
> > it is not a trivial job, certainly not if you want to stay within the
> > current design of the display engine.  You will probably need to
> > design and implement various new fields of the window and buffer
> > objects that will cache more data about the previous state of buffer
> > and display than what they already do, make sure these caches are
> > reset when no longer valid, then build new redisplay optimizations
> > based on those, find the conditions where the optimizations can and
> > cannot be used, etc.
> 
> So I wouldn't be correct to say that the display is only dependent on 
> the current window configuration (shown buffers, window sizes, window 
> scroll positions), buffer text which can be checked via 
> buffer-modified-tick and the overlays which... might have a similar 
> "counter" added?

I don't think I understand the question, but the display engine
already uses the buffer-modified-tick and overlay-modified tick.

> But mode-lines and header-lines aren't amenable to caching, right? That 
> complicates things.

Header line is almost never a problem, it hardly ever changes IME.
Mode line, OTOH, changes almost all the time (at least with its
default format), so I think caching it would be quite futile.  And you
will anyway need to somehow compare the cached value with the updated
one, something that the display engine already does, using the results
of the previous display as "the cache".

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That would mean the last line we scroll into the window will be only
  > partially visible, and that is a no-no in scrolling.

It is somewhat undesirable, but I am not convinced it must be
absolutely rejected.  Perhaps in some cases it will be the lesser
of the implementable evils.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: monnier@iro.umontreal.ca, dgutov@yandex.ru, acm@muc.de,
> 	rrandresf@gmail.com, emacs-devel@gnu.org, rudalics@gmx.at
> Date: Tue, 21 Apr 2020 23:14:07 -0400
> 
>   > That would mean the last line we scroll into the window will be only
>   > partially visible, and that is a no-no in scrolling.
> 
> It is somewhat undesirable, but I am not convinced it must be
> absolutely rejected.

That's currently a very significant requirement throughout
redisplay_window and its subroutine: any result that leaves the
current line (i.e. the line where point is) not completely visible is
rejected, and we select some other window-start point to avoid that.
It's possible to toss this requirement, of course, but then a lot of
existing code in the display engine will have to be rewritten, because
I won't be surprised if the assumption that this holds is implicitly
made by some parts of the display code.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That's currently a very significant requirement throughout
  > redisplay_window and its subroutine: any result that leaves the
  > current line (i.e. the line where point is) not completely visible is
  > rejected, and we select some other window-start point to avoid that.

Ok.

The text I responded to

>   > That would mean the last line we scroll into the window will be only
>   > partially visible, and that is a no-no in scrolling.

talks about "the last line we scroll into the window", and _in general_ that
is not always the current line.

Is it always the current line in the case under discussion?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Stefan Monnier]

> Is it always the current line in the case under discussion?

No, never.  The case under discussion was about choosing between
scrolling and recentering based on the value of `scroll-conservatively`.


        Stefan

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org, rudalics@gmx.at, rrandresf@gmail.com,
> 	monnier@iro.umontreal.ca, dgutov@yandex.ru, acm@muc.de
> Date: Wed, 22 Apr 2020 22:37:14 -0400
> 
> The text I responded to
> 
> >   > That would mean the last line we scroll into the window will be only
> >   > partially visible, and that is a no-no in scrolling.
> 
> talks about "the last line we scroll into the window", and _in general_ that
> is not always the current line.
> 
> Is it always the current line in the case under discussion?

When scroll-conservatively is in effect (which is the context of this
discussion), yes.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

On 21.04.2020 17:02, Eli Zaretskii wrote:

>> A certain amount of work that's also part of redisplay.
> 
> I suggest to use a common terminology, otherwise we will just confuse
> each other.  Let's agree to call "redisplay" only what happens when
> Emacs calls one of the two functions: redisplay_internal and
> echo_area_display.  OK?  Redisplay calls many functions, including
> fontification-functions, but those functions themselves are not to be
> called "redisplay", because they don't display anything.

Do scrolling commands and/or code that calls posn-at-point do anything 
else common with redisplay other than fontification functions?

For instance, converting the buffer text into some other structure that 
rendering the window will be based on. I think you called it a "glyphs 
table" or something like that.

If *that* work (together with fontification) is usually what takes the 
most time during redisplay, there could be some simple cache added on to 
of it, which would make skipping redisplay unnecessary in cases when the 
command would pre-fill such cache.

Otherwise, this idea is a dud.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > When scroll-conservatively is in effect (which is the context of this
  > discussion), yes.

In that case I think you are right.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Eli Zaretskii]

> Cc: acm@muc.de, rudalics@gmx.at, rrandresf@gmail.com, rms@gnu.org,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 24 Apr 2020 04:23:57 +0300
> 
> > I suggest to use a common terminology, otherwise we will just confuse
> > each other.  Let's agree to call "redisplay" only what happens when
> > Emacs calls one of the two functions: redisplay_internal and
> > echo_area_display.  OK?  Redisplay calls many functions, including
> > fontification-functions, but those functions themselves are not to be
> > called "redisplay", because they don't display anything.
> 
> Do scrolling commands and/or code that calls posn-at-point do anything 
> else common with redisplay other than fontification functions?

"Common" is too general a word to give an intelligent and helpful
answer to your question.  I have no other way except explaining a lot
of how relevant parts of the display engine work (below), or else we
will be forever trapped in misunderstandings.  (I do suggest to read
the commentary and the code in xdisp.c, to facilitate better
communications.  It takes time and non-trivial effort to write these
descriptions.)

> For instance, converting the buffer text into some other structure that 
> rendering the window will be based on. I think you called it a "glyphs 
> table" or something like that.

You mean the glyph matrices.  There are two matrices for each window
on display: the "current" matrix and the "desired" matrix.  The
current matrix describes what's currently on the glass.  The desired
matrix is created during each redisplay cycle and describes what
_should_ be on the glass; it is generally sparse, i.e. holds only
parts of the window's screen lines, because the display engine decided
only those parts need to be updated for this cycle.  The desired
matrix is used to update what's on the glass, and then its "glyph
rows" (each row corresponds to a screen line) are copied into the
current matrix (thus making the current matrix up-to-date), and then
the desired matrix is discarded.

Glyph rows in the desired matrix are produced by the functions
display_line (which handles buffer text and any display/overlay
strings put on buffer text via text properties and overlays), and
display_string (which handles Lisp or C strings unrelated to any
buffer, like what we do with mode/header/tab-line, menu-bar on TTY and
non-toolkit GUI frames, etc.)  These two functions use the "iterator"
object ('struct it', see dispextern.h) to iterate the object, whether
a buffer or a string, one character position at a time, and produce
the glyph structures which are elements of glyph rows.  The iteration
includes a lot of logic, but basically is built on 3 low-level
subroutines: set_iterator_to_next (which moves the iterator to the
next "display element"), get_next_display_element (which figures out
what kind of display element -- a character, a character composition,
an image, a stretch of white space, an xwidget -- is that element, and
fills the relevant fields of the iterator object with the data for
that element), and PRODUCE_GLYPHS, which is a macro that invokes a
produce_glyphs method of the window's frame, and fills up the glyph
structure of the next element of the glyph row.  The produce_glyphs
method also updates the running row metrics fields within the iterator
object, ones that keep track of the current pixel position, the
height, ascent, and descent of the row, etc.  JIT fontification
happens inside get_next_display_element, when it detects that it came
upon buffer text that doesn't have a non-nil 'fontified' property.

Scrolling commands, posn-at-point, and others that need layout
information of any kind, use the functions that simulate the display.
Those functions do NOT produce any glyph rows.  They use the
move_it_in_display_line function, which invokes the same 3 subroutines
as display_line and display_string, and with a similar surrounding
logic, but their only purpose is to move the iterator object in the
same way as redisplay would, keeping the metrics, but without producing
any glyphs.

I hope you can now figure out the answer to your question about the
"common" parts.

> If *that* work (together with fontification) is usually what takes the 
> most time during redisplay

This remains to be shown.  Someone™ should profile redisplay in
relevant use cases and present the profiles, then we could decide
whether this is or isn't the case.  Personally, I think it's roughly
40%-60% or 30%-70%, with the actual redisplay being the larger part.
But humans are notoriously bad in guessing this stuff, so I won't be
surprised if mine turns out to be wrong.

> there could be some simple cache added on to of it, which would make
> skipping redisplay unnecessary in cases when the command would
> pre-fill such cache.

I have my doubts that it will turn out to be simple.  Besides my
humble experience of hacking the display code, I have a much more
significant evidence: that such a cache was not implemented when the
current display engine was developed for Emacs 21.  Gerd Moellmann is
an extremely capable and talented programmer, and most of the work he
did on the display engine during Emacs 21 development was precisely to
make it fast enough.  All the redisplay optimizations we have now were
designed and implemented for that purpose, because without them
redisplay was unbearably slow on the machines we had at that time.  At
some point Gerd wrote that he implemented these optimizations one by
one, until he got satisfactory redisplay speed.  If there was a simple
way of caching more data to speed this up, I have no doubt we'd have
that already.

Of course, no one is perfect, so it could be that some opportunities
were missed.  Ideas and patches are welcome, together with timings and
profiles to show their utility.

Re: Scrolling commands and skipping redisplay, was: Re: emacs rendering comparisson between emacs23 and emacs26.3

[Dmitry Gutov]

Hi Eli,

On 24.04.2020 12:51, Eli Zaretskii wrote:

>> Do scrolling commands and/or code that calls posn-at-point do anything
>> else common with redisplay other than fontification functions?
> 
> "Common" is too general a word to give an intelligent and helpful
> answer to your question.  I have no other way except explaining a lot
> of how relevant parts of the display engine work (below), or else we
> will be forever trapped in misunderstandings.  (I do suggest to read
> the commentary and the code in xdisp.c, to facilitate better
> communications.  It takes time and non-trivial effort to write these
> descriptions.)

Thank you for the explanation. I see you also updated the commentary. 
The people who will come after will surely benefit.

> I hope you can now figure out the answer to your question about the
> "common" parts.

I think so, and the answer is "no". That means performance revolution 
will have to wait. But this was seriously educational.

>> If *that* work (together with fontification) is usually what takes the
>> most time during redisplay
> 
> This remains to be shown.  Someone™ should profile redisplay in
> relevant use cases and present the profiles, then we could decide
> whether this is or isn't the case.  Personally, I think it's roughly
> 40%-60% or 30%-70%, with the actual redisplay being the larger part.
> But humans are notoriously bad in guessing this stuff, so I won't be
> surprised if mine turns out to be wrong.

Indeed.

>> there could be some simple cache added on to of it, which would make
>> skipping redisplay unnecessary in cases when the command would
>> pre-fill such cache.
> 
> I have my doubts that it will turn out to be simple.  Besides my
> humble experience of hacking the display code, I have a much more
> significant evidence: that such a cache was not implemented when the
> current display engine was developed for Emacs 21.  Gerd Moellmann is
> an extremely capable and talented programmer, and most of the work he
> did on the display engine during Emacs 21 development was precisely to
> make it fast enough.  All the redisplay optimizations we have now were
> designed and implemented for that purpose, because without them
> redisplay was unbearably slow on the machines we had at that time.  At
> some point Gerd wrote that he implemented these optimizations one by
> one, until he got satisfactory redisplay speed.  If there was a simple
> way of caching more data to speed this up, I have no doubt we'd have
> that already.

But 20 years have passed now, and I imagine the complexity of building a 
glyph matrix and then rendering it couldn't have changed much (maybe 
proportional to the screen resolutions). The font-lock rules have most 
likely became more complex in the meantime (meaning, fontification is 
possible always the slowest part by far). So actually giving up on some 
optimizations (such as skipping redisplay) might keep Emacs about the 
same in terms of performance on modern machines, yet improve 
responsiveness in those cases when it can't keep up with input.

But it's a difficult argument to make given we must have some users with 
20-years old machines still. And while our developers compile with 
optimizations disabled (the current master might help improve on this).

Seems I should look into throttling repeated keyboard input when 
redisplay can't keep up instead after all. In theory, it should give 
about the same increase in responsiveness in the scenarios we've been 
discussing here.